Really slow IMAP performance
Please could you help me to understand and fix why my dovecot IMAP performance is so bad? I've read through a lot of the performance-related material on the website, but I don't think that any of it could account for slowness at the level that I am seeing.
The simplest scenario is moving a message from my Inbox to another IMAP folder. Using Gnus as the client, the whole UI freezes for about 2 minutes (which I assume is until the move is complete), and journalctl on the dovecot server says:
Feb 24 10:24:24 arudy dovecot[1712]: imap(neil): Warning: Transaction log file /home/neil/dovecot-mail/.imap/INBOX/dovecot.index.log was locked for 98 seconds (Mailbox was synchronized)
or the same message with (rotating while syncing).
There must be something badly wrong in my setup, or perhaps in the spec of the server that dovecot is running on. What should I look at to start understanding this better?
Many thanks - Neil
Aki Tuomi <aki.tuomi@dovecot.fi> writes:
On 24 February 2018 at 12:45 Neil Jerram < neil@ossau.homelinux.net> wrote:
Please could you help me to understand and fix why my dovecot IMAP performance is so bad? I've read through a lot of the performance-related material on the website, but I don't think that any of it could account for slowness at the level that I am seeing.
The simplest scenario is moving a message from my Inbox to another IMAP folder. Using Gnus as the client, the whole UI freezes for about 2 minutes (which I assume is until the move is complete), and journalctl on the dovecot server says:
Feb 24 10:24:24 arudy dovecot[1712]: imap(neil): Warning: Transaction log file /home/neil/dovecot-mail/.imap/INBOX/dovecot.index.log was locked for 98 seconds (Mailbox was synchronized)
or the same message with (rotating while syncing).
There must be something badly wrong in my setup, or perhaps in the spec of the server that dovecot is running on. What should I look at to start understanding this better?
Many thanks - Neil
Can you tell a bit more about your environment? Sounds like io issue
Thanks for your reply. I'm not sure exactly what you have in mind, but here are some starting points:
arudy:~# uname -a Linux arudy 4.13.0-1-686-pae #1 SMP Debian 4.13.4-2 (2017-10-15) i686 GNU/Linux
arudy:~# dovecot -n # 2.2.32 (dfbe293d4): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.20 (7cd71ba) # OS: Linux 4.13.0-1-686-pae i686 Debian buster/sid auth_mechanisms = plain login auth_username_format = %Ln auth_verbose = yes login_trusted_networks = 192.168.11.8 mail_access_groups = mail mail_fsync = never mail_location = mbox:~/dovecot-mail:INBOX=/var/mail/%u namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = pam } plugin { antispam_backend = dspam antispam_dspam_args = --deliver;--user;%u antispam_dspam_binary = /usr/bin/dspam antispam_signature = X-DSPAM-Signature antispam_signature_missing = error antispam_spam = Spam antispam_trash = trash;Trash;Deleted Items; Deleted Messages fts = solr fts_solr = url=http://localhost:8080/solr/ sieve = file:~/sieve;active=~/.dovecot.sieve } postmaster_address = postmaster@ossau.homelinux.net protocols = " imap lmtp" service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } } ssl_cert = </etc/dovecot/cert.pem ssl_key = # hidden, use -P to show it userdb { driver = passwd } protocol lmtp { mail_fsync = optimized } protocol lda { mail_fsync = optimized }
arudy:~# cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 8 model name : Pentium III (Coppermine) stepping : 6 microcode : 0x8 cpu MHz : 935.497 cache size : 256 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fdiv_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca cmov pse36 mmx fxsr sse cpuid bugs : bogomips : 1870.99 clflush size : 32 cache_alignment : 32 address sizes : 36 bits physical, 32 bits virtual power management:
arudy:~# free total used free shared buff/cache available Mem: 248808 31572 4080 2576 213156 205880 Swap: 763052 20480 742572
On 24 February 2018 at 15:47 Neil Jerram <neil@ossau.homelinux.net> wrote:
Aki Tuomi <aki.tuomi@dovecot.fi> writes:
On 24 February 2018 at 12:45 Neil Jerram < neil@ossau.homelinux.net> wrote:
Please could you help me to understand and fix why my dovecot IMAP performance is so bad? I've read through a lot of the performance-related material on the website, but I don't think that any of it could account for slowness at the level that I am seeing.
The simplest scenario is moving a message from my Inbox to another IMAP folder. Using Gnus as the client, the whole UI freezes for about 2 minutes (which I assume is until the move is complete), and journalctl on the dovecot server says:
Feb 24 10:24:24 arudy dovecot[1712]: imap(neil): Warning: Transaction log file /home/neil/dovecot-mail/.imap/INBOX/dovecot.index.log was locked for 98 seconds (Mailbox was synchronized)
or the same message with (rotating while syncing).
There must be something badly wrong in my setup, or perhaps in the spec of the server that dovecot is running on. What should I look at to start understanding this better?
Many thanks - Neil
Can you tell a bit more about your environment? Sounds like io issue
Thanks for your reply. I'm not sure exactly what you have in mind, but here are some starting points:
arudy:~# uname -a Linux arudy 4.13.0-1-686-pae #1 SMP Debian 4.13.4-2 (2017-10-15) i686 GNU/Linux
arudy:~# dovecot -n # 2.2.32 (dfbe293d4): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.20 (7cd71ba) # OS: Linux 4.13.0-1-686-pae i686 Debian buster/sid auth_mechanisms = plain login auth_username_format = %Ln auth_verbose = yes login_trusted_networks = 192.168.11.8 mail_access_groups = mail mail_fsync = never mail_location = mbox:~/dovecot-mail:INBOX=/var/mail/%u
You are using mbox format. This is ... bit slow. =)
When you move mails between mbox files, it has to rewrite the entire mbox file every time. You should probably start using maildir or sdbox instead.
Aki
Aki Tuomi <aki.tuomi@dovecot.fi> writes:
On 24 February 2018 at 15:47 Neil Jerram <neil@ossau.homelinux.net> wrote:
[...]
Feb 24 10:24:24 arudy dovecot[1712]: imap(neil): Warning: Transaction log file /home/neil/dovecot-mail/.imap/INBOX/dovecot.index.log was locked for 98 seconds (Mailbox was synchronized) [...]
You are using mbox format. This is ... bit slow. =)
When you move mails between mbox files, it has to rewrite the entire mbox file every time. You should probably start using maildir or sdbox instead.
Ah, right, thanks.
My INBOX file has 22990 messages. Is the slowness that I am seeing definitely expected for an mbox of that size? (It may also be relevant that the HDD it's stored on is pretty old now, and has been known to report SMART errors...)
If so, I'll start looking at how to migrate, given that my system is Postfix + Dovecot. If you have any particular recommendations or migration pointers for a system like that, I'd appreciate them.
Best wishes - Neil
On 24 February 2018 at 16:04 Neil Jerram <neil@ossau.homelinux.net> wrote:
Aki Tuomi <aki.tuomi@dovecot.fi> writes:
On 24 February 2018 at 15:47 Neil Jerram <neil@ossau.homelinux.net> wrote:
[...]
Feb 24 10:24:24 arudy dovecot[1712]: imap(neil): Warning: Transaction log file /home/neil/dovecot-mail/.imap/INBOX/dovecot.index.log was locked for 98 seconds (Mailbox was synchronized) [...]
You are using mbox format. This is ... bit slow. =)
When you move mails between mbox files, it has to rewrite the entire mbox file every time. You should probably start using maildir or sdbox instead.
Ah, right, thanks.
My INBOX file has 22990 messages. Is the slowness that I am seeing definitely expected for an mbox of that size? (It may also be relevant that the HDD it's stored on is pretty old now, and has been known to report SMART errors...)
If so, I'll start looking at how to migrate, given that my system is Postfix + Dovecot. If you have any particular recommendations or migration pointers for a system like that, I'd appreciate them.
Best wishes - Neil
Yes. You deffo are looking at several reasons for slowness.
I can only recommend moving into maildir or sdbox format, and probably new HDD too.
https://wiki2.dovecot.org/Tools/Doveadm/Sync here is example of 'converting' between mailbox formats using dsync. You should also read https://wiki2.dovecot.org/Migration/MailFormat
mbox format has been known to act up with dsync occasionally, so I recommend using
doveadm backup maildir:~/Maildir
if you want to give it a try, instead of doveadm sync. Backup does dsync too, but it only works one way.
Aki
Aki Tuomi <aki.tuomi@dovecot.fi> writes:
Yes. You deffo are looking at several reasons for slowness.
I can only recommend moving into maildir or sdbox format, and probably new HDD too.
https://wiki2.dovecot.org/Tools/Doveadm/Sync here is example of 'converting' between mailbox formats using dsync. You should also read https://wiki2.dovecot.org/Migration/MailFormat
mbox format has been known to act up with dsync occasionally, so I recommend using
doveadm backup maildir:~/Maildir
Thanks, I've done that now, and things are looking much better.
I rediscovered that I've configured postfix to deliver locally using dovecot-lmtp - which meant that I then only needed to change dovecot's mail_location setting, and nothing at all in the postfix config.
Many thanks! - Neil
On 2018-02-24 (07:14 MST), Aki Tuomi <aki.tuomi@dovecot.fi> wrote:
That didn't show up when searching wiki2 for "Migration" :/
-- "...Life is not a journey to the grave with the intention of arriving safely in one pretty and well-preserved piece, but to slide across the finish line broadside, thoroughly used up, worn out, leaking oil, and shouting GERONIMO!!!" -- Bill McKenna
On Sat Feb 24 2018 17:01:01 GMT-0500 (Eastern Standard Time), @lbutlr <kremels@kreme.com> wrote:
On 2018-02-24 (07:14 MST), Aki Tuomi <aki.tuomi@dovecot.fi> wrote:
That didn't show up when searching wiki2 for "Migration" :/
Don't search 'wiki2', search just wiki.dovecot.org
I never liked the way Timo rolled out the wiki for the new version 2 when he did, I knew it would do nothing but create confusion...
Really, he should just redirect all references to wiki2 to wiki and kill the old content...
Den 24. feb. 2018 15:04, skrev Neil Jerram:
Aki Tuomi <aki.tuomi@dovecot.fi> writes:
On 24 February 2018 at 15:47 Neil Jerram <neil@ossau.homelinux.net> wrote:
[...]
Feb 24 10:24:24 arudy dovecot[1712]: imap(neil): Warning: Transaction log file /home/neil/dovecot-mail/.imap/INBOX/dovecot.index.log was locked for 98 seconds (Mailbox was synchronized) [...]
You are using mbox format. This is ... bit slow. =)
When you move mails between mbox files, it has to rewrite the entire mbox file every time. You should probably start using maildir or sdbox instead.
Ah, right, thanks.
My INBOX file has 22990 messages. Is the slowness that I am seeing definitely expected for an mbox of that size? (It may also be relevant that the HDD it's stored on is pretty old now, and has been known to report SMART errors...)
If so, I'll start looking at how to migrate, given that my system is Postfix + Dovecot. If you have any particular recommendations or migration pointers for a system like that, I'd appreciate them.
I'd go with whatever tools you are familiar with. If you don't know where to start, formail(1) can read an mbox and do whatever for each mail contained therein. This, together with procmail, used to be the go-to tools in the days before IMAP.
Theese days you'd probably want to involve your local delivery agent on the output from formail. The lda would invoke sieve instead of procmail if that is your thing.
Whatever you do, try to set up so you can do some tests before you blast 22000 mails to somewhere you do not want them :-)
On 2018-02-24 (07:04 MST), Neil Jerram <neil@ossau.homelinux.net> wrote:
My INBOX file has 22990 messages. Is the slowness that I am seeing definitely expected for an mbox of that size? (It may also be relevant that the HDD it's stored on is pretty old now, and has been known to report SMART errors...)
back int he dark ages I would send and alert to users if their inbox for over 1,000 messages because an mbox that large drove the server to its knees and made merely logging in to mail take an excruciating amount of time. If they didn't fix it I'd archive their inbox and start over.
I am astonished your machine can process an mbox with over 22 times that many messages.
https://wiki1.dovecot.org/Migration/MailFormat
That page doesn't exist on the wiki for dovecot 2, but that script to convert mbox to milder should still work. Obviously, keep backups and such.
-- Did they get you to trade your heroes for ghosts? Hot ashes for trees? Hot air for a cool breeze? Cold comfort for change?
On Sat, 24 Feb 2018, Neil Jerram wrote:
My INBOX file has 22990 messages. Is the slowness that I am seeing definitely expected for an mbox of that size? (It may also be relevant that the HDD it's stored on is pretty old now, and has been known to report SMART errors...)
Yeah, a copy of that mailbox will be that slow, esp. if the messages have large attachments. Even a simple operation like deleting/expunging the the first message will cause data shuffling of the entire mailbox.
Joseph Tam <jtam.home@gmail.com>
participants (6)
-
＠lbutlr
-
Aki Tuomi
-
Håkon Alstadheim
-
Joseph Tam
-
Neil Jerram
-
Tanstaafl