We're using dovecot (via LMTP) as a backup for all incoming mail.
I upgraded from 2.3.6 to 2.3.7 on the 12th: 2019-07-12 14:35:44 upgrade dovecot-imapd:amd64 2:2.3.6-2~bionic 2:2.3.7-8~bionic
and it seems that 2.3.7 is slower than 2.3.6 -- mail to the backup IMAP box is suddenly taking quite some time to get delivered and is piling up in the queue.
doveconf -n:
default_vsz_limit = 2 G lmtp_user_concurrency_limit = 10000 mail_attachment_dir = /home/copymail/attachments mail_fsync = never mail_location = mdbox:~/mdbox mail_plugins = zlib fts mdbox_rotate_size = 128 M ... namespaces ... plugin { fts_autoindex = yes fts_languages = de,en zlib_save = gz zlib_save_level = 5 } protocols = " imap lmtp" userdb { args = username_format=%u /etc/dovecot/passwd driver = passwd-file } protocol lmtp { mail_plugins = zlib fts }
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | https://www.charite.de
- Ralf Hildebrandt via dovecot dovecot@dovecot.org:
We're using dovecot (via LMTP) as a backup for all incoming mail.
I upgraded from 2.3.6 to 2.3.7 on the 12th: 2019-07-12 14:35:44 upgrade dovecot-imapd:amd64 2:2.3.6-2~bionic 2:2.3.7-8~bionic
and it seems that 2.3.7 is slower than 2.3.6 -- mail to the backup IMAP box is suddenly taking quite some time to get delivered and is piling up in the queue.
And alas, I had a look at the monitoring and found that disk IO has increase A LOT after the update: https://www.arschkrebs.de/images/dovecot-disk-io-increase.png
I zoomed in and found that the sudden increace in IO coincides with the update to 2.3.7 (14:35): https://www.arschkrebs.de/images/dovecot-io-2.png
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | https://www.charite.de
On 15 Jul 2019, at 23.17, Ralf Hildebrandt via dovecot dovecot@dovecot.org wrote:
- Ralf Hildebrandt via dovecot dovecot@dovecot.org:
We're using dovecot (via LMTP) as a backup for all incoming mail.
I upgraded from 2.3.6 to 2.3.7 on the 12th: 2019-07-12 14:35:44 upgrade dovecot-imapd:amd64 2:2.3.6-2~bionic 2:2.3.7-8~bionic
and it seems that 2.3.7 is slower than 2.3.6 -- mail to the backup IMAP box is suddenly taking quite some time to get delivered and is piling up in the queue.
And alas, I had a look at the monitoring and found that disk IO has increase A LOT after the update: https://www.arschkrebs.de/images/dovecot-disk-io-increase.png
I zoomed in and found that the sudden increace in IO coincides with the update to 2.3.7 (14:35): https://www.arschkrebs.de/images/dovecot-io-2.png https://www.arschkrebs.de/images/dovecot-io-2.png
Do the different disks have different kinds of data? Like you seem to be using external attachments and fts? Also your doveconf -n doesn't show what fts backend you're using. Or is it just half-configured and not really used?
- Timo Sirainen via dovecot dovecot@dovecot.org:
And alas, I had a look at the monitoring and found that disk IO has increase A LOT after the update: https://www.arschkrebs.de/images/dovecot-disk-io-increase.png
I zoomed in and found that the sudden increace in IO coincides with the update to 2.3.7 (14:35): https://www.arschkrebs.de/images/dovecot-io-2.png https://www.arschkrebs.de/images/dovecot-io-2.png
Do the different disks have different kinds of data?
All of the data is an /dev/sda1
Like you seem to be using external attachments and fts?
Jup, using that, but on one disk
Also your doveconf -n doesn't show what fts backend you're using. Or is it just half-configured and not really used?
I used to use lucene, but then I had issues building it and commented it out.
Looking now, it turns out that your packages come with lucene support... Reenabling:
mail_plugins = zlib fts fts_lucene
...
plugin { fts = lucene fts_autoindex = yes fts_languages = de,en fts_lucene = whitespace_chars=@. zlib_save = gz zlib_save_level = 5 }
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | https://www.charite.de
On 16 Jul 2019, at 11.15, Ralf Hildebrandt via dovecot dovecot@dovecot.org wrote:
- Timo Sirainen via dovecot dovecot@dovecot.org:
And alas, I had a look at the monitoring and found that disk IO has increase A LOT after the update: https://www.arschkrebs.de/images/dovecot-disk-io-increase.png
I zoomed in and found that the sudden increace in IO coincides with the update to 2.3.7 (14:35): https://www.arschkrebs.de/images/dovecot-io-2.png https://www.arschkrebs.de/images/dovecot-io-2.png
Do the different disks have different kinds of data?
All of the data is an /dev/sda1
What filesystem is this?
I did a bunch of testing, and after initially thinking I saw an increase it was just due to bad testing because the new TCP_NODELAY changes allowed imaptest to do more work. So I can't see that there is any disk IO difference.
There are a few intentional changes to lib-index and also mdbox code. One of those is that dovecot.index.cache files are being recreated more often now. In some cases they were wasting a lot of disk space because expunged mails hadn't been removed from them. I guess it's a possibility that all the disk IO was because in your system Dovecot started recreating all those files at the same time. Maybe you could try upgrading again during slower hours and see if the disk IO normalizes after a while?
All of the data is an /dev/sda1
What filesystem is this?
ext4
I did a bunch of testing, and after initially thinking I saw an increase it was just due to bad testing because the new TCP_NODELAY changes allowed imaptest to do more work. So I can't see that there is any disk IO difference.
There are a few intentional changes to lib-index and also mdbox code. One of those is that dovecot.index.cache files are being recreated more often now. In some cases they were wasting a lot of disk space because expunged mails hadn't been removed from them. I guess it's a possibility that all the disk IO was because in your system Dovecot started recreating all those files at the same time. Maybe you could try upgrading again during slower hours and see if the disk IO normalizes after a while?
Still running 2.3.7.
Looking at the disk IO graphs today's IO usage looks OK!
I restarted dovecot yesterday after fixing the FTS settings: Jul 16 10:14:45 mail-cbf dovecot: master: Dovecot v2.3.7 (494d20bdc) starting up for imap, lmtp (core dumps disabled)
after that I saw quite a bit of IO on sda1 (a bit of data read, lot of write), which subsided at11:30
After that diskio was low, up to the present day.
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | https://www.charite.de
participants (2)
-
Ralf Hildebrandt
-
Timo Sirainen