[Dovecot] migration 1.2.5 -> 2.0.5 load increased dramatically
During norml operation this morning we're seeing a significantly increased load on the IMAP server (factor 5, from 5 to 25).
Does dovecot need to rebuild index files or perform other IO intensive operations after the upgrade?
We're using Maildir
Our monitoring shows increased "system" time. We're runnign fewer processes, the IO rate ist identical to yesterday, memory usage is identical.
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | http://www.charite.de
- Ralf Hildebrandt Ralf.Hildebrandt@charite.de:
During norml operation this morning we're seeing a significantly increased load on the IMAP server (factor 5, from 5 to 25).
Does dovecot need to rebuild index files or perform other IO intensive operations after the upgrade?
We're using Maildir
Our monitoring shows increased "system" time. We're runnign fewer processes, the IO rate ist identical to yesterday, memory usage is identical.
I found the reason: My excessive number of ca-certificates!
dovecot-lda invoke doveconf, which in turn needs a whooping 1.2s wallclock time to parse the config!
I pruned the ca-certificates file and removed ALL ca-certificates but the ones that belong to the certificate chain for postamt.charite.de
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | http://www.charite.de
On 06/10/2010 13:16, Ralf Hildebrandt wrote:
- Ralf HildebrandtRalf.Hildebrandt@charite.de:
During norml operation this morning we're seeing a significantly increased load on the IMAP server (factor 5, from 5 to 25).
Does dovecot need to rebuild index files or perform other IO intensive operations after the upgrade?
We're using Maildir
Our monitoring shows increased "system" time. We're runnign fewer processes, the IO rate ist identical to yesterday, memory usage is identical.
I found the reason: My excessive number of ca-certificates!
dovecot-lda invoke doveconf, which in turn needs a whooping 1.2s wallclock time to parse the config!
I pruned the ca-certificates file and removed ALL ca-certificates but the ones that belong to the certificate chain for postamt.charite.de
Hello Ralf,
Glad to see you found out what the problem was.
I was just wondering if this episode suggested any changes which may be helpful. I am not an expert on the issue you discovered, so I came up with some questions which might help guide us.
Do you feel that the problem was more that you were feeding Dovecot an unusually large certificate pack, or that your certificate pack was reasonable and Dovecot should be able to cope better with packs of this size?
Should we write something in the Dovecot wiki about it?
Should we need to suggest an enhancement to Dovecot?
Regards,
Bill
- William Blunn bill+dovecot@blunn.org:
Glad to see you found out what the problem was.
Turns out I was wrong! This needs more analysis.
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | http://www.charite.de
- Ralf Hildebrandt Ralf.Hildebrandt@charite.de:
- Ralf Hildebrandt Ralf.Hildebrandt@charite.de:
During norml operation this morning we're seeing a significantly increased load on the IMAP server (factor 5, from 5 to 25).
Does dovecot need to rebuild index files or perform other IO intensive operations after the upgrade?
We're using Maildir
Our monitoring shows increased "system" time. We're runnign fewer processes, the IO rate ist identical to yesterday, memory usage is identical.
I found the reason: My excessive number of ca-certificates!
At least I think I did :(
dovecot-lda still invokes doveconf, which still takes about 1.5s wallclock time.
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | http://www.charite.de
- Ralf Hildebrandt Ralf.Hildebrandt@charite.de:
dovecot-lda invoke doveconf, which in turn needs a whooping 1.2s wallclock time to parse the config!
strace -c /usr/dovecot-2/bin/doveconf
reports:
% time seconds usecs/call calls errors syscall
91.02 0.504031 504031 1 wait4 3.61 0.020001 833 24 brk 2.17 0.012001 12001 1 execve 1.44 0.008001 8001 1 clone 0.80 0.004443 114 39 27 open 0.69 0.003833 348 11 read 0.27 0.001471 134 11 fstat64 0.00 0.000000 0 60 write 0.00 0.000000 0 17 close ...
So what is doveconf waiting for?
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | http://www.charite.de
- Timo Sirainen tss@iki.fi:
On Wed, 2010-10-06 at 14:58 +0200, Ralf Hildebrandt wrote:
% time seconds usecs/call calls errors syscall
91.02 0.504031 504031 1 wait4 So what is doveconf waiting for?
See if it's faster when disabling Pigeonhole's settings plugin? (Remove /usr/local/lib/dovecot/settings/*)
Yes, it's MUCH faster:
# strace -c /usr/dovecot-2/bin/doveconf
# 2.0.5: /usr/dovecot-2/etc/dovecot/dovecot.conf
doveconf: Fatal: Error in configuration file /usr/dovecot-2/etc/dovecot/dovecot.conf line 11: Unknown setting: managesieve_notify_capability
% time seconds usecs/call calls errors syscall
32.24 0.004000 4000 1 execve 23.13 0.002870 2870 1 pipe 16.10 0.001997 333 6 6 access 14.52 0.001801 164 11 fcntl64 10.08 0.001251 156 8 fstat64 2.37 0.000294 59 5 read 0.91 0.000113 4 27 19 open 0.64 0.000080 5 15 15 stat64 0.00 0.000000 0 2 write 0.00 0.000000 0 8 close ...
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | http://www.charite.de
On Wed, 2010-10-06 at 15:56 +0200, Ralf Hildebrandt wrote:
Yes, it's MUCH faster:
# strace -c /usr/dovecot-2/bin/doveconf
# 2.0.5: /usr/dovecot-2/etc/dovecot/dovecot.conf doveconf: Fatal: Error in configuration file /usr/dovecot-2/etc/dovecot/dovecot.conf line 11: Unknown setting: managesieve_notify_capability
The improvement could be simply from it failing too early because of this error. :)
On Wed, 2010-10-06 at 14:49 +0100, Timo Sirainen wrote:
On Wed, 2010-10-06 at 14:58 +0200, Ralf Hildebrandt wrote:
% time seconds usecs/call calls errors syscall
91.02 0.504031 504031 1 wait4 So what is doveconf waiting for?
See if it's faster when disabling Pigeonhole's settings plugin? (Remove /usr/local/lib/dovecot/settings/*)
Oh, actually that shouldn't be the problem. A bit strange why it would take 0.5 secs there, but when calling via dovecot-lda it shouldn't do this at all. You should be testing with:
doveconf -m lda
Maybe easiest if you make a copy of your config and keep deleting stuff from it until doveconf runs fast so you can find what settings exactly are causing it. Or is it slow even with a zero byte config file?
- Timo Sirainen tss@iki.fi:
Oh, actually that shouldn't be the problem. A bit strange why it would take 0.5 secs there, but when calling via dovecot-lda it shouldn't do this at all. You should be testing with:
doveconf -m lda
Should I split my config file in different config file (or does dovecot always parse them all)?
Maybe easiest if you make a copy of your config and keep deleting stuff from it until doveconf runs fast so you can find what settings exactly are causing it. Or is it slow even with a zero byte config file?
time doveconf -m lda takes about
real 0m0.231s user 0m0.008s sys 0m0.220s
sometimes: real 0m0.652s user 0m0.012s sys 0m0.304s
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | http://www.charite.de
On Wed, 2010-10-06 at 16:14 +0200, Ralf Hildebrandt wrote:
- Timo Sirainen tss@iki.fi:
Oh, actually that shouldn't be the problem. A bit strange why it would take 0.5 secs there, but when calling via dovecot-lda it shouldn't do this at all. You should be testing with:
doveconf -m lda
Should I split my config file in different config file (or does dovecot always parse them all)?
I'm not really sure what you mean. It starts from the dovecot.conf and reads and parses everything that gets included there. For testing you could just do:
doveconf > dovetest.conf
And then try with:
time doveconf -m lda -c dovetest.conf > /dev/null
And keep removing stuff until it finishes quickly (without giving any errors).
time doveconf -m lda takes about
real 0m0.231s
That's a long time. It takes 0.02 in my somewhat old machine.
- Timo Sirainen tss@iki.fi:
On Wed, 2010-10-06 at 16:14 +0200, Ralf Hildebrandt wrote:
- Timo Sirainen tss@iki.fi:
Oh, actually that shouldn't be the problem. A bit strange why it would take 0.5 secs there, but when calling via dovecot-lda it shouldn't do this at all. You should be testing with:
doveconf -m lda
Should I split my config file in different config file (or does dovecot always parse them all)?
I'm not really sure what you mean. It starts from the dovecot.conf and reads and parses everything that gets included there. For testing you could just do:
doveconf > dovetest.conf
doveconf -n > dovetest.conf maybe? (it's much shorter)
Or do you think a default setting would bite me?
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | http://www.charite.de
On Wed, 2010-10-06 at 16:52 +0200, Ralf Hildebrandt wrote:
I'm not really sure what you mean. It starts from the dovecot.conf and reads and parses everything that gets included there. For testing you could just do:
doveconf > dovetest.conf
doveconf -n > dovetest.conf maybe? (it's much shorter)
Yeah.
Or do you think a default setting would bite me?
No.
participants (3)
-
Ralf Hildebrandt
-
Timo Sirainen
-
William Blunn