[Dovecot] critical performance problem
I have been using dovecot 1.0alpha4 + deliver/sieve in production for a few accounts (400'000).
After testing 1.0.rc22 I decided to upgrade one of the imap servers...
And... what a mess !
The server just gets unusable. I don't know why but with new release, it just overloads the system with I/O. It turns that with 1.0.rc22 I get 15Mo/s read on disk average, without any reason. What did really change ? How come a system (with a few thousand accounts) works perfectly and then with 1.0.rc22 gets unusable with a load approaching 300 ?
If you have any ideas...
On Tue, 2007-02-20 at 09:53 +0100, Rene Luria wrote:
The server just gets unusable. I don't know why but with new release, it just overloads the system with I/O. It turns that with 1.0.rc22 I get 15Mo/s read on disk average, without any reason. What did really change ? How come a system (with a few thousand accounts) works perfectly and then with 1.0.rc22 gets unusable with a load approaching 300 ?
Show dovecot -n output? What's your operating system? What filesystem? Does it do IMAP, POP3 or both?
Timo Sirainen wrote:
Show dovecot -n output? What's your operating system? What filesystem? Does it do IMAP, POP3 or both?
dovecot -n here above, for the 1.0.rc22, as it's not supporter by 1.0alpha4
it uses both pop3 and imap and, if I need to repeat, same server works nice with 1.0alpha4. Now, I had to downgrade.
# /opt/dovecot/etc/dovecot.conf protocols: imap imaps pop3 pop3s ssl_disable: yes ssl_cert_file: /opt/dovecot/etc/dovecot.pem ssl_key_file: /opt/dovecot/etc/dovecot.pem disable_plaintext_auth: no login_dir: /opt/dovecot/var/run/dovecot/login login_executable(default): /opt/dovecot/libexec/dovecot/imap-login login_executable(imap): /opt/dovecot/libexec/dovecot/imap-login login_executable(pop3): /opt/dovecot/libexec/dovecot/pop3-login login_greeting: Let's read your email. login_process_per_connection: no login_greeting_capability(default): yes login_greeting_capability(imap): yes login_greeting_capability(pop3): no verbose_proctitle: yes first_valid_uid: 8 mail_location: maildir:%h maildir_stat_dirs: yes mail_executable(default): /opt/dovecot/libexec/dovecot/imap mail_executable(imap): /opt/dovecot/libexec/dovecot/imap mail_executable(pop3): /opt/dovecot/libexec/dovecot/pop3 mail_plugins(default): quota imap_quota mail_plugins(imap): quota imap_quota mail_plugins(pop3): quota mail_plugin_dir(default): /opt/dovecot/lib/dovecot/imap mail_plugin_dir(imap): /opt/dovecot/lib/dovecot/imap mail_plugin_dir(pop3): /opt/dovecot/lib/dovecot/pop3 pop3_uidl_format(default): pop3_uidl_format(imap): pop3_uidl_format(pop3): %f namespace: type: private separator: / inbox: yes auth default: mechanisms: plain login digest-md5 cram-md5 apop username_chars: abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@& username_translation: %@ passdb: driver: ldap args: /opt/dovecot/etc/dovecot-ldap.conf userdb: driver: ldap args: /opt/dovecot/etc/dovecot-ldap.conf socket: type: listen client: master: path: /opt/dovecot/var/run/dovecot/auth-master mode: 438
And to get the right picture: are mailbox indexes compatible between 1.0alpha4 and 1.0.rc22 ? Because it just seems that it recomputes every index file for every pop3 or imap connection
On Tue, 2007-02-20 at 11:02 +0100, Rene Luria wrote:
And to get the right picture: are mailbox indexes compatible between 1.0alpha4 and 1.0.rc22 ? Because it just seems that it recomputes every index file for every pop3 or imap connection
They should be compatible. And even if they weren't it shouldn't recompute it for every connection.
Looks like dovecot -n output doesn't show the plugin section. Do you use maildir++ quota?
I don't see anything special in the configuration, and I don't think there are any generic major problems in rc22, so it's probably somehow specific to your system. Or maybe because you're upgrading from alpha4. I guess it's possible that rc22 somehow messes up because the dovecot.index* files are somewhat different than it likes.
Anyway there have been so many changes since alpha4 (it's almost 1,5 years old) that it's pretty much impossible to guess what could be the problem without knowing what is causing the I/O..
On Tue, 2007-02-20 at 12:13 +0200, Timo Sirainen wrote:
And to get the right picture: are mailbox indexes compatible between 1.0alpha4 and 1.0.rc22 ? Because it just seems that it recomputes every index file for every pop3 or imap connection
They should be compatible. And even if they weren't it shouldn't recompute it for every connection.
Just tried installing alpha4 and then upgrading to rc22. rc22 reads alpha4's index files just fine, although cache file got truncated (that should happen only with 64bit systems).
Timo Sirainen wrote:
On Tue, 2007-02-20 at 12:13 +0200, Timo Sirainen wrote:
And to get the right picture: are mailbox indexes compatible between 1.0alpha4 and 1.0.rc22 ? Because it just seems that it recomputes every index file for every pop3 or imap connection They should be compatible. And even if they weren't it shouldn't recompute it for every connection.
Just tried installing alpha4 and then upgrading to rc22. rc22 reads alpha4's index files just fine, although cache file got truncated (that should happen only with 64bit systems).
As I began this thread, in terms of operations or functionalities, the upgrade is transparent. Everything works.
Moving to 1.0.rc22 makes load raise up to 200 or more Downgrading back to 1.0alpha4 gets the load back to normal (after a few hours)
On Tue, 2007-02-20 at 11:41 +0100, Rene Luria wrote:
Just tried installing alpha4 and then upgrading to rc22. rc22 reads alpha4's index files just fine, although cache file got truncated (that should happen only with 64bit systems).
As I began this thread, in terms of operations or functionalities, the upgrade is transparent. Everything works.
Moving to 1.0.rc22 makes load raise up to 200 or more Downgrading back to 1.0alpha4 gets the load back to normal (after a few hours)
Well, since I'm out of ideas I can't do anything about it without knowing more specifically what is causing the problem.
One thing you could try is what happens to load if you add :INDEX=MEMORY to default_mail_env in alpha4. If it doesn't raise the load too high (if you're not using webmail it shouldn't) try upgrading to rc22 with the :INDEX=MEMORY also and see if the load stays the same or if it's still raising to 200. If it's the same then the problem is clearly related to index files.
Or you could try doing some test rc22 installation running in a different port, then try with some test users if you can figure out (eg. with strace) what it's doing wrong (constantly rebuilding index files?).
The problem might also be related to deliver and not to imap/pop3 at all. You could try running the old deliver with its own config file and :INDEX=MEMORY and upgrade the rest of the Dovecot to see if that makes a difference.
Ok, everything is about indexes.
What happens, as I thought, is that indexes get recomputed. Once per account/login, of course, but with a high loaded system it just make the situation unbearable in normal use.
There is then nothing wrong about upgrading, in terms of performance. It's just that a high load is to be expected 'till indexes have been recreated. (erasing them before won't help that much)
The only way to mitigate it in some way is to prevent concurrent intents to recreate indexes. pop3_lock_session can help then.
Thank you for your answers.
p.s. on a little less loaded server, load rised but then situation got back to normal after an hour.
On Tue, 2007-02-20 at 12:27 +0100, Rene Luria wrote:
What happens, as I thought, is that indexes get recomputed. Once per account/login, of course, but with a high loaded system it just make the situation unbearable in normal use.
Oh, okay. Do you use 64bit system? I'm not sure why it would have rebuilt anything with 32bit systems.
Timo Sirainen wrote:
On Tue, 2007-02-20 at 12:27 +0100, Rene Luria wrote:
What happens, as I thought, is that indexes get recomputed. Once per account/login, of course, but with a high loaded system it just make the situation unbearable in normal use.
Oh, okay. Do you use 64bit system? I'm not sure why it would have rebuilt anything with 32bit systems.
nope, it's a xeon 3ghz
now, locking session is the real fix. otherwise, stracing two concurrent pop3 sessions on the same account showed every message was read (to rebuild the index probably)
On Tue, Feb 20, 2007 at 12:27:04PM +0100, Rene Luria wrote:
Ok, everything is about indexes.
What happens, as I thought, is that indexes get recomputed. Once per account/login, of course, but with a high loaded system it just make the situation unbearable in normal use.
There is then nothing wrong about upgrading, in terms of performance. It's just that a high load is to be expected 'till indexes have been recreated. (erasing them before won't help that much)
Is there a way to automatically (re)create all indexes, e.g. after an upgrade or a mailstore migration?
GeertThe migration from alpha to the latest should include a step where you delete all old caches: find . -name \.imap -exec rm -rf {} \;
-- Leroy
On Tue 20 Feb 2007 at 11:22AM, Leroy van Logchem wrote:
The migration from alpha to the latest should include a step where you delete all old caches: find . -name \.imap -exec rm -rf {} \;
If that is actually true, then the daemon should be taught to do this itself-- presumably these files are versioned?
-dp-- Daniel Price - Solaris Kernel Engineering - dp@eng.sun.com - blogs.sun.com/dp
On Tue, 2007-02-20 at 10:12 -0800, Dan Price wrote:
The migration from alpha to the latest should include a step where you delete all old caches: find . -name \.imap -exec rm -rf {} \;
If that is actually true, then the daemon should be taught to do this itself-- presumably these files are versioned?
They are versioned (we're at v7.0), and as far as I know there's no need to do that. If someone can show me when/why it's needed, it's a bug and I'll fix it.
On Tue, 2007-02-20 at 09:53 +0100, Rene Luria wrote:
I have been using dovecot 1.0alpha4 + deliver/sieve in production for a few accounts (400'000).
How did you upgrade deliver/sieve? I guess you were before using the separate deliver binary from dovecot-sieve CVS tree. But did you upgrade to Dovecot's own deliver with cmusieve plugin? (again looks like dovecot -n doesn't print protocol lda {}..)
Timo Sirainen wrote:
On Tue, 2007-02-20 at 09:53 +0100, Rene Luria wrote:
I have been using dovecot 1.0alpha4 + deliver/sieve in production for a few accounts (400'000).
How did you upgrade deliver/sieve? I guess you were before using the separate deliver binary from dovecot-sieve CVS tree. But did you upgrade to Dovecot's own deliver with cmusieve plugin? (again looks like dovecot -n doesn't print protocol lda {}..)
that's right, from dovecot-sieve cvs to dovecot and cmusieve plugin and everything works, it's only about perf
participants (5)
-
Dan Price
-
Geert Hendrickx
-
Leroy van Logchem
-
Rene Luria
-
Timo Sirainen