[Dovecot] Investigating suspected cache corruption
Hi All,
I'm running Dovecot 1.1.11 for a site of about 7000 active users with
a Squirrelmail frontend. We migrated from Courier (big shout to
everyone who worked on the wonderful part of the Dovecot wiki that
deals with Courier migration) and I have been extremely happy with the
performance gains Dovecot has given us. Not only is the IMAP part of
Squirrelmail faster, but our same hardware is under far less stress
than it was before the migration.
To date, the only problem we have experienced is a seemingly random
but significant slowdown in Squirrelmail's loading of the messages in
a user's Inbox. Each time, I have been able to fix the problem by
deleting the user's dovecot.index, dovecot.index.cache and
dovecot.index.log files in the user's INDEX directory for their
dovecot Inbox.
I tried having users experiencing this problem log in to a test
Squirrelmail box with mail_debug on, but I didn't see anything out of
the ordinary and they report the same slowness on the test server. I
don't see anything but normal login/logout IMAP lines in the
production dovecot logs.
I have hesitated a bit in posting to the list since this seems to be
happening to a small percentage of users and it technically may be a
problem with Squirrelmail, but the fact that I'm able to fix the
problem by nuking Dovecot's index files has me concerned - does anyone
have any suggestions for troubleshooting? My first idea when I get
another one of these is to do a telnet IMAP session to Dovecot on one
of the Squirrelmail boxes and see if I can get it to slow down when
retrieving messages from the Inbox.
We are using Maildir over NFS with the control/index files stored on
NFS as well. We're doing FS quota over NFS using rquotad to query the
NFS server for each user's quota. Here's the output of dovecot -n:
# 1.1.11: /etc/dovecot.conf
Warning: fd limit 1024 is lower than what Dovecot can use under full
load (more than 1536). Either grow the limit or change
login_max_processes_count and max_mail_processes settings
# OS: Linux 2.6.18-128.1.1.0.1.el5 x86_64 Red Hat Enterprise Linux
Server release 5.3 (Tikanga)
login_dir: /var/run/dovecot/login
login_executable(default): /usr/libexec/dovecot/imap-login
login_executable(imap): /usr/libexec/dovecot/imap-login
login_executable(pop3): /usr/libexec/dovecot/pop3-login
login_process_per_connection: no
login_processes_count: 10
login_max_processes_count: 512
verbose_proctitle: yes
first_valid_uid: 0
mail_location: maildir:~/Maildir:CONTROL=/Mail/mailhome/
%u/.dovecot:INDEX=/Mail/mailhome/%u/.dovecot
mmap_disable: yes
mail_nfs_storage: yes
mail_nfs_index: yes
mail_drop_priv_before_exec: yes
mail_executable(default): /usr/libexec/dovecot/imap
mail_executable(imap): /usr/libexec/dovecot/imap
mail_executable(pop3): /usr/libexec/dovecot/pop3
mail_plugins(default): quota imap_quota
mail_plugins(imap): quota imap_quota
mail_plugins(pop3):
mail_plugin_dir(default): /usr/lib64/dovecot/imap
mail_plugin_dir(imap): /usr/lib64/dovecot/imap
mail_plugin_dir(pop3): /usr/lib64/dovecot/pop3
imap_client_workarounds(default): outlook-idle delay-newmail
imap_client_workarounds(imap): outlook-idle delay-newmail
imap_client_workarounds(pop3):
imap_logout_format(default): bytes(in/out)=%i/%o
imap_logout_format(imap): bytes(in/out)=%i/%o
imap_logout_format(pop3): bytes=%i/%o
pop3_uidl_format(default): %08Xu%08Xv
pop3_uidl_format(imap): %08Xu%08Xv
pop3_uidl_format(pop3): UID%u-%v
pop3_client_workarounds(default):
pop3_client_workarounds(imap):
pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh
pop3_logout_format(default): top=%t/%p, retr=%r/%b, del=%d/%m, size=%s
pop3_logout_format(imap): top=%t/%p, retr=%r/%b, del=%d/%m, size=%s
pop3_logout_format(pop3): top(count/bytes)=%t/%p, retr(count/bytes)=%r/
%b, del(deleted/pre-deleted)=%d/%m, mailbox-size(bytes)=%s
namespace:
type: private
prefix: INBOX.
inbox: yes
list: yes
subscriptions: yes
auth default:
mechanisms: plain login
passdb:
driver: pam
userdb:
driver: passwd
plugin:
quota: fs:Mail Quota:user
Many thanks, David Warden
On Wed, 2009-10-14 at 16:17 -0400, David Warden wrote:
To date, the only problem we have experienced is a seemingly random
but significant slowdown in Squirrelmail's loading of the messages in
a user's Inbox. Each time, I have been able to fix the problem by
deleting the user's dovecot.index, dovecot.index.cache and
dovecot.index.log files in the user's INDEX directory for their
dovecot Inbox.
First thing to try next time: Does it help if you only delete dovecot.index.cache file? How large is the cache file? And take a copy of the dovecot.index* files before doing that, I can then ask a few more questions about them.
I tried having users experiencing this problem log in to a test
Squirrelmail box with mail_debug on, but I didn't see anything out of
the ordinary and they report the same slowness on the test server. I
don't see anything but normal login/logout IMAP lines in the
production dovecot logs.
mail_debug doesn't really help with performance debugging.
My first idea when I get
another one of these is to do a telnet IMAP session to Dovecot on one
of the Squirrelmail boxes and see if I can get it to slow down when
retrieving messages from the Inbox.
That sounds like a good idea to try. And use the exact same IMAP commands as Squirrelmail to do it. strace -tt output of of the slow imap process could also show something interesting.
participants (2)
-
David Warden
-
Timo Sirainen