[Dovecot] constant mailbox rebuilding with dovecot 1.0.10
I've recently upgraded to 1.0.10 in hopes that this problem would go away but no luck. I've got some users at my company with large inboxes on maildir (say 22,000 messages). Their mailboxes are hosted on nfs and dovecot is configured to do dotlocking. Still I see messages like this continually in the logs (sometimes every hour or so):
server dovecot: IMAP(user): dotlock /home/user/Maildir/dovecot.index.log.lock was immediately recreated under us server dovecot: IMAP(user): file_dotlock_create() failed with file /home/user/Maildir/dovecot.index.log: No such file or directory server dovecot: IMAP(user): Our dotlock file /home/user/Maildir/dovecot-uidlist.lock was overridden (kept it 0 secs) server dovecot: IMAP(user): Duplicate file in uidlist file /home/user/Maildir/dovecot-uidlist: 1162416499.010493.mbox:2,Sa server dovecot: IMAP(user): Maildir /home/user/Maildir sync: UID inserted in the middle of mailbox (48573 > 1, file = 1162416499.010493.mbox:2,Sa)
or this:
server dovecot: IMAP(user): dotlock /home/user/Maildir/dovecot.index.log.lock was immediately recreated under us server dovecot: IMAP(user): file_dotlock_create() failed with file /home/user/Maildir/dovecot.index.log: No such file or directory server dovecot: IMAP(user): Duplicate file in uidlist file /home/user/Maildir/dovecot-uidlist: 1167674195.83666_2.tandoori.schrodinger.com:2,Sklwv server dovecot: IMAP(user): Maildir /home/user/Maildir sync: UID inserted in the middle of mailbox (34969 > 1, file = 1167674195.83666_2.server.com:2,Sklwv) server dovecot: IMAP(user): Corrupted transaction log file /home/user/Maildir/dovecot.index.log.2: invalid indexid (1204046496 != 1204055346)
it seems that every time this sort of thing happens the mail client (thunderbird) sits and rebuilds it's mailbox info for as long as 20-30 minutes, during which time it is unusable. I suspect this happens with other users as well but since they have smaller inboxes they don't notice the long delays.
Also this is not linked to version 1.0.10, I've been seeing on and off with the previous version (1.0rc17).
Here's the 'dovecot -n' output. I should add that the default is for users to use mbox but some users (like the above one) use maildir via a separate dovecot.passwd file. Server is FreeBSD 4.7.
# 1.0.10: /usr/local/etc/dovecot.conf base_dir: /var/run/dovecot/ protocols: imaps ssl_cert_file: /usr/local/openssl/certs/server.crt ssl_key_file: /usr/local/openssl/private/server.key login_dir: /var/run/dovecot//login login_executable: /usr/local/libexec/dovecot/imap-login login_greeting_capability: yes verbose_proctitle: yes first_valid_uid: 700 last_valid_uid: 9999 first_valid_gid: 100 last_valid_gid: 200 mail_extra_groups: mail default_mail_env: mbox:%h/mail:INBOX=/var/mail/%u mail_location: mbox:%h/mail:INBOX=/var/mail/%u mmap_disable: yes lock_method: dotlock mbox_read_locks: dotlock mbox_write_locks: dotlock imap_client_workarounds: delay-newmail outlook-idle netscape-eoh tb-extra-mailbox-sep auth default: passdb: driver: pam passdb: driver: passwd-file args: /usr/local/etc/dovecot.passwd passdb: driver: passwd userdb: driver: passwd-file args: /usr/local/etc/dovecot.passwd userdb: driver: passwd
-- Phil Hollenback hollenba@schrodinger.com Schrodinger, Inc.
On Tue, 2008-02-26 at 15:56 -0500, Philip J. Hollenback wrote:
I've recently upgraded to 1.0.10 in hopes that this problem would go away but no luck. I've got some users at my company with large inboxes on maildir (say 22,000 messages). Their mailboxes are hosted on nfs and dovecot is configured to do dotlocking.
Are there multiple Dovecot servers accessing the mailboxes? Can the same user have different connections to different servers at the same time?
This sounds like NFS caching problems that are described in http://wiki.dovecot.org/NFS
On 02/26/08, Timo Sirainen wrote:
On Tue, 2008-02-26 at 15:56 -0500, Philip J. Hollenback wrote:
I've recently upgraded to 1.0.10 in hopes that this problem would go away but no luck. I've got some users at my company with large inboxes on maildir (say 22,000 messages). Their mailboxes are hosted on nfs and dovecot is configured to do dotlocking.
Are there multiple Dovecot servers accessing the mailboxes? Can the same user have different connections to different servers at the same time?
This sounds like NFS caching problems that are described in http://wiki.dovecot.org/NFS
The real problem seemed to be when users run multiple imap clients and they stomp on each others' index files on the nfs filesystem.
I believe I have this fixed. I should clarify that there is only one dovecot server which in turn nfs-mounts the mailbox filesystem from a netapp. I realized after reading all the dovecot nfs documentation that I could point the INDEX variable to a local filesystem on the dovecot server. I did that with the following dovecot.conf default_mail_env:
default_mail_env = mbox:%h/mail:INBOX=/var/mail/%u:INDEX=/opt/dcache/%u
because the default at my company is mbox. Then for each maildir user I set a dovecot.passwd entry as follows:
user:XXXXX:501:100::/home/user:/bin/false::Maildir:/home/%u/Maildir:INDEX=/opt/dcache/%u
finally I set my dovecot.conf locking/nfs options as follows:
mmap_disable=yes dotlock_use_excl = no lock_method = fcntl mbox_read_locks = dotlock mbox_write_locks = dotlock
since I'm using dovecot 1.0 I believe this is the correct way to do dotlocking on the mbox files and fcntl locking on the index files (which remember are on the local filesystem under /opt/dcache).
Since I've made this change I've seen no more corrupted index files.
P.
-- Phil Hollenback hollenba@schrodinger.com Schrodinger, Inc.
On Tue, 2008-02-26 at 15:56 -0500, Philip J. Hollenback wrote:
server dovecot: IMAP(user): Our dotlock file /home/user/Maildir/dovecot-uidlist.lock was overridden (kept it 0 secs) .. server dovecot: IMAP(user): dotlock /home/user/Maildir/dovecot.index.log.lock was immediately recreated under us
These could also mean that your clocks aren't synchronized.
participants (2)
-
Philip J. Hollenback
-
Timo Sirainen