[Dovecot] File locking problems and duplicate emails with the snarf plugin
Two weeks ago we switched all our IMAP users (around 25,000) from the Washington IMAP server to Dovecot, and since then there have been several occasions when reading new email just hangs, and duplicate emails appear. We are using the 'snarf' plugin to copy new email to the inbox, and it looks as though the problems are caused by file locking. We have been using the Washington IMAP server for a number of years without any file locking problems, so the underlying file locking mechanism is robust. Has anyone else seen this problem ? Our IMAP servers (5 of them) are a mixture of Sparc Solaris 8 and Solaris 10 machines, and they use NFS to a central mailspool running on AIX. We are using a mixture of Dovecot 1.0.3 and 1.0.5.
Any help would be appreciated, as the problem has caused a lot of disruption, but we are reluctant to revert to the Washington IMAP server because of the performance improvements with Dovecot - there has been a reduction of over 60% in NFS accesses to the filestore.
dovecot -n gives:
# 1.0.3: /usr/local/etc/dovecot.conf base_dir: /var/run/dovecot/ listen: *:143 ssl_listen: *:993 ssl_ca_file: /usr/local/ssl/certs/sureserveredu.pem ssl_cert_file: /usr/local/ssl/certs/imap-server_ucl_ac_uk.crt ssl_key_file: /usr/local/ssl/certs/imap.key ssl_cipher_list: MEDIUM:!LOW disable_plaintext_auth: no login_dir: /var/run/dovecot//login login_executable: /usr/local/libexec/dovecot/imap-login login_user: pp login_process_per_connection: no login_processes_count: 30 max_mail_processes: 1500 first_valid_uid: 200 mail_location: mbox:%h/imap-mail:INBOX=/nfs/rcs/mail-spool/mail/%u mmap_disable: yes mbox_lock_timeout: 55 mbox_dotlock_change_timeout: 50 mail_drop_priv_before_exec: yes mail_executable: /usr/local/sbin/dovecot-login.csh mail_plugins: mbox_snarf mail_plugin_dir: /usr/local/lib/dovecot namespace: type: private separator: / inbox: yes auth default: passdb: driver: pam passdb: driver: passwd userdb: driver: passwd plugin: mbox_snarf: ~/imap-mail/mbox
--
Adrian Barker, Internet Technology Section Information Systems University College London, Gower Street, London WC1E 6BT External phone: +44 20 7679 5140, Fax (+44) 20 7388 5406 Internal phone: x 25140 Email: A.Barker@ucl.ac.uk
On Wed, 2007-11-21 at 20:29 +0000, Adrian Barker wrote:
Two weeks ago we switched all our IMAP users (around 25,000) from the Washington IMAP server to Dovecot, and since then there have been several occasions when reading new email just hangs, and duplicate emails appear. We are using the 'snarf' plugin to copy new email to the inbox, and it looks as though the problems are caused by file locking.
How do you deliver mails to the mboxes? If not using Dovecot's deliver, the mbox locking order could be different between them and they'd deadlock. mbox_read/write_locks settings affect this.
We have been using the Washington IMAP server for a number of years without any file locking problems, so the underlying file locking mechanism is robust.
Did UW-IMAP really use fcntl locks with mboxes? Could it be that it just didn't, and now the problem is with fcntl locking in general?
Timo Sirainen wrote:
On Wed, 2007-11-21 at 20:29 +0000, Adrian Barker wrote:
Two weeks ago we switched all our IMAP users (around 25,000) from the Washington IMAP server to Dovecot, and since then there have been several occasions when reading new email just hangs, and duplicate emails appear. We are using the 'snarf' plugin to copy new email to the inbox, and it looks as though the problems are caused by file locking.
How do you deliver mails to the mboxes? If not using Dovecot's deliver, the mbox locking order could be different between them and they'd deadlock. mbox_read/write_locks settings affect this.
We have been using the Washington IMAP server for a number of years without any file locking problems, so the underlying file locking mechanism is robust.
Did UW-IMAP really use fcntl locks with mboxes? Could it be that it just didn't, and now the problem is with fcntl locking in general?
We use Exim to deliver the email, which uses both fcntl and lock files. We cannot use the Dovecot deliver process, as we have a mixture of Unix, POP and IMAP clients. We used the default file locking for Dovecot, but on Friday turned off the use of lock files (after running various file locking tests) and now just use fcntl. This seems to have improved the performance substantially. We will know in a day or two whether the problem is completely solved.
--
Adrian Barker Internet Technology Section Information Systems University College London, Gower Street, London WC1E 6BT External phone: +44 20 7679 5140, Fax (+44) 20 7388 5406 Internal phone: x 25140 Email: A.Barker@ucl.ac.uk
participants (2)
-
Adrian Barker
-
Timo Sirainen