[Dovecot] Stale NFS file handle and duplicated messages
We recently moved all our users from the Washington IMAP server to Dovecot. There has been a substantial improvement in performance, but some users are reporting duplicate messages, and we see 'Stale NFS file handle' errors in the syslog. Does anyone know what would cause this ? The errors relate to the index file, eg.
Nov 29 00:49:47 imap-e.ucl.ac.uk dovecot: [ID 107833 mail.error] IMAP(ccaazzz): close() failed with index file /home/ccaazzz/imap-mail/.imap/mbox/dovecot.index: Stale NFS file handle
Our IMAP servers (5 of them) are a mixture of Sparc Solaris 8 and Solaris 10 machines, using NFS to a central mailspool running on AIX. We are using a mixture of Dovecot 1.0.3 and 1.0.5.
dovecot -n gives:
# 1.0.3: /usr/local/etc/dovecot.conf base_dir: /var/run/dovecot/ listen: *:143 ssl_listen: *:993 ssl_ca_file: /usr/local/ssl/certs/sureserveredu.pem ssl_cert_file: /usr/local/ssl/certs/imap-server_ucl_ac_uk.crt ssl_key_file: /usr/local/ssl/certs/imap.key ssl_cipher_list: MEDIUM:!LOW disable_plaintext_auth: no login_dir: /var/run/dovecot//login login_executable: /usr/local/libexec/dovecot/imap-login login_user: pp login_process_per_connection: no login_processes_count: 30 max_mail_processes: 1500 first_valid_uid: 200 mail_location: mbox:%h/imap-mail:INBOX=/nfs/rcs/mail-spool/mail/%u mmap_disable: yes mbox_lock_timeout: 55 mbox_dotlock_change_timeout: 50 mail_drop_priv_before_exec: yes mail_executable: /usr/local/sbin/dovecot-login.csh mail_plugins: mbox_snarf mail_plugin_dir: /usr/local/lib/dovecot namespace: type: private separator: / inbox: yes auth default: passdb: driver: pam passdb: driver: passwd userdb: driver: passwd plugin: mbox_snarf: ~/imap-mail/mbox
--
Adrian Barker, Internet Technology Section Information Systems University College London, Gower Street, London WC1E 6BT External phone: +44 20 7679 5140, Fax (+44) 20 7388 5406 Internal phone: x 25140 Email: A.Barker@ucl.ac.uk
On Sat, 2007-12-08 at 10:21 +0000, Adrian Barker wrote:
We recently moved all our users from the Washington IMAP server to Dovecot. There has been a substantial improvement in performance, but some users are reporting duplicate messages, and we see 'Stale NFS file handle' errors in the syslog. Does anyone know what would cause this ? The errors relate to the index file, eg.
Nov 29 00:49:47 imap-e.ucl.ac.uk dovecot: [ID 107833 mail.error] IMAP(ccaazzz): close() failed with index file /home/ccaazzz/imap-mail/.imap/mbox/dovecot.index: Stale NFS file handle
Our IMAP servers (5 of them) are a mixture of Sparc Solaris 8 and Solaris 10 machines, using NFS to a central mailspool running on AIX. We are using a mixture of Dovecot 1.0.3 and 1.0.5.
Have you read http://wiki.dovecot.org/NFS? v1.0 still has some problems with NFS if you try to access the mailboxes from multiple servers at the same time. I've done a lot of work on v1.1 to get this working right.
I'd suggest either
a) Move index files to local disks
b) Try how v1.1 works with mail_nfs_*=yes settings. I'm going to make the first v1.1 "release candidate" soon, maybe within a few days. Beta10 still has one performance issue unfixed, so might not be a good idea to use that yet. :)
Timo Sirainen wrote:
On Sat, 2007-12-08 at 10:21 +0000, Adrian Barker wrote:
We recently moved all our users from the Washington IMAP server to Dovecot. There has been a substantial improvement in performance, but some users are reporting duplicate messages, and we see 'Stale NFS file handle' errors in the syslog. Does anyone know what would cause this ? The errors relate to the index file, eg.
Nov 29 00:49:47 imap-e.ucl.ac.uk dovecot: [ID 107833 mail.error] IMAP(ccaazzz): close() failed with index file /home/ccaazzz/imap-mail/.imap/mbox/dovecot.index: Stale NFS file handle
Our IMAP servers (5 of them) are a mixture of Sparc Solaris 8 and Solaris 10 machines, using NFS to a central mailspool running on AIX. We are using a mixture of Dovecot 1.0.3 and 1.0.5.
Have you read http://wiki.dovecot.org/NFS? v1.0 still has some problems with NFS if you try to access the mailboxes from multiple servers at the same time. I've done a lot of work on v1.1 to get this working right.
I'd suggest either
a) Move index files to local disks
b) Try how v1.1 works with mail_nfs_*=yes settings. I'm going to make the first v1.1 "release candidate" soon, maybe within a few days. Beta10 still has one performance issue unfixed, so might not be a good idea to use that yet. :)
Thanks for replying. I have read the wiki page on using NFS. In general, our users should only be accessing a single IMAP server at any one time
- we have a hardware load-balancer in front of the IMAP servers, but it is possible that some have IMAP clients on multiple machines that are left running. This can be checked from the logs. Would keeping the index files locally on the IMAP servers have an impact on performance ? Before moving to Dovecot, IMAP performance was a very fraught issue, and the benefits of Dovecot have been substantial - the load on some of our file servers is 1/4 of what it was before the change, so we are very grateful for all the work that has gone into Dovecot. Unless the problem gets worse, we can probably wait until 1.1 is released.
--
Adrian Barker, Internet Technology Section Information Systems University College London, Gower Street, London WC1E 6BT External phone: +44 20 7679 5140, Fax (+44) 20 7388 5406 Internal phone: x 25140 Email: A.Barker@ucl.ac.uk
On Sat, 2007-12-08 at 11:29 +0000, Adrian Barker wrote:
Nov 29 00:49:47 imap-e.ucl.ac.uk dovecot: [ID 107833 mail.error] IMAP(ccaazzz): close() failed with index file /home/ccaazzz/imap-mail/.imap/mbox/dovecot.index: Stale NFS file handle .. Thanks for replying. I have read the wiki page on using NFS. In general, our users should only be accessing a single IMAP server at any one time
- we have a hardware load-balancer in front of the IMAP servers, but it is possible that some have IMAP clients on multiple machines that are left running. This can be checked from the logs.
The "Stale NFS file handle" means that another server had deleted the file while this server was still trying to read/write it. So every time it happens there must have been two servers accessing the mailbox at the same time.
Would keeping the index files locally on the IMAP servers have an impact on performance ?
If users are usually sent to the same server, it should only improve the performance. If users are randomly sent to different servers then the indexes will have to be updated in multiple servers, which probably makes the performance worse.
participants (2)
-
Adrian Barker
-
Timo Sirainen