[Dovecot] nfs locking issues...
I'm running Dovecot 1.1 RC1. I believe I've done all the due diligence for making things working correctly over nfs. But I run into locking issues if I run over nfs.
procmail is doing the delivery over nfs. uw-imap was ruining over nfs.
dovecot is fine if its on the nfs server (i.e. it has local access to
the disk, no nfs)
I run into lock deadlocks if I run dovecot over nfs
Users are directed to different servers, but a given users will always
get the same server (at least until there's no activity for that user
for over an hour). This only applies to when running dovecot over nfs.
Everybody goes to the nfs server otherwise.
Sometimes I can kill all the processes, remove the dot locks, and recover. Something I need to reboot the nfs server to recover.
Let me give a run down of the particulars:
All the mail is in mbox format. everybody runs linux with kernel 2.6.24.2 (any known nfs locking issues there?). delivery is done via procmail, its locking and dovecot's are both dotlock, fcntl. The nfs mount options are actimeo=3,hard,noatime,rsize=32768,wsize=32768,nfsvers=3 The clocks are in sync.
Dovecot.conf: protocols = imap imaps pop3 pop3s login_greeting = imap ready. mail_location = mbox:~/:INBOX=/var/mail/%u mmap_disable = yes mail_nfs_storage = yes mail_nfs_index = yes protocol imap { } protocol pop3 { pop3_uidl_format = %08Xu%08Xv } auth default { mechanisms = plain passdb pam { } userdb passwd { } user = root } dict { } plugin { }
Any help would be greatly appreciated.
Thanks, John
-- John Gray gray@agora-net.com AgoraNet, Inc. (302) 224-2475 314 E. Main Street, Suite 1 (302) 224-2552 (fax) Newark, De 19711 http://www.agora-net.com
A few questions...
- What are you running as the underlying FS for your NFS share?
- What are you using for the NFS server?
- Do you see any messages about lockd in your messages file?
- When you strace a deadlocked process, what do you see?
- Do these locks appear to deadlock on the same machine, or is one lock on one machine and another lock on another machine?
-Aaron
On Tue, Feb 26, 2008 at 11:03 AM, John Gray gray@agora-net.com wrote:
I'm running Dovecot 1.1 RC1. I believe I've done all the due diligence for making things working correctly over nfs. But I run into locking issues if I run over nfs.
procmail is doing the delivery over nfs. uw-imap was ruining over nfs. dovecot is fine if its on the nfs server (i.e. it has local access to the disk, no nfs) I run into lock deadlocks if I run dovecot over nfs Users are directed to different servers, but a given users will always get the same server (at least until there's no activity for that user for over an hour). This only applies to when running dovecot over nfs. Everybody goes to the nfs server otherwise.
Sometimes I can kill all the processes, remove the dot locks, and recover. Something I need to reboot the nfs server to recover.
Let me give a run down of the particulars:
All the mail is in mbox format. everybody runs linux with kernel 2.6.24.2 (any known nfs locking issues there?). delivery is done via procmail, its locking and dovecot's are both dotlock, fcntl. The nfs mount options are actimeo=3,hard,noatime,rsize=32768,wsize=32768,nfsvers=3 The clocks are in sync.
Dovecot.conf: protocols = imap imaps pop3 pop3s login_greeting = imap ready. mail_location = mbox:~/:INBOX=/var/mail/%u mmap_disable = yes mail_nfs_storage = yes mail_nfs_index = yes protocol imap { } protocol pop3 { pop3_uidl_format = %08Xu%08Xv } auth default { mechanisms = plain passdb pam { } userdb passwd { } user = root } dict { } plugin { }
Any help would be greatly appreciated.
Thanks, John
-- John Gray gray@agora-net.com AgoraNet, Inc. (302) 224-2475 314 E. Main Street, Suite 1 (302) 224-2552 (fax) Newark, De 19711 http://www.agora-net.com
On Tue, 2008-02-26 at 11:03 -0500, John Gray wrote:
I run into lock deadlocks if I run dovecot over nfs
As Aaron asked, it would be useful to strace the process to see what exactly is waiting: mbox file or index files, dotlocking or fcntl locking.
Sometimes I can kill all the processes, remove the dot locks, and recover. Something I need to reboot the nfs server to recover.
Maybe you could just try disabling fcntl locking for index files and see how it works then:
lock_method = dotlock
Also if that doesn't help, try how mboxes work with only dotlocking:
mbox_read_locks = dotlock mbox_write_locks = dotlock
The locking issue is on the mbox files. When it happens, I get a bunch procmail processes backing up waiting for the lock.
The filesystem is reiserfs (version 3).
The NFS server is debian sarge box (which will be going to etch this weekend). And its using the nfs-kernel-server.
The dead locks appear to be the same nfs client machine. Though I'm not
dead sure on it. The mail delivery does come in on several machines.
And the imap/pop to just one for a given user.
I'm not seeing any lockd messages. I'll look into making sure its running.
I'll try to get an strace if I see it again. And I'll try the dotlock suggestion.
Thanks Timo and Aaron!
John
Timo Sirainen wrote:
On Tue, 2008-02-26 at 11:03 -0500, John Gray wrote:
I run into lock deadlocks if I run dovecot over nfs
As Aaron asked, it would be useful to strace the process to see what exactly is waiting: mbox file or index files, dotlocking or fcntl locking.
Sometimes I can kill all the processes, remove the dot locks, and recover. Something I need to reboot the nfs server to recover.
Maybe you could just try disabling fcntl locking for index files and see how it works then:
lock_method = dotlock
Also if that doesn't help, try how mboxes work with only dotlocking:
mbox_read_locks = dotlock mbox_write_locks = dotlock
Just for the record, I took Timo's suggestion, and went to just using dotlocks. The problem hasn't return (at least not yet).
John
John Gray wrote:
The locking issue is on the mbox files. When it happens, I get a bunch procmail processes backing up waiting for the lock.
The filesystem is reiserfs (version 3).
The NFS server is debian sarge box (which will be going to etch this weekend). And its using the nfs-kernel-server.
The dead locks appear to be the same nfs client machine. Though I'm not dead sure on it. The mail delivery does come in on several machines. And the imap/pop to just one for a given user.
I'm not seeing any lockd messages. I'll look into making sure its running.
I'll try to get an strace if I see it again. And I'll try the dotlock suggestion.
Thanks Timo and Aaron!
John
Timo Sirainen wrote:
On Tue, 2008-02-26 at 11:03 -0500, John Gray wrote:
I run into lock deadlocks if I run dovecot over nfs
As Aaron asked, it would be useful to strace the process to see what exactly is waiting: mbox file or index files, dotlocking or fcntl locking.
Sometimes I can kill all the processes, remove the dot locks, and recover. Something I need to reboot the nfs server to recover.
Maybe you could just try disabling fcntl locking for index files and see how it works then:
lock_method = dotlock
Also if that doesn't help, try how mboxes work with only dotlocking:
mbox_read_locks = dotlock mbox_write_locks = dotlock
-- John Gray gray@agora-net.com AgoraNet, Inc. (302) 224-2475 314 E. Main Street, Suite 1 (302) 224-2552 (fax) Newark, De 19711 http://www.agora-net.com
participants (3)
-
Aaron Wiebe
-
John Gray
-
Timo Sirainen