[Dovecot] Multi Server Exim/Dovecot and NFS

[David Warden]

On Jul 16, 2012, at 8:58 AM, Robert Blayzor wrote:

> In reading the Dovecot NFS WIKI entry:
> 
> http://wiki2.dovecot.org/NFS
> 
> It mentions a FreeBSD NFS client caching bug.  I believe that bug only exists pre-8.x ?  It seems that a patch in the PR notes was MFC'd and made it into 8.x at some point.  (at least I checked the source in 8.3 and it was there).  So I assume that FreeBSD 8.x mitigates the issue?  Maybe not?
> 
> 
> So our deployment is multi server Dovecot/Exim running on FreeBSD servers, frontended by a layer3 load balancing switch and a NetApp NFSv3 datastore.

Our deployment is similar. Dovecot/Exim on Oracle Linux 6.2 (RHEL derivative), layer3 load balancing (pacemaker/ldirectord managing LVS, "sticky" sessions per client IP) and NetApp NFSv3 datastore (maildir).

> 
> The only real concern we have right now is race conditions and file write contention on the Dovecot index files.  Our original plan was to use Dovecot LMTP on the servers to deliver to the inbox, but that doesn't seem to be recommended as we'll have a number of servers all acting as primary MX for domains and delivering to the same set of mailboxes.  So are the Dovecot dotlock file locking mechanisms not enough for multiple servers trying to write/access index files in Maildirs?

I can't speak to FreeBSD but we have had no problems using the NetApp NFS lock manager (lock_method = fcntl) instead of dotlock. Previously we were using dotlock and any skew in clocks on the Dovecot servers would cause higher CPU usage and tons of extra logging. If it works for you, try to use the NetApp NFS lock manager instead of dotlock.

It sounds like you have a bunch of hosts running both Exim and Dovecot and they are all your MX hosts? If so, it sounds like you could use Dovecot's LDA binary or LMTP. Since you are running Dovecot also on your Exim hosts and thus don't need to go over the network to a different host to see the mailstore/deliver the message, I would stick with LDA but both options will work.

> 
> If we have to deliver straight to the ~/Maildir/new out of Exim, that's fine, also.  We're just wondering if there is a added benefit of actually using LMTP. (other than making sure the index are kept the most recent).
> 

I believe the latest word from Timo is that the benefit of index updating from LDA/LMTP delivery for maildir is minimal, especially if your indexes are on NFS. We do use LDA for delivery with indexes on NFS, and have not had problems (~7000 user site). The important settings are:

# Don't use mmap() at all. This is required if you store indexes to shared
# filesystems (NFS or clustered filesystem).
#mmap_disable = no
mmap_disable = yes

# When to use fsync() or fdatasync() calls:
#   optimized (default): Whenever necessary to avoid losing important data
#   always: Useful with e.g. NFS when write()s are delayed
#   never: Never use it (best performance, but crashes can lose data)
#mail_fsync = optimized
mail_fsync = always

# Mail storage exists in NFS. Set this to yes to make Dovecot flush NFS caches
# whenever needed. If you're using only a single mail server this isn't needed.
#mail_nfs_storage = no
mail_nfs_storage = yes
# Mail index files also exist in NFS. Setting this to yes requires
# mmap_disable=yes and fsync_disable=no.
#mail_nfs_index = no
mail_nfs_index = yes

> The other concern we have is client mailbox access.  While we can make sessions "sticky" from one IP so they hit the same server, it doesn't help with the person that checks their email via POP from their PC while having their mobile phone access it via IMAP. (obviously from different IP addresses).  So I guess there is some possibly index file write contention there as well.  (though hopefully this should be rare).

We make sessions "sticky" and have the same "different client IP, same username" pattern. We haven't had problems with this setup since switching away from dotlock to NFS locks.

> 
> I'm curious as to know others experiences with a similar deployment.
> 

I should mention we are hoping to change this deployment soon. As others have said, you would probably be better served at least with Dovecot Director (to ensure client access and Exim deliveries using LMTP for user X always hits server Y) and possibly mdbox instead of maildir. We have been testing Director and I feel good enough about it to move indexes off of NFS onto local disk and setting a scheduled task to update indexes on every server at some reasonable interval so a server failure doesn't require a complete index rebuild when users tied to the failed server get sent to a different one. Migrating from maildir to mdbox is on my to-investigate list once Director is implemented but without having tested a migration there is a lot to like in terms of Alternate Storage and reduced IOPS v Maildir. You just have to *really* trust those index files because they are the only place flags/keywords are stored.

> TIA
> 
> -- 
> Robert Blayzor
> INOC, LLC
> rblayzor at inoc.net
> http://www.inoc.net/~rblayzor/
> 
> 
> 


HTH,
David Warden