[Dovecot] Multi Server Exim/Dovecot and NFS

[Robert Blayzor]

On Jul 16, 2012, at 11:34 AM, David Warden wrote:
> I can't speak to FreeBSD but we have had no problems using the NetApp NFS lock manager (lock_method = fcntl) instead of dotlock. Previously we were using dotlock and any skew in clocks on the Dovecot servers would cause higher CPU usage and tons of extra logging. If it works for you, try to use the NetApp NFS lock manager instead of dotlock.

Right.  I believe Pre 7.x on FreeBSD, NFS client side locking was rather dodgy or non-existent.  I believe in more recent version 8.x+ (and perhaps even some in 7.x), NFS locks client to server locks are supported.  But this is rather encouraging and we'll do some testing with it.  I wasn't a fan of going with dotlock...


> It sounds like you have a bunch of hosts running both Exim and Dovecot and they are all your MX hosts? If so, it sounds like you could use Dovecot's LDA binary or LMTP. Since you are running Dovecot also on your Exim hosts and thus don't need to go over the network to a different host to see the mailstore/deliver the message, I would stick with LDA but both options will work.

Yes, Exim/Dovecot same servers, multiple servers.  I had LDA/LMTP working fine until I read the notes that if mail is being delivered simultaneously from multiple servers that this should not be used as it's not safe for the indexing.  If fcntl locking fixes this, then we'll look at it again.  I was weighing the benefits of doing index updating at STMP time vs just doing it when the user logs in, and I'm leaning toward just having Exim deliver directly to the ~/Maildir/new instead of using LDA.  If a user gets 100 emails in their box, but doesn't login for days, no sense in updating all of the index files 100 times at SMTP time, and just do it once when they login.  Flawed logic?  Also LMTP is a bit more protocol over and is another database lookup on the Dovecot side, etc.  


> I believe the latest word from Timo is that the benefit of index updating from LDA/LMTP delivery for maildir is minimal, especially if your indexes are on NFS. We do use LDA for delivery with indexes on NFS, and have not had problems (~7000 user site). The important settings are: 

And that's exactly what I'm thinking.


> # Don't use mmap() at all. This is required if you store indexes to shared
> # filesystems (NFS or clustered filesystem).
> #mmap_disable = no
> mmap_disable = yes
> 
> # When to use fsync() or fdatasync() calls:
> #   optimized (default): Whenever necessary to avoid losing important data
> #   always: Useful with e.g. NFS when write()s are delayed
> #   never: Never use it (best performance, but crashes can lose data)
> #mail_fsync = optimized
> mail_fsync = always
> 
> # Mail storage exists in NFS. Set this to yes to make Dovecot flush NFS caches
> # whenever needed. If you're using only a single mail server this isn't needed.
> #mail_nfs_storage = no
> mail_nfs_storage = yes
> # Mail index files also exist in NFS. Setting this to yes requires
> # mmap_disable=yes and fsync_disable=no.
> #mail_nfs_index = no
> mail_nfs_index = yes


Righto, we've got all that.  Except right now we're just testing with dotlocking...  I'm going to have to try fcntl locking and see what that gets me.  Perhaps someone here is using FreeBSD with nfs locking now and can share experiences as well.


> We make sessions "sticky" and have the same "different client IP, same username" pattern. We haven't had problems with this setup since switching away from dotlock to NFS locks.

This is encouraging news.  Keeping a client sticky is important I think, especially for IMAP... no so much for POP.  I guess in the load balancer you could create a client protocol group and make them all hit the same server, but doing pop/IMAP from the same client would be unusual unless it was several different client types behind NAT.


> I should mention we are hoping to change this deployment soon. As others have said, you would probably be better served at least with Dovecot Director (to ensure client access and Exim deliveries using LMTP for user X always hits server Y) and possibly mdbox instead of maildir. We have been testing Director and I feel good enough about it to move indexes off of NFS onto local disk and setting a scheduled task to update indexes on every server at some reasonable interval so a server failure doesn't require a complete index rebuild when users tied to the failed server get sent to a different one. Migrating from maildir to mdbox is on my to-investigate list once Director is implemented but without having tested a migration there is a lot to like in terms of Alternate Storage and reduced IOPS v Maildir. You just have to *really* trust those index files because they are the only place flags/keywords are stored.

I'm not really up to speed on the dbox format, but I did look at it briefly.  I'd like to stick with maildir if at all possible.  It's worked for us for years, even over NFS, with little or no problems.  I think your always going to have problems with clients accessing from multiple locations/servers, etc... It's just how complex you want to get.  Director seems like a good step, but also adds another layer for us to test and maintain.  We're looking at that too..

Thanks for your input!

-- 
Robert Blayzor
INOC, LLC
rblayzor at inoc.net
http://www.inoc.net/~rblayzor/