[Dovecot] Newbie questions: Load-balanced Dovecot with NFS storage

[Dean Brooks]

On Fri, Mar 02, 2007 at 04:12:10PM +0000, Mike Brudenell wrote:
> *  Because sessions could be on different servers memory mapping of  
> index
>    files doesn't work well with NFS, so set
>        mmap_disable=yes
>   in dovecot.conf.

If your load balancer is set up to have persistent servers based upon
user criteria if some sort, you could actually store the indexes on
local drives on each machine.  Worst case scenario, if user flipped to
a different box in the cluster, Dovecot would have to rebuild its
index increasing CPU and I/O.  Best case, you see a performance gain
by using local store and reducing NFS traffic.

> *  Dovecot relies on the mtime timestamp of mailbox files so the
>    NFS-mounted mailstore needs to be mounted with these options on each
>    server:
>        actimeo=0
>    (Are there any other mount options we should use too?)

This is absolutely critical, yes.  Without this, all sorts of nasty
things can happen.  You can also use the "noac" attribute, at least
as of Solaris 8.

> *  We will be using filesystem quotas on users mailboxes.  We understand
>    that Dovecot's index files are best on no-quota filestores so  
> will store these separately.

The only issue here is dotlocking.  If your MTA or if Dovecot uses
dotlocks of any kind, you'll need to be sure those are also outside
of your quota-restricted filestore.  Otherwise, you are going to need
to be sure that the filesystem quota is never actually reached, and
that the limit is artificially set lower than the filesystem quota
in some other manner.

> Q1. Would it be better to store the index files on NFS-shared  
> filestore and
>     direct users to any of the IMAP server machines?  Or to store  
> the index
>     files on local disk and direct each user to their 'preferred' IMAP
>     server machine?

Our plan is to store index files on local store and load balance to
persistent servers.  Sure, the persistent cache table expires over
time, but then again, the indexes get out of date over time anyway.

> Q2. Does Dovecot (or "something") clean out old index files that haven't
>     been accessed for a while?  Eg, when a user has temporarily come
>     through on a different IMAP server to normal.  Or do the index  
> files
>     sit there untouched for evermore?

They sit untouched forever.  Feel free to remove them after they get to
be of certain age.

> Q3. Storing the index files on the NetApp filer would give us the  
> ability
>     to share them between servers and grow their volume as need be,  
> but at
>     the cost of performance.  How big do the index files get?  Are they
>     typically a few kBytes per message? Per mailbox? Per user? Or more?

I considered this, but my concern is reliability.  Dovecot's index
files seem a bit "delicate" in recent patches and I'm afraid of
possible issues of sharing them between servers, especially if
there are multiple IMAP sessions open on different servers.  I'm not
familiar with ultimate size at this point, but it depends on the
maximum size of the mailboxes and folders I suppose.

> Q4. We will be using Exim as the MTA, which can deliver direct to  
> Maildir
>     mailboxes.  However I understand that Dovecot's "deliver" LDA  
> adds the
>     benefit of updating the index files as each message is  
> delivered.  Is
>     this a significant gain?  Or is there little difference in  
> actual use?

No experience with the LDA component.

> Q5. We have around 20,000 mail accounts and will therefore be seeing  
> lots
>     of concurrent IMAP sessions, usually secure (SSL) ones.  I have  
> seen
>     mention that this can give rise to "Too many open files" errors  
> under
>     Solaris.  How do we avoid this when we are likely to have several
>     thousand concurrent IMAP sessions per server machine?

Yow.  Thousands of concurrent IMAP sessions *per* server?  All using
SSL?  With only 20,000 mail accounts?  Are you sure about that?  That
seems like an awfully high active-reader ratio given the low number of
accounts.  Still, if true, it is what it is and needs to be accomodated.

Obviously increasing the number of systems in your cluster is one way
to fight it.  I know there was a recent bug in Dovecot that was
causing file descriptor leaks, but if I recall it was fixed in a
recent patch.

Still, if you are going to have thousands of concurrent IMAP sessions,
I would consider making sure you have a good number of systems in
your cluster.

I'm not sure if you are using a webmail client, such as Squirrelmail, 
but if so you may also want to consider running an IMAP proxy server
to keep sessions open and persistent between page loads.

--
Dean Brooks
dean at iglou.com