[Dovecot] nfs director

[Ariel Biener]

We're a similar installation (60-70k users, FAS3050 cluster).

We have been using "perdition" (IMAP/POP redirector) software
for a while. The IMAP/POP.ourdomain A records point to 2 front ends,
which all they do is to redirect the IMAP/POP session to the a specific
mail server for each user, based on their LDAP mailhost entry.

We use postfix to deliver mail, and procmail is the LDA. We are running
a background
process on each mail server (3 mailservers which do SMTP/POP/IMAP, and
barely sweat
at it - 2xquad core Xeons with 8gig each), which monitors the maillog,
and if dovecot
sees a index corruption, the monitor fixes the problem (we used to see
these errors
when we still used mbox, not anymore though).

We run a periodic process, which gets statistics of usage from the mail
servers, and reassign
the users to mail servers in order to better distribute the load. Each
new user that is created gets
his mail server by a random function which choses one of the three.

Each mail server in the user LDAP entry is in fact a virtual address on
a load balancer, pointing to
the real mail server behind it, BUT also having a backup server for each
in case the real
server crashes, so assuming mailsrv1 crashes, mailsrv2 will take its
clients.

The setup works rather well, within the limitations of maildir and
netapp (mainly
full body search being slowish with very large mailboxes made out of 10s
of thousands
of files).

We used to not use the "perdition" directors in the past, and once we
started using them,
we saved alot of problems on a few fronts:

1. Index corruption issues
2. SSL termination - since the front ends to the SSL termination, the
backend servers access
    from the front ends is clear text, saving CPU cycles from the
backends servers.

I haven't taken a look yet at Dovecot's solution for the director, but I
am writing this since I do think that it is
addressing a real life problem for any medium++ or larger installation
that uses NFS.

Oh, and Timo, I don't think we are just "a couple of NFS users". Maildir
and NFS are not as uncommon as
you'd think, even in very large installations.

--Ariel

Brandon Davidson wrote:
> Noel,
>
> On 8/26/10 9:59 PM, "Noel Butler" <noel.butler at ausics.net> wrote:
>
>   
>>> I fail to see advantage if anything it add in more point of failure, with
>>>       
>> i agree with this and it is why we dont use it
>>
>> we use dovecots deliver with postfix and have noticed no problems, not
>> to say there was none, but if so, we dont notice it.
>>     
>
> We might be a slightly larger install than you (60k users, mail on FAS 3170
> Metrocluster), but we have noticed corruption issues and the director is
> definitely going to see use in our shop. We still use Sendmail+procmail for
> delivery, so no issue there... but we've got hordes of IMAP users that will
> leave a client running at home, at their desk, on their phone, and then will
> use Webmail on their laptop.
>
> Without the director, all of these sessions end up on different backend
> mailservers, and it's basically a crapshoot which Dovecot instance notices a
> new message first. NFS locking being what it is, odds are an index will get
> corrupted sooner or later, and when this happens the user's mail
> 'disappears' until Dovecot can reindex it. The users inevitably freak out
> and call the helpdesk, who tells them to close and reopen their mail client.
> Maybe you're small enough to not run into problems, or maybe your users just
> have lower expectations or a higher pain threshold than ours. Either way,
> it's unpleasant for everyone involved, and quite easy to solve with the
> director proxy.
>
> Timo has been saying for YEARS that you need user-node affinity if you're
> doing NFS, and now he's done something about it. If you've already got a
> load balancer, then just point the balancer at a pool of directors, and then
> point the directors at your existing mailserver pool.
>
> <shameless plug>
> For health monitoring on the directors, check out:
> http://github.com/brandond/poolmon
> </shameless plug>
>
> -Brad
>
>   

-- 
 --
 Ariel Biener
 e-mail: ariel at post.tau.ac.il
 PGP: http://www.tau.ac.il/~ariel/pgp.html