[Dovecot] Big sites using Dovecot [long]

[Chris Wakelin]

Jonathan wrote:
> Hi,
> 
> Having asked if there are any big sites (50,000-100,000 users) it 
> seems there are a few.  I'd like to ask some fairly general
> questions.
> 
> I have inherited responsibility for a Cyrus mail store, at a UK
> university.

And I have responsibility for a Dovecot mail store at a UK university :)

> 
> It is front-ended by a pair of mail gateways running Exim which
> handle spam, A/V etc.

Similarly, though we have four (two for internal-only mail, two doing
the spam-scoring for externally-originated mail).

> 
> Local delivery is a dedicated Suse Linux box running Postfix feeding
> Cyrus over LMTP.  There are around 80,000 accounts, with around
> 20,000 active (one or (many) more messages per day).  I suspect we
> peak at around 500 simultaneous users.  The message store is around
> 600Gb.

We've got about 30,000 accounts, about half of which are "active", I
guess, though I haven't counted. The message store is about 1.2 TB,
served from some Sun T3 and T4 RAID5 fibre-channel arrays. There is no
backup store, yet, but we're expecting to do something soon. Our inbox
backups are nearly 24 hours to DLT tape too, but we hope to replace that
with backup-to-disk (a NetApp R200) soon. Folders take a couple of days,
but we do incremental backups of them during the week.

> 
> A small number of users are on an Exchange server instead of Cyrus. 
> They will not be moving.  User authentication runs over LDAP and
> there is an attribute in LDAP which identifies whether the user is a
> Cyrus user or an Exchange user, so that Exim knows which mail store
> to send their mail to, and Webmail knows whether to redirect them to
> Horde or to a Microsoft Outlook Web server.

> 
> It is time for a refresh which needs to take place seamlessly, and in
>  short order (complete roll-out in the next couple of months).  We
> need to add a few extras into the equation...
> 
> 1. It is corporate policy to move all storage to a NetApp filer which
> is replicated using frequent snap-mirrors to a second site over a
> shared 1Gb link. (Due to possible bandwidth issues, the two filers do
> not update synchronously, but the backup NetApp should be no more
> than a couple of minutes behind, and this much loss of data would be 
> tolerated in the event of a disaster recovery deployment.)
> 
> 2. NFS is preferred over iSCSI, due to file recovery and disk space 
> utilisation on the NetApp.

We too have two NetApps, now (at last, two years late!) in different
machine rooms, replicating asynchronously via snapmirror. Only one of
them is "live", though I've considered dividing the load between them.

Our Exchange service is served off the live NetApp via iSCSI (with the
file recovery/disk space issues!), but it has deep pockets as far as
management are concerned.

We're supposed to be migrating all our staff users to Exchange over the
next year, hopefully without them noticing. Redundancy of the remaining
Dovecot "student" service is considered much less important (my boss has
even been heard to mutter about outsourcing the student service :o )

Having said that, I still have an aspiration to move the Dovecot
mailstore to the NetApps, probably converting to Maildir in the process.

> 
> 3. The two servers (or two clusters, if we go that way) will be sited
>  one at each site.  In the event of a data centre failure, we need to
>  have quick and effective fail over to the other site (manual
> intervention is acceptable).  It is possible that the redundant link
> between the sites could fail, leading to the servers losing touch
> with each other but both still running.

Similarly, failover is expected to be manual and we'd be prepared to
lose a few minutes worth of data.

> 5. Clustered servers would be preferred so we can do rolling upgrades
> by removing individual machines for OS patches etc.  We have layer 4 
> load balancers available.

I guess we could do this with Dovecot, subject to NFS limitations. Our
load balancers can remember which server an IP was sent to, so as long
as users didn't log in from two different machines simultaneously, we'd
probably get away even with having the Dovecot indexes on the NetApp.
Even if they did get connected through more than one server, I'd expect
Dovecot not to get too upset. See the Dovecot Wiki page on NFS.

> 
> 6. Our preferred corporate platform is Suse Linux Enterprise Server 9
>  running on Intel hardware.

For Unix, this is becoming the preferred platform, but the Dovecot
service is currently running on Solaris 8 on a Sun V480. Management
prefers Windows :(

> 
> Cyrus generally is seen as a very competent solution, and greatly 
> preferred to the UW Imap server it replaced (this may be to do with
> the NFS servers UW used).

We too migrated from UW, but we kept the mail in mbox format (I managed
to make Dovecot behave identically to UW so we could just switch between
them as necessary).

Cyrus has more features (ACLs, quotas, Sieve), which Dovecot is
gradually implementing as plugins, but I'd expect the Cyrus versions to
be more fully-featured/robust (e.g. the IMAP ACL extension isn't yet
supported in Dovecot, so clients can't see/change the ACLs).

> Reasons for leaving Cyrus are (1) NFS and (2) replication - although
> I understand the Cyrus 2.3 tree has some good support for keeping
> multiple servers loosely synchronised over a WAN.

Before Dovecot came to my attention, I was planning a move to Cyrus,
similar to the migration that Cambridge went through. Cyrus is probably
more "mature" (though the code seemed quite messy) but the transparency
of the move to Dovecot was worth the risk!

> I am very nervous about comments on this list concerning NFS
> lock-ups. This system has to be bullet-proof 24/7.  I would consider
> SolarisX86 (or possibly FreeBSD) if the NFS implementation is robust
> out-of-the-box.  Management would like the warm feeling that a
> vendor-supported operating system would give them (so Suse and Sun
> are preferred).

Likewise, I'm nervous, but some NFS patches went in to SuSE Enterprise 9
this week; presumably the ones mentioned for kernel 2.6.16. I'd also
expect the NetApp to be more forgiving than most servers (you might even
be able to force Cyrus onto it). I've wondered whether NFSv4 would be
better in any way (but that would probably have to be Solaris 10 as the
client).

> My gut feeling is that I would like to split the users into two 
> communities, with half on each NetApp, and with the two NetApps
> mirroring to each other.  In practice users will work from both sites
> (and remotely) but each one has a "home" site in terms of their home
> directory, etc. At each site, I'd like 2 identical Dovecot boxes.
> I'll call this a 2 x 2 solution.
> 
> All users (Exchange users excepted) have the same address wired into
> their e-mail client for IMAP/SSL and SMTP/SSL, so there would have to
> be some magic to ensure that the user ended up talking to a Dovecot
> server which could see the appropriate NetApp.  I don't think the
> load balancers are clever enough to be able to do this.  I think I've
> read it's possible for an IMAP server to hand a user off to a
> different IMAP server, but can Dovecot do this and is there client
> support.  Or should I just proxy users who hit the wrong server.  Or
> should I just put everyone on the same NetApp and use 4 servers? I'll
> call this a 4 x 1 solution.

Yes Dovecot can act as proxy. I set up a test one this afternoon, as it
happens! You may also want to look at Perdition, which is the "market
leader" in this sort of proxying. Both support various databases, but
you'd probably need to modify the code to look at your existing LDAP
attribute.

> 
> If we lost a site with a live NetApp, I would expect the surviving
> site to mount the latest snap-mirror and serve it.  In the case we
> are running 2 x 2 it would become 1 x 2.  If we were running 4 x 1 it
> would become 2 x 1 which is arguably more robust.
> 
> Does anyone have any comments on any of this.  If it were your site, 
> what would you be doing?  What kit would you use?  Which operating
> system?  How will it play with our load balancers. 4 x 1 or 2 x 2?
> Would anyone else in UK academia like to compare notes?
> 
> Many thanks, Jonathan.

Best Wishes,
Chris

-- 
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
Christopher Wakelin,                           c.d.wakelin at reading.ac.uk
IT Services Centre, The University of Reading,  Tel: +44 (0)118 378 8439
Whiteknights, Reading, RG6 2AF, UK              Fax: +44 (0)118 975 3094