Jonathan wrote:
Hi,
Having asked if there are any big sites (50,000-100,000 users) it seems there are a few. I'd like to ask some fairly general questions.
I have inherited responsibility for a Cyrus mail store, at a UK university.
And I have responsibility for a Dovecot mail store at a UK university :)
It is front-ended by a pair of mail gateways running Exim which handle spam, A/V etc.
Similarly, though we have four (two for internal-only mail, two doing the spam-scoring for externally-originated mail).
Local delivery is a dedicated Suse Linux box running Postfix feeding Cyrus over LMTP. There are around 80,000 accounts, with around 20,000 active (one or (many) more messages per day). I suspect we peak at around 500 simultaneous users. The message store is around 600Gb.
We've got about 30,000 accounts, about half of which are "active", I guess, though I haven't counted. The message store is about 1.2 TB, served from some Sun T3 and T4 RAID5 fibre-channel arrays. There is no backup store, yet, but we're expecting to do something soon. Our inbox backups are nearly 24 hours to DLT tape too, but we hope to replace that with backup-to-disk (a NetApp R200) soon. Folders take a couple of days, but we do incremental backups of them during the week.
A small number of users are on an Exchange server instead of Cyrus. They will not be moving. User authentication runs over LDAP and there is an attribute in LDAP which identifies whether the user is a Cyrus user or an Exchange user, so that Exim knows which mail store to send their mail to, and Webmail knows whether to redirect them to Horde or to a Microsoft Outlook Web server.
It is time for a refresh which needs to take place seamlessly, and in short order (complete roll-out in the next couple of months). We need to add a few extras into the equation...
It is corporate policy to move all storage to a NetApp filer which is replicated using frequent snap-mirrors to a second site over a shared 1Gb link. (Due to possible bandwidth issues, the two filers do not update synchronously, but the backup NetApp should be no more than a couple of minutes behind, and this much loss of data would be tolerated in the event of a disaster recovery deployment.)
NFS is preferred over iSCSI, due to file recovery and disk space utilisation on the NetApp.
We too have two NetApps, now (at last, two years late!) in different machine rooms, replicating asynchronously via snapmirror. Only one of them is "live", though I've considered dividing the load between them.
Our Exchange service is served off the live NetApp via iSCSI (with the file recovery/disk space issues!), but it has deep pockets as far as management are concerned.
We're supposed to be migrating all our staff users to Exchange over the next year, hopefully without them noticing. Redundancy of the remaining Dovecot "student" service is considered much less important (my boss has even been heard to mutter about outsourcing the student service :o )
Having said that, I still have an aspiration to move the Dovecot mailstore to the NetApps, probably converting to Maildir in the process.
- The two servers (or two clusters, if we go that way) will be sited one at each site. In the event of a data centre failure, we need to have quick and effective fail over to the other site (manual intervention is acceptable). It is possible that the redundant link between the sites could fail, leading to the servers losing touch with each other but both still running.
Similarly, failover is expected to be manual and we'd be prepared to lose a few minutes worth of data.
- Clustered servers would be preferred so we can do rolling upgrades by removing individual machines for OS patches etc. We have layer 4 load balancers available.
I guess we could do this with Dovecot, subject to NFS limitations. Our load balancers can remember which server an IP was sent to, so as long as users didn't log in from two different machines simultaneously, we'd probably get away even with having the Dovecot indexes on the NetApp. Even if they did get connected through more than one server, I'd expect Dovecot not to get too upset. See the Dovecot Wiki page on NFS.
- Our preferred corporate platform is Suse Linux Enterprise Server 9 running on Intel hardware.
For Unix, this is becoming the preferred platform, but the Dovecot service is currently running on Solaris 8 on a Sun V480. Management prefers Windows :(
Cyrus generally is seen as a very competent solution, and greatly preferred to the UW Imap server it replaced (this may be to do with the NFS servers UW used).
We too migrated from UW, but we kept the mail in mbox format (I managed to make Dovecot behave identically to UW so we could just switch between them as necessary).
Cyrus has more features (ACLs, quotas, Sieve), which Dovecot is gradually implementing as plugins, but I'd expect the Cyrus versions to be more fully-featured/robust (e.g. the IMAP ACL extension isn't yet supported in Dovecot, so clients can't see/change the ACLs).
Reasons for leaving Cyrus are (1) NFS and (2) replication - although I understand the Cyrus 2.3 tree has some good support for keeping multiple servers loosely synchronised over a WAN.
Before Dovecot came to my attention, I was planning a move to Cyrus, similar to the migration that Cambridge went through. Cyrus is probably more "mature" (though the code seemed quite messy) but the transparency of the move to Dovecot was worth the risk!
I am very nervous about comments on this list concerning NFS lock-ups. This system has to be bullet-proof 24/7. I would consider SolarisX86 (or possibly FreeBSD) if the NFS implementation is robust out-of-the-box. Management would like the warm feeling that a vendor-supported operating system would give them (so Suse and Sun are preferred).
Likewise, I'm nervous, but some NFS patches went in to SuSE Enterprise 9 this week; presumably the ones mentioned for kernel 2.6.16. I'd also expect the NetApp to be more forgiving than most servers (you might even be able to force Cyrus onto it). I've wondered whether NFSv4 would be better in any way (but that would probably have to be Solaris 10 as the client).
My gut feeling is that I would like to split the users into two communities, with half on each NetApp, and with the two NetApps mirroring to each other. In practice users will work from both sites (and remotely) but each one has a "home" site in terms of their home directory, etc. At each site, I'd like 2 identical Dovecot boxes. I'll call this a 2 x 2 solution.
All users (Exchange users excepted) have the same address wired into their e-mail client for IMAP/SSL and SMTP/SSL, so there would have to be some magic to ensure that the user ended up talking to a Dovecot server which could see the appropriate NetApp. I don't think the load balancers are clever enough to be able to do this. I think I've read it's possible for an IMAP server to hand a user off to a different IMAP server, but can Dovecot do this and is there client support. Or should I just proxy users who hit the wrong server. Or should I just put everyone on the same NetApp and use 4 servers? I'll call this a 4 x 1 solution.
Yes Dovecot can act as proxy. I set up a test one this afternoon, as it happens! You may also want to look at Perdition, which is the "market leader" in this sort of proxying. Both support various databases, but you'd probably need to modify the code to look at your existing LDAP attribute.
If we lost a site with a live NetApp, I would expect the surviving site to mount the latest snap-mirror and serve it. In the case we are running 2 x 2 it would become 1 x 2. If we were running 4 x 1 it would become 2 x 1 which is arguably more robust.
Does anyone have any comments on any of this. If it were your site, what would you be doing? What kit would you use? Which operating system? How will it play with our load balancers. 4 x 1 or 2 x 2? Would anyone else in UK academia like to compare notes?
Many thanks, Jonathan.
Best Wishes, Chris
-- --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- Christopher Wakelin, c.d.wakelin@reading.ac.uk IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439 Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094