[Dovecot] HA Mailbox Design
Hi,
We would like to implement a Highly Available Mail Server and would like to ask advice on how to architect this.
Some details on our setup:
Currently we have only one internal mail server (Postfix/Dovecot 2.0 - planning to move to 2.1), receiving mail from a gateway server (filtering spam/viruses) - a Cisco Ironport - which we are considering to replace with one (or a set of) cloud-based Postfix/Amavis-new/SpamAssassin/ClamAV VMs (currently in testing mode).
Delivery uses Dovecot LDA. User accounts are LDAP-based.
We use Maildir and the load is low (aside spam). Only about 250 users/mailboxes (4G each). All servers are CentOS 5.8 (planning move to 6.3) KVM VMs (on a cloud where we don't have control on the host, but on highly reliable hardware/networks).
We can have VMs on two different clouds and we also have at least two different connections (routes) to the cloud(s) (to support HA).
Any directions will be appreciated. Hoping to design an HA architecture but aiming to keep it simple and (as much as possible) easily maintainable one.
Thanks for your response and help, Nick
On 8/9/2012 9:58 AM, Nikolaos Milas wrote:
Hi,
We would like to implement a Highly Available Mail Server and would like to ask advice on how to architect this.
Some details on our setup:
Currently we have only one internal mail server (Postfix/Dovecot 2.0 - planning to move to 2.1), receiving mail from a gateway server (filtering spam/viruses) - a Cisco Ironport - which we are considering to replace with one (or a set of) cloud-based Postfix/Amavis-new/SpamAssassin/ClamAV VMs (currently in testing mode).
First, please don't use the term "cloud". That's a marketing buzzword that variously covers many actual network computing architectures. This is a technical mailing list. We use technical terms here, and precise detail of the architecture matters greatly, buzzwords are irrelevant.
Delivery uses Dovecot LDA. User accounts are LDAP-based. We use Maildir and the load is low (aside spam). Only about 250 users/mailboxes (4G each). All servers are CentOS 5.8 (planning move to 6.3) KVM VMs (on a cloud where we don't have control on the host, but on highly reliable hardware/networks).
If you don't control the host you don't control the storage (disks), thus making a traditional "HA" mail server system difficult. You state the "cloud" infrastructure is highly reliable. That begs the question, what is your definition of a "Highly Available Mail Server"? What is it that you actually want to accomplish? In some detail please. Do you mean "POP/IMAP service is always available to clients even when something fails"?
We can have VMs on two different clouds and we also have at least two different connections (routes) to the cloud(s) (to support HA).
Again, please stop talking about "clouds". None of what you just stated above means anything without actual architectural details. When designing/implementing an HA setup, details are absolutely critical, required to make it work. Details, details, details. Also, you've mentioned absolutely nothing about storage. Storage is the pivot pin in an HA setup, the central consideration. HA is built around the storage architecture, not around applications and servers (hosts or VMs).
Any directions will be appreciated. Hoping to design an HA architecture but aiming to keep it simple and (as much as possible) easily maintainable one.
HA architectures are never simple. That's the nature of the beast. And they're typically not cheap to implement as they absolutely require fault tolerant shared storage of one kind or another, either a disk array with dual active/active controllers and PSUs with a cluster filesystem on Dovecot nodes, or storage appliance that offers full redundancy and NFS protocol for Dovecot node access. This prevents single point of failure, which is what HA is all about.
To be quite frank, based upon the level of technical acumen you've demonstrated here, and the general financial position Greece finds itself in, and the fact you're a public institution, it seems you're a much better candidate for a Gmail hosted infrastructure than a VPS infrastructure with some manner of ad hoc software only HA measures bolted on, which is all you can do with VPS servers--you don't control the storage.
Gmail hosted mail instantly gives you a high availability email infrastructure for "free". Cost is negligible for your number of mailboxes. Antispam and all the standard stuff is built in and fairly decent. You get both POP and IMAP connectivity options, allowing access with desktop/mobile MUAs. If you prefer a different web interface you can setup a Horde or Roundcube VM at your "cloud" provider and access the Gmail accounts.
You'll have less "control" of the system, but it will save far more money than a VPS "cloud" solution, be far easier to administer, and the setup can be done in a day, with almost zero administration required afterward. If you have many users with personal Gmail accounts they'll take to it like a duck to water.
If you're looking for a "cloud" based HA email infrastructure at very low cost, you simply can't beat hosted Gmail. Or Google Apps, whatever they call it these days. The downside is it may eliminate your job if you're strictly a dedicated email system administrator.
-- Stan
Torsdag 9. august 2012 20.47.50 skrev Stan Hoeppner:
[snip]
To be quite frank, based upon the level of technical acumen you've demonstrated here, and the general financial position Greece finds itself in, and the fact you're a public institution, it seems you're a much better candidate for a Gmail hosted infrastructure than a VPS infrastructure with some manner of ad hoc software only HA measures bolted on, which is all you can do with VPS servers--you don't control the storage.
If they are a public institution, then they may be prohibited from hosting on Google, simply because possibly sensitive data would then be hosted in another country.
As for HA I agree with Stan in that it is both very expensive and difficult to do right, but I would also ask if do you *really* need it?
Arne
Arne K. Haaje http://www.drlinux.no/ Twitter: drlinuxno LinkedIn: http://no.linkedin.com/pub/arne-haaje/27/189/bb
On 10/8/2012 4:47 πμ, Stan Hoeppner wrote:
That begs the question, what is your definition of a "Highly Available Mail Server"? What is it that you actually want to accomplish? In some detail please.
OK, I'll make it as much as possible accurate.
Let's skip all the network stuff and see a particular scenario (as we have drafted it).
We have an incoming gateway server (gw.example.com) accepting mail and filtering viruses/spam. Then it relays all (clean) mail to mail1.example.com, which uses Postfix/Dovecot (2.0 or 2.1) and provides Maildir mailboxes (POP/IMAP) to users.
Now, let us assume we are deploying another server, mail2.example.com (also Postfix/Dovecot), which we want to function as follows:
Under normal conditions, mail2.example.com is a full mirror of mail1.example.com; when any mail message is added/viewed/moved/removed etc. to any user's folder or any folder is added/viewed/moved/removed etc. at mail1.example.com, we want it to be automatically and directly (in real time) added/viewed/moved/removed etc. to mail2.example.com too. In other words, we need continuous, real-time sync.
If mail1.example.com for some reason is unavailable, then we will be able to manually redirect relaying (of incoming messages) to mail2.example.com. Then, users will be able to use mail2.example.com to access their mail. Now, when mail1.example.com becomes available again, we want to: a. inform users (by sending them a mail on mail2.example.com) that mail1.example.com is available again, b. stop relaying to mail2.example.com c. sync once mailboxes on mail1.example.com to mail2.example.com (because mail2.example.com is now more current) d. redirect relaying to mail1.example.com e. switch to normal operation (see §1 above)
Can I do this and how?
I would call this pseudo-HA, since users have to switch servers in case of failures. To use the above as "true" HA (as I view it), there could be a mail.example.com functioning as a proxy and automatically redirecting users to mail1 or mail2, depending on admins' choice. Can I do this too? (How?)
[Google mail is not an option, we don't want external hosting. We can have as many high-performance, highly-reliable VMs as we want for free on our ISP's network - it's a service to the Greek educational/research community. They use two different specialized high-end enterprise-grade dedicated virtualization clusters of host hardware (which I -not being very accurate- called clouds) on their networks, each of which uses dedicated high-end enterprise-grade SAN-based storage. Practically we have never had VM outages due to hardware failures, only due to software (rarely) or network (mainly) ones. mail1.example.com would be deployed on the main virtualization cluster and mail2.example.com would be on the the other cluster. KVM is used as host virtualization software.]
Alternative suggestions on design approaches would be welcome.
Thanks and regards, Nick
Nikolaos Milas wrote:
On 10/8/2012 4:47 πμ, Stan Hoeppner wrote:
That begs the question, what is your definition of a "Highly Available Mail Server"? What is it that you actually want to accomplish? In some detail please.
- Under normal conditions, mail2.example.com is a full mirror of mail1.example.com; when any mail message is added/viewed/moved/removed etc. to any user's folder or any folder is added/viewed/moved/removed etc. at mail1.example.com, we want it to be automatically and directly (in real time) added/viewed/moved/removed etc. to mail2.example.com too. In other words, we need continuous, real-time sync.
Can I do this and how?
You might have a look at DRBD (distributed replicated block device) which provides a high available block device with fully synchronous mirroring:
http://www.drbd.org/home/mirroring
Dovecot can then simply work with the filesystem residing on the highly avilable DRBD volume.
Regards Daniel
On 8/11/2012 11:52 AM, Daniel Parthey wrote:
Nikolaos Milas wrote:
On 10/8/2012 4:47 πμ, Stan Hoeppner wrote:
That begs the question, what is your definition of a "Highly Available Mail Server"? What is it that you actually want to accomplish? In some detail please.
- Under normal conditions, mail2.example.com is a full mirror of mail1.example.com; when any mail message is added/viewed/moved/removed etc. to any user's folder or any folder is added/viewed/moved/removed etc. at mail1.example.com, we want it to be automatically and directly (in real time) added/viewed/moved/removed etc. to mail2.example.com too. In other words, we need continuous, real-time sync.
Can I do this and how?
You might have a look at DRBD (distributed replicated block device) which provides a high available block device with fully synchronous mirroring:
http://www.drbd.org/home/mirroring
Dovecot can then simply work with the filesystem residing on the highly avilable DRBD volume.
But to be clear, for a true HA setup with full active/active nodes, this must be a cluster filesystem (GFS2/OCFS2).
-- Stan
On 08/11/2012 01:18 PM, Stan Hoeppner wrote:
On 8/11/2012 11:52 AM, Daniel Parthey wrote:
Nikolaos Milas wrote:
On 10/8/2012 4:47 πμ, Stan Hoeppner wrote:
That begs the question, what is your definition of a "Highly Available Mail Server"? What is it that you actually want to accomplish? In some detail please.
- Under normal conditions, mail2.example.com is a full mirror of mail1.example.com; when any mail message is added/viewed/moved/removed etc. to any user's folder or any folder is added/viewed/moved/removed etc. at mail1.example.com, we want it to be automatically and directly (in real time) added/viewed/moved/removed etc. to mail2.example.com too. In other words, we need continuous, real-time sync.
Can I do this and how? You might have a look at DRBD (distributed replicated block device) which provides a high available block device with fully synchronous mirroring:
http://www.drbd.org/home/mirroring
Dovecot can then simply work with the filesystem residing on the highly avilable DRBD volume. But to be clear, for a true HA setup with full active/active nodes, this must be a cluster filesystem (GFS2/OCFS2).
A good solution for kvm + drbd is this:
http://pve.proxmox.com/wiki/Two-Node_High_Availability_Cluster
On 8/11/2012 8:02 AM, Nikolaos Milas wrote:
- Under normal conditions, mail2.example.com is a full mirror of mail1.example.com; when any mail message is added/viewed/moved/removed etc. to any user's folder or any folder is added/viewed/moved/removed etc. at mail1.example.com, we want it to be automatically and directly (in real time) added/viewed/moved/removed etc. to mail2.example.com too. In other words, we need continuous, real-time sync.
There are 3 ways to do this, all require a form of shared storage:
- A cluster filesystem atop shared storage
- DRBD mirroring with a cluster filesystem atop
- NFS
#1 won't work in a VPS "cloud" unless the provider gives you direct access to an iSCSI SAN LUN. #2 might but you'll be in uncharted waters. #3 will work in a VPS cloud, but the host serving NFS is a single point of failure.
DRBD mirroring is typically done with an X-over cable between the two nodes. If the packets must traverse a WAN link, or an internal network that experiences any packet loss, DRBD will have problems. You'll need to make sure fencing is working properly which entails lots of testing before going into production.
Pick DRBD expert's brains to determine if doing it over a VPS LAN is possible/feasible.
- If mail1.example.com for some reason is unavailable, then we will be able to manually redirect relaying (of incoming messages) to mail2.example.com.
When setup properly no manual intervention is necessary. Everything is automatic. If one of the two Dovecot/DRBD nodes is down mail is automatically relayed to the other. This is done by putting putting a Postfix instance on each cluster node with equal priority MX records for both.
So instead of having two MX relay/AS/AV servers and two Dovecot mailbox servers, you have once instance of everything on each of two nodes. I.e. Postfix+amavisd+SA+Dovecot+etc on each node. Equal priority MX will route inbound mail fairly evenly between both nodes. If one MX is unreachable, remote SMTP clients will retry the other MX. It's fully automatic. You may want to install Dovecot director on one of the nodes so IMAP connections are spread evenly and to avoid mailbox locking/delay issues.
Then, users will be able to use mail2.example.com to access their mail. Now, when mail1.example.com becomes available again, we want to: a. inform users (by sending them a mail on mail2.example.com) that mail1.example.com is available again, b. stop relaying to mail2.example.com c. sync once mailboxes on mail1.example.com to mail2.example.com (because mail2.example.com is now more current) d. redirect relaying to mail1.example.com e. switch to normal operation (see §1 above)
Again, all of this is unnecessary, as it's fully automatic when done properly.
Can I do this and how?
Writing you a complete design document for doing this is beyond the scope of a mailing list thread. There exists plenty of documentation on the web. You will have to do your own research, but I've pointed you in the right direction. There are people on this list using a configuration almost identical to this, but not with VPS, though I am not one of them. When you are far enough along in the process and have specific questions I'm sure others will be glad to help.
I would call this pseudo-HA, since users have to switch servers in case of failures. To use the above as "true" HA (as I view it), there could be a mail.example.com functioning as a proxy and automatically redirecting users to mail1 or mail2, depending on admins' choice. Can I do this too? (How?)
See Dovecot director: http://wiki2.dovecot.org/Director
[Google mail is not an option, we don't want external hosting. We can have as many high-performance, highly-reliable VMs as we want for free on our ISP's network - it's a service to the Greek educational/research community. They use two different specialized high-end enterprise-grade dedicated virtualization clusters of host hardware (which I -not being very accurate- called clouds) on their networks, each of which uses dedicated high-end enterprise-grade SAN-based storage. Practically we have never had VM outages due to hardware failures, only due to software (rarely) or network (mainly) ones. mail1.example.com would be deployed on the main virtualization cluster and mail2.example.com would be on the the other cluster. KVM is used as host virtualization software.]
The approach I outlined above may work, as long as the network is reliable enough to keep DRBD happy. Once built you will obviously want to test the setup with a simulated real world workload for a few weeks to a month in order to confirm the network is reliable enough to support DRBD, before attempting full live deployment. But this is true of any new back end services deployment.
I dunno if Eric Rostetter is still around. He's got a very similar setup running at UT Austin, but on two dedicated servers, not VPS. He could probably give you some pointers if you run into design/config trouble: http://www.ph.utexas.edu/person/rostetter_eric
I'm sure there are others with a very similar setup on this list.
-- Stan
On 11/8/2012 8:11 μμ, Stan Hoeppner wrote:
Writing you a complete design document for doing this is beyond the scope of a mailing list thread. There exists plenty of documentation on the web. You will have to do your own research, but I've pointed you in the right direction.
Thank you Stan and everyone else who contributed on this subject.
I'll research further. Your info has been very enlightening.
Thanks again, Nick
participants (5)
-
Arne K. Haaje
-
Daniel Parthey
-
Nikolaos Milas
-
rob
-
Stan Hoeppner