Andrzej Adam Filip put forth on 11/2/2010 4:36 AM:

Stan Hoeppner stan@hardwarefreak.com wrote:

...

Taking a survey.

1. How many of you have a remote site hot backup Dovecot IMAP server?

2. How are you replicating mailbox data to the hot backup system? A. rsync B. DRBD+GFS2 C. Other

Do you need IMAP or POP3 would be sufficient? IMHO POP3 offers interesting options beyond what you have mentioned.

IMAP.

-- Stan

On Tue, 2010-11-02 at 10:36 +0100, Andrzej Adam Filip wrote:

Stan Hoeppner stan@hardwarefreak.com wrote:

...

Taking a survey.

1. How many of you have a remote site hot backup Dovecot IMAP server?

2. How are you replicating mailbox data to the hot backup system? A. rsync B. DRBD+GFS2 C. Other

Do you need IMAP or POP3 would be sufficient? IMHO POP3 offers interesting options beyond what you have mentioned.

Agreed, it shouldn't matter, its nothing but stored data

And hardly call rsync "hot backup"

my vote is snapmirror, same DC protected by many CCT cameras, area dependant swipe and biometric entry, VESDA, tonnes of FM200, redundant generators, is 200 mtrs from the second largest fire station in the state, and 90 mtrs from police HQ. not on a flood plain, chances earthquakes pretty much zero, redundant fibre, carriers and microwave, the only thing without redundancy is the espresso machine, but we working on that one! :) so who needs off-site ... wont get close enough with a bomb, and many taller buildings in the way for a plane, and besides, terrorists likely have better targets.

Stefan G. Weichinger put forth on 11/2/2010 1:15 PM:

A bit off-topic, sorry ... I want to set up a hot backup dovecot in a VM, aside the physical server, so I am very interested in the "best practise" to do so ...

There isn't one. If there was Timo would have pointed you to the wiki.

Doing server fail over is inherently problematic for a large number of reasons. The easiest way to implement it is to literally turn on the backup server (power on) when the primary fails. The backup comes up with the same hostname and IP address as the primary and mounts the same physical storage.

The storage must be either a SAN LUN, NFS directories, or a local disk that has been mirrored over the network during normal operations. But, you can't have the same hostname and IP if the machine is running allowing the mirroring to take place.

Thus, for a "standby" server, it must be powered off and take ownership of the storage when powered on. You _can_ do realtime mirroring to the standby while it's running, but then you have some really complex issues to deal with as far as hostname and IP assignments when the primary host dies and you have to take over the name and IP on the spare server. This can be done with a reboot and using alternate config files, and might actually work better in a virtual environment than with a physical machine as VM guests tend to boot faster than physical hosts due to things like long pauses caused by hardware BIOS routines.

The key to all of the above is proper identification of primary host failure. The biggest problem with this setup is the "two brains" issue. There are a number of network scenarios that can cause your backup server or monitoring software to think the primary host is offline when it's really not. The secondary thus comes up, and now you have two hosts of the same name and IP address on the network. This situation can cause a number of serious problems

IMO, the best way to do high availability is to use an active/active cluster of any number of nodes you see fit to meet your performance and reliability needs. All hosts are live all the time and share he load. When one goes down client performance may simply drops a bit, but that's about the extent of the downside.

It's inherently more straight forward to setup than the previous scenario, especially if you're using NFS storage. In this case, you'd build two identical Dovecot servers and have each mount the same NFS mail directory. Read the list archives for ways to mitigate the index file issue. Timo wrote a new director specifically to meet this need.

Two other options for the shared storage are a fiber channel or iSCSI SAN, or using DRBD to mirror disks (or logical devices--RAID) over the network. Both of these solutions require using a cluster filesystem such as GFS2. These can be quite a bit more difficult to setup and get working properly than the NFS method, especially for less experienced sysadmins. They can also be more difficult to troubleshoot, especially for sysadmins lacking sufficient knowledge or aptitude with regard to storage hardware and low level Linux device drivers.

Hope this helps you a bit. You probably won't find a "how to" document that spoon feeds you the steps for an exact build/setup of this. If you choose the DRBD route you might be able to get Eric to write you up a step-by-step of how he did his two node DRBD Dovecot cluster. Maybe he's already written one. :)

-- Stan

Am 03.11.2010 09:15, schrieb Stefan G. Weichinger:

Am 03.11.2010 07:17, schrieb Stan Hoeppner:

...

Hope this helps you a bit.

Thanks a lot, Stan, for this explanation ... !

additional explanation of my current point of view:

I run a dovecot server with postfix (and postfixadmin, so mysql) for a customer. It is located in their LAN and only communicates with another server of mine which is doing the whole filtering etc (located far away).

My main concern is the availability of the mailboxes. Lately this server has started to simply crash out of a sudden, without any visible reasons (so far).

We speak of around 180 mailboxes here, and you know, email is crucial for today's business.

So I start to set up a second server within a VM, duplicate the physical server. Right now I rsync the mail-data to a second physical server, just to have the data quickly available, just in case (tape-backups are done as well, sure).

I will restore the mysql-db on the VM and try to get the virtual dovecot working, just to have a quick backup server at hand. It doesn't really have to be "zero downtime", I just want to be prepared to get something up within hours ...

Stefan

Johan Hendriks put forth on 11/3/2010 3:32 AM:

Hello, i am working primarly with FreeBSD, and the latest release has a service called HAST. See it as a mirrored disk over the network. You can install both machines with dovecot, and use the hast disk as the data storage. With CARP in the mix, when the master machine fails, it starts dovecot on the slave. This way you have a failover without user interference.

1. How do you automatically redirect clients to the IP address of the slave when the master goes down? Is this seamless? What is the duration of "server down" seen by clients? Seconds, minutes?

2. When you bring the master back up after repairing the cause of the failure, does it automatically and correctly resume mirroring of the HAST device so it obtains the new emails that were saved to the slave while it was offline? How do you then put the master back into service and make the slave offline again?

-- Stan

Op 3-11-2010 12:16, Stan Hoeppner schreef:

Johan Hendriks put forth on 11/3/2010 3:32 AM:

...

Hello, i am working primarly with FreeBSD, and the latest release has a service called HAST. See it as a mirrored disk over the network. You can install both machines with dovecot, and use the hast disk as the data storage. With CARP in the mix, when the master machine fails, it starts dovecot on the slave. This way you have a failover without user interference.

1. How do you automatically redirect clients to the IP address of the slave when the master goes down? Is this seamless? What is the duration of "server down" seen by clients? Seconds, minutes?

2. When you bring the master back up after repairing the cause of the failure, does it automatically and correctly resume mirroring of the HAST device so it obtains the new emails that were saved to the slave while it was offline? How do you then put the master back into service and make the slave offline again?

The servers work with an virtual ip. Carp does this, i use it for the firewalls on our location. Server 1 has ip adres 192.168.1.1, server 2 gets 192.168.1.2, and the virtual addres is 192.168.1.3 The clients connect to the virtual address 192.168.1.3, and contact the machine wich is master at that moment.

I do not know how the rebuild goes with hast, if the master provider goes down, like i said, i need to try and test it. Maybe an question on the freebsd-fs mailing list will answer this.

More about HAST http://wiki.freebsd.org/HAST More about Carp http://www.freebsd.org/doc/nl/books/handbook/carp.html

regards, Johan Hendriks

Quoting Johan Hendriks joh.hendriks@gmail.com:

I do not know how the rebuild goes with hast, if the master provider
goes down, like i said, i need to try and test it. Maybe an question on the freebsd-fs mailing list will answer this.

More about HAST http://wiki.freebsd.org/HAST

Sounds/looks a lot like DRBD. The config snip shown is about the same for both.. But I don't think it is DRBD, but rather based on DRBD...

DRBD generally (configurable) will restart a rebuild as soon as a host comes up and contacts the other host(s), assuming it can. If it can't for any reason, then it of course sits and waits for manual intervention or for something to change so it can continue.

-- Eric Rostetter The Department of Physics The University of Texas at Austin

This message is provided "AS IS" without warranty of any kind, either expressed or implied. Use this message at your own risk.

Stan Hoeppner wrote:

IMO, the best way to do high availability is to use an active/active cluster of any number of nodes you see fit to meet your performance and reliability needs. All hosts are live all the time and share he load. When one goes down client performance may simply drops a bit, but that's about the extent of the downside.

It's inherently more straight forward to setup than the previous scenario, especially if you're using NFS storage. In this case, you'd build two identical Dovecot servers and have each mount the same NFS mail directory. Read the list archives for ways to mitigate the index file issue. Timo wrote a new director specifically to meet this need.

Two other options for the shared storage are a fiber channel or iSCSI SAN, or using DRBD to mirror disks (or logical devices--RAID) over the network. Both of these solutions require using a cluster filesystem such as GFS2. These can be quite a bit more difficult to setup and get working properly than the NFS method, especially for less experienced sysadmins. They can also be more difficult to troubleshoot, especially for sysadmins lacking sufficient knowledge or aptitude with regard to storage hardware and low level Linux device drivers.

Stan, what do you think about GlusterFS or Ceph? I am looking for a high available AND scalable solution based on distributed/replicated storages (maildirs). I am not talking about a simple NFS export or something like this.

A real-time replication managed by dovecot would surely be the best way to keep data in sync and balance clients.

-- Sven