[Dovecot] Transparent Migration from cyrus to dovecot
Hi dovecot people,
We are in the process of preparing the migration from a cyrus 2.1 installation to dovecot. Dovecot will be installed on new hardware, so we have separated servers that can/will exist in parallel for a while.
Our goal is to do the migration without interrupting the service for our users too much. Currently we tend to using dsync. So I am asking for best practice suggestions, tips and hints from people who have done such a thing before.
Curiously awaiting your replies ;)
Cheers! PS: I am subscribed to the list. So no need to include my address in replies. Thanks!
j.hofmüller
Optimism doesn't alter the laws of physics. - Subcommander T'Pol
Make use of the proxy feature. You can add a "server" entry into your userdb, that way you can literally move users over one by one and flip their server location. You can easily test individual users and move them over individually.
Works brilliantly
Ed W
On 06/10/2013 11:39, Jogi Hofmüller wrote:
Hi dovecot people,
We are in the process of preparing the migration from a cyrus 2.1 installation to dovecot. Dovecot will be installed on new hardware, so we have separated servers that can/will exist in parallel for a while.
Our goal is to do the migration without interrupting the service for our users too much. Currently we tend to using dsync. So I am asking for best practice suggestions, tips and hints from people who have done such a thing before.
Curiously awaiting your replies ;)
Cheers! PS: I am subscribed to the list. So no need to include my address in replies. Thanks!
On 10/6/2013 1:56 PM, Ed W wrote:
Make use of the proxy feature. You can add a "server" entry into your userdb, that way you can literally move users over one by one and flip their server location. You can easily test individual users and move them over individually.
Works brilliantly
Second this. Pair it with a snapshot-capable FS and you can migrate the bulk of data in the background, then do the stop cyrus delivery, offline cyrus, copy remaining differences, online dovecot, start delivery to dovecot steps in a matter of seconds.
Hi Ed,
Thanks for the encouragement!
Am 2013-10-06 22:56, schrieb Ed W:
Make use of the proxy feature. You can add a "server" entry into your userdb, that way you can literally move users over one by one and flip their server location. You can easily test individual users and move them over individually.
One question still remains in my head. The migration/dsync page [1] states that 'The source IMAP/POP3 mailboxes shouldn't be modified while dsync is running. Also "dsync backup" means that if the destination has any changes that don't exist in source IMAP server, the changes are deleted.' So how does the setup behave *while* I migrate a user's mail?
I figured that I would start with a proxy entry for every user. Then disabling proxy for the first mailbox and start migrating it. So new mail would be delivered to the newly created dovecot mailbox while all the mail from the old server would start appearing. From the quote above I take it that new mail *could* disappear.
OK, this is all still theory since I have not done any tests. However, the more I know beforehand, the better the process will work, I hope ;)
[1] http://wiki2.dovecot.org/Migration/Dsync
Cheers!
j.hofmüller
Optimism doesn't alter the laws of physics. - Subcommander T'Pol
Hi Jogi
"Jogi Hofmüller" jogi@mur.at schrieb:
One question still remains in my head. The migration/dsync page [1] states that 'The source IMAP/POP3 mailboxes shouldn't be modified while dsync is running. Also "dsync backup" means that if the destination has any changes that don't exist in source IMAP server, the changes are deleted.' So how does the setup behave *while* I migrate a user's mail?
We disabled the login (IMAP, POP3 and SMTP as well) of the user to be migrated and kicked the user from the system. Then we deferred the delivery to the user mailbox in the MTA with a temporary failure and a message like "Mailbox being migrated", with the help of a lookup table in the database. This avoids changes to the affected mailbox during the sync.
If something goes terribly wrong during sync, you can simply switch the proxy back to the old mailbox and re-enable delivery.
No mail will be lost, since it should remain in the remote MTA's mail queue for a while in order to be retried and delivered later.
Regards Daniel
On 12/10/2013 19:22, Daniel Parthey wrote:
No mail will be lost, since it should remain in the remote MTA's mail queue for a while in order to be retried and delivered later.
No guarantee there, some services are broken and do not retry, hotmail used to, and I've heard in some cases, still does, do this, some marketing system (ok, so thats no loss) do this - there reasoning is because of such high outbound queues, it would only delay first runs and upset their clients, again, no loss to me, but one persons spam can be anothers ham.
It is after all why we have secondary MX's, on network, and if need be, off network.
On 10/12/2013 3:43 AM, Noel Butler wrote:
On 12/10/2013 19:22, Daniel Parthey wrote:
No mail will be lost, since it should remain in the remote MTA's mail queue for a while in order to be retried and delivered later.
No guarantee there, some services are broken and do not retry, hotmail used to, and I've heard in some cases, still does, do this, some marketing system (ok, so thats no loss) do this - there reasoning is because of such high outbound queues, it would only delay first runs and upset their clients, again, no loss to me, but one persons spam can be anothers ham.
It is after all why we have secondary MX's, on network, and if need be, off network.
Instead of deferring the message and returning a 4xx to the remote client, accept it normally and put it into a hold queue or defer the delivery transport. After the switchover, requeue the message.
Hey Jogi,
On 06.10.2013 12:39, Jogi Hofmüller wrote:
Our goal is to do the migration without interrupting the service for our users too much. Currently we tend to using dsync. So I am asking for best practice suggestions, tips and hints from people who have done such a thing before.
I work for NetCologne GmbH, an ISP in Cologne, Germany. I did a talk "Austausch einer ISP-Mailplattform ohne Downtime" at the mail server conference the Heinlein-Support company held in Berlin in 2011.
https://www.youtube.com/watch?v=kLQOkiBebU0
It's sure a bit dated and we started with dovecot 1.2.x, so no dsync available. But maybe it's a least somewhat entertaining to watch how we did things and avoided downtime.
Regards
Christian
Dear Christian,
Am 2013-10-10 17:06, schrieb Christian Rohmann:
I work for NetCologne GmbH, an ISP in Cologne, Germany. I did a talk "Austausch einer ISP-Mailplattform ohne Downtime" at the mail server conference the Heinlein-Support company held in Berlin in 2011.
Thanks for the video! Unfortunately most things that allowed you to do migration in the file system don't apply for us (e.g. we have mailboxes in the GB range). So I think we will go for dsync and dovecot's proxy features ;)
Regards,
j.hofmüller
mur.sat -- a space art project http://sat.mur.at/
On Thursday 10 Oct 2013 19:34:51 Jogi Hofmüller wrote:
Dear Christian,
Am 2013-10-10 17:06, schrieb Christian Rohmann:
I work for NetCologne GmbH, an ISP in Cologne, Germany. I did a talk "Austausch einer ISP-Mailplattform ohne Downtime" at the mail server conference the Heinlein-Support company held in Berlin in 2011.
Thanks for the video! Unfortunately most things that allowed you to do migration in the file system don't apply for us (e.g. we have mailboxes in the GB range). So I think we will go for dsync and dovecot's proxy features ;)
Perhaps rather 'old-school' now, but I've used Perdition for large transparent migrations in the past very successfully,
http://horms.net/projects/perdition/with Dovecot being your target platform, it makes sense to explore the Dovecot [proxy] approach first; Perdition may be handy to have as a backup strategy.
cheers,
Andrew.
participants (7)
-
Andrew Richards
-
Christian Rohmann
-
Daniel Parthey
-
Darren Pilgrim
-
Ed W
-
Jogi Hofmüller
-
Noel Butler