Server migration
Hi i have to migrate, online, a dovecot 1.2.15 to a new server. Which is the best way to accomplish this?
I have 2 possibility:
- migrate from the very old server to a newer server with the same dovecot version
- migrate from the very old server to a new server with the latest dovecot version
can i simply use rsync to sync everything and, when the sync is quick, move the mailbox from the old server to the new server? My biggest concern is how to manage the the emails that are coming during the server switch.
Let's assume a 50gb maildir , the first sync would require hours to complete (tons of very small files) do i can't shutdown the mailbox. The second sync would require much less time and would also sync the email received during the first sync (but the mailbox is still receiving new emails) now, as third phase, i can move the mailbox to the new server (by changing the postfix configuration) so that all new emails are received on the new server and then start the last rsync (by removing the --delete flag or any new emails would be deleted as not existsnt on the older server)
Any better solution?
On 2016-10-24 8:00, Gandalf Corvotempesta wrote:
can i simply use rsync to sync everything and, when the sync is quick, move the mailbox from the old server to the new server? My biggest concern is how to manage the the emails that are coming during the server switch.
Let's assume a 50gb maildir , the first sync would require hours to complete (tons of very small files) do i can't shutdown the mailbox. The second sync would require much less time and would also sync the email received during the first sync (but the mailbox is still receiving new emails) now, as third phase, i can move the mailbox to the new server (by changing the postfix configuration) so that all new emails are received on the new server and then start the last rsync (by removing the --delete flag or any new emails would be deleted as not existsnt on the older server)
Any better solution?
When I am doing this I just turn off both servers for the third sync. Its short enough to not cause much problem. And then after third sync I start the new server and all clients can connect to it so I also mitigate any problems resulting from clients that would be still connected to the old server. The last issue depends on the way you force everyone to use new server (DNS, routing, etc).
Remember that beside the new emails that could arrive during sync you have also all sorts of user-generated operations as move, delete etc. So if you just do 3rd rsync without --delete you can end up duplicating users' emails if they move them during procedure.
Best, Karol
2016-10-24 11:23 GMT+02:00 Karol Augustin karol@augustin.pl:
When I am doing this I just turn off both servers for the third sync. Its short enough to not cause much problem. And then after third sync I start the new server and all clients can connect to it so I also mitigate any problems resulting from clients that would be still connected to the old server. The last issue depends on the way you force everyone to use new server (DNS, routing, etc).
The speed for third sync depends on the number of files to be scanned. I have mailboxes with tons of very small emails, thus even if the first two sync has transferred all datas, the scan made by rsync to check which files needs to be transferred requires many hours.
My own mailbox has 80GB of mails. I can sync everything on a new server and then start a new rsync phase. this new phase requires exactly 1 hours and 49 minutes (as I can see from the last night backup). Transferred data: 78MB. 1 hours and 49 minutes to transfer only 78MB.
Remember that beside the new emails that could arrive during sync you have also all sorts of user-generated operations as move, delete etc. So if you just do 3rd rsync without --delete you can end up duplicating users' emails if they move them during procedure.
By shutting down both servers, the "--delete" argument could be used with no issues.
Am 24.10.2016 um 09:00 schrieb Gandalf Corvotempesta:
Hi i have to migrate, online, a dovecot 1.2.15 to a new server. Which is the best way to accomplish this?
I have 2 possibility:
- migrate from the very old server to a newer server with the same dovecot version
- migrate from the very old server to a new server with the latest dovecot version
can i simply use rsync to sync everything and, when the sync is quick, move the mailbox from the old server to the new server? My biggest concern is how to manage the the emails that are coming during the server switch.
Let's assume a 50gb maildir , the first sync would require hours to complete (tons of very small files) do i can't shutdown the mailbox. The second sync would require much less time and would also sync the email received during the first sync (but the mailbox is still receiving new emails) now, as third phase, i can move the mailbox to the new server (by changing the postfix configuration) so that all new emails are received on the new server and then start the last rsync (by removing the --delete flag or any new emails would be deleted as not existsnt on the older server)
Any better solution?
If your server OS supports newer Dovecot versions then I would highly suggest you to upgrade to Dovecot 2.2.xx (or at least to the latest 2.1) and set up Dovecot's replication[1] feature.
With this method you can actually archieve a smooth migration while your current server replicates all emails in real time to your new server, including new incoming emails and also mailbox changes to your new server and when the migration is done you'll just have to change your DNS and disable the Replication service.
If you don't want or cannot set up replication you could still do a one-shot migration via Dovecot's dsync[2] on the new server, pulling the mails from the old. 50GB isn't that much as long as your two servers are at least connected with 100 Mbit to the inet. You may want to block for the time of the migration via iptables your users accessing Dovecot. However, under the bottom-line, if this is really necessary depends on you and the needs of your mailusers/customers.
Best regards Michael Seevogel
P.S. You should think about to use on the new server mdbox as mailbox format. That's kinda a hybrid of mbox and maildir and benefits of features of both its predecessors. However, backup and restoring is in case of mdbox "a bit" different. Just have a read...
[1] http://wiki.dovecot.org/Replication [2] http://wiki2.dovecot.org/Migration/Dsync
2016-10-24 14:47 GMT+02:00 Michael Seevogel ms@ddnetservice.de:
If your server OS supports newer Dovecot versions then I would highly suggest you to upgrade to Dovecot 2.2.xx (or at least to the latest 2.1) and set up Dovecot's replication[1] feature.
Are you talking about the new server or the older one that I have to replace? The new server has to be installed from scratch, so, yes, I can use Dovecot 2.2 from Jessie
The "old" server is based on Squeeze, I can upgrade that to Wheezy and install Dovecot 2.2 from wheezy-backports but I have huge trouble when I've tried to do the same on a similiar server. I was unable to upgrade the dovecot configuration by following the documentation as this didn't work:
doveconf -n -c /etc/dovecot/dovecot.conf > dovecot-2.conf
I had an empty dovecot-2.conf file, no warning or output at all. It did nothing.
With this method you can actually archieve a smooth migration while your current server replicates all emails in real time to your new server, including new incoming emails and also mailbox changes to your new server and when the migration is done you'll just have to change your DNS and disable the Replication service.
Cool. Any guide about this ? Should I start the replication on one side and wait for finish before pointing the mailbox to the new server?
If you don't want or cannot set up replication you could still do a one-shot migration via Dovecot's dsync[2] on the new server, pulling the mails from the old. 50GB isn't that much as long as your two servers are at least connected with 100 Mbit to the inet. You may want to block for the time of the migration via iptables your users accessing Dovecot. However, under the bottom-line, if this is really necessary depends on you and the needs of your mailusers/customers.
I can't block the whole server. I have to migrate 1 user at once. But I can disable the pop3/imap access for that user, so noone is changing the files during the migration (except for the postfix/exim delivery agent)
P.S. You should think about to use on the new server mdbox as mailbox format. That's kinda a hybrid of mbox and maildir and benefits of features of both its predecessors. However, backup and restoring is in case of mdbox "a bit" different. Just have a read...
[1] http://wiki.dovecot.org/Replication [2] http://wiki2.dovecot.org/Migration/Dsync
Thank you
Am 24.10.2016 um 15:25 schrieb Gandalf Corvotempesta:
2016-10-24 14:47 GMT+02:00 Michael Seevogel ms@ddnetservice.de:
If your server OS supports newer Dovecot versions then I would highly suggest you to upgrade to Dovecot 2.2.xx (or at least to the latest 2.1) and set up Dovecot's replication[1] feature.
Are you talking about the new server or the older one that I have to replace? The new server has to be installed from scratch, so, yes, I can use Dovecot 2.2 from Jessie
I meant your old server. With "old" I was expecting something like Debian Sarge or SuSE Linux 9.3. That would have been really old, but since you are on Debian Squeeze, I would definitely choose the way with an upgraded Dovecot version and its replication service.
The "old" server is based on Squeeze, I can upgrade that to Wheezy and install Dovecot 2.2 from wheezy-backports but I have huge trouble when I've tried to do the same on a similiar server. I was unable to upgrade the dovecot configuration by following the documentation as this didn't work:
doveconf -n -c /etc/dovecot/dovecot.conf > dovecot-2.conf
I had an empty dovecot-2.conf file, no warning or output at all. It did nothing.
Well, I'am not too familiar with Debian since I'am a Red Hatter but perhaps you could use the binaries from there: http://wiki2.dovecot.org/PrebuiltBinaries
Dunno if you have to rebuild the binaries, or if you can install them straight on Squeeze. You could also try to convert your old dovecot.conf on a different machine (maybe your new server?) and then just copy it back to your old server.
As a last straw you could certainly adapt the dovecot.conf for Dovecot 2.2 manually, it shouldn't be too complicated, but this is totally up to you.
With this method you can actually archieve a smooth migration while your current server replicates all emails in real time to your new server, including new incoming emails and also mailbox changes to your new server and when the migration is done you'll just have to change your DNS and disable the Replication service.
Cool. Any guide about this ? Should I start the replication on one side and wait for finish before pointing the mailbox to the new server?
How to setup and start replication is described here: http://wiki2.dovecot.org/Replication
Also make sure that you migrate/copy your userdb from the old server to the new server and that you properly test the user-mailbox access on the new server before you start the replication process.
Regarding replication: I would wait with adjusting the DNS records until the replication has finished and you know that the new server works as expected.
However, you may want to keep the replication process running for one or two more days to catch emails still arriving due to DNS caching times on your old server. The same may apply to mailusers that still access your old server via POP3/IMAP.
Best regards Michael Seevogel
Il 24 ott 2016 5:11 PM, "Michael Seevogel" ms@ddnetservice.de ha scritto:
I meant your old server. With "old" I was expecting something like Debian Sarge or SuSE Linux 9.3. That would have been really old, but since you are on Debian Squeeze, I would definitely choose the way with an upgraded Dovecot version and its replication service.
Is 2.1 from squeeze-backports enough to start the replication over a newer server with dovecot 2.2? Is this supported or both server must run the same version?
I've looked around but the replication system is still not clear to me. Any howto explaining this in details?
On 26.10.2016 09:27, Gandalf Corvotempesta wrote:
Il 24 ott 2016 5:11 PM, "Michael Seevogel" ms@ddnetservice.de ha scritto:
I meant your old server. With "old" I was expecting something like Debian Sarge or SuSE Linux 9.3. That would have been really old, but since you are on Debian Squeeze, I would definitely choose the way with an upgraded Dovecot version and its replication service.
Is 2.1 from squeeze-backports enough to start the replication over a newer server with dovecot 2.2? Is this supported or both server must run the same version?
I've looked around but the replication system is still not clear to me. Any howto explaining this in details?
Hi!
I would recommend using same major release with replication.
If you are using maildir++ format, it should be enough to copy all the maildir files over and start dovecot on new server.
Aki Tuomi Dovecot oy
Il 26 ott 2016 8:30 AM, "Aki Tuomi" aki.tuomi@dovecot.fi ha scritto:
I would recommend using same major release with replication.
If you are using maildir++ format, it should be enough to copy all the maildir files over and start dovecot on new server.
This is much easier than dovecot replication as i can start immedialy with no need to upgrade the old server
my only question is: how to manage the email received on the new server during the last rsync phase? As i wrote previously, i have some huge maildirs where rsync take hours to scan all files i can't keep the server down for hours or customers won't receive any new emails, so, after the initial sync i have to move the mailbox on the new server (only for deliveries) . In this way I'll not loose any emails but the new servers as newer data than the old server. When doing rsync with --delete, the news mails would be removed
A solution could be to disable customer access to the new server and put "new" directory in rsync exclude. Doing this won't delete the newly received emails as the "new" directory isn't synced. and no one osd able to move from new to cur as users are blocked for login.
On 26.10.2016 09:38, Gandalf Corvotempesta wrote:
Il 26 ott 2016 8:30 AM, "Aki Tuomi" aki.tuomi@dovecot.fi ha scritto:
I would recommend using same major release with replication.
If you are using maildir++ format, it should be enough to copy all the maildir files over and start dovecot on new server.
This is much easier than dovecot replication as i can start immedialy with no need to upgrade the old server
my only question is: how to manage the email received on the new server during the last rsync phase? As i wrote previously, i have some huge maildirs where rsync take hours to scan all files i can't keep the server down for hours or customers won't receive any new emails, so, after the initial sync i have to move the mailbox on the new server (only for deliveries) . In this way I'll not loose any emails but the new servers as newer data than the old server. When doing rsync with --delete, the news mails would be removed
A solution could be to disable customer access to the new server and put "new" directory in rsync exclude. Doing this won't delete the newly received emails as the "new" directory isn't synced. and no one osd able to move from new to cur as users are blocked for login.
If you are moving from 1.x to 2.x, I think you should make some trials first, and preferably move the user one at a time, blocking access to old server/new server during move. It is very forklift upgrade, much danger.
Aki
2016-10-26 8:57 GMT+02:00 Aki Tuomi aki.tuomi@dovecot.fi:
If you are moving from 1.x to 2.x, I think you should make some trials first, and preferably move the user one at a time, blocking access to old server/new server during move. It is very forklift upgrade, much danger.
Yes, I'll do some test migration before moving the whole server. Maildir structure isn't changed between 1.x and 2.x, thus all emails should be safe. I have to test the new 2.2 configuration to see if existing users are able to log-in but how can I test if existing client would be able to preserve the mail ids without downloading everything again?
On 10/26/2016 2:38 AM, Gandalf Corvotempesta gandalf.corvotempesta@gmail.com wrote:
This is much easier than dovecot replication as i can start immedialy with no need to upgrade the old server
my only question is: how to manage the email received on the new server during the last rsync phase?
Use IMAPSync - much better than rsync for this.
On 27 Oct 2016, at 15:29, Tanstaafl tanstaafl@libertytrek.org wrote:
On 10/26/2016 2:38 AM, Gandalf Corvotempesta gandalf.corvotempesta@gmail.com wrote:
This is much easier than dovecot replication as i can start immedialy with no need to upgrade the old server
my only question is: how to manage the email received on the new server during the last rsync phase?
Use IMAPSync - much better than rsync for this.
imapsync will change IMAP UIDs and cause clients to redownload all mails. http://wiki2.dovecot.org/Migration/Dsync should work though.
2016-10-27 14:36 GMT+02:00 Timo Sirainen tss@iki.fi:
imapsync will change IMAP UIDs and cause clients to redownload all mails. http://wiki2.dovecot.org/Migration/Dsync should work though.
Just to be sure: dsync from the *new* node would connect via IMAP to the older node and transfer everything ? By running this:
doveadm -o mail_fsync=never sync -1 -R -u user@domain imapc:
should be OK if newer mails are arrived on the new server ?
On 27 Oct 2016, at 15:58, Gandalf Corvotempesta gandalf.corvotempesta@gmail.com wrote:
2016-10-27 14:36 GMT+02:00 Timo Sirainen tss@iki.fi:
imapsync will change IMAP UIDs and cause clients to redownload all mails. http://wiki2.dovecot.org/Migration/Dsync should work though.
Just to be sure: dsync from the *new* node would connect via IMAP to the older node and transfer everything ? By running this:
doveadm -o mail_fsync=never sync -1 -R -u user@domain imapc:
should be OK if newer mails are arrived on the new server ?
Yes.
On 10/27/2016 8:36 AM, Timo Sirainen tss@iki.fi wrote:
On 27 Oct 2016, at 15:29, Tanstaafl tanstaafl@libertytrek.org wrote:
On 10/26/2016 2:38 AM, Gandalf Corvotempesta
my only question is: how to manage the email received on the new server during the last rsync phase?
Use IMAPSync - much better than rsync for this.
imapsync will change IMAP UIDs and cause clients to redownload all mails. http://wiki2.dovecot.org/Migration/Dsync should work though.
Oh... I thought the --useuid option eliminated this problem?
On 28 Oct 2016, at 16.54, Tanstaafl tanstaafl@libertytrek.org wrote:
On 10/27/2016 8:36 AM, Timo Sirainen tss@iki.fi wrote:
On 27 Oct 2016, at 15:29, Tanstaafl tanstaafl@libertytrek.org wrote:
On 10/26/2016 2:38 AM, Gandalf Corvotempesta
my only question is: how to manage the email received on the new server during the last rsync phase?
Use IMAPSync - much better than rsync for this.
imapsync will change IMAP UIDs and cause clients to redownload all mails. http://wiki2.dovecot.org/Migration/Dsync should work though.
Oh... I thought the --useuid option eliminated this problem?
It does not. There is no option at IMAP level to set the UID.
In this case —useuid seems to keep track on source:uid -> dest:uid pairs on multiple syncs and uses uid numbers to avoid syncing mails as duplicates instead of using headers to do that.
Sami
On 10/30/2016 5:32 AM, Sami Ketola sami.ketola@dovecot.fi wrote:
On 28 Oct 2016, at 16.54, Tanstaafl tanstaafl@libertytrek.org wrote:
Oh... I thought the --useuid option eliminated this problem?
It does not. There is no option at IMAP level to set the UID.
In this case —useuid seems to keep track on source:uid -> dest:uid pairs on multiple syncs and uses uid numbers to avoid syncing mails as duplicates instead of using headers to do that.
Ok, interesting. So... how does dsync do it? Or would it only work between two dovecot servers?
I'm interested in migrating from other servers (Office 365 in one case).
Thanks,
Charles
On 31 Oct 2016, at 13.11, Tanstaafl tanstaafl@libertytrek.org wrote:
On 10/30/2016 5:32 AM, Sami Ketola sami.ketola@dovecot.fi wrote:
On 28 Oct 2016, at 16.54, Tanstaafl tanstaafl@libertytrek.org wrote:
Oh... I thought the --useuid option eliminated this problem?
It does not. There is no option at IMAP level to set the UID.
In this case —useuid seems to keep track on source:uid -> dest:uid pairs on multiple syncs and uses uid numbers to avoid syncing mails as duplicates instead of using headers to do that.
Ok, interesting. So... how does dsync do it? Or would it only work between two dovecot servers?
I'm interested in migrating from other servers (Office 365 in one case).
Dsync does not use IMAP protocol to store the mails to storage but instead uses the dovecot storage API to do that. Internally we can set what ever properties we want to including IMAP UIDs and POP3 UIDLs.
Migrating from legacy system should then be done by pulling the mails from the legacy platform by using the imapc connector and storing them by using the internal apis.
We can also store mails to imapc: location but in that case there is many properties that will be lost due to limitations of the IMAP protocol.
Sami
On 11/1/2016 3:58 AM, Sami Ketola sami.ketola@dovecot.fi wrote:
On 31 Oct 2016, at 13.11, Tanstaafl tanstaafl@libertytrek.org wrote:
Ok, interesting. So... how does dsync do it? Or would it only work between two dovecot servers?
I'm interested in migrating from other servers (Office 365 in one case).
Dsync does not use IMAP protocol to store the mails to storage but instead uses the dovecot storage API to do that. Internally we can set what ever properties we want to including IMAP UIDs and POP3 UIDLs.
Migrating from legacy system should then be done by pulling the mails from the legacy platform by using the imapc connector and storing them by using the internal apis.
We can also store mails to imapc: location but in that case there is many properties that will be lost due to limitations of the IMAP protocol.
Thanks Sami, but I don't see a definitive answer top my question in the above...
So, when migrating from legacy system (legacy = non-dovecot) using imapc, is dovecot able to preserver the UIDs?
Thanks, and my apologies for being a bit dense...
On 1 Nov 2016, at 14.10, Tanstaafl tanstaafl@libertytrek.org wrote:
On 11/1/2016 3:58 AM, Sami Ketola sami.ketola@dovecot.fi wrote:
On 31 Oct 2016, at 13.11, Tanstaafl tanstaafl@libertytrek.org wrote:
Ok, interesting. So... how does dsync do it? Or would it only work between two dovecot servers?
I'm interested in migrating from other servers (Office 365 in one case).
Dsync does not use IMAP protocol to store the mails to storage but instead uses the dovecot storage API to do that. Internally we can set what ever properties we want to including IMAP UIDs and POP3 UIDLs.
Migrating from legacy system should then be done by pulling the mails from the legacy platform by using the imapc connector and storing them by using the internal apis.
We can also store mails to imapc: location but in that case there is many properties that will be lost due to limitations of the IMAP protocol.
Thanks Sami, but I don't see a definitive answer top my question in the above...
So, when migrating from legacy system (legacy = non-dovecot) using imapc, is dovecot able to preserver the UIDs?
If you fetch emails over imapc and store to dovecot dsync will preserve IMAP UIDs.
Sami
2016-10-24 14:47 GMT+02:00 Michael Seevogel ms@ddnetservice.de:
P.S. You should think about to use on the new server mdbox as mailbox format. That's kinda a hybrid of mbox and maildir and benefits of features of both its predecessors. However, backup and restoring is in case of mdbox "a bit" different. Just have a read...
No, I don't like that format, for this: This also means that you must not lose the dbox index files, they can't be regenerated without data loss
additionally, this means to change even our LDA, as neither Exim or Postfix are able to deliver messages.
participants (7)
-
Aki Tuomi
-
Gandalf Corvotempesta
-
Karol Augustin
-
Michael Seevogel
-
Sami Ketola
-
Tanstaafl
-
Timo Sirainen