[Dovecot] Complete migration from Cyrus on remote server
Hi All!
I am working on a complete migration from an old solaris machine running Cyrus IMAP v2.3.14 and wish to migrate all users to a new ubuntu vm running Dovecot 2.0.19.
What I have so far is a fully functional dovecot installation with LDAP / Dovecot SASL auth using Maildir++.
My plan is to use Perdition IMAP proxy on a third host and migrate users a few at a time (see attached picture of my whiteboard).
The proxy is in place and already used by webmail clients to connect to the old server, shortly I will change the dns to point smtp.domain.com and imap.domain.com etc to the proxy instead of the old server but I haven't finished testing yet. Once that is done it is a simple matter of changing the proxy configuration per user to make them hit the new server instead.
The problem I am having with this is that I am using Imapsync (http://imapsync.lamiral.info) to perform the initial synchronization (i have scripted it to maintain sync every 10 minutes with the addtion of "--maxage 1" option) using the following command line options:
/usr/local/bin/imapsync --noreleasecheck --useheader ALL --delete2 --expunge2 --tmpdir /var/vmail/cache --subscribe --subscribed --tls1 --tls2 --usecache --nofoldersizes --nocheckmessageexists --host1 oldserver.domain.com --authuser1 adminuser --user1 testuser@domain.com --authmech1 PLAIN --password1 <HIDDEN> --host2 newserver.domain.com --authuser2 adminuser --user2 testuser@domain.com --authmech2 PLAIN --password2 <HIDDEN>
All email is copied over and stored in dovecot as you would expect. Then I switch the user to the new server on the imapproxy and restart the client (thunderbird or whatever). This is where all email headers are downloaded again as if they were new! There are *Many* users here who would complain intensely if this happened and I was really wanting the migration to be as invisible as possible.
Is there a way to do what I have described without the clients redownloading headers / full messages?
I have been working on this for some time now but have hit a roadblock and don't know where to turn, any help would be really appreciated!
Notes:
- The proxy is not used during imapsync.
- Some users have 20 years worth of mail running into 10's of GB
- I have looked at dsync but the docs state that the mailbox should not be accessed during sync, this is not possible for me (it doesn't say why this is the case however)
Zitat von Alan McGinlay - SICS <alanm@sics.se>:
Hi All!
I am working on a complete migration from an old solaris machine running Cyrus IMAP v2.3.14 and wish to migrate all users to a new ubuntu vm running Dovecot 2.0.19.
What I have so far is a fully functional dovecot installation with LDAP / Dovecot SASL auth using Maildir++.
My plan is to use Perdition IMAP proxy on a third host and migrate users a few at a time (see attached picture of my whiteboard).
The proxy is in place and already used by webmail clients to connect to the old server, shortly I will change the dns to point smtp.domain.com and imap.domain.com etc to the proxy instead of the old server but I haven't finished testing yet. Once that is done it is a simple matter of changing the proxy configuration per user to make them hit the new server instead.
The problem I am having with this is that I am using Imapsync (http://imapsync.lamiral.info) to perform the initial synchronization (i have scripted it to maintain sync every 10 minutes with the addtion of "--maxage 1" option) using the following command line options:
/usr/local/bin/imapsync --noreleasecheck --useheader ALL --delete2 --expunge2 --tmpdir /var/vmail/cache --subscribe --subscribed --tls1 --tls2 --usecache --nofoldersizes --nocheckmessageexists --host1 oldserver.domain.com --authuser1 adminuser --user1 testuser@domain.com --authmech1 PLAIN --password1 <HIDDEN> --host2 newserver.domain.com --authuser2 adminuser --user2 testuser@domain.com --authmech2 PLAIN --password2 <HIDDEN>
All email is copied over and stored in dovecot as you would expect. Then I switch the user to the new server on the imapproxy and restart the client (thunderbird or whatever). This is where all email headers are downloaded again as if they were new! There are *Many* users here who would complain intensely if this happened and I was really wanting the migration to be as invisible as possible.
Is there a way to do what I have described without the clients redownloading headers / full messages?
I have been working on this for some time now but have hit a roadblock and don't know where to turn, any help would be really appreciated!
Notes:
- The proxy is not used during imapsync.
- Some users have 20 years worth of mail running into 10's of GB
- I have looked at dsync but the docs state that the mailbox should not be accessed during sync, this is not possible for me (it doesn't say why this is the case however)
Sounds like you have not adjusted the message UID to not change on the
new server?
http://wiki2.dovecot.org/Migration
Regards
Andreas
On 3.12.2013, at 16.01, Alan McGinlay - SICS <alanm@sics.se> wrote:
I am working on a complete migration from an old solaris machine running Cyrus IMAP v2.3.14 and wish to migrate all users to a new ubuntu vm running Dovecot 2.0.19.
v2.0 is rather old.
All email is copied over and stored in dovecot as you would expect. Then I switch the user to the new server on the imapproxy and restart the client (thunderbird or whatever). This is where all email headers are downloaded again as if they were new! There are *Many* users here who would complain intensely if this happened and I was really wanting the migration to be as invisible as possible.
Is there a way to do what I have described without the clients redownloading headers / full messages?
I don’t think imapsync supports Dovecot’s dovecot-uidlist, so it can’t preserve the IMAP UIDs.
- I have looked at dsync but the docs state that the mailbox should not be accessed during sync, this is not possible for me (it doesn't say why this is the case however)
dsync is intended for these kinds of migrations. I’m not sure where you read that mailbox shouldn’t be accessed during it? Anyway v2.0 dsync is rather bad. Even if you end up using Dovecot v2.0, for the migration you could use Dovecot v2.2 dsync binary (installed e.g. under /tmp/dovecot-2.2). v2.2.9 dsync+imapc performance is very good when mail_prefetch_count=10 (or more).
On 12/ 3/13 02:17 PM, Timo Sirainen wrote:
On 3.12.2013, at 16.01, Alan McGinlay - SICS <alanm@sics.se> wrote:
- I have looked at dsync but the docs state that the mailbox should not be accessed during sync, this is not possible for me (it doesn't say why this is the case however) dsync is intended for these kinds of migrations. I’m not sure where you read that mailbox shouldn’t be accessed during it?
http://wiki2.dovecot.org/Migration/Dsync "The source IMAP/POP3 mailboxes shouldn't be modified while dsync is running."
On 3.12.2013, at 23.55, Oscar del Rio <delrio@mie.utoronto.ca> wrote:
On 12/ 3/13 02:17 PM, Timo Sirainen wrote:
On 3.12.2013, at 16.01, Alan McGinlay - SICS <alanm@sics.se> wrote:
- I have looked at dsync but the docs state that the mailbox should not be accessed during sync, this is not possible for me (it doesn't say why this is the case however) dsync is intended for these kinds of migrations. I’m not sure where you read that mailbox shouldn’t be accessed during it?
http://wiki2.dovecot.org/Migration/Dsync "The source IMAP/POP3 mailboxes shouldn't be modified while dsync is running.”
I’m not sure why I wrote that. Maybe it didn’t work as well in earlier versions. I removed the text now from the page.
Zitat von Timo Sirainen <tss@iki.fi>:
On 3.12.2013, at 23.55, Oscar del Rio <delrio@mie.utoronto.ca> wrote:
On 12/ 3/13 02:17 PM, Timo Sirainen wrote:
On 3.12.2013, at 16.01, Alan McGinlay - SICS <alanm@sics.se> wrote:
- I have looked at dsync but the docs state that the mailbox should not be accessed during sync, this is not possible for me (it doesn't say why this is the case however) dsync is intended for these kinds of migrations. I’m not sure
where you read that mailbox shouldn’t be accessed during it?http://wiki2.dovecot.org/Migration/Dsync "The source IMAP/POP3 mailboxes shouldn't be modified while dsync
is running.”I’m not sure why I wrote that. Maybe it didn’t work as well in
earlier versions. I removed the text now from the page.
We have done it by simply copy the Cyrus Mailbox, use the
"cyrus2dovecot" script listed here
http://wiki2.dovecot.org/Migration/Cyrus and afterwards apply dsync to
convert if maildir is not the intended target format for Dovecot. The
2.0.19 Dovecot version is the one included in Ubuntu 12.04 LTS BTW.
Regards
Andreas
On 2013-12-04 11:35, lst_hoe02@kwsoft.de wrote:
Zitat von Timo Sirainen <tss@iki.fi>:
On 3.12.2013, at 23.55, Oscar del Rio <delrio@mie.utoronto.ca> wrote:
On 12/ 3/13 02:17 PM, Timo Sirainen wrote: On 3.12.2013, at 16.01, Alan McGinlay - SICS <alanm@sics.se> wrote:
- I have looked at dsync but the docs state that the mailbox should not be accessed during sync, this is not possible for me (it doesn't say why this is the case however) dsync is intended for these kinds of migrations. I’m not sure where you read that mailbox shouldn’t be accessed during it?
http://wiki2.dovecot.org/Migration/Dsync "The source IMAP/POP3 mailboxes shouldn't be modified while dsync is running.”
I’m not sure why I wrote that. Maybe it didn’t work as well in
earlier versions. I removed the text now from the page.We have done it by simply copy the Cyrus Mailbox, use the "cyrus2dovecot" script listed here http://wiki2.dovecot.org/Migration/Cyrus and afterwards apply dsync to convert if maildir is not the intended target format for Dovecot. The 2.0.19 Dovecot version is the one included in Ubuntu 12.04 LTS BTW.
Regards
Andreas
Yeah unfortunately ubuntu / debian packages are somewhat outdated but the Dovecot docs say them selves not to use the later (nightly built) packages on systems that need to be stable.
On 2013-12-03 22:55, Oscar del Rio wrote:
On 12/ 3/13 02:17 PM, Timo Sirainen wrote: On 3.12.2013, at 16.01, Alan McGinlay - SICS <alanm@sics.se> wrote:
- I have looked at dsync but the docs state that the mailbox should not be accessed during sync, this is not possible for me (it doesn't say why this is the case however) dsync is intended for these kinds of migrations. I’m not sure where you read that mailbox shouldn’t be accessed during it?
http://wiki2.dovecot.org/Migration/Dsync "The source IMAP/POP3 mailboxes shouldn't be modified while dsync is running."
Thanks, yes indeed that is where I read that factoid. So what does that mean, it works but might produce corrupt messages on the receiver if the other side is accessed simultaneously? I did actually use Imapsync without first blocking access and it seemed to work (aside from the UID issues i mentioned)
participants (4)
-
Alan McGinlay - SICS
-
lst_hoe02＠kwsoft.de
-
Oscar del Rio
-
Timo Sirainen