Migrating from Cyrus to Dovecot
Hi,
we are migrating from Cyrus 2.3.7 to Dovecot 2.2.13. We have ~7000 maildirs with ~500GB. Our goal is to do the migration without users have notice and with the shortest service downtime. The users use IMAP (with shared folders and ACL), POP3 and sieve filters.
As a first choice, we tried the Dovecot's dsync tool. First tests were great, but we are not able to change the Cyrus auth backend for migration. Moreover, this migration seems too slow for us.
As a second try, we tried the cyrus2dovecot migrating Perl scripts (and their derivates) from Wiki2. More or less they works but we found we need more control during the migration.
So, as a third try, we wrote own migrating scripts. And thanks to the cyrus2dovecot it wasn't too much complicated. And there are my questions:
A) Files and dirs timestamps The mtime of email file is important as an internal date as I found on Wiki2. But what about timestamps of cur/new/tmp directories or Dovecot's internal files line dovecot-uidlist? Do they play some role here?
B) The 128 bit mailbox UID The Wiki2 speaks about 128 bit mailbox UID at first line of dovecot-uidlist. Cyrus preserves only 64 bit UID. Is this mailbox UID required by Dovecot? If so, can we use 0000000000000000501100008c4a11c1 (Cyrus UID padded by zeros)?
C) Format of dovecot-uidlist records Wiki2 shows two examples: 25006 :1276528487.M364837P9451.kurkku,S=1355,W=1394:2, 25017 W2481 :1276533073.M242911P3632.kurkku:2,F
Which format is preferred? Or what the benefits are?
D) Converting between CRLF and LF If I understand correctly, Dovecot stores emails with LF only. We have all emails with CRLF now on Cyrus and converting them to LF only is a little more time consuming. Is there any benefit to do that? Or can we live with 'mail_save_crlf' without problems?
E) POP3 backend I found many informations about IMAP internals but few on POP3 internals. What do I need to do POP3 migration transparent for user?
Many thanks for any answers.
Regards, Milo
On 27 Mar 2015, at 10:19, Miloslav Hůla miloslav.hula@gmail.com wrote:
Hi,
we are migrating from Cyrus 2.3.7 to Dovecot 2.2.13. We have ~7000 maildirs with ~500GB. Our goal is to do the migration without users have notice and with the shortest service downtime. The users use IMAP (with shared folders and ACL), POP3 and sieve filters.
As a first choice, we tried the Dovecot's dsync tool. First tests were great, but we are not able to change the Cyrus auth backend for migration. Moreover, this migration seems too slow for us.
As a second try, we tried the cyrus2dovecot migrating Perl scripts (and their derivates) from Wiki2. More or less they works but we found we need more control during the migration.
So, as a third try, we wrote own migrating scripts. And thanks to the cyrus2dovecot it wasn't too much complicated. And there are my questions:
A) Files and dirs timestamps The mtime of email file is important as an internal date as I found on Wiki2. But what about timestamps of cur/new/tmp directories or Dovecot's internal files line dovecot-uidlist? Do they play some role here?
No.
B) The 128 bit mailbox UID The Wiki2 speaks about 128 bit mailbox UID at first line of dovecot-uidlist. Cyrus preserves only 64 bit UID. Is this mailbox UID required by Dovecot? If so, can we use 0000000000000000501100008c4a11c1 (Cyrus UID padded by zeros)?
The mailbox GUID is internal to Dovecot. There's no standard IMAP way to see it, so there's no need to migrate it. Better not to set it and let Dovecot generate it automatically.
C) Format of dovecot-uidlist records Wiki2 shows two examples: 25006 :1276528487.M364837P9451.kurkku,S=1355,W=1394:2, 25017 W2481 :1276533073.M242911P3632.kurkku:2,F
Which format is preferred? Or what the benefits are?
If W=size is in the filename, it never needs to be recalculated if dovecot-uidlist is lost. Of course, dovecot-uidlist should never be lost. So I don't think it makes a huge difference. If you care about performance, sdbox/mdbox mailbox format would behave much better. sdbox is a close match to Cyrus - so with Maildir you're actually likely making the disk I/O performance somewhat slower in Dovecot than in Cyrus, although that also depends on other things.
D) Converting between CRLF and LF If I understand correctly, Dovecot stores emails with LF only. We have all emails with CRLF now on Cyrus and converting them to LF only is a little more time consuming. Is there any benefit to do that? Or can we live with 'mail_save_crlf' without problems?
Dovecot can automatically handle both mixed CRLF and LF mails, you can keep old mails as CRLF and new mails as LF. mail_save_crlf setting only controls what is used for new emails. If you want to save more disk space you can enable compression.
E) POP3 backend I found many informations about IMAP internals but few on POP3 internals. What do I need to do POP3 migration transparent for user?
Just preserve the UIDL. See the pop3_uidl_format setting in http://wiki2.dovecot.org/Migration/Cyrus
Hi Timo,
thank you for the valuable answers!
Milo
Dne 27.3.2015 v 22:12 Timo Sirainen napsal(a):
On 27 Mar 2015, at 10:19, Miloslav Hůla miloslav.hula@gmail.com wrote:
Hi,
we are migrating from Cyrus 2.3.7 to Dovecot 2.2.13. We have ~7000 maildirs with ~500GB. Our goal is to do the migration without users have notice and with the shortest service downtime. The users use IMAP (with shared folders and ACL), POP3 and sieve filters.
As a first choice, we tried the Dovecot's dsync tool. First tests were great, but we are not able to change the Cyrus auth backend for migration. Moreover, this migration seems too slow for us.
As a second try, we tried the cyrus2dovecot migrating Perl scripts (and their derivates) from Wiki2. More or less they works but we found we need more control during the migration.
So, as a third try, we wrote own migrating scripts. And thanks to the cyrus2dovecot it wasn't too much complicated. And there are my questions:
A) Files and dirs timestamps The mtime of email file is important as an internal date as I found on Wiki2. But what about timestamps of cur/new/tmp directories or Dovecot's internal files line dovecot-uidlist? Do they play some role here?
No.
B) The 128 bit mailbox UID The Wiki2 speaks about 128 bit mailbox UID at first line of dovecot-uidlist. Cyrus preserves only 64 bit UID. Is this mailbox UID required by Dovecot? If so, can we use 0000000000000000501100008c4a11c1 (Cyrus UID padded by zeros)?
The mailbox GUID is internal to Dovecot. There's no standard IMAP way to see it, so there's no need to migrate it. Better not to set it and let Dovecot generate it automatically.
C) Format of dovecot-uidlist records Wiki2 shows two examples: 25006 :1276528487.M364837P9451.kurkku,S=1355,W=1394:2, 25017 W2481 :1276533073.M242911P3632.kurkku:2,F
Which format is preferred? Or what the benefits are?
If W=size is in the filename, it never needs to be recalculated if dovecot-uidlist is lost. Of course, dovecot-uidlist should never be lost. So I don't think it makes a huge difference. If you care about performance, sdbox/mdbox mailbox format would behave much better. sdbox is a close match to Cyrus - so with Maildir you're actually likely making the disk I/O performance somewhat slower in Dovecot than in Cyrus, although that also depends on other things.
D) Converting between CRLF and LF If I understand correctly, Dovecot stores emails with LF only. We have all emails with CRLF now on Cyrus and converting them to LF only is a little more time consuming. Is there any benefit to do that? Or can we live with 'mail_save_crlf' without problems?
Dovecot can automatically handle both mixed CRLF and LF mails, you can keep old mails as CRLF and new mails as LF. mail_save_crlf setting only controls what is used for new emails. If you want to save more disk space you can enable compression.
E) POP3 backend I found many informations about IMAP internals but few on POP3 internals. What do I need to do POP3 migration transparent for user?
Just preserve the UIDL. See the pop3_uidl_format setting in http://wiki2.dovecot.org/Migration/Cyrus
participants (2)
-
Miloslav Hůla
-
Timo Sirainen