[Dovecot] cyrus2courrier mass migration
Hello,
So we've chosen to use dovecot as our imap mail server and replace our old cyrus server ! BUT :
I would like to do a conversion of my imap accounts (and keep all theirs characteristics) from cyrus-imapd-2.2.12 in cyrus format to dovecot 1.1.3 in courrier format.
I've tried those 3 tools :
cyrus2courrier.1.5.ts cyrus2courier-1.3z4 imapsync (works well but very slow and should not be a true solution)
But nothing is working to keep the seen/unseen flag (expect imapsync) ! I know my cyrus version is too "recent" but do you have any idea how could I do this migration ?
A second problem is that I have 600 Go of imap account to migrate, if you've any clue on how to do a migration without loosing a lot of availability, I would be happy to read you !
Tanks in advance.
- Mathieu Kretchner <mathieu.kretchner@sophia.inria.fr> [2008-09-22 16:11]:
I would like to do a conversion of my imap accounts (and keep all theirs characteristics) from cyrus-imapd-2.2.12 in cyrus format to dovecot 1.1.3 in courrier format.
I've tried those 3 tools :
cyrus2courrier.1.5.ts cyrus2courier-1.3z4 imapsync (works well but very slow and should not be a true solution)
But nothing is working to keep the seen/unseen flag (expect imapsync) ! I know my cyrus version is too "recent" but do you have any idea how could I do this migration ?
cyrus2courier 1.5.ts should work with Cyrus 2.2.12 if the seen (and subscription) states are available in flat text files; otherwise, they must first be converted using cvt_cyrusdb(8).
However, for our conversion, we've written our own "cyrus2dovecot" which also supports skiplist databases (amongst another few features not provided by the other tools):
http://cyrus2dovecot.sw.fu-berlin.de/Holger
Holger Weiss wrote:
cyrus2courier 1.5.ts should work with Cyrus 2.2.12 if the seen (and subscription) states are available in flat text files; otherwise, they must first be converted using cvt_cyrusdb(8).
However, for our conversion, we've written our own "cyrus2dovecot" which also supports skiplist databases (amongst another few features not provided by the other tools):
Great! Im think it should uploaded to Timo`s migration wiki page.
-- Best regards, Proskurin Kirill
Holger Weiss a écrit :
- Mathieu Kretchner <mathieu.kretchner@sophia.inria.fr> [2008-09-22 16:11]:
I would like to do a conversion of my imap accounts (and keep all theirs characteristics) from cyrus-imapd-2.2.12 in cyrus format to dovecot 1.1.3 in courrier format.
I've tried those 3 tools :
cyrus2courrier.1.5.ts cyrus2courier-1.3z4 imapsync (works well but very slow and should not be a true solution)
But nothing is working to keep the seen/unseen flag (expect imapsync) ! I know my cyrus version is too "recent" but do you have any idea how could I do this migration ?
cyrus2courier 1.5.ts should work with Cyrus 2.2.12 if the seen (and subscription) states are available in flat text files; otherwise, they must first be converted using cvt_cyrusdb(8).
However, for our conversion, we've written our own "cyrus2dovecot" which also supports skiplist databases (amongst another few features not provided by the other tools):
http://cyrus2dovecot.sw.fu-berlin.de/
Holger
Thank you, I'm just going to try !! And I keep you informed about my experience!
Holger Weiss a écrit :
- Mathieu Kretchner <mathieu.kretchner@sophia.inria.fr> [2008-09-22 16:11]:
I would like to do a conversion of my imap accounts (and keep all theirs characteristics) from cyrus-imapd-2.2.12 in cyrus format to dovecot 1.1.3 in courrier format.
I've tried those 3 tools :
cyrus2courrier.1.5.ts cyrus2courier-1.3z4 imapsync (works well but very slow and should not be a true solution)
But nothing is working to keep the seen/unseen flag (expect imapsync) ! I know my cyrus version is too "recent" but do you have any idea how could I do this migration ?
cyrus2courier 1.5.ts should work with Cyrus 2.2.12 if the seen (and subscription) states are available in flat text files; otherwise, they must first be converted using cvt_cyrusdb(8).
My last try has nothing good :
cyrus2courier-1.5-dev : segfault cyrus2courier-1.5-ts : segfault
I don"t understand why ? But it's happening only with -e my_seen_dir option!
However, for our conversion, we've written our own "cyrus2dovecot" which also supports skiplist databases (amongst another few features not provided by the other tools):
I've tried this one too and I'm stuck too :
[root@test testimap]# ./cyrus2dovecot -C /root/Mail/%u -D dovecot-mail/ mketchn
cyrus2dovecot [mketchn]: (error) No Cyrus INBOX at: /root/Mail/mketchn
I've red the perl code and found this :
if (-e "$boxpath/cyrus.header" and -e "$boxpath/cyrus.index") {//code } else { # # The folder we're in is not in a Cyrus mailbox. However, we # don't simply return here because this folder might contain # other folders which might be Cyrus mailboxes (unless we're in # the INBOX, in which case something is going wrong). # error('No Cyrus INBOX at:', $boxpath) if $box eq 'INBOX'; debug("Skipping $boxpath as it's not a Cyrus mailbox."); $mailfolder = 0; }
But I'm sure cyrus.header and cyrus.index exist.
Any idea (maybe this is a format problem because even cyrus2courier is doing segfault)
Holger
- Mathieu Kretchner <mathieu.kretchner@sophia.inria.fr> [2008-09-23 12:59]:
Holger Weiss a écrit :
However, for our conversion, we've written our own "cyrus2dovecot" which also supports skiplist databases (amongst another few features not provided by the other tools):
I've tried this one too and I'm stuck too :
[root@test testimap]# ./cyrus2dovecot -C /root/Mail/%u -D dovecot-mail/ mketchn
cyrus2dovecot [mketchn]: (error) No Cyrus INBOX at: /root/Mail/mketchn
[...]
But I'm sure cyrus.header and cyrus.index exist.
Are you really sure? That is, the following commands don't complain?
$ ls /root/Mail/mketchn/cyrus.header
$ ls /root/Mail/mketchn/cyrus.indexHolger
Holger Weiss a écrit :
- Mathieu Kretchner <mathieu.kretchner@sophia.inria.fr> [2008-09-23 12:59]:
Holger Weiss a écrit :
However, for our conversion, we've written our own "cyrus2dovecot" which also supports skiplist databases (amongst another few features not provided by the other tools):
http://cyrus2dovecot.sw.fu-berlin.de/ I've tried this one too and I'm stuck too :
[root@test testimap]# ./cyrus2dovecot -C /root/Mail/%u -D dovecot-mail/ mketchn
cyrus2dovecot [mketchn]: (error) No Cyrus INBOX at: /root/Mail/mketchn
[...]
But I'm sure cyrus.header and cyrus.index exist.
Are you really sure? That is, the following commands don't complain?
$ ls /root/Mail/mketchn/cyrus.header $ ls /root/Mail/mketchn/cyrus.index
Holger
Sorry for the noise it was my fault : mkretchn and not mketchn :
./cyrus2dovecot -C /root/Mail/%u -S ../seendir/%u.seen -U ../seendir/%u.sub -D dovecot-mail/ mkretchn cyrus2dovecot [mkretchn]: 7619 messages in 52 folders (168.4 MiB, 25 s)
great but I still have all my e-mail unseen (maybe mkretchn.seen is desynchronized, because I' working on a copy of my account) ! However reply, forward and TODO falgs (maybe more) are kept.
Timo,
Is it enough for you ? I'm not familiar with gdb...
Starting program: /root/testimap/cyrus2courier-1.5.ts/src/cyrus2courier -vvv /root/Mail/ /root/Mail/dovecot-mail/ -e /root/seendir/m/ -d mkretchn (no debugging symbols found) (no debugging symbols found) (no debugging symbols found)
Program received signal SIGSEGV, Segmentation fault. 0x00628b1a in strcmp () from /lib/libc.so.6 (gdb) backtrace full #0 0x00628b1a in strcmp () from /lib/libc.so.6 No symbol table info available. #1 0x08048fab in setfolder_seenfile () No symbol table info available. #2 0x08049f76 in cydump () No symbol table info available. #3 0x0804ab26 in traverse () No symbol table info available. #4 0x0804b399 in main () No symbol table info available. (gdb) info registers eax 0x31 49 ecx 0xbfce92eb -1076981013 edx 0x0 0 ebx 0x5 5 esp 0xbfce92bc 0xbfce92bc ebp 0xbfce9308 0xbfce9308 esi 0xbfce92eb -1076981013 edi 0x8c271a8 146960808 eip 0x628b1a 0x628b1a <strcmp+10> eflags 0x10202 [ IF RF ] cs 0x73 115 ss 0x7b 123 ds 0x7b 123 es 0x7b 123 fs 0x0 0 gs 0x33 51
Mathieu Kretchner a écrit :
Holger Weiss a écrit :
- Mathieu Kretchner <mathieu.kretchner@sophia.inria.fr> [2008-09-23 12:59]:
Holger Weiss a écrit :
However, for our conversion, we've written our own "cyrus2dovecot" which also supports skiplist databases (amongst another few features not provided by the other tools):
http://cyrus2dovecot.sw.fu-berlin.de/ I've tried this one too and I'm stuck too :
[root@test testimap]# ./cyrus2dovecot -C /root/Mail/%u -D dovecot-mail/ mketchn
cyrus2dovecot [mketchn]: (error) No Cyrus INBOX at: /root/Mail/mketchn
[...]
But I'm sure cyrus.header and cyrus.index exist. Are you really sure? That is, the following commands don't complain?
$ ls /root/Mail/mketchn/cyrus.header $ ls /root/Mail/mketchn/cyrus.index
Holger
Sorry for the noise it was my fault : mkretchn and not mketchn :
./cyrus2dovecot -C /root/Mail/%u -S ../seendir/%u.seen -U ../seendir/%u.sub -D dovecot-mail/ mkretchn cyrus2dovecot [mkretchn]: 7619 messages in 52 folders (168.4 MiB, 25 s)
great but I still have all my e-mail unseen (maybe mkretchn.seen is desynchronized, because I' working on a copy of my account) ! However reply, forward and TODO falgs (maybe more) are kept.
Timo,
Is it enough for you ? I'm not familiar with gdb...
Starting program: /root/testimap/cyrus2courier-1.5.ts/src/cyrus2courier -vvv /root/Mail/ /root/Mail/dovecot-mail/ -e /root/seendir/m/ -d mkretchn (no debugging symbols found) (no debugging symbols found) (no debugging symbols found)
Program received signal SIGSEGV, Segmentation fault. 0x00628b1a in strcmp () from /lib/libc.so.6 (gdb) backtrace full #0 0x00628b1a in strcmp () from /lib/libc.so.6 No symbol table info available. #1 0x08048fab in setfolder_seenfile () No symbol table info available. #2 0x08049f76 in cydump () No symbol table info available. #3 0x0804ab26 in traverse () No symbol table info available. #4 0x0804b399 in main () No symbol table info available. (gdb) info registers eax 0x31 49 ecx 0xbfce92eb -1076981013 edx 0x0 0 ebx 0x5 5 esp 0xbfce92bc 0xbfce92bc ebp 0xbfce9308 0xbfce9308 esi 0xbfce92eb -1076981013 edi 0x8c271a8 146960808 eip 0x628b1a 0x628b1a <strcmp+10> eflags 0x10202 [ IF RF ] cs 0x73 115 ss 0x7b 123 ds 0x7b 123 es 0x7b 123 fs 0x0 0 gs 0x33 51 I've compiled c2c with debuging symbols and here is more understandable my backtrace :
(gdb) bt #0 0x00628b1a in strcmp () from /lib/libc.so.6 #1 0x08048f8b in setfolder_seenfile (_sf=0xb7f4e008, _mailbox=0x997dcc3 "mkretchn", _uidvalidity=1204103732) at c2c.c:936 #2 0x08049f56 in cydump (path=0x804e1e0 "/root/Mail/mkretchn", dest=0x9973018 "/root/Mail/dovecot-mail/mkretchn/", is_root=1) at c2c.c:660 #3 0x0804ab06 in traverse (cy_root=0xbfb2cb5e "/root/Mail", co_root=0xbfb2cb69 "/root/Mail/dovecot-mail", dir=0xbfb2cb98 "mkretchn") at c2c.c:359 #4 0x0804b379 in main (argc=862205238, argv=<value optimized out>) at c2c.c:162
it's done with c2c-1.5-dev !
On Wed, 2008-09-24 at 16:16 +0200, Mathieu Kretchner wrote:
(gdb) bt #0 0x00628b1a in strcmp () from /lib/libc.so.6 #1 0x08048f8b in setfolder_seenfile (_sf=0xb7f4e008, _mailbox=0x997dcc3 "mkretchn", _uidvalidity=1204103732) at c2c.c:936
I released 1.6.ts version that fixes this crash. Although I didn't look at it closely to find out if there could have been a more correct fix.
Holger Weiss <holger@CIS.FU-Berlin.DE>:
However, for our conversion, we've written our own "cyrus2dovecot" which also supports skiplist databases (amongst another few features not provided by the other tools):
The wiki page says "cyrus2dovecot doesn't preserve timestamps or flags" http://wiki.dovecot.org/Migration/Cyrus
However, the cyrus2dovecot home page says: "Virtually all available metadata is preserved during the conversion. This includes the UID and INTERNALDATE of each message, the IMAP folder subscriptions, the UIDVALIDITY and UIDNEXT values for each folder, as well as all IMAP flags (i.e., \Recent, \Seen, \Answered, \Flagged, \Draft, \Deleted) and the first 26 user-defined keywords."
Which to me sounds like the opposite of what the dovecot wiki says.
Perhaps the wiki should be updated?
On 9/24/2008 11:47 AM, Steinar Bang wrote:
Holger Weiss <holger@CIS.FU-Berlin.DE>:
However, for our conversion, we've written our own "cyrus2dovecot" which also supports skiplist databases (amongst another few features not provided by the other tools):
The wiki page says "cyrus2dovecot doesn't preserve timestamps or flags" http://wiki.dovecot.org/Migration/Cyrus
However, the cyrus2dovecot home page says: "Virtually all available metadata is preserved during the conversion. This includes the UID and INTERNALDATE of each message, the IMAP folder subscriptions, the UIDVALIDITY and UIDNEXT values for each folder, as well as all IMAP flags (i.e., \Recent, \Seen, \Answered, \Flagged, \Draft, \Deleted) and the first 26 user-defined keywords."
Which to me sounds like the opposite of what the dovecot wiki says.
Perhaps the wiki should be updated?
Looks like there are 2 different scripts. If you click the link referenced on the dovecot wiki, it takes you to a different 'homepage' for cyrus2dovecot.
But yeah, if the one you referenced above works well, the wiki should be changed to reflect use of this new one (with the permission of the author of course)...
--
Best regards,
Charles
- Charles Marcus <CMarcus@Media-Brokers.com> [2008-09-24 12:12]:
On 9/24/2008 11:47 AM, Steinar Bang wrote:
The wiki page says "cyrus2dovecot doesn't preserve timestamps or flags" http://wiki.dovecot.org/Migration/Cyrus
However, the cyrus2dovecot home page says: "Virtually all available metadata is preserved during the conversion. This includes the UID and INTERNALDATE of each message, the IMAP folder subscriptions, the UIDVALIDITY and UIDNEXT values for each folder, as well as all IMAP flags (i.e., \Recent, \Seen, \Answered, \Flagged, \Draft, \Deleted) and the first 26 user-defined keywords."
Which to me sounds like the opposite of what the dovecot wiki says.
Perhaps the wiki should be updated?
Looks like there are 2 different scripts.
Yes, indeed. We (Freie Universität Berlin) published our Cyrus2Dovecot just a few days ago. Unlike the older script, ours _does_ try hard to preserve all available metadata, and this should actually work with all Cyrus versions (tested with both the oldest and the newest available Cyrus release, that is, 1.4 and 2.3.12p2, and with various database formats). Apart from that, ours provides a few additional features we needed for our conversion which might also be useful to others, such as renaming (or moving) folders using s/regex/replacement/ substitutions.
But yeah, if the one you referenced above works well, the wiki should be changed to reflect use of this new one (with the permission of the author of course)...
We now added our Cyrus2Dovecot to the wiki page.
Holger
participants (6)
-
Charles Marcus
-
Holger Weiss
-
Mathieu Kretchner
-
Proskurin Kirill
-
Steinar Bang
-
Timo Sirainen