[Dovecot] MBOX to Maildir preserving the UIDL (outlook-tested, no redownloading)
TESTED AND RUNNING, MIGRATING from MBOX to MAILDIR, PRESERVE UID and OUTLOOK sees the email as downloaded. I hope this helps a lot of people out there. Feel free to post this somewhere else too. By: Horn Wijaya (hwijaya@nodens.biz) 8th May 2008
I have been searching for weeks for a foolproof method to migrate all my users mbox to maildir/dovecot without having the pain of downloading all emails again, especially for heavy users. It would cause frustration to thousands of my users.
So.... eventually I found out why Outlook will keep downloading the emails despite all the configuration done in the dovecot.conf. Apparently the UID is generated realtime. When it first generated, it took the time the process runs.
Before I migrate, here's the email sample which I have downloaded to my outlook (and keep email on server), and then I use telnet to pop to get the uidl list. +OK POP3 localhost.localdomain v2003.83rh server ready user ***** +OK User name accepted, password please pass ***** +OK Mailbox open, 444 messages uidl +OK Unique-ID listing follows 1 47b1303d00004c2d 2 47b1303d00004c2e 3 47b1303d00004c2f ...
47b1303d = 1202794557 = 12 Feb 2008 13:35:57 the date and time the mbox was created 00004c2d = 19501 the IMAP counter so far.... sequently added as you could see with email #2 and email #3
Then I switch my POP server to dovecot, and migrate my MBOX to Maildir, and I do a telnet to get the UIDL +OK Dovecot ready. user ***** +OK pass ***** +OK Logged in. uidl +OK 1 4822815000000001 2 4822815000000002 3 4822815000000003 ...
48228150 = 1210220880 = NOW 00000001 = counter starts at 1 No wonder outlook sees the emails are NOT-DOWNLOADED-YET status.
Then I ponder for another few days and the key is in the dovecot-uidlist...... How to fill it with the correct info.... and there's a way. Which I hope useful for converter developer, and could be implemented in the next version of the converter.
Here's the step I do to reach my objective which is to get UID 47b1303d00004c2d in the first email:
- mb2md-3.20.pl -s /var/mail/hwijaya -d home/hwijaya/Maildir/ that's converting 444 emails from mbox to 444 files in the Maildir/cur folder
- cd /home/hwijaya/Maildir/cur
- grep IMAPbase * and I get 1210219860.000000.mbox:2,S:X-IMAPbase: 1202794557 19944 The last 2 strings are the KEY to the ORIGINAL UID
- So in order to prepare the environment to fool it, some calculations are needed: How do we get 19501 as the first counter? as expected in the first POP TEST above: 19944 - 444 emails + 1 = 19501
- cd /home/hwijaya/Maildir
- echo '3 V1202794557 N19501' > dovecot-uidlist
- DONE....... pop it via outlook and nothing will be downloaded, after pop or imap is processed, the dovecot-uidlist is auto-filled, index auto generated.
Proof: +OK Dovecot ready. user ***** +OK pass ***** +OK Logged in. uidl +OK 1 47b1303d00004c2d 2 47b1303d00004c2e 3 47b1303d00004c2f ...
I assume the same method could be used for individual folders in the imap (under .mailboxlist or .subscriptions)
Please do let us know if there's any cases/environment that it won't work.
Thanks very much, and a big migration job is in front of me now...
Warmest Regards From: Horn Wijaya Consultant hwijaya@nodens.biz http://www.nodens.biz/
Nodens Solutions Pte Ltd "where there is freedom in your choice"
A small amendment regarding how to obtain the UID and COUNTER
I have tried on IMAP Folder. To get IMAP info like the IMAPbase, apparently it does not exist in the files in cur folder of Drafts and Sent Items after I converted them to Maildir
mb2md-3.20.pl -s /var/mail/hwijaya/Drafts -d home/hwijaya/Maildir/.Drafts mb2md-3.20.pl -s "/var/mail/hwijaya/Sent Items" -d "/home/hwijaya/Maildir/.Sent Items" /home/hwijaya/Maildir/subscriptions has to be created based on existing folders, in this case: Drafts Sent Items
cd /home/hwijaya/Maildir/.Drafts/cur grep IMAP * and I get nothing Apparently, I have to grep IMAP /home/hwijaya/Drafts (mbox format) that returns X-IMAP: 1210246344 0000000444 Hence: echo '3 V1210246344 N1' > /home/hwijaya/Maildir/.Drafts/dovecot-uidlist
Same procedure with my Sent / Sent Items folder
so it's safe to just grep the IMAP from the mbox format (inbox, drafts, sent, etc) itself before migration. the rest of the previous procedure stay.
I have successfully migrated an mbx imap account into a Maildir imap account (accessed by OUtlook and Squirrelmail). No outlook redownloading.
Cheers.
Warmest Regards From: Horn Wijaya Consultant hwijaya@nodens.biz http://www.nodens.biz/
Nodens Solutions Pte Ltd "where there is freedom in your choice"
Horn Wijaya wrote:
TESTED AND RUNNING, MIGRATING from MBOX to MAILDIR, PRESERVE UID and OUTLOOK
- mb2md-3.20.pl -s /var/mail/hwijaya -d home/hwijaya/Maildir/ that's converting 444 emails from mbox to 444 files in the Maildir/cur folder
- cd /home/hwijaya/Maildir/cur
- grep IMAPbase * and I get 1210219860.000000.mbox:2,S:X-IMAPbase: 1202794557 19944 The last 2 strings are the KEY to the ORIGINAL UID
- So in order to prepare the environment to fool it, some calculations are needed: How do we get 19501 as the first counter? as expected in the first POP TEST above: 19944 - 444 emails + 1 = 19501
- cd /home/hwijaya/Maildir
- echo '3 V1202794557 N19501' > dovecot-uidlist
Hello,
First of all, I was looking into this problem of getting the right converter for couple years. You can search list archives for that - there are more problems with perfect migration, than you know of. Second, your calculations assume two things, that are not always true - puts these headers)
- original mailbox already has X-IMAPbase header (not every mail server
- first message UID is last message UID-<number of messages>, which is not true, if you have deleted some messages in between. Third, usually servers use different UIDL format, not time+number. Last, if you are migrating from mbox to Maildir on IMAP server, then you have more than one folder. Probably you have "/" as folder separator on mbox-based system. And will have "." on Maildir. That means, Outlook will need to refresh it's folder list anyway. And that involves re-downloading all messages.
Yes, in your case you might be able to make a perfect migration tool, but this won't be ideal for everybody.
P.S. After couple years of thinking, planning and looking for a ideal solution I have modified mb2md-3.20.pl script a little and was able to migrate about 200Gb of mail with no problem. Yes, most clients had to re-download all their messages (I had to force POP3 clients to empty their server mailboxes), but it wasn't that big of a deal, as I was afraid of. And now, even with much slower hardware I see about 10 times lower LA than with mbox.
FiLHi Dmitry,
Understood.
My outlook in regards to imap, even though it changed folder, it didn't ask for redownload, because Dovecot UID is the same. I have tried 444 messages on Drafts, Sent Items and INBOX, all change folder to Maildir folder.
Anyway, point taken. Thanks for the input.
Warmest Regards From: Horn Wijaya Consultant hwijaya@nodens.biz http://www.nodens.biz/
Nodens Solutions Pte Ltd "where there is freedom in your choice" -----Original Message----- From: dovecot-bounces+hwijaya=nodens.biz@dovecot.org [mailto:dovecot-bounces+hwijaya=nodens.biz@dovecot.org] On Behalf Of Dmitry Filonov Sent: Thursday, May 08, 2008 11:42 PM To: dovecot@dovecot.org Subject: Re: [Dovecot] MBOX to Maildir preserving the UIDL (outlook-tested, no redownloading)
Horn Wijaya wrote:
TESTED AND RUNNING, MIGRATING from MBOX to MAILDIR, PRESERVE UID and OUTLOOK
- mb2md-3.20.pl -s /var/mail/hwijaya -d home/hwijaya/Maildir/ that's converting 444 emails from mbox to 444 files in the Maildir/cur folder
- cd /home/hwijaya/Maildir/cur
- grep IMAPbase * and I get 1210219860.000000.mbox:2,S:X-IMAPbase: 1202794557 19944 The last 2 strings are the KEY to the ORIGINAL UID
- So in order to prepare the environment to fool it, some calculations are needed: How do we get 19501 as the first counter? as expected in the first POP TEST above: 19944 - 444 emails + 1 = 19501
- cd /home/hwijaya/Maildir
- echo '3 V1202794557 N19501' > dovecot-uidlist
Hello,
First of all, I was looking into this problem of getting the right converter for couple years. You can search list archives for that - there are more problems with perfect migration, than you know of. Second, your calculations assume two things, that are not always true - puts these headers)
- original mailbox already has X-IMAPbase header (not every mail server
- first message UID is last message UID-<number of messages>, which is not true, if you have deleted some messages in between. Third, usually servers use different UIDL format, not time+number. Last, if you are migrating from mbox to Maildir on IMAP server, then you have more than one folder. Probably you have "/" as folder separator on mbox-based system. And will have "." on Maildir. That means, Outlook will need to refresh it's folder list anyway. And that involves re-downloading all messages.
Yes, in your case you might be able to make a perfect migration tool, but this won't be ideal for everybody.
P.S. After couple years of thinking, planning and looking for a ideal solution I have modified mb2md-3.20.pl script a little and was able to migrate about 200Gb of mail with no problem. Yes, most clients had to re-download all their messages (I had to force POP3 clients to empty their server mailboxes), but it wasn't that big of a deal, as I was afraid of. And now, even with much slower hardware I see about 10 times lower LA than with mbox.
FiLhttp://dovecot.org/list/dovecot/2008-March/029736.html points to a
script that should preserve the UIDs. It's also linked to in http://wiki.dovecot.org/Migration/MailFormat
Thanks for the input, will try that out see whether it will work on my environment.
Warmest Regards From: Horn Wijaya Consultant hwijaya@nodens.biz http://www.nodens.biz/
Nodens Solutions Pte Ltd "where there is freedom in your choice" -----Original Message----- From: Timo Sirainen [mailto:tss@iki.fi] Sent: Thursday, May 08, 2008 11:55 PM To: Horn Wijaya Cc: dovecot@dovecot.org Subject: Re: [Dovecot] MBOX to Maildir preserving the UIDL (outlook-tested, no redownloading)
http://dovecot.org/list/dovecot/2008-March/029736.html points to a
script that should preserve the UIDs. It's also linked to in
http://wiki.dovecot.org/Migration/MailFormat
on second thought, after I search through my tests, That's the first script I tested, and it failed. .py tested .pl tested they didn't support the preservation of UID.
The posting was meant for informative purposes, who knows someone who stuck like me could find something out of it. So yeah, I guess I will stick to my own method for my own environment, at least I know it would work :)
Warmest Regards From: Horn Wijaya Consultant hwijaya@nodens.biz http://www.nodens.biz/
Nodens Solutions Pte Ltd "where there is freedom in your choice" -----Original Message----- From: dovecot-bounces+hwijaya=nodens.biz@dovecot.org [mailto:dovecot-bounces+hwijaya=nodens.biz@dovecot.org] On Behalf Of Timo Sirainen Sent: Thursday, May 08, 2008 11:55 PM To: Horn Wijaya Cc: dovecot@dovecot.org Subject: Re: [Dovecot] MBOX to Maildir preserving the UIDL (outlook-tested,no redownloading)
http://dovecot.org/list/dovecot/2008-March/029736.html points to a
script that should preserve the UIDs. It's also linked to in
http://wiki.dovecot.org/Migration/MailFormat
participants (3)
-
Dmitry Filonov
-
Horn Wijaya
-
Timo Sirainen