[Dovecot] MBOX to Maildir preserving the UIDL (outlook-tested, no redownloading)

[Horn Wijaya]

TESTED AND RUNNING, MIGRATING from MBOX to MAILDIR, PRESERVE UID and OUTLOOK
sees the email as downloaded.
   I hope this helps a lot of people out there.
   Feel free to post this somewhere else too.
   By: Horn Wijaya (hwijaya at nodens.biz) 8th May 2008 

I have been searching for weeks for a foolproof method to migrate all my
users mbox to maildir/dovecot without having the pain of downloading all
emails again, especially for heavy users. It would cause frustration to
thousands of my users. 

So.... eventually I found out why Outlook will keep downloading the emails
despite all the configuration done in the dovecot.conf.
Apparently the UID is generated realtime. When it first generated, it took
the time the process runs.

Before I migrate, here's the email sample which I have downloaded to my
outlook (and keep email on server), and then I use telnet to pop to get the
uidl list. 
   +OK POP3 localhost.localdomain v2003.83rh server ready
   user *****
   +OK User name accepted, password please
   pass *****
   +OK Mailbox open, 444 messages
   uidl
   +OK Unique-ID listing follows
   1 47b1303d00004c2d
   2 47b1303d00004c2e
   3 47b1303d00004c2f
   ... 

47b1303d = 1202794557 = 12 Feb 2008 13:35:57 the date and time the mbox was
created 
00004c2d = 19501 the IMAP counter so far.... sequently added as you could
see with email #2 and email #3 

Then I switch my POP server to dovecot, and migrate my MBOX to Maildir, and
I do a telnet to get the UIDL 
   +OK Dovecot ready.
   user *****
   +OK
   pass *****
   +OK Logged in.
   uidl
   +OK
   1 4822815000000001
   2 4822815000000002
   3 4822815000000003
   ... 

48228150 = 1210220880 = NOW 
00000001 = counter starts at 1 No wonder outlook sees the emails are
NOT-DOWNLOADED-YET status. 

Then I ponder for another few days and the key is in the
dovecot-uidlist...... How to fill it with the correct info.... and there's a
way. Which I hope useful for converter developer, and could be implemented
in the next version of the converter.

Here's the step I do to reach my objective which is to get UID
47b1303d00004c2d in the first email: 

1. mb2md-3.20.pl -s /var/mail/hwijaya -d home/hwijaya/Maildir/
   that's converting 444 emails from mbox to 444 files in the Maildir/cur
folder
2. cd /home/hwijaya/Maildir/cur
3. grep IMAPbase *
   and I get
      1210219860.000000.mbox:2,S:X-IMAPbase: 1202794557 19944
   The last 2 strings are the KEY to the ORIGINAL UID 
4. So in order to prepare the environment to fool it, some calculations are
needed:
   How do we get 19501 as the first counter? as expected in the first POP
TEST above:
      19944 - 444 emails + 1 = 19501 
5. cd /home/hwijaya/Maildir
6. echo '3 V1202794557 N19501' > dovecot-uidlist 
7. DONE....... pop it via outlook and nothing will be downloaded, after pop
or imap is processed, the dovecot-uidlist is auto-filled, index auto
generated. 

Proof: 
   +OK Dovecot ready.
   user *****
   +OK
   pass *****
   +OK
   Logged in.
   uidl
   +OK
   1 47b1303d00004c2d
   2 47b1303d00004c2e
   3 47b1303d00004c2f
   ...

I assume the same method could be used for individual folders in the imap
(under .mailboxlist or .subscriptions) 

Please do let us know if there's any cases/environment that it won't work. 

Thanks very much, and a big migration job is in front of me now...  

------------------------------------------
Warmest Regards From:
Horn Wijaya
Consultant
hwijaya at nodens.biz
http://www.nodens.biz/

Nodens Solutions Pte Ltd
      "where there is freedom in your choice"