[Dovecot] Cyrus MailDir to Dovecot mbox
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Anyone out there know of a tool that can convert a Cyrus MailDir format to Dovecot mbox? I have a user that I need to move over from a mail server running cyrus using MailDir format. At first I told him to setup second account with Thunderbird to our mail server and then use Thunderbird to move the messages over. But he has over 100K messages so not as easy to do. Is there a tool out there for doing this sort of conversion? I tried to simply cat all the messages into a single file, but just got a "Not valid mbox format" when I tried to read the mail folder.
C. J. Keist Email: cj.keist@colostate.edu UNIX/Network Manager Phone: 970-491-0630 Engineering Network Services Fax: 970-491-5569 College of Engineering, CSU Ft. Collins, CO 80523-1301
All I want is a chance to prove 'Money can't buy happiness' -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFI63pnA29OFr7C6jcRAkBSAKCU61WhK1R2FwZEE4B0/3kFzEeu6ACeLgw+ 4qabCHV1qe7YInhcnqsdKQk= =rjlB -----END PGP SIGNATURE-----
On Oct 7, 2008, at 6:04 PM, CJ Keist wrote:
Anyone out there know of a tool that can convert a Cyrus MailDir
format to Dovecot mbox? I have a user that I need to move over from a mail server running cyrus using MailDir format. At first I told him to
setup second account with Thunderbird to our mail server and then use Thunderbird to move the messages over. But he has over 100K
messages so not as easy to do. Is there a tool out there for doing this sort of conversion? I
tried to simply cat all the messages into a single file, but just got a "Not valid mbox format" when I tried to read the mail folder.
You could use either cyrus2dovecot or cyrus2courier tools to convert
to Maildir format (http://wiki.dovecot.org/Migration/Cyrus). From
there use some maildir to mbox conversion tool (e.g. Dovecot's convert-
tool).
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Timo, Thanks. The cyrus2dovecot looks to have worked. So if I understand the process I now just move the MailDir folder that was created to the users home directory on our mail server. I will try the dovecot convert plugin. Just make sure I'm doing this correct in the config file:
mail_location = mbox:~/:INBOX=/userM/mail/%u protocol imap { mail_plugins = convert } plugin { convert_mail = Maildir:~/Maildir:INBOX=/userM/mail/%u }
Is this correct? The example on the website only shows how to convert from mbox to MailDir.
Timo Sirainen wrote:
On Oct 7, 2008, at 6:04 PM, CJ Keist wrote:
Anyone out there know of a tool that can convert a Cyrus MailDir format to Dovecot mbox? I have a user that I need to move over from a mail server running cyrus using MailDir format. At first I told him to setup second account with Thunderbird to our mail server and then use Thunderbird to move the messages over. But he has over 100K messages so not as easy to do. Is there a tool out there for doing this sort of conversion? I tried to simply cat all the messages into a single file, but just got a "Not valid mbox format" when I tried to read the mail folder.
You could use either cyrus2dovecot or cyrus2courier tools to convert to Maildir format (http://wiki.dovecot.org/Migration/Cyrus). From there use some maildir to mbox conversion tool (e.g. Dovecot's convert-tool).
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFI685lA29OFr7C6jcRAj5TAJ4pFrKhGVhLA/87U31oiEMNWW3jSQCfVfGM JhEJPPZTdgwpX/0QzpyOZug= =VpaM -----END PGP SIGNATURE-----
On Tue, 2008-10-07 at 15:02 -0600, CJ Keist wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Timo, Thanks. The cyrus2dovecot looks to have worked. So if I understand the process I now just move the MailDir folder that was created to the users home directory on our mail server. I will try the dovecot convert plugin. Just make sure I'm doing this correct in the config file:
If you do it only once, you don't need to use convert plugin. There's also a convert-tool binary included.
mail_location = mbox:~/:INBOX=/userM/mail/%u protocol imap { mail_plugins = convert } plugin { convert_mail = Maildir:~/Maildir:INBOX=/userM/mail/%u }
Is this correct? The example on the website only shows how to convert from mbox to MailDir.
That looks also correct.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Timo, Thanks again for the tip on the convert-tool. That did work to convert this guys 121K worth of messages. The built in convert plugin with dovecot was not able to do it. I believe due to the fact that this guy has folders with messages and then sub-folders with more messages. I'm migrating to mbox format which doesn't allow that feature. The convert-tool showed me this and it was easy fix to just move all sub-folders up to a flat directory structure. Then the convert-tool ripped right through all the messages.
Timo Sirainen wrote:
On Tue, 2008-10-07 at 15:02 -0600, CJ Keist wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Timo, Thanks. The cyrus2dovecot looks to have worked. So if I understand the process I now just move the MailDir folder that was created to the users home directory on our mail server. I will try the dovecot convert plugin. Just make sure I'm doing this correct in the config file:
If you do it only once, you don't need to use convert plugin. There's also a convert-tool binary included.
mail_location = mbox:~/:INBOX=/userM/mail/%u protocol imap { mail_plugins = convert } plugin { convert_mail = Maildir:~/Maildir:INBOX=/userM/mail/%u }
Is this correct? The example on the website only shows how to convert from mbox to MailDir.
That looks also correct.
C. J. Keist Email: cj.keist@colostate.edu UNIX/Network Manager Phone: 970-491-0630 Engineering Network Services Fax: 970-491-5569 College of Engineering, CSU Ft. Collins, CO 80523-1301
All I want is a chance to prove 'Money can't buy happiness' -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFI7haeA29OFr7C6jcRAkGIAJ9WhyE0Q20Gwymb7C9OcAI6ECGOSgCgm0TO SPGT7Li6m5m6neusNvpEtqA= =kCOX -----END PGP SIGNATURE-----
On 10/9/2008, CJ Keist (cj.keist@colostate.edu) wrote:
The built in convert plugin with dovecot was not able to do it. I believe due to the fact that this guy has folders with messages and then sub-folders with more messages. I'm migrating to mbox format which doesn't allow that feature. The convert-tool showed me this and it was easy fix to just move all sub-folders up to a flat directory structure. Then the convert-tool ripped right through all the messages.
So, I'm curious why you didn't just migrate to maildir format - sounds like it would have been much easier...
--
Best regards,
Charles
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Charles, It's to do with how our current file services are setup and a personal preference and familiarity with the mbox format. Right now other than users inboxes, all mail folders are stored in the users home directory on central file server which the mail server mounts via NFS. Maildir is a big change from mbox and would give a confusing view to users that decided to look into their Maildir folder. I don't even want to think about what they might try mucking around with in there ;). mbox format is just cleaner and users can easily see that these are the same folders they see on their mail client. But if I ever get the money to add enough local disk storage on our mail server and can move all users mail folders to the mail server, is when I would probably think about switching over to the Maildir format. This way the users only way to access their Maildir folder would be through a mail client.
Charles Marcus wrote:
On 10/9/2008, CJ Keist (cj.keist@colostate.edu) wrote:
The built in convert plugin with dovecot was not able to do it. I believe due to the fact that this guy has folders with messages and then sub-folders with more messages. I'm migrating to mbox format which doesn't allow that feature. The convert-tool showed me this and it was easy fix to just move all sub-folders up to a flat directory structure. Then the convert-tool ripped right through all the messages.
So, I'm curious why you didn't just migrate to maildir format - sounds like it would have been much easier...
C. J. Keist Email: cj.keist@colostate.edu UNIX/Network Manager Phone: 970-491-0630 Engineering Network Services Fax: 970-491-5569 College of Engineering, CSU Ft. Collins, CO 80523-1301
All I want is a chance to prove 'Money can't buy happiness' -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFI7ixiA29OFr7C6jcRAl15AJ98vUKIeIlMHw0108xJ0JtW55S1QgCgppwy DB0RY6UcqhkE/rUXXeDmdGw= =asZV -----END PGP SIGNATURE-----
On 10/9/2008 12:08 PM, CJ Keist wrote:
Charles, It's to do with how our current file services are setup and a personal preference and familiarity with the mbox format. Right now other than users inboxes, all mail folders are stored in the users home directory on central file server which the mail server mounts via NFS. Maildir is a big change from mbox and would give a confusing view to users that decided to look into their Maildir folder. I don't even want to think about what they might try mucking around with in there ;). mbox format is just cleaner and users can easily see that these are the same folders they see on their mail client.
<shudder>
I wouldn't be able to sleep at night if my users' mail was directly accessible by the users... ;)
But if I ever get the money to add enough local disk storage on our mail server and can move all users mail folders to the mail server, is when I would probably think about switching over to the Maildir format. This way the users only way to access their Maildir folder would be through a mail client.
I feel your pain...
--
Best regards,
Charles
CJ Keist cj.keist@colostate.edu:
[On why mbox and not maildir]
It's to do with how our current file services are setup and a personal preference and familiarity with the mbox format.
FWIW it was familiarity with mbox, and that maildir was a big unknown for me, that almost made me choose mbox.
But then I read up on maildir and found it quite elegant, simple to understand, and robust (ie. no more huge mbox file filled with nulls, which has happened to me when something failed during mbox file write).
I can see one issue with it, when using rsync to back up the mailboxes to a different machine: since the file names of the articles change when they are read, and when the state changes, rsync may ending up transferring the same article more than once (unless it has some clever checksumming to recognize files that may be the same when the name has changed), but in practice I don't think that will be much of a problem.
Steinar Bang sb@dod.no:
I can see one issue with it, when using rsync to back up the mailboxes to a different machine: since the file names of the articles change when they are read, and when the state changes, rsync may ending up transferring the same article more than once (unless it has some clever checksumming to recognize files that may be the same when the name has changed), but in practice I don't think that will be much of a problem.
I googled for rsync and maildir, and found this: http://www.nabble.com/Rsync-snapshots,-Maildir,-and-Sarbanes-Oxley-td1191416...
It mentions using the --fuzzy flag. From the rsync man page: -y, --fuzzy This option tells rsync that it should look for a basis file for any destination file that is missing. The current algorithm looks in the same directory as the destination file for either a file that has an identical size and modified-time, or a simi‐ larly-named file. If found, rsync uses the fuzzy basis file to try to speed up the transfer.
Note that the use of the --delete option might get rid of any
potential fuzzy-match files, so either use --delete-after or
specify some filename exclusions if you need to prevent this.(and a note to myself on the last bit: I'm currently using --archive --delete, so that's something to look out for. I guess that my new arguments will be --archive --fuzzy --delete-after)
CJ Keist cj.keist@colostate.edu:
Anyone out there know of a tool that can convert a Cyrus MailDir format to Dovecot mbox? I have a user that I need to move over from a mail server running cyrus using MailDir format. At first I told him to setup second account with Thunderbird to our mail server and then use Thunderbird to move the messages over. But he has over 100K messages so not as easy to do.
From server to server sync'ing, there's imapsync, which avoids having to pull the messages over manually. http://wiki.dovecot.org/Migration
I used it to pull around 30-40k messages from a cyrus imapd to a dovecot on a different machine (my dovecot used maildir, not mbox, but that shouldn't make any difference).
The UIDs aren't preserved, but everything else seems to be. My MUA (Gnus with the nnimap backend) uses custom flags to store state and all of that state was preserved.
Losing the UIDs in Gnus' case, only meant that the cached article information was useless, and that I would have to freshly download everything I wanted cached. And it also turned out to be mostly irrelevant, since the folder structure changed with dovecot. The only folder for which I could have preserved cached information would have been the INBOX.
participants (4)
-
Charles Marcus
-
CJ Keist
-
Steinar Bang
-
Timo Sirainen