[Dovecot] Migrating mail from mbox to maildir using dovecot
Hello,
Currently our mail server is setup to use postfix and dovecot, which is fine except it is setup to use mbox format mail. That and procmail as we need to move to sieve but that is a different issue. I have searched until blue in the face and cannot find a 100% working solution that doesn't require alot of manual interaction to move the mail boxes from mbox to maildir. I have tried the mb2md.pl stuff but with over 150+ users that can get tedious having to log in as each one of them to migrate them. I also tried the convert plugin located here: http://wiki.dovecot.org/Plugins/Convert but it kept erroring about not able to convert the box and exiting. We have approx 200GB of mail total that needs to be migrated. Maybe I am just simply missing something but is there a way that we can migrate all the users with minimal end user interaction and minimal administration time? We need this done asap as our backups are killing us with the large mbox style files.
Thanks in advance for any and all input to this.
Donny B.
On 2010-09-11 3:14 AM, Donny Brooks dbrooks@mdah.state.ms.us wrote:
Currently our mail server is setup to use postfix and dovecot,
Version of DC?
I have searched until blue in the face and cannot find a 100% working solution that doesn't require alot of manual interaction to move the mail boxes from mbox to maildir.
Dovecot 2.0 would use dsync, and it should be really easy and avoids most problems the other methods can have (haven't used it yet myself, but will on a system I'll be upgrading soon)...
http://wiki2.dovecot.org/Migration/MailFormat
But either way, to avoid doing each one manually you'll need to use some scripting magic. If I were doing it, I'd probably write a script (or more likely hire someone to do it for me, since my scripting skills are rudimentary at best) that would run at the least busy time of day for your system, and iterate through the user:
disable login kill any existing processes for that user run dsync to convert everything re-enable login rinse, repeat.
I also tried the convert plugin located here: http://wiki.dovecot.org/Plugins/Convert but it kept erroring about not able to convert the box and exiting.
Did you ask here about the errors? Timo could probably address any problems your having very quickly.
--
Best regards,
Charles
On Saturday, September 11, 2010 10:28 AM CDT, Charles Marcus CMarcus@Media-Brokers.com wrote:
On 2010-09-11 3:14 AM, Donny Brooks dbrooks@mdah.state.ms.us wrote:
Currently our mail server is setup to use postfix and dovecot,
Version of DC?
On the old mail server it is: dovecot-1.2.11-3.fc11.x86_64 On the new mail server we are migrating to it is: dovecot-1.2.14-1.fc13.x86_64 currently
Is there a repo out there that I can install 2.0 from? The only one I have found is atrpms and it is a pain to get conflict worked out with other packages.
I have searched until blue in the face and cannot find a 100% working solution that doesn't require alot of manual interaction to move the mail boxes from mbox to maildir.
Dovecot 2.0 would use dsync, and it should be really easy and avoids most problems the other methods can have (haven't used it yet myself, but will on a system I'll be upgrading soon)...
http://wiki2.dovecot.org/Migration/MailFormat
But either way, to avoid doing each one manually you'll need to use some scripting magic. If I were doing it, I'd probably write a script (or more likely hire someone to do it for me, since my scripting skills are rudimentary at best) that would run at the least busy time of day for your system, and iterate through the user:
disable login kill any existing processes for that user run dsync to convert everything re-enable login rinse, repeat.
I also tried the convert plugin located here: http://wiki.dovecot.org/Plugins/Convert but it kept erroring about not able to convert the box and exiting.
Did you ask here about the errors? Timo could probably address any problems your having very quickly.
--
Best regards,
Charles
Most likely I will need to stick for now to the 1.x branch and just use the convert plugin. I will dig for the error again and see what it was. I will send a new message with the details on that. Thanks for the input.
--
Donny B.
On 11/09/2010 20:17, Donny Brooks wrote:
On the old mail server it is: dovecot-1.2.11-3.fc11.x86_64 On the new mail server we are migrating to it is: dovecot-1.2.14-1.fc13.x86_64 currently
Is there a repo out there that I can install 2.0 from? The only one I have found is atrpms and it is a pain to get conflict worked out with other packages.
If you use Dovecot 2.0 you also get the choice of multi-dbox.
http://wiki2.dovecot.org/MailboxFormat/dbox
This would give you some advantages:
With multi-dbox, multiple messages are stored in the same file. But once a mail file contains 2 megabytes (configurable), Dovecot moves on to a new file and leaves the old files unchanging. This should be quite nice for your backup strategy because you would only backup the "current" file for each user, limited to 2 megabytes per user (or whatever amount you think appropriate).
With multi-dbox, if you have many small messages, these are all stored in the same file. This vastly reduces the number of files required to store mail, which means fewer inodes consumed, and I think backup systems often work better with a smaller number of larger files than a larger number of smaller files.
If you use multi-dbox, you will need to use a Dovecot delivery software (e.g. dovecot-lda).
Bill
I think I will have to stay with the 1.x branch since we have to stick to using approved rpm's (internal policy) and fedora 13 does not have a 2.x branch that I see yet. So maildir it is for now.
I see my error on the inbox not getting the new mail. That was one of those "all nighter" errors that I just plain missed due to lack of sleep. I forgot to point postfix to the proper place to deliver the mail.
So once I setup postfix to deliver the mail properly and the dovecot convert plugin is setup, is there a way to gradually migrate the users so I don't kill the server? We have about 220GB or so of mail between about 160+/- users. What would be the best way to migrate it all to maildir?
--
Donny B.
On Sunday, September 12, 2010 02:56 AM CDT, William Blunn bill@blunn.org wrote:
On 11/09/2010 20:17, Donny Brooks wrote:
On the old mail server it is: dovecot-1.2.11-3.fc11.x86_64 On the new mail server we are migrating to it is: dovecot-1.2.14-1.fc13.x86_64 currently
Is there a repo out there that I can install 2.0 from? The only one I have found is atrpms and it is a pain to get conflict worked out with other packages.
If you use Dovecot 2.0 you also get the choice of multi-dbox.
http://wiki2.dovecot.org/MailboxFormat/dbox
This would give you some advantages:
With multi-dbox, multiple messages are stored in the same file. But once a mail file contains 2 megabytes (configurable), Dovecot moves on to a new file and leaves the old files unchanging. This should be quite nice for your backup strategy because you would only backup the "current" file for each user, limited to 2 megabytes per user (or whatever amount you think appropriate).
With multi-dbox, if you have many small messages, these are all stored in the same file. This vastly reduces the number of files required to store mail, which means fewer inodes consumed, and I think backup systems often work better with a smaller number of larger files than a larger number of smaller files.
If you use multi-dbox, you will need to use a Dovecot delivery software (e.g. dovecot-lda).
Bill
On 2010-09-14 11:51 AM, Donny Brooks dbrooks@mdah.state.ms.us wrote:
So once I setup postfix to deliver the mail properly and the dovecot convert plugin is setup, is there a way to gradually migrate the users so I don't kill the server? We have about 220GB or so of mail between about 160+/- users. What would be the best way to migrate it all to maildir?
I can't give you precise instructions, but from the bottom of the Plugins wiki page:
"To enable / disable plugins per user you can make your userdb return mail_plugins extra field. See UserDatabase/ExtraFields for examples."
http://wiki1.dovecot.org/UserDatabase/ExtraFields
So you could enable the convert plugin on a per user basis and do them one at a time or in small groups...
--
Best regards,
Charles
This actually looks like the way I will look into doing it. That way I can migrate the ones that have close to 8GB of mail on a weekend and the rest that are "typical" with a few hundred MB any old time.
Now just to figure out the proper syntax for ldap. ;)
--
Donny B.
On Tuesday, September 14, 2010 11:02 AM CDT, Charles Marcus CMarcus@Media-Brokers.com wrote:
On 2010-09-14 11:51 AM, Donny Brooks dbrooks@mdah.state.ms.us wrote:
So once I setup postfix to deliver the mail properly and the dovecot convert plugin is setup, is there a way to gradually migrate the users so I don't kill the server? We have about 220GB or so of mail between about 160+/- users. What would be the best way to migrate it all to maildir?
I can't give you precise instructions, but from the bottom of the Plugins wiki page:
"To enable / disable plugins per user you can make your userdb return mail_plugins extra field. See UserDatabase/ExtraFields for examples."
http://wiki1.dovecot.org/UserDatabase/ExtraFields
So you could enable the convert plugin on a per user basis and do them one at a time or in small groups...
--
Best regards,
Charles
On 14/09/2010 16:51, Donny Brooks wrote:
I think I will have to stay with the 1.x branch since we have to stick to using approved rpm's (internal policy) and fedora 13 does not have a 2.x branch that I see yet.
I agree sticking to the standard binary packages where possible is generally a good idea. But having a carefully chosen handful of packages built from source, where it offers a significant benefit, could be overall better than sticking slavishly to an overly-simplistic rule.
If your organisation is only 160 people there can not be that many people (2 or 3?) involved in making that policy. You should be able to get them all in a meeting and sort it out.
Rules are not usually written in stone. Rules may not be /wrong/ per se, rather they may benefit from some /refinement/.
So maildir it is for now.
I see my error on the inbox not getting the new mail. That was one of those "all nighter" errors that I just plain missed due to lack of sleep. I forgot to point postfix to the proper place to deliver the mail.
So once I setup postfix to deliver the mail properly and the dovecot convert plugin is setup, is there a way to gradually migrate the users so I don't kill the server? We have about 220GB or so of mail between about 160+/- users. What would be the best way to migrate it all to maildir?
Have mail location configured on a per-user basis via the userdb.
You can then pull individual users mail over to Maildir and update their mail location configurations.
Bill
Op 14-9-2010 17:51, Donny Brooks schreef:
I think I will have to stay with the 1.x branch since we have to stick to using approved rpm's (internal policy) and fedora 13 does not have a 2.x branch that I see yet. So maildir it is for now. I see my error on the inbox not getting the new mail. That was one of those "all nighter" errors that I just plain missed due to lack of sleep. I forgot to point postfix to the proper place to deliver the mail. So once I setup postfix to deliver the mail properly and the dovecot convert plugin is setup, is there a way to gradually migrate the users so I don't kill the server? We have about 220GB or so of mail between about 160+/- users. What would be the best way to migrate it all to maildir?
In my test to convert a mbox situation (sendmail) to Dovecot, I used script similar to that below. The conversion is done with the mb2md.pl script and all takes place when a user logs in. The fact that $@ is destroyed by the awk does not seem to harm. YMMV!
I used the hooks provided in the dovecot config file (thanks Timo) where you can call a script instead of the IMAP exec. Details are in the 1.x dovecot wiki.
HTH Egbert Jan (NL)
#!/bin/sh # # split $USER in User and Domain
var=$(echo $USER | awk -F"@" '{print $1,$2}') set -- $var
# # If /var/mail/<user> does not exist : skip to exec # if [ -f /var/mail/$1 ] then set > /etc/dovecot/dovecot-$1 #/etc/dovecot/mb2md.pl -S -W -s /var/mail/egbert -d $HOME #chown vmail:vmail $HOME/cur/* mv /var/mail/$1 /var/mail/$1-converted fi exec /usr/lib/dovecot/imap "$@"
Donny Brooks put forth on 9/11/2010 2:14 AM:
We need this done asap as our backups are killing us with the large mbox style files.
Does your backup software not offer data deduplication? I was under the impression that all commercial backup software offered this feature in modern times. In a nutshell, if a file changes from one backup to the next, only the part of that file which has changed is then backed up, not the entire file. The "master" copy and the "fragments" are then merged during a restore.
If you don't have this feature, may I ask what backup solution you're using?
I'm not pointing you away from maildir. It is still preferable for some reasons in some environments. I'm simply saying, AFAIK, maildir simply doesn't have much advantage over mbox WRT modern (commercial) backup software.
-- Stan
participants (5)
-
Charles Marcus
-
Donny Brooks
-
Egbert Jan van den Bussche
-
Stan Hoeppner
-
William Blunn