[Dovecot] batch convert mbox to maildir
Hello, I have used mb2md for an individual user like this:
su ~username cd ~ ./mb2md -m
This converted /var/spool/mail/username mbox to /home/users/username/Maildir just as I expected.
To convert 100s of users the same way would be a chore (su as each user for permissions, etc.).
Is there a way to batch process all /var/spool/mail/$USER to /home/users/$USER/Maildir with the correct permissions/file owners for each user?
This weekend I migrated the university I work for to Dovecot and maildir from UW and mbox. I made some pretty significant changes to mb2md such that it converts keywords (aka, custom flags) and builds the dovecot-uidlist file based on X-IMAP/X-IMAPbase and X-UID headers. Both of these additions are optional though. I was also tasked with converting not just /var/mail/%u, but also /home/%u and all mbox folders for all of users...~30,000 people. The uidlist file it generates is for 0.99.14. I don't know if 1.0 has a different list syntax. I also added the UW uidl patch so Dovecot generates uid numbers the same way UW did. This prevented, most of, our users from noticing we even did anything (except for the 12 hour email downtime ;).
In order to convert as quickly as possible we created a wrapper around mb2md that forks mb2md processes for a list of users. The list is in a Berkeley db file. You can specify the maximum number of threads you want to use so you don't overload your system. We ran our conversion on 4 machines, converting 2 file systems (we have 8) per machine, with 32 "threads" per master converter. The backend is an EMC Celerra storage array that we read mboxes from and wrote new maildirs to. We converted all /var/mail and user folders, ~13,000,000 (yes, million) files/messages in about 3.5 to 4 hours.
If you would like the source for this system I would be happy to provide it. There is no documentation right now as it was a one time only thing, but I can try to explain. If others are interested just let me know. The whole thing is written in Perl and expects to be run as root. It chowns everything in the final steps.
Regards,
| Todd Piket | Email: todd@mtu.edu | | Programmer/Analyst | Phone: (906) 487-1720 | | Distributed Computing Services | | | Michigan Technological University | |
Netlink Tech wrote:
Hello, I have used mb2md for an individual user like this:
su ~username cd ~ ./mb2md -m
This converted /var/spool/mail/username mbox to /home/users/username/Maildir just as I expected.
To convert 100s of users the same way would be a chore (su as each user for permissions, etc.).
Is there a way to batch process all /var/spool/mail/$USER to /home/users/$USER/Maildir with the correct permissions/file owners for each user?
Since you only have a few hundred user's to convert, why just just write a quick shell wrapper on the command line. Something like this (this is bourne or bash):
# for user in ls /var/spool/mail|egrep -v "root|daemon"; do
echo md2md -s /var/spool/mail/$user -d /home/users/$user;
echo chown -R $user /home/users/$user/Maildir;
echo chown -R $user /home/users/$user/Maildir;
echo chmod -R 700 /home/users/$user/Maildir; done
Logged in as 'root', that will loop through all files in /var/spool/mail (assuming the filenames are the same as the username) and convert them to Maildir in the userhome, then set the owner to themselves and make them 700 permissions.
I haven't tested that, I just typed it out. In fact I've never even used 'md2md', but the a 'for' wrapper is simple enough for anything. So you may want to double check the md2md arguments.
You can add user's you don't want to the 'egrep -v' list to ignore, OR, if you'd prefer, take the '-v' out and make it a list of user's to only do. The 'echos' are there to just print what it will do for testing. Remove them when you have it like you want it.
Hope that helps.
~Adam
On Mon, 3 Apr 2006, Netlink Tech wrote:
Hello, I have used mb2md for an individual user like this:
su ~username cd ~ ./mb2md -m
This converted /var/spool/mail/username mbox to /home/users/username/Maildir just as I expected.
To convert 100s of users the same way would be a chore (su as each user for permissions, etc.).
Is there a way to batch process all /var/spool/mail/$USER to /home/users/$USER/Maildir with the correct permissions/file owners for each user?
I found this and it appears as though it will do exactly what I want: http://people.redhat.com/rkeech/#dovecot
I am sure that writing the shell wrapper would also work, but someone else (Richard Keech and Philip Mak) already wrote the scripts to do this.
Thanks!
On Mon, 3 Apr 2006, Adam M. Dunn wrote:
Since you only have a few hundred user's to convert, why just just write a quick shell wrapper on the command line. Something like this (this is bourne or bash):
# for user in
ls /var/spool/mail|egrep -v "root|daemon"; do
echo md2md -s /var/spool/mail/$user -d /home/users/$user;
echo chown -R $user /home/users/$user/Maildir;
echo chown -R $user /home/users/$user/Maildir;
echo chmod -R 700 /home/users/$user/Maildir; doneLogged in as 'root', that will loop through all files in /var/spool/mail (assuming the filenames are the same as the username) and convert them to Maildir in the userhome, then set the owner to themselves and make them 700 permissions.
I haven't tested that, I just typed it out. In fact I've never even used 'md2md', but the a 'for' wrapper is simple enough for anything. So you may want to double check the md2md arguments.
You can add user's you don't want to the 'egrep -v' list to ignore, OR, if you'd prefer, take the '-v' out and make it a list of user's to only do. The 'echos' are there to just print what it will do for testing. Remove them when you have it like you want it.
Hope that helps.
~Adam
On Mon, 3 Apr 2006, Netlink Tech wrote:
Hello, I have used mb2md for an individual user like this:
su ~username cd ~ ./mb2md -m
This converted /var/spool/mail/username mbox to /home/users/username/Maildir just as I expected.
To convert 100s of users the same way would be a chore (su as each user for permissions, etc.).
Is there a way to batch process all /var/spool/mail/$USER to /home/users/$USER/Maildir with the correct permissions/file owners for each user?
participants (3)
-
Adam M. Dunn
-
Netlink Tech
-
Todd Piket