[Dovecot] Migrate from Maildir to mdbox
Looking at http://wiki2.dovecot.org/Migration/MailFormat, I do not see much on how to do a migration from Maildir to mdbox. That page isn't quite updated it looks.
I have several domains (virtual) with the following storage layout:
/var/spool/virtual/$domain.
So the users have their mail stored as follows:
/var/spool/virtual/domain1/$user/Maildir /var/spool/virtual/domain2/$user/Maildir /var/spool/virtual/domain3/$user/Maildir /var/spool/virtual/domainN/$user/Maildir
From the wiki, I am told the steps to take are:
- Set mail_location=mdbox:~/mdbox
- Run "doveadm mirror -u username maildir:~/Maildir"
The instruction above confuses me because in my situation, I don't have a "username" per se. dovecot-deliver knows how to figure out my usernames and their mail_location settings. How do I derive my "username" when users are virtual, stored in MySQL back end?
Again, the right command seems to rely on dsync, not "doveadm mirror".
I could manually get all the values for the users for each domain and run:
dysnc -u $u -o "mail_location=maildir:/var/spool/virtual/domain1/$u/Maildir" mirror "mdbox:/var/spool/virtual/domain1/$u/mdbox"
This is so manual, especially because I have to (if I remember well) create /var/spool/virtual/domain1/$u/mdbox (and have the permissions right) before running dysnc.
Is there an easier way out, perhaps??
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254733744121/+254722743223
Damn!!
Yeah, i guess the right ocmando would be dsync
dsync get the user from mysql, with user query. I guess.
Are you looking to do it without any downtime ?
Are you getting the mail_location from mysql or in conf file ?
I would use dsync backup. I had some trouble using mirror when a box was corrupted, it messed up both sides.
[]'sf.rique
On Fri, Dec 24, 2010 at 5:42 AM, Odhiambo Washington odhiambo@gmail.comwrote:
Looking at http://wiki2.dovecot.org/Migration/MailFormat, I do not see much on how to do a migration from Maildir to mdbox. That page isn't quite updated it looks.
I have several domains (virtual) with the following storage layout:
/var/spool/virtual/$domain.
So the users have their mail stored as follows:
/var/spool/virtual/domain1/$user/Maildir /var/spool/virtual/domain2/$user/Maildir /var/spool/virtual/domain3/$user/Maildir /var/spool/virtual/domainN/$user/Maildir
From the wiki, I am told the steps to take are:
- Set mail_location=mdbox:~/mdbox
- Run "doveadm mirror -u username maildir:~/Maildir"
The instruction above confuses me because in my situation, I don't have a "username" per se. dovecot-deliver knows how to figure out my usernames and their mail_location settings. How do I derive my "username" when users are virtual, stored in MySQL back end?
Again, the right command seems to rely on dsync, not "doveadm mirror".
I could manually get all the values for the users for each domain and run:
dysnc -u $u -o "mail_location=maildir:/var/spool/virtual/domain1/$u/Maildir" mirror "mdbox:/var/spool/virtual/domain1/$u/mdbox"
This is so manual, especially because I have to (if I remember well) create /var/spool/virtual/domain1/$u/mdbox (and have the permissions right) before running dysnc.
Is there an easier way out, perhaps??
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254733744121/+254722743223
Damn!!
http://wiki2.dovecot.org/Tools/Dsync
http://wiki2.dovecot.org/Tools/Dsynctake a look.
[]'sf.rique
On Fri, Dec 24, 2010 at 12:56 PM, Henrique Fernandes sf.rique@gmail.comwrote:
Yeah, i guess the right ocmando would be dsync
dsync get the user from mysql, with user query. I guess.
Are you looking to do it without any downtime ?
Are you getting the mail_location from mysql or in conf file ?
I would use dsync backup. I had some trouble using mirror when a box was corrupted, it messed up both sides.
[]'sf.rique
On Fri, Dec 24, 2010 at 5:42 AM, Odhiambo Washington odhiambo@gmail.comwrote:
Looking at http://wiki2.dovecot.org/Migration/MailFormat, I do not see much on how to do a migration from Maildir to mdbox. That page isn't quite updated it looks.
I have several domains (virtual) with the following storage layout:
/var/spool/virtual/$domain.
So the users have their mail stored as follows:
/var/spool/virtual/domain1/$user/Maildir /var/spool/virtual/domain2/$user/Maildir /var/spool/virtual/domain3/$user/Maildir /var/spool/virtual/domainN/$user/Maildir
From the wiki, I am told the steps to take are:
- Set mail_location=mdbox:~/mdbox
- Run "doveadm mirror -u username maildir:~/Maildir"
The instruction above confuses me because in my situation, I don't have a "username" per se. dovecot-deliver knows how to figure out my usernames and their mail_location settings. How do I derive my "username" when users are virtual, stored in MySQL back end?
Again, the right command seems to rely on dsync, not "doveadm mirror".
I could manually get all the values for the users for each domain and run:
dysnc -u $u -o "mail_location=maildir:/var/spool/virtual/domain1/$u/Maildir" mirror "mdbox:/var/spool/virtual/domain1/$u/mdbox"
This is so manual, especially because I have to (if I remember well) create /var/spool/virtual/domain1/$u/mdbox (and have the permissions right) before running dysnc.
Is there an easier way out, perhaps??
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254733744121/+254722743223
Damn!!
On Fri, Dec 24, 2010 at 5:56 PM, Henrique Fernandes sf.rique@gmail.comwrote:
Yeah, i guess the right ocmando would be dsync
dsync get the user from mysql, with user query. I guess.
Are you looking to do it without any downtime ?
Yes. It's a server with a few hundred users and some of them have mail on the server, to the tune of 2GB..
Are you getting the mail_location from mysql or in conf file ?
mail_location is from 10-mail.conf
I would use dsync backup. I had some trouble using mirror when a box was corrupted, it messed up both sides.
Kindly provide more details on what you did. Maybe a good headstart,
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254733744121/+254722743223
Damn!!
To do without any down time, i start to use mail_location from the sql.
Like, regular user still using mail_location from 10-mail.
Lets give an example:
User1@domain1.com user1@domain2.com User2@domain1.com User2@domail2.com
All them have mail_location in 10-mail
Gona firt migrate User1@domain1.com
# chmod a-w /path/to/thatuserhome/
Now, dovecot can not write any emails there. Will will result in temporary failure, at least in my conf it does.
So now, User1@domain1 can not receie any email only ready it.
now gona migrate!
# dsync -v -u user1@domain1.com backup mdbox:~/mdbox
Alright see if it doe snot have any erros.
So now you have 2 options, or you get each user a nemsapce or you select mail_location from mysql. I use mysql.
First have to set it up
In user query, i supose you already have it. add an collum in or mysql setup called mailbox and anothe rone called home
you can call it whatever you want, but in user query you have to use some thing like this
user_query = SELECT home AS home, mailbox AS mail
I uses home as home in case you change the table name! In mailbox collum you have the mail_location path! Each user can have it owns! And home, you have the home path. I use diferens home path cause have many storages and diferentes locations.
So, once you have mail_location coming from mysql, all you need to do after convertion is change in mysql the path of mailbox table that now will be for user1@domain1.com mdbox: ~/mdbox same you use in the comand above!
Don't forget to give the right permissions so dovecot can write in it before you change in mysql.
Now user1@domain1.com is working with mdbox and others are working with maildir.
After you migreate all, if you want to you can change the mail_location in 10-mail to mdbox:~/mdbox or keep using mysql.
When you select mailbox and it comes back as null, it will use the conf from 10-mail
Sorry about my english.
If you have any questions i would be glad to help!
Just to be sure, after migrate each user, make sure to kill all imap process that they are loggin on. ( i did not, but the wiki says you should )
About the mirror problems. I was using mbox, some boxes were broken, and when dsync crash, this messy up both sides. Using backup you can only messy up the second argument. So this will not give you any problems. Mirror can edit the source email.
[]'sf.rique
On Fri, Dec 24, 2010 at 1:55 PM, Odhiambo Washington odhiambo@gmail.comwrote:
On Fri, Dec 24, 2010 at 5:56 PM, Henrique Fernandes sf.rique@gmail.comwrote:
Yeah, i guess the right ocmando would be dsync
dsync get the user from mysql, with user query. I guess.
Are you looking to do it without any downtime ?
Yes. It's a server with a few hundred users and some of them have mail on the server, to the tune of 2GB..
Are you getting the mail_location from mysql or in conf file ?
mail_location is from 10-mail.conf
I would use dsync backup. I had some trouble using mirror when a box was corrupted, it messed up both sides.
Kindly provide more details on what you did. Maybe a good headstart,
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254733744121/+254722743223
Damn!!
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Am 24.12.2010 um 18:11 schrieb Henrique Fernandes:
Gona firt migrate User1@domain1.com
# chmod a-w /path/to/thatuserhome/
Now, dovecot can not write any emails there. Will will result in temporary failure, at least in my conf it does.
That seems a bit brute and will throw a good amount of log entries for sure.
How about this. Implement a header check in Postfix and put the mail for the account to be maintained on HOLD:
[ master.cf ] cleanup unix n - - - 0 cleanup -o header_checks=pcre:/etc/postfix/header_checks_maintenance
$ cat /etc/postfix/header_checks_maintenance /^To:.*user@domain.tld.*/ HOLD Planned maintenance for account […]
All mail matching the regex will be put on hold. This will allow other users to resume operation.
Disable new logins for the user by adding 'allow_net=" to his record, effectively disallowing all nets. Then kill his open sessions if any:
$ doveadm who <user> $ doveadm kick <user>
Perform the mdbox backup:
$ dsync -v -u <user> backup mdbox:~/mdbox
- adjust userdb entries if required to reflect new 'mail_location'
- clean up/revoke/uncomment 'header_checks_maintenance' and 'allow_net' entries
- requeue/release HELD mails and flush queue
$ postsuper -r <message> … $ postsuper -f
Regards Thomas
-----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.16 (Darwin)
iEYEARECAAYFAk0Yl38ACgkQ+meF/S97aXiijACgnArXmWOhnAkXoEtMJeqg8jjv GGsAn3L43ojC6npaWUFzwJpsir95fghn =SqwN -----END PGP SIGNATURE-----
- Thomas Leuxner tlx@leuxner.net:
Am 24.12.2010 um 18:11 schrieb Henrique Fernandes:
Gona firt migrate User1@domain1.com
# chmod a-w /path/to/thatuserhome/
Now, dovecot can not write any emails there. Will will result in temporary failure, at least in my conf it does.
That seems a bit brute and will throw a good amount of log entries for sure.
How about this. Implement a header check in Postfix and put the mail for the account to be maintained on HOLD:
[ master.cf ] cleanup unix n - - - 0 cleanup -o header_checks=pcre:/etc/postfix/header_checks_maintenance
Usually you do this in main.cf
$ cat /etc/postfix/header_checks_maintenance /^To:.*user@domain.tld.*/ HOLD Planned maintenance for account […]
And usually one uses "check_recipient_access" for this, not header_checks
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | http://www.charite.de
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Am 27.12.2010 um 15:04 schrieb Ralf Hildebrandt:
How about this. Implement a header check in Postfix and put the mail for the account to be maintained on HOLD:
[ master.cf ] cleanup unix n - - - 0 cleanup -o header_checks=pcre:/etc/postfix/header_checks_maintenance
Usually you do this in main.cf
$ cat /etc/postfix/header_checks_maintenance /^To:.*user@domain.tld.*/ HOLD Planned maintenance for account […]
And usually one uses "check_recipient_access" for this, not header_checks
Aren't the smtp restrictions enforced before accepting the mail? How would one hold something he hasn't got yet? :)
Regards Thomas -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.16 (Darwin)
iEYEARECAAYFAk0YtwkACgkQ+meF/S97aXgriQCeLY3NCh/rnCE6/oz4ft8N+/5D aWsAoLIcP+DLbdmiypyeFBogWxZ6JLPS =QdLM -----END PGP SIGNATURE-----
Thomas Leuxner put forth on 12/27/2010 9:55 AM:
Am 27.12.2010 um 15:04 schrieb Ralf Hildebrandt:
How about this. Implement a header check in Postfix and put the mail for the account to be maintained on HOLD:
[ master.cf ] cleanup unix n - - - 0 cleanup -o header_checks=pcre:/etc/postfix/header_checks_maintenance
Usually you do this in main.cf
$ cat /etc/postfix/header_checks_maintenance /^To:.*user@domain.tld.*/ HOLD Planned maintenance for account [&]
And usually one uses "check_recipient_access" for this, not header_checks
Aren't the smtp restrictions enforced before accepting the mail? How would one hold something he hasn't got yet? :)
Yes, they are.
man 5 access
HOLD is a valid action for access tables. In Ralf's example the HOLD action is what is "enforced", causing the mail to be accepted queued.
-- Stan
And usually one uses "check_recipient_access" for this, not header_checks
Aren't the smtp restrictions enforced before accepting the mail?
while, not before. They actually lead to mail being accepted or rejected.
How would one hold something he hasn't got yet? :)
See "man 5 access".
HOLD optional text...
Place the message on the hold queue, where it will
sit until someone either deletes it or releases it
for delivery. Log the optional text if specified,
otherwise log a generic message.
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | http://www.charite.de
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Am 27.12.2010 um 21:06 schrieb Ralf Hildebrandt:
See "man 5 access".
HOLD optional text... Place the message on the hold queue, where it will sit until someone either deletes it or releases it for delivery. Log the optional text if specified, otherwise log a generic message
Thanks, working as expected now. Was a bit tricky to adapt in a live system:
smtpd_recipient_restrictions = reject_unknown_recipient_domain, reject_non_fqdn_recipient, reject_unverified_recipient, permit_mynetworks, reject_unauth_destination, check_recipient_access hash:/etc/postfix/recipient_maintenance
Recipients are easier set up here compared to pcre, yes :)
user@domain.tld HOLD Planned maintenance […]
Regards Thomas
-----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.16 (Darwin)
iEYEARECAAYFAk0Y9XEACgkQ+meF/S97aXj16QCggpahCYtj03HLmPGGNkQ3/gFu MwYAoJqCYAMC442QPKZfl6kvhvTSfsR3 =ATLD -----END PGP SIGNATURE-----
Well, chmod i guess it is the simpler, do not need to change anyconf. After you give it back permission it will be ok.
Not a lot of erros, just gona be erros if get any mails at that time, i suposed will not be that long. And this process will be at some time with litle access.
But use whatever works for you!
[]'sf.rique
On Mon, Dec 27, 2010 at 6:22 PM, Thomas Leuxner tlx@leuxner.net wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Am 27.12.2010 um 21:06 schrieb Ralf Hildebrandt:
See "man 5 access".
HOLD optional text... Place the message on the hold queue, where it will sit until someone either deletes it or releases it for delivery. Log the optional text if specified, otherwise log a generic message
Thanks, working as expected now. Was a bit tricky to adapt in a live system:
smtpd_recipient_restrictions = reject_unknown_recipient_domain, reject_non_fqdn_recipient, reject_unverified_recipient, permit_mynetworks, reject_unauth_destination, check_recipient_access hash:/etc/postfix/recipient_maintenance
Recipients are easier set up here compared to pcre, yes :)
user@domain.tld HOLD Planned maintenance […]
Regards Thomas
-----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.16 (Darwin)
iEYEARECAAYFAk0Y9XEACgkQ+meF/S97aXj16QCggpahCYtj03HLmPGGNkQ3/gFu MwYAoJqCYAMC442QPKZfl6kvhvTSfsR3 =ATLD -----END PGP SIGNATURE-----
participants (5)
-
Henrique Fernandes
-
Odhiambo Washington
-
Ralf Hildebrandt
-
Stan Hoeppner
-
Thomas Leuxner