Hi all,
What is the process to convert "sdbox" installation to a "mdbox" installation ?
Thanks
Joan
Joan Moreau put forth on 12/23/2010 5:03 PM:
What is the process to convert "sdbox" installation to a "mdbox" installation ?
http://wiki2.dovecot.org/Tools/Dsync
-- Stan
Thanks for the link, but the page is not really speaking about the matter.
I clarify my question:
How to convert a "sdbox" system (in production) to "mdbox" without loosing any email ?
On Thu, 23 Dec 2010 21:26:18 -0600, Stan Hoeppner wrote:
Joan Moreau put forth on 12/23/2010 5:03 PM:
What is the process to convert "sdbox" installation to a "mdbox" installation ?
http://wiki2.dovecot.org/Tools/Dsync [1]
Links:
Joan Moreau wrote on 12/24/2010:
Thanks for the link, but the page is not really speaking about the matter.
I clarify my question:
How to convert a "sdbox" system (in production) to "mdbox" without loosing any email ?
you can use this command for example (without loosing emails):
dsync -o mail_location=sdbox:/path_to_old_mailbox/sdbox mirror mdbox:/path_to_new_mailbox/mdbox
You can stop Dovecot, convert your emails, change the Dovecot configuration to mdbox (instead of sdbox) and restart Dovecot.
Maybe you could also use a script which runs after the user login. See http://wiki.dovecot.org/PostLoginScripting
-- Daniel
Thanks, but this keeps very unclear for me.
Concretely, I have in my dovecot.conf:
mail_gid = mail mail_location = dbox:/data/mail/%d/%n mail_uid = mailusers
What you are saying is that, I shall :
1 - plan to have all the new emails in a different folder (say "/data/mail2" ) 2 - make a script that go through all users & domains 3 - for each user (and therefore dbox folder), run "dsync -o mail_location=sdbox:/data/mail/%d/%n mirror mdbox:/data/mail2/%d/%n" (where %d and %n are updated correctly for each user) 4 - move /data/mail to /data/mailold and move /data/mail2 /data/mail 5 - update dovecot.conf with "mail_location = mdbox:/data/mail/%d/%n" 6 - Only then, restart dovecot in production ?
ISn't there a way to do that keeping dovecot running ?
You kind help is very much appreciated
Joan
On Fri, 24 Dec 2010 13:01:46 +0100, Daniel Luttermann wrote:
Joan Moreau wrote on 12/24/2010:
Thanks for the link, but the page is not really speaking about the matter.
I clarify my question: convert a "sdbox" system (in production) to "mdbox" without loosing any email ? an use this command for example (without loosing emails): dsync -o mail_location=sdbox:/path_to_old_mailbox/sdbox mirror mdbox:/path_to_new_mailbox/mdbox You can stop Dovecot, convert your emails, change the Dovecot configuration to mdbox (instead of sdbox) and restart Dovecot. Maybe you could also use a script which runs after the user login. See http://wiki.dovecot.org/PostLoginScripting [1] -- Daniel
Links:
Joan Moreau put forth on 12/24/2010 5:41 PM:
1 - plan to have all the new emails in a different folder (say "/data/mail2" ) 2 - make a script that go through all users & domains 3 - for each user (and therefore dbox folder), run "dsync -o mail_location=sdbox:/data/mail/%d/%n mirror mdbox:/data/mail2/%d/%n" (where %d and %n are updated correctly for each user) 4 - move /data/mail to /data/mailold and move /data/mail2 /data/mail
You don't have to move the mail "back".
5 - update dovecot.conf with "mail_location = mdbox:/data/mail/%d/%n"
Simply change the mail location to the new directory in dovecot.conf. After you've confirmed all the users' mail is safely in the new location you can delete all the old mail folders/structure.
6 - Only then, restart dovecot in production ?
You can perform all of this without stopping Dovecot. IMAP commands are used for the mirror operation to the new directory/mailboxes not some direct file manipulation that requires the mailboxes be offline.
Shutting down dovecot simply makes sure you don't miss new mail that's written by users or your MTA (new deliveries) while the mirror operation is running (which can take a LONG time). If you do it live, you simply have to make a second run, which will be MUCH faster due to fewer "new" mails to mirror. Think of how rsync works--only new files are copied. So the seconds run will be quick. Immediately after the second run, stop dovecot, change the mail location in dovecot.conf to the new directory with the mdbox files and start dovecot. Or, better, stop dovecot before the 2nd mirror run, run the mirror op, change dovecot.conf with new mail location, start dovecot. Down time during the 2nd run should/will be minimal.
ISn't there a way to do that keeping dovecot running ?
Yes, see above.
-- Stan
Gonna start over and try to be more simply.
On Fri, Dec 24, 2010 at 9:57 PM, Stan Hoeppner stan@hardwarefreak.comwrote:
Joan Moreau put forth on 12/24/2010 5:41 PM:
1 - plan to have all the new emails in a different folder (say "/data/mail2" )
No. Do not need to do anything.
2 - make a script that go through all users & domains
Yes.
3 - for each user (and therefore dbox folder), run "dsync -o mail_location=sdbox:/data/mail/%d/%n mirror mdbox:/data/mail2/%d/%n" (where %d and %n are updated correctly for each user)
I did not, but you could!
4 - move /data/mail to /data/mailold and move /data/mail2 /data/mail
You don't have to move the mail "back".
No.
5 - update dovecot.conf with "mail_location = mdbox:/data/mail/%d/%n"
Simply change the mail location to the new directory in dovecot.conf. After you've confirmed all the users' mail is safely in the new location you can delete all the old mail folders/structure.
You could, or use it from mysql.
6 - Only then, restart dovecot in production ?
You can perform all of this without stopping Dovecot. IMAP commands are used for the mirror operation to the new directory/mailboxes not some direct file manipulation that requires the mailboxes be offline.
Shutting down dovecot simply makes sure you don't miss new mail that's written by users or your MTA (new deliveries) while the mirror operation is running (which can take a LONG time). If you do it live, you simply have to make a second run, which will be MUCH faster due to fewer "new" mails to mirror. Think of how rsync works--only new files are copied. So the seconds run will be quick. Immediately after the second run, stop dovecot, change the mail location in dovecot.conf to the new directory with the mdbox files and start dovecot. Or, better, stop dovecot before the 2nd mirror run, run the mirror op, change dovecot.conf with new mail location, start dovecot. Down time during the 2nd run should/will be minimal.
I did with down time most part of it. But a few hundres i did live.
I have about 9k acount on my server.
ISn't there a way to do that keeping dovecot running ?
Yes, see above.
As i will try to explain again!
Let see.
First some info.
Dovecot can ether get this informations about gid and uid and mail location from mysql using user_query. If the return of query is an result this will be used, if it is null the conf file will be use.
I don't know if what i used is the best way, but it worked for me.
Right now, you can run this comand.
# dsync -v -u user@domain.com backup mdbox:/Converted/mail/user1/mdbox
And it will make exacly the same mail from user@domain.com in /Converted/mail/user/mdbox
but with only this, you can not do anything, becase dovecot for imap process sees the location of mail in the conf file. and it still the other path. This is why mysql with field "mail" goes. It tells dovecot that for that user, the maillocation is /Converted/mail/user/mdbox
My user query tells me for each user. uid gid home and mail_location
Usualy my mail_location is maildir: ~/mail/ and home is /some-storage/domain/user/
And for other users it uses mail_location from the file that is mbox:/domaim/user:INBOX:/var/spoll/mail
I mean, like this i have total control what each user will have it for conf.
So once you have configured dovecot to user mysql you can run this steps.
1 - make the home of user unwriteble to dovecot. 2 - conver with dsync like this. # dsync -v -u user@domain.commdbox:~/mdbox/ 5 - make mdbox:~/mdbox writible to dovecot 4 - Change mail location of this user in mysql ( before it was null, so dovecot uses the file ) change it to mdbox:~/mdbox/ 5 - Done. Now dovecot is already writing ( LDA ) and reading ( IMAP POP3 ) from mdbox
I guess it would be all
Still any questions ?
Hey, sorry i confused in other post that is going on the list, but i guess will be helpfull anyway!
[]'sf.rique
On Fri, Dec 24, 2010 at 10:43 PM, Henrique Fernandes sf.rique@gmail.comwrote:
Gonna start over and try to be more simply.
On Fri, Dec 24, 2010 at 9:57 PM, Stan Hoeppner stan@hardwarefreak.comwrote:
Joan Moreau put forth on 12/24/2010 5:41 PM:
1 - plan to have all the new emails in a different folder (say "/data/mail2" )
No. Do not need to do anything.
2 - make a script that go through all users & domains
Yes.
3 - for each user (and therefore dbox folder), run "dsync -o mail_location=sdbox:/data/mail/%d/%n mirror mdbox:/data/mail2/%d/%n" (where %d and %n are updated correctly for each user)
I did not, but you could!
4 - move /data/mail to /data/mailold and move /data/mail2 /data/mail
You don't have to move the mail "back".
No.
5 - update dovecot.conf with "mail_location = mdbox:/data/mail/%d/%n"
Simply change the mail location to the new directory in dovecot.conf. After you've confirmed all the users' mail is safely in the new location you can delete all the old mail folders/structure.
You could, or use it from mysql.
6 - Only then, restart dovecot in production ?
You can perform all of this without stopping Dovecot. IMAP commands are used for the mirror operation to the new directory/mailboxes not some direct file manipulation that requires the mailboxes be offline.
Shutting down dovecot simply makes sure you don't miss new mail that's written by users or your MTA (new deliveries) while the mirror operation is running (which can take a LONG time). If you do it live, you simply have to make a second run, which will be MUCH faster due to fewer "new" mails to mirror. Think of how rsync works--only new files are copied. So the seconds run will be quick. Immediately after the second run, stop dovecot, change the mail location in dovecot.conf to the new directory with the mdbox files and start dovecot. Or, better, stop dovecot before the 2nd mirror run, run the mirror op, change dovecot.conf with new mail location, start dovecot. Down time during the 2nd run should/will be minimal.
I did with down time most part of it. But a few hundres i did live.
I have about 9k acount on my server.
ISn't there a way to do that keeping dovecot running ?
Yes, see above.
As i will try to explain again!
Let see.
First some info.
Dovecot can ether get this informations about gid and uid and mail location from mysql using user_query. If the return of query is an result this will be used, if it is null the conf file will be use.
I don't know if what i used is the best way, but it worked for me.
Right now, you can run this comand.
# dsync -v -u user@domain.com backup mdbox:/Converted/mail/user1/mdbox
And it will make exacly the same mail from user@domain.com in /Converted/mail/user/mdbox
but with only this, you can not do anything, becase dovecot for imap process sees the location of mail in the conf file. and it still the other path. This is why mysql with field "mail" goes. It tells dovecot that for that user, the maillocation is /Converted/mail/user/mdbox
My user query tells me for each user. uid gid home and mail_location
Usualy my mail_location is maildir: ~/mail/ and home is /some-storage/domain/user/
And for other users it uses mail_location from the file that is mbox:/domaim/user:INBOX:/var/spoll/mail
I mean, like this i have total control what each user will have it for conf.
So once you have configured dovecot to user mysql you can run this steps.
1 - make the home of user unwriteble to dovecot. 2 - conver with dsync like this. # dsync -v -u user@domain.commdbox:~/mdbox/ 5 - make mdbox:~/mdbox writible to dovecot 4 - Change mail location of this user in mysql ( before it was null, so dovecot uses the file ) change it to mdbox:~/mdbox/ 5 - Done. Now dovecot is already writing ( LDA ) and reading ( IMAP POP3 ) from mdbox
I guess it would be all
Still any questions ?
Thanks a lot.
That works for me
On Fri, 24 Dec 2010 17:57:35 -0600, Stan Hoeppner wrote:
- for each user (and therefore dbox folder), run "dsync -o mail_location=sdbox:/data/mail/%d/%n mirror mdbox:/data/mail2/%d/%n" (where %d and %n are updated correctly for each user) 4 - move /data/mail to /data/mailold and move /data/mail2 /data/mail
Joan Moreau put forth on 12/24/2010 5:41 PM:
1 - plan to have all the new emails in a different folder (say "/data/mail2" ) 2 - make a script that go through all users & domains 3
You don't have to move the mail "back".
5 - update dovecot.conf with "mail_location = mdbox:/data/mail/%d/%n" Simply change the mail location to the new directory in dovecot.conf. After you've confirmed all the users' mail is safely in the new location you can delete all the old mail folders/structure.
Actually, the directory created by dsync contains correctly all the folders of the original IMAP account.
but they are empty ! (just one file "dovecot.index.log" )
What do I do wrong ?
Thanks again
On Fri, 24 Dec 2010 17:57:35 -0600, Stan Hoeppner wrote:
Joan Moreau put forth on 12/24/2010 5:41 PM:
1 - plan to have all the new emails in a different folder (say "/data/mail2" ) 2 - make a script that go through all users & domains 3 - for each user (and therefore dbox folder), run "dsync -o mail_location=sdbox:/data/mail/%d/%n mirror mdbox:/data/mail2/%d/%n" (where %d and %n are updated correctly for each user) 4 - move /data/mail to /data/mailold and move /data/mail2 /data/mail You don't have to move the mail "back".
5 - update dovecot.conf with "mail_location = mdbox:/data/mail/%d/%n" Simply change the mail location to the new directory in dovecot.conf. After you've confirmed all the users' mail is safely in the new location you can delete all the old mail folders/structure.
Can you email the output of
# dovecont -n
and
# dsync -D ( rest of your comand )
Rememer, first dsync does not need to edit anyconf on dovecot!
[]'sf.rique
On Thu, Dec 30, 2010 at 2:34 PM, Joan Moreau jom@grosjo.net wrote:
Actually, the directory created by dsync contains correctly all the folders of the original IMAP account.
but they are empty ! (just one file "dovecot.index.log" )
What do I do wrong ?
Thanks again
On Fri, 24 Dec 2010 17:57:35 -0600, Stan Hoeppner wrote:
Joan Moreau put forth on 12/24/2010 5:41 PM:
1 - plan to have all the new emails in a different folder (say "/data/mail2" ) 2 - make a script that go through all users & domains 3 - for each user (and therefore dbox folder), run "dsync -o mail_location=sdbox:/data/mail/%d/%n mirror mdbox:/data/mail2/%d/%n" (where %d and %n are updated correctly for each user) 4 - move /data/mail to /data/mailold and move /data/mail2 /data/mail You don't have to move the mail "back".
5 - update dovecot.conf with "mail_location = mdbox:/data/mail/%d/%n" Simply change the mail location to the new directory in dovecot.conf. After you've confirmed all the users' mail is safely in the new location you can delete all the old mail folders/structure.
Ok, sorry, some bug on my side.
Now, when doing the sync for all users, I get this every time:
dsync[30214]: segfault at 0 ip 00007f02990349eb sp 00007fff56c2cc80 error 4 in libdovecot-storage.so.0.0.0[7f0299001000+d3000]
Any idea ?
On Mon, 3 Jan 2011 00:13:20 -0200, Henrique Fernandes wrote:
Can you email the output of
# dovecont -n
and
# dsync -D ( rest of your comand )
Rememer, first dsync does not need to edit anyconf on dovecot!
[]'sf.rique
On Thu, Dec 30, 2010 at 2:34 PM, Joan Moreau wrote:
Actually, the directory created by dsync contains correctly all the folders of the original IMAP account.
but they are empty ! (just one file "dovecot.index.log" )
What do I do wrong ?
Thanks again
On
Fri, 24 Dec 2010 17:57:35 -0600, Stan Hoeppner wrote:
Joan Moreau put forth on 12/24/2010 5:41 PM:
1 - plan to have all the new emails in a different folder (say "/data/mail2" ) 2 - make a script that go through all users & domains 3 - for each user (and therefore dbox
folder), run "dsync -o mail_location=sdbox:/data/mail/%d/%n mirror
mdbox:/data/mail2/%d/%n" (where %d and %n are updated correctly for each
user) 4 - move /data/mail to /data/mailold and move /data/mail2 /data/mail
You don't have to move the mail "back".
5 - update dovecot.conf with "mail_location = mdbox:/data/mail/%d/%n" Simply change the mail location to the new directory in dovecot.conf. After you've confirmed all the users' mail is safely in the new location you can delete all the old mail folders/structure.
Links:
[1] mailto:jom@grosjo.net
Not sure how do I get the "core" (core dumped are disabled in my kernel)
On Tue, 04 Jan 2011 09:49:40 +0200, Timo Sirainen wrote:
On Mon, 2011-01-03 at 20:42 -0500, Joan Moreau wrote:
dsync[30214]: segfault at 0 ip 00007f02990349eb sp 00007fff56c2cc80 error 4 in libdovecot-storage.so.0.0.0[7f0299001000+d3000]
Get gdb backtrace. ulimit -c unlimited dsync ... gdb dsync core bt full
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Tue, Jan 04, 2011 at 11:01:25AM -0500, Joan Moreau wrote:
Not sure how do I get the "core" (core dumped are disabled in my kernel)
I think this is the "ulimit -c unlimited" part in Timo's mail. But we can't know for sure if we don't know more about your kernel.
Regards
- -- tomás -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux)
iD8DBQFNJBVJBcgs9XrR2kYRAvvoAJ94xleMGwY7Q2QP1M+iRUFPytzi9ACfcmMy N4JDzxELS7vpDnZ41I236iI= =v8hm -----END PGP SIGNATURE-----
On Tue, 2011-01-04 at 11:01 -0500, Joan Moreau wrote:
Not sure how do I get the "core" (core dumped are disabled in my kernel) .. backtrace. ulimit -c unlimited dsync ... gdb dsync core bt full
Well, you could probably run the whole thing in gdb, but in some situations it doesn't work:
gdb --args dsync mirror etc run bt full
this is what I get:
# gdb --args dsync -f -D -v -u 'jom@grosjo.net' -o mail_location=sdbox:/data/mail/grosjo.net/jom mirror mdbox:/data/mail2/grosjo.net/jom GNU gdb (GDB) SUSE (7.1-3.12) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-suse-linux". For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/... Reading symbols from /usr/bin/dsync...done. (gdb) run Starting program: /usr/bin/dsync -f -D -v -u jom@grosjo.net -o mail_location=sdbox:/data/mail/grosjo.net/jom mirror mdbox:/data/mail2/grosjo.net/jom [Thread debugging using libthread_db enabled] dsync(jom@grosjo.net): Error: open(/data/mail2/grosjo.net/jom/storage/.temp.1294246748.P26952Q8500M325824.server, O_CREAT) failed: Too many open files dsync(jom@grosjo.net): Error: Can't save message to mailbox Business/GJ-Pro/ParkingRoissy/Log: Internal error occurred. Refer to server log for more information. [2011-01-05 17:59:08]
Program received signal SIGSEGV, Segmentation fault. mailbox_save_cancel (_ctx=0x7fffffffdc78) at mail-storage.c:1419 1419 struct mailbox *box = ctx->transaction->box;
(gdb) bt full
#0 mailbox_save_cancel (_ctx=0x7fffffffdc78) at mail-storage.c:1419
ctx = 0x0
box = <value optimized out>
keywords = 0x7ffff7ffe148
#1 0x000000000040fee7 in local_worker_msg_save (_worker=0x648d80,
msg=<value optimized out>, data=<value optimized out>, callback=0x40a400
On Wed, 05 Jan 2011 14:31:59 +0200, Timo Sirainen wrote:
On Tue, 2011-01-04 at 11:01 -0500, Joan Moreau wrote:
Not sure how do I get the "core" (core dumped are disabled in my kernel) ..
backtrace. ulimit -c unlimited dsync ... gdb dsync core bt full Well, you could probably run the whole thing in gdb, but in some situations it doesn't work: gdb --args dsync mirror etc run bt full
On 5.1.2011, at 19.33, Joan Moreau wrote:
dsync(jom@grosjo.net): Error: open(/data/mail2/grosjo.net/jom/storage/.temp.1294246748.P26952Q8500M325824.server, O_CREAT) failed: Too many open files
Hmm. This is the main problem. How much messages do you have?.. I thought mdbox was supposed to keep only a few files open. Maybe there's a bug in here..
Program received signal SIGSEGV, Segmentation fault. mailbox_save_cancel (_ctx=0x7fffffffdc78) at mail-storage.c:1419 1419 struct mailbox *box = ctx->transaction->box;
This crash is also now fixed by: http://hg.dovecot.org/dovecot-2.0/rev/7bdd06de6f5a
I tested this on my own mailbox. I have about 50 IMAP folders, about 25Gb of mails, and I do not know how many thousands of emails.
I think the dsync shall not keep the file pointers opened, but process one ofter another somehow, so there shall be no limit in mailboxes size.
On Wed, 5 Jan 2011 20:24:08 +0200, Timo Sirainen wrote:
On 5.1.2011, at 19.33, Joan Moreau wrote:
dsync(jom@grosjo.net [1]): Error: open(/data/mail2/grosjo.net/jom/storage/.temp.1294246748.P26952Q8500M325824.server, O_CREAT) failed: Too many open files Hmm. This is the main problem. How much messages do you have?.. I thought mdbox was supposed to keep only a few files open. Maybe there's a bug in here..
Program received signal SIGSEGV, Segmentation fault. mailbox_save_cancel (_ctx=0x7fffffffdc78) at mail-storage.c:1419 1419 struct mailbox *box = ctx->transaction->box; This crash is also now fixed by: http://hg.dovecot.org/dovecot-2.0/rev/7bdd06de6f5a [2]
Links:
[1] mailto:jom@grosjo.net [2] http://hg.dovecot.org/dovecot-2.0/rev/7bdd06de6f5a
participants (6)
-
Daniel Luttermann
-
Henrique Fernandes
-
Joan Moreau
-
Stan Hoeppner
-
Timo Sirainen
-
tomas＠tuxteam.de