Hi all,
What is the process to convert "sdbox" installation to a "mdbox" installation ?
Thanks
Joan
Joan Moreau put forth on 12/23/2010 5:03 PM:
What is the process to convert "sdbox" installation to a "mdbox" installation ?
http://wiki2.dovecot.org/Tools/Dsync
-- Stan
Thanks for the link, but the page is not really speaking about the matter.
I clarify my question:
How to convert a "sdbox" system (in production) to "mdbox" without loosing any email ?
On Thu, 23 Dec 2010 21:26:18 -0600, Stan Hoeppner wrote:
http://wiki2.dovecot.org/Tools/Dsync [1]
Links:
Joan Moreau wrote on 12/24/2010:
Thanks for the link, but the page is not really speaking about the matter.
I clarify my question:
How to convert a "sdbox" system (in production) to "mdbox" without loosing any email ?
you can use this command for example (without loosing emails):
dsync -o mail_location=sdbox:/path_to_old_mailbox/sdbox mirror mdbox:/path_to_new_mailbox/mdbox
You can stop Dovecot, convert your emails, change the Dovecot configuration to mdbox (instead of sdbox) and restart Dovecot.
Maybe you could also use a script which runs after the user login. See http://wiki.dovecot.org/PostLoginScripting
-- Daniel
Thanks, but this keeps very unclear for me.
Concretely, I have in my dovecot.conf:
mail_gid = mail mail_location = dbox:/data/mail/%d/%n mail_uid = mailusers
What you are saying is that, I shall :
1 - plan to have all the new emails in a different folder (say "/data/mail2" ) 2 - make a script that go through all users & domains 3 - for each user (and therefore dbox folder), run "dsync -o mail_location=sdbox:/data/mail/%d/%n mirror mdbox:/data/mail2/%d/%n" (where %d and %n are updated correctly for each user) 4 - move /data/mail to /data/mailold and move /data/mail2 /data/mail 5 - update dovecot.conf with "mail_location = mdbox:/data/mail/%d/%n" 6 - Only then, restart dovecot in production ?
ISn't there a way to do that keeping dovecot running ?
You kind help is very much appreciated
Joan
On Fri, 24 Dec 2010 13:01:46 +0100, Daniel Luttermann wrote:
Links:
Joan Moreau put forth on 12/24/2010 5:41 PM:
You don't have to move the mail "back".
5 - update dovecot.conf with "mail_location = mdbox:/data/mail/%d/%n"
Simply change the mail location to the new directory in dovecot.conf. After you've confirmed all the users' mail is safely in the new location you can delete all the old mail folders/structure.
6 - Only then, restart dovecot in production ?
You can perform all of this without stopping Dovecot. IMAP commands are used for the mirror operation to the new directory/mailboxes not some direct file manipulation that requires the mailboxes be offline.
Shutting down dovecot simply makes sure you don't miss new mail that's written by users or your MTA (new deliveries) while the mirror operation is running (which can take a LONG time). If you do it live, you simply have to make a second run, which will be MUCH faster due to fewer "new" mails to mirror. Think of how rsync works--only new files are copied. So the seconds run will be quick. Immediately after the second run, stop dovecot, change the mail location in dovecot.conf to the new directory with the mdbox files and start dovecot. Or, better, stop dovecot before the 2nd mirror run, run the mirror op, change dovecot.conf with new mail location, start dovecot. Down time during the 2nd run should/will be minimal.
ISn't there a way to do that keeping dovecot running ?
Yes, see above.
-- Stan
Gonna start over and try to be more simply.
On Fri, Dec 24, 2010 at 9:57 PM, Stan Hoeppner <stan@hardwarefreak.com>wrote:
No. Do not need to do anything.
2 - make a script that go through all users & domains
Yes.
I did not, but you could!
4 - move /data/mail to /data/mailold and move /data/mail2 /data/mail
You don't have to move the mail "back".
No.
You could, or use it from mysql.
I did with down time most part of it. But a few hundres i did live.
I have about 9k acount on my server.
ISn't there a way to do that keeping dovecot running ?
Yes, see above.
As i will try to explain again!
Let see.
First some info.
Dovecot can ether get this informations about gid and uid and mail location from mysql using user_query. If the return of query is an result this will be used, if it is null the conf file will be use.
I don't know if what i used is the best way, but it worked for me.
Right now, you can run this comand.
dsync -v -u user@domain.com backup mdbox:/Converted/mail/user1/mdbox
And it will make exacly the same mail from user@domain.com in /Converted/mail/user/mdbox
but with only this, you can not do anything, becase dovecot for imap process sees the location of mail in the conf file. and it still the other path. This is why mysql with field "mail" goes. It tells dovecot that for that user, the maillocation is /Converted/mail/user/mdbox
My user query tells me for each user. uid gid home and mail_location
Usualy my mail_location is maildir: ~/mail/ and home is /some-storage/domain/user/
And for other users it uses mail_location from the file that is mbox:/domaim/user:INBOX:/var/spoll/mail
I mean, like this i have total control what each user will have it for conf.
So once you have configured dovecot to user mysql you can run this steps.
1 - make the home of user unwriteble to dovecot. 2 - conver with dsync like this. # dsync -v -u user@domain.commdbox:~/mdbox/ 5 - make mdbox:~/mdbox writible to dovecot 4 - Change mail location of this user in mysql ( before it was null, so dovecot uses the file ) change it to mdbox:~/mdbox/ 5 - Done. Now dovecot is already writing ( LDA ) and reading ( IMAP POP3 ) from mdbox
I guess it would be all
Still any questions ?
Thanks a lot.
That works for me
On Fri, 24 Dec 2010 17:57:35 -0600, Stan Hoeppner wrote:
- for each user (and therefore dbox folder), run "dsync -o mail_location=sdbox:/data/mail/%d/%n mirror mdbox:/data/mail2/%d/%n" (where %d and %n are updated correctly for each user) 4 - move /data/mail to /data/mailold and move /data/mail2 /data/mail
Ok, sorry, some bug on my side.
Now, when doing the sync for all users, I get this every time:
dsync[30214]: segfault at 0 ip 00007f02990349eb sp 00007fff56c2cc80 error 4 in libdovecot-storage.so.0.0.0[7f0299001000+d3000]
Any idea ?
On Mon, 3 Jan 2011 00:13:20 -0200, Henrique Fernandes wrote:
Links:
[1] mailto:jom@grosjo.net
Not sure how do I get the "core" (core dumped are disabled in my kernel)
On Tue, 04 Jan 2011 09:49:40 +0200, Timo Sirainen wrote:
On Mon, 2011-01-03 at 20:42 -0500, Joan Moreau wrote:
dsync[30214]: segfault at 0 ip 00007f02990349eb sp 00007fff56c2cc80 error 4 in libdovecot-storage.so.0.0.0[7f0299001000+d3000]
Get gdb backtrace. ulimit -c unlimited dsync ... gdb dsync core bt full
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Tue, Jan 04, 2011 at 11:01:25AM -0500, Joan Moreau wrote:
Not sure how do I get the "core" (core dumped are disabled in my kernel)
I think this is the "ulimit -c unlimited" part in Timo's mail. But we can't know for sure if we don't know more about your kernel.
Regards
- -- tomás -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux)
iD8DBQFNJBVJBcgs9XrR2kYRAvvoAJ94xleMGwY7Q2QP1M+iRUFPytzi9ACfcmMy N4JDzxELS7vpDnZ41I236iI= =v8hm -----END PGP SIGNATURE-----
this is what I get:
gdb --args dsync -f -D -v -u 'jom@grosjo.net' -o
mail_location=sdbox:/data/mail/grosjo.net/jom mirror mdbox:/data/mail2/grosjo.net/jom GNU gdb (GDB) SUSE (7.1-3.12) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-suse-linux". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/bin/dsync...done. (gdb) run Starting program: /usr/bin/dsync -f -D -v -u jom@grosjo.net -o mail_location=sdbox:/data/mail/grosjo.net/jom mirror mdbox:/data/mail2/grosjo.net/jom [Thread debugging using libthread_db enabled] dsync(jom@grosjo.net): Error: open(/data/mail2/grosjo.net/jom/storage/.temp.1294246748.P26952Q8500M325824.server, O_CREAT) failed: Too many open files dsync(jom@grosjo.net): Error: Can't save message to mailbox Business/GJ-Pro/ParkingRoissy/Log: Internal error occurred. Refer to server log for more information. [2011-01-05 17:59:08]
Program received signal SIGSEGV, Segmentation fault. mailbox_save_cancel (_ctx=0x7fffffffdc78) at mail-storage.c:1419 1419 struct mailbox *box = ctx->transaction->box;
(gdb) bt full #0 mailbox_save_cancel (_ctx=0x7fffffffdc78) at mail-storage.c:1419 ctx = 0x0 box = <value optimized out> keywords = 0x7ffff7ffe148 #1 0x000000000040fee7 in local_worker_msg_save (_worker=0x648d80, msg=<value optimized out>, data=<value optimized out>, callback=0x40a400 <msg_save_callback>, context=0x8a6110) at dsync-worker-local.c:1719 storage = <value optimized out> worker = 0x648d80 dest_box = 0x8c0df0 save_ctx = 0x0 __FUNCTION__ = "local_worker_msg_save" #2 0x000000000040f289 in dsync_worker_msg_save (worker=0x648d80, msg=<value optimized out>, data=0x7fffffffdd50, callback=0x40a400 <msg_save_callback>, context=0x8a6110) at dsync-worker.c:234 _data_stack_cur_id = 3 #3 0x000000000040a4cd in msg_get_callback (result=<value optimized out>, data=0x7fffffffdd50, context=0x8a6110) at dsync-brain-msgs-new.c:78 ctx = 0x8a6110 mailbox = <value optimized out> input = 0x4eb7d50 __FUNCTION__ = "msg_get_callback" #4 0x000000000040f858 in local_worker_msg_get_next (worker=0x63ee20, get=0x7fffffffdd90) at dsync-worker-local.c:1826 data = {pop3_uidl = 0x619fd0 "", received_date = 1260352593, input = 0x4eb7d50} trans = <value optimized out> box = 0x18 __FUNCTION__ = "local_worker_msg_get_next" #5 0x000000000040fa38 in local_worker_msg_get (_worker=<value optimized out>, mailbox=<value optimized out>, uid=<value optimized out>, callback=<value optimized out>, context=<value optimized out>) at dsync-worker-local.c:1847 worker = <value optimized out> get = {mailbox = {guid = "\rɚ\"\224\233\301L\277W\000\000֣\t-"}, uid = 9898, callback = 0x40a450 <msg_get_callback>, context = 0x8a6110} #6 0x000000000040f306 in dsync_worker_msg_get (worker=0x63ee20, mailbox=0x704ec1, uid=9898, callback=0x40a450 <msg_get_callback>, context=<value optimized out>) at dsync-worker.c:261 _data_stack_cur_id = 2 __FUNCTION__ = "dsync_worker_msg_get" #7 0x000000000040a371 in dsync_brain_msg_sync_add_new_msg (iter=0x70b440) at dsync-brain-msgs-new.c:180 inst = <value optimized out> inst_box = <value optimized out> save_ctx = <value optimized out> copy_ctx = <value optimized out> src_iter = 0x70b3b8 #8 dsync_brain_mailbox_add_new_msgs (iter=0x70b440) at dsync-brain-msgs-new.c:210 msg = 0x7ffff3ef8330 msgs = 0x7ffff3ed2010 msg_count = 414226 ret = true #9 dsync_brain_msg_sync_add_new_msgs (iter=0x70b440) at dsync-brain-msgs-new.c:309 mailbox = 0x704ea8 mailbox_guid = 0x704ec1 #10 0x0000000000409a3b in dsync_brain_msg_sync_more (sync=0x704870) at dsync-brain-msgs.c:424 mailboxes = 0x7048f8 count = 244 mailbox_idx = 244 #11 0x0000000000408449 in dsync_brain_sync_msgs (brain=<value optimized out>) at dsync-brain.c:726 mailboxes = {arr = {buffer = 0x6d4050, element_size = 112}, v = 0x6d4050, v_modifiable = 0x6d4050} pool = 0x6d4030 ret = <value optimized out> #12 dsync_brain_sync (brain=<value optimized out>) at dsync-brain.c:847 __FUNCTION__ = "dsync_brain_sync" #13 0x0000000000409381 in dsync_brain_subs_list_finished (context=0x663210) at dsync-brain.c:170 No locals. #14 dsync_worker_subs_input (context=0x663210) at dsync-brain.c:223 list = 0x663210 subs = {vname = 0x0, storage_name = 0x0, ns_prefix = 0x0, last_change = 0} unsubs = {name_sha1 = {guid = '\000' <repeats 15 times>}, ns_prefix = 0x0, last_change = 0} ret = <value optimized out> #15 0x0000000000408820 in dsync_brain_sync (brain=0x64c990) at dsync-brain.c:832 __FUNCTION__ = "dsync_brain_sync" #16 0x000000000040921c in dsync_brain_mailbox_list_finished (context=0x65a9e0) at dsync-brain.c:99 No locals. #17 dsync_worker_mailbox_input (context=0x65a9e0) at dsync-brain.c:126 list = 0x65a9e0 dsync_box = {name = 0x0, name_sep = 0 '\000', name_sha1 = {guid = '\000' <repeats 15 times>}, mailbox_guid = {guid = '\000' <repeats 15 times>}, uid_validity = 0, uid_next = 0, message_count = 0, highest_modseq = 0, last_change = 0, flags = 0, cache_fields = {arr = {buffer = 0x0, element_size = 0}, v = 0x0, v_modifiable = 0x0}} dup_box = 0x672c58 ret = <value optimized out> #18 0x0000000000408861 in dsync_brain_sync (brain=0x64c990) at dsync-brain.c:823 __FUNCTION__ = "dsync_brain_sync" #19 0x00000000004090e8 in dsync_brain_sync_all (brain=0x64c990) at dsync-brain.c:887 old_state = DSYNC_STATE_GET_MAILBOXES __FUNCTION__ = "dsync_brain_sync_all" #20 0x00000000004074d7 in main (argc=10, argv=0x621370) at dsync.c:303 set_parser = <value optimized out> ssflags = <value optimized out> brain_flags = 3 storage_service = 0x623140 service_user = 0x623ab8 input = {module = 0x41419f "mail", service = 0x4129eb "dsync", username = 0x6213e3 "jom@grosjo.net", local_ip = {family = 0, u = {ip6 = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, ip4 = {s_addr = 0}}}, remote_ip = {family = 0, u = {ip6 = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, ip4 = { s_addr = 0}}}, userdb_fields = 0x0, no_userdb_lookup = 0} mail_user = 0x63ae10 mail_user2 = 0x642280 worker1 = 0x63ee20 worker2 = 0x648d80 workertmp = <value optimized out> error = <value optimized out> username = 0x6213e3 "jom@grosjo.net" cmd_name = <value optimized out> mailbox = 0x0 local_location = 0x619e40 "/data/mail/grosjo.net/jom/mailboxes" remote_cmd_args = 0x0 path1 = 0x619e40 "/data/mail/grosjo.net/jom/mailboxes" path2 = <value optimized out> dsync_server = false unexpected_changes = false dsync_debug = <value optimized out> reverse_workers = false alt_char = 95 '_' c = <value optimized out> ret = <value optimized out> fd_in = 0 fd_out = 1 __FUNCTION__ = "main"
On Wed, 05 Jan 2011 14:31:59 +0200, Timo Sirainen wrote:
On 5.1.2011, at 19.33, Joan Moreau wrote:
dsync(jom@grosjo.net): Error: open(/data/mail2/grosjo.net/jom/storage/.temp.1294246748.P26952Q8500M325824.server, O_CREAT) failed: Too many open files
Hmm. This is the main problem. How much messages do you have?.. I thought mdbox was supposed to keep only a few files open. Maybe there's a bug in here..
This crash is also now fixed by: http://hg.dovecot.org/dovecot-2.0/rev/7bdd06de6f5a
I tested this on my own mailbox. I have about 50 IMAP folders, about 25Gb of mails, and I do not know how many thousands of emails.
I think the dsync shall not keep the file pointers opened, but process one ofter another somehow, so there shall be no limit in mailboxes size.
On Wed, 5 Jan 2011 20:24:08 +0200, Timo Sirainen wrote:
Links:
[1] mailto:jom@grosjo.net [2] http://hg.dovecot.org/dovecot-2.0/rev/7bdd06de6f5a
participants (6)
-
Daniel Luttermann
-
Henrique Fernandes
-
Joan Moreau
-
Stan Hoeppner
-
Timo Sirainen
-
tomas＠tuxteam.de