[Dovecot] maildir-dbox hybrid
Hi!
How can I convert old mailboxes in "maildir-dbox hybrid" to a current dbox?
I have a Dovecot installation that was upgraded from 1.2.x to 2.0.x. Having used the old convert plugin during the 1.2-times I am now stuck with mailboxes which are partially converted (maildir-dbox hybrid state).
On http://wiki2.dovecot.org/Upgrading/2.0 it just says to contact the mailinglist. :)
| If you have any Maildir files in your dbox, you need to convert them | somehow (contact Dovecot mailing list)
Regards,
Roland.
P.S.: I am currently using version: 2.0.5 (b8803a1e2dc2)
On 17.10.2010, at 17.52, Roland Stuehmer wrote:
How can I convert old mailboxes in "maildir-dbox hybrid" to a current dbox?
Does this help? http://dovecot.org/list/dovecot/2010-September/053012.html
On Sun, 17 Oct 2010 22:23:29 +0100, Timo Sirainen tss@iki.fi wrote:
On 17.10.2010, at 17.52, Roland Stuehmer wrote:
How can I convert old mailboxes in "maildir-dbox hybrid" to a current dbox?
Does this help? http://dovecot.org/list/dovecot/2010-September/053012.html
If I understand it correctly, dbox/maildir hybrid is when there are some messages stored as maildir msgs and some messages in dbox format, isn't it?
The conversion, as described in the suggested post seems like a lot of trouble. So, I have some questions:
In the first scenario, if I redeliver maildir-format messages, does the users that use POP3 and store their mail on the server will re-download all redelivered messages? (if it does matter, my pop3_uidl_format = %08Xu%08Xv).
The other option - convert the dbox-es back to maildir, move the old maildir messages in the new maildir, manually add them to the new uid-list and finally convert the new maildir to sdbox - seems quite fragile.
Best regards
-- Luben Karavelov Research and development Spectrum Net JSC
36, D-r G. M. Dimitrov Blvd. 1797 Sofia Mobile: +359 884332840 url: www.spnet.net
On Mon, 2010-10-18 at 21:56 +0300, Luben Karavelov wrote:
On Sun, 17 Oct 2010 22:23:29 +0100, Timo Sirainen tss@iki.fi wrote:
On 17.10.2010, at 17.52, Roland Stuehmer wrote:
How can I convert old mailboxes in "maildir-dbox hybrid" to a current dbox?
Does this help? http://dovecot.org/list/dovecot/2010-September/053012.html
If I understand it correctly, dbox/maildir hybrid is when there are some messages stored as maildir msgs and some messages in dbox format, isn't it?
Right.
The conversion, as described in the suggested post seems like a lot of trouble.
Yeah.
So, I have some questions:
In the first scenario, if I redeliver maildir-format messages, does the users that use POP3 and store their mail on the server will re-download all redelivered messages? (if it does matter, my pop3_uidl_format = %08Xu%08Xv).
Yes, pop3 users will redownload mails.
The other option - convert the dbox-es back to maildir, move the old maildir messages in the new maildir, manually add them to the new uid-list and finally convert the new maildir to sdbox - seems quite fragile.
Yeah. Not very nice either. Here's a 3rd option: Apply the attached patch to v1.2 and then open all users' all mailboxes (write a script or something). It should convert all maildir files to dbox files. Try with one account first to make sure it works. :)
On Tue, 19 Oct 2010 19:11:20 +0100, Timo Sirainen tss@iki.fi wrote:
On Mon, 2010-10-18 at 21:56 +0300, Luben Karavelov wrote:
On Sun, 17 Oct 2010 22:23:29 +0100, Timo Sirainen tss@iki.fi wrote:
On 17.10.2010, at 17.52, Roland Stuehmer wrote:
How can I convert old mailboxes in "maildir-dbox hybrid" to a current dbox?
Does this help? http://dovecot.org/list/dovecot/2010-September/053012.html
If I understand it correctly, dbox/maildir hybrid is when there are some messages stored as maildir msgs and some messages in dbox format, isn't it?
Right.
The conversion, as described in the suggested post seems like a lot of trouble.
Yeah.
So, I have some questions:
In the first scenario, if I redeliver maildir-format messages, does the users that use POP3 and store their mail on the server will re-download all redelivered messages? (if it does matter, my pop3_uidl_format = %08Xu%08Xv).
Yes, pop3 users will redownload mails.
The other option - convert the dbox-es back to maildir, move the old maildir messages in the new maildir, manually add them to the new uid-list and finally convert the new maildir to sdbox - seems quite fragile.
Yeah. Not very nice either. Here's a 3rd option: Apply the attached patch to v1.2 and then open all users' all mailboxes (write a script or something). It should convert all maildir files to dbox files. Try with one account first to make sure it works. :)
Thanks Timo, I will try the patch and report back.
-- Luben Karavelov Research and development Spectrum Net JSC
36, D-r G. M. Dimitrov Blvd. 1797 Sofia Mobile: +359 884332840 url: www.spnet.net
I have tried the patch with a single dbox and it works. But when I try to access it via POP3 I get this error:
MAIL=dbox:/var/www/210870/mail/156897-dbox /usr/lib/dovecot/pop3 pop3(areonfresh): Error: Getting size of message UID=2428 failed -ERR [IN-USE] Couldn't sync mailbox. pop3(areonfresh): Error: Couldn't init INBOX: Can't sync mailbox: Messages keep getting expunged pop3(areonfresh): Info: Mailbox init failed top=0/0, retr=0/0, del=0/2055, size=286255398
With IMAP there is no such a problem. This is offline machine, just for test, so nobody is accessing the mail. I have seen such a messages on other boxes also. How could I clean this type of errors?
Best regards and thanks for the great work.
-- Luben Karavelov Research and development Spectrum Net JSC
36, D-r G. M. Dimitrov Blvd. 1797 Sofia Mobile: +359 884332840 url: www.spnet.net
On Wed, 2010-10-20 at 16:34 +0300, Luben Karavelov wrote:
I have tried the patch with a single dbox and it works. But when I try to access it via POP3 I get this error:
MAIL=dbox:/var/www/210870/mail/156897-dbox /usr/lib/dovecot/pop3 pop3(areonfresh): Error: Getting size of message UID=2428 failed -ERR [IN-USE] Couldn't sync mailbox.
With IMAP there is no such a problem. This is offline machine, just for test, so nobody is accessing the mail. I have seen such a messages on other boxes also. How could I clean this type of errors?
Does it matter since it works with imap? The idea was to just select all mailboxes once to get the mails converted, doesn't matter if it's done with imap or pop3. You shouldn't use that binary for real clients, because it keeps writing to each dbox file all the time, making the performance just horrible.
Timo Sirainen wrote:
Yeah. Not very nice either. Here's a 3rd option: Apply the attached patch to v1.2 and then open all users' all mailboxes (write a script or something). It should convert all maildir files to dbox files. Try with one account first to make sure it works. :)
Hi Timo, list,
your 3rd option sounded best, but I got some problems and had to revert to my backup:
I patched Dovecot 1.2.15 with your diff (and with debian patches). I then accessed all mailboxes. For some it went OK, but for some I now only see old mail and am missing the newest ones starting from some time in February (I probably did a dovecot upgrade then).
What I got is thousands of these:
Oct 21 21:29:30 atrus dovecot: IMAP(roland): /var/spool/dovecot/mail/roland/dbox/mailboxes/INBOX/dbox-Mails/u.8922 corrupted: Invalid dbox version
...and some occasional these:
Oct 21 21:32:59 atrus dovecot: IMAP(roland): Panic: file mail-index-transaction.c: line 1273 (mail_index_update_ext): assertion failed: (seq > 0 && (seq <= mail_index_view_get_messages_count(t->view) || seq <= t->last_new_seq))
Oct 21 21:32:59 atrus dovecot: IMAP(roland): Raw backtrace: imap [0x4a6e32] -> imap [0x4a6eb3] -> imap [0x4a6596] -> imap(mail_index_update_ext+0x15d) [0x47d7bd] -> imap(dbox_sync_index_rebuild+0x1da) [0x43c87a] -> imap(dbox_sync_begin+0x481) [0x43ae01] -> imap(dbox_sync+0x18) [0x43b198] -> imap(dbox_storage_sync_init+0x36) [0x43b206] -> imap(mailbox_sync+0x30) [0x46d380] -> imap(cmd_select_full+0x37c) [0x42129c] -> imap [0x423adc] -> imap [0x423b8d] -> imap(client_handle_input+0x45) [0x423d05] -> imap(client_input+0x5f) [0x42469f] -> imap(io_loop_handler_run+0xcb) [0x4aefeb] -> imap(io_loop_run+0x18) [0x4ae468] -> imap(main+0x537) [0x42c727] -> /lib/libc.so.6(__libc_start_main+0xe6) [0x7fe1edc081a6] -> imap [0x41c689]
Oct 21 21:32:59 atrus dovecot: dovecot: child 2920 (imap) killed with signal 6 (core dumps disabled)
...where the last one is not always there.
Was there a dbox format change?
Best regards,
Roland.
Roland Stuehmer wrote:
I patched Dovecot 1.2.15 with your diff (and with debian patches). I then accessed all mailboxes. For some it went OK, but for some I now only see old mail and am missing the newest ones starting from some time in February (I probably did a dovecot upgrade then).
What I got is thousands of these:
Oct 21 21:29:30 atrus dovecot: IMAP(roland): /var/spool/dovecot/mail/roland/dbox/mailboxes/INBOX/dbox-Mails/u.8922 corrupted: Invalid dbox version
[...]
Was there a dbox format change?
Hi,
are there any opinions on this? :)
Cheers,
Roland.
On Tue, 2010-11-09 at 23:57 +0100, Roland Stuehmer wrote:
are there any opinions on this? :)
Attached a program to de-maildirify a dbox. You have to run it separately for each directory containing dbox-Mails:
find ~/dbox/ -name dbox-Mails -type d -exec ./dbox-dehybrid {} \;
I tested that it seemed to convert a test maildir ok, but backups would be a good idea in any case :) The converter deletes all of the unnecessary files while doing the conversion.
Am 25.11.2010 17:45, schrieb Timo Sirainen:
On Thu, 2010-11-25 at 15:33 +0000, Timo Sirainen wrote:
Attached a program to de-maildirify a dbox. You have to run it separately for each directory containing dbox-Mails:
Minor update: it shouldn't warn about non-existing maildir files.
Hi Timo, list,
I tried the updated version of the program, but I got a number of "Fatal" errors, see below.
After that I retuned to the previos method of using the patched dovecot 1.2.16, which (after some error I reported here) left me with a running instance with only the recovered maildir messages. Together with an instance with just the new dbox mails I used imapsync to merge them and have all my mail in one place now. :)
$ gcc -g -Wall -I$DOVECOT -I$DOVECOT/src/lib -I$DOVECOT/src/lib-storage/index/dbox-common -DHAVE_CONFIG_H dbox-dehybrid.c -o dbox-dehybrid $DOVECOT/src/lib/.libs/liblib.a
$ find /var/spool/dovecot -name dbox-Mails -type d -exec ./dbox-dehybrid {} \; Fatal: stat(/var/spool/dovecot/mail/roland/dbox/mailboxes/INBOX/dbox-Mails/1114431075.R4ed46e60Q8777.atrus,S=4331,W=4418:2,So) failed: No such file or directory Fatal: stat(/var/spool/dovecot/mail/roland/dbox/mailboxes/SPAM/dbox-Mails/1249480272.R72231c08Q2552.atrus,S=3328,W=3394:2,Sc) failed: No such file or directory
Best,
Roland.
participants (3)
-
Luben Karavelov
-
Roland Stuehmer
-
Timo Sirainen