[Dovecot] Migrating to mdbox

[Chris Wilson]

Hi Timo,

On Thu, 10 Mar 2011, Timo Sirainen wrote:
> On 9.3.2011, at 18.02, Chris Wilson wrote:
> 
> > I think it would be a good idea for me to move from mbox to mdbox, as I 
> > have 57,000 messages in my inbox, and clients hang for a minute whenever 
> > they request a CHECK, or Dovecot feels the need to run one itself.
> 
> mbox_very_dirty_syncs=yes should get rid of this problem even without switching to mdbox.

That option sounds scary :) I thought it would be safer to switch to a 
format where messages are read-only, especially as I currently have some 
strange symptoms (blank or nearly blank messages, messages cut in two, 
etc.) with mbox and I know it's not good in general, or good at handling 
large mailboxes in particular.

> > dsync(chris): Error: open() failed with mbox file /home/chris/mail/admin: 
> > Permission denied
> 
> So that is intentional?..

Not exactly, it's a symlink to /var/spool/mail/admin which for some reason 
I don't have permission to open at the moment.

> > chris at one-mail(~)$ sudo dsync -v -u chris backup mdbox:/home/chris/mdbox
> > dsync(chris): Error: Next message unexpectedly corrupted in mbox file 
> > /home/chris/mail/spam at 404513741
> 
> mbox bug. Same would happen if you tried to read that same mail via 
> imap. It fixed itself automatically though.

Okay good, that probably explains why when I tried to debug it (suspecting 
a bad Content-Length header in the previous messages) I couldn't see any 
problem.

> > dsync(chris): Info: INBOX.Sent: only in source
> > dsync(chris): Info: sent-mail: only in source
> > dsync(chris): Info: INBOX.Trash: only in source
> 
> Just saying that destination doesn't have these mailboxes yet.

OK, good.

> > Note: how is it possible for the same folder to be "only in source" 
> > and "only in dest"?
> 
> I think they have different GUIDs, or maybe Dovecot failed to get one of 
> their GUID. Dsync works with mailbox GUIDs, not names. Anyway, I don't 
> really know why it would suddenly lose one of them.

Perhaps it's worth printing the guids in that case? Is the guid stored in 
the first message in the mailbox?

Note that the mdbox for spam was created (maybe not completely) by a 
previous run of dsync, and I really can't see how the guid of the spam 
mbox could have changed (it definitely wasn't recreated yesterday). 

> > chris at one-mail(~)$ sudo dsync -v -u chris backup mdbox:/home/chris/mdbox
> > dsync(chris): Info: spam: highest_modseq changed: 2 != 3
> > 
> > Now the latter is all I get whenever I run dsync. Is it broken?
> 
> Nope. The modseq syncing isn't really perfect. So everything should be 
> synced now.

I'm a bit surprised by how much less disk space it's taking:

chris at one-mail(~)$ du -sk /var/spool/mail/chris mail mdbox
1010516	/var/spool/mail/chris
2083872	mail
1016664	mdbox

It doesn't seem to have backed up most of my mailboxes:

chris at one-mail(~)$ ls mdbox/mailboxes/

addressbook INBOX.Sent INBOX.Trash Sent sent-mail sent-mail-oct-2010 
sent-mail-sep-2008 spam temp

chris at one-mail(~)$ ls mail

addressbook archive INBOX.Drafts mail postponed-msgs sent-mail 
sent-mail-nov-2010 spam vcn admin badbox INBOX.Sent payslips 
saved-messages sent-mail-aug-2006 sent-mail-oct-2010 temp aidbot Drafts 
INBOX.Trash pinerc Sent sent-mail-jan-2009 sent-mail-sep-2008 Trash

Any ideas how I can get it to backup the rest of my mailboxes to mdbox?

Cheers, Chris.
-- 
Aptivate | http://www.aptivate.org | Phone: +44 1223 760887
The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES

Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.