[Dovecot] Need fast Maildir to mdbox conversion

[Robin]

On 3/27/2012 3:40 PM, Jeff Gustafson wrote:
> 	I looked around the 'Net to see if there might be a custom program for
> offline Maildir to mdbox conversion. So far I haven't turned up
> anything. The problem for us is that the dsync program simply takes a
> lot of time to convert mailboxes.

Is it slower than doing an IMAP APPEND over an authenticated dovecot 
connection?

I've used a simple PERL script based on Mail::IMAPClient and Mail::Box 
to import 180,000+ mailboxes into dovecot's mdbox at fairly high speed, 
and all it does is IMAP APPENDs.  (I had to shard the mailboxes because 
these PERL based tools exhaust RAM when run with mailboxes larger than 
about 600MB).

On my development VM test box (32 bit Slack 13.37, 2G/2G split kernel, 
no RAID, Q6600 with only two cores allocated to the VM) and 8GB of DDR2 
RAM does

Emails=180,044
real    237m28.485s  (12.5 emails/second)
user    94m50.425s
sys     10m09.389s
21,984,824  /mail/home

I'm writing a swiss-army (C-based, no bytecode crap languages) mailbox 
"transcoding" tool, since none appear to exist.  To keep it simple, I/O 
to/from "remote" mailbox (connections) are not pipelined.  It won't 
require more than MAXEMAILSIZE's worth of RAM (if one of the directions 
involves a remote connection), and so far when processing MIX, Maildir, 
and Mbox files, it's extremely fast.

Adding support for [sm]dbox wouldn't appear to be problematic.  At the 
moment, it supports everything Panda's c-client supports plus 
Maildir/Maildir++ (including Panda's "MIX").

Write support for Maildir's extremely UNDER-tested so far, as I've 
mainly used it to import Maildir hives.

I've experimented with Maildir as a format, and while the one email to a 
file model seems like a sensible idea, it seems to simply transfer 
stress from one part of the system to another, mainly filesystems, and 
not many of those are really up for handling that many files in one 
directory very efficiently.

None of my users have mailboxes with fewer than 100K emails in them, 
some have more than a million.

=R=