[Dovecot] Performance of Maildir vs sdbox/mdbox

[Timo Sirainen]

On Wed, 2012-01-18 at 22:36 +0800, Lee Standen wrote:

> How about this... are there any tools available (that you know of) to 
> capture real live customer POP3/IMAP traffic and replay it against a 
> separate system?  That might be a feasible option for doing a 
> like-for-like comparison in our environment?  We could probably get 
> something in place to simulate the load if we can do something like 
> that...

I've thought about that too before, but with IMAP traffic it doesn't
work very well. Even if the storages were 100% synchronized at startup,
the session states could easily become desynced. For example if client
does a NOOP at the same time when two mails are being delivered to the
mailbox, serverA might show only one of them while serverB would show
two of them because it was executed a tiny bit later. All of the
client's future commands could then be affected by this desync.

(OK, I wrote the above thinking about a real-time system where you could
redirect the client's traffic to two systems, but basically same
problems exist for offline replays too. Although it would be easier to
fix the replays to handle this.)

> > You're going to run into NFS caching troubles with the above split
> > setup. I don't recommend it. You will see error messages about index
> > corruption with it, and with dbox it can cause metadata loss.
> > http://wiki2.dovecot.org/NFS http://wiki2.dovecot.org/Director
> 
> That might be the one thing (unfortunately) which prevents us from 
> going with the dbox format.  I understand the same issue can actually 
> occur on Dovecot Maildir as well, but because Maildir works without 
> these index files, we were willing to just go with it.  

Are you planning on also redirecting POP3/IMAP connections to somewhat
randomly to the different servers? I really don't recommend that, even
with Maildir.. Some of the errors will be user visible, even if no
actual data loss happens. Users may get disconnected, and sometimes
might have to clean their client's cache.

> I will raise it 
> again, but there has been a lot of push back about introducing a single 
> point of failure, even though this is a perceived one.

What is a single point of failure there?

> > It's at least safer to first switch to Dovecot+Maildir to make sure 
> > that
> > any problems you might find aren't related to the mailbox format..
> 
> Yep, I'm considering that.  The flip side is that it's actually going 
> to be difficult for us to change mail format once we've migrated into 
> this system, but we have an opportunity for (literally) a month long 
> testing phase beginning in Feb/March which will let us test as many 
> possibilities as we can.

The mailbox format switching can be done one user at a time with zero
downtime with dsync.