On May 12, 2009, at 6:41 AM, Richard Hobbs wrote:
Single-dbox is the highest performing, but note that it's not as
much tested as mbox and Maildir code. I think it should work ok, but
I'm not aware of any larger installations using dbox currently. So in case
you find a problem, you might have to upgrade/patch Dovecot to get it
fixed and that would require compiling from sources.In that case (and with a little further investigation which i've just done) we've decided to go with maildir! That is still going to be significantly better performing than mbox, right?
Depends on the usage, but it's significantly better performing than UW- IMAP. Dovecot+mbox is also significantly faster than UW-IMAP+mbox.
Also... do you know how uw-imapd & maildir compares to dovecot &
maildir in terms of performance?
Maildir is a patch on top of the official UW-IMAP distribution. I
don't know how well it performs, but it doesn't use any indexes and
indexes are what makes Dovecot fast.
Does dovecot still use indices with maildir?
Yes.
mbox -> Maildir conversion can preserve both IMAP and POP3 UIDLs
using an external script. Maildir -> dbox conversion can also preserve
both, but that causes Dovecot to use this "hybrid Maildir-dbox format",
which is slower than the full native dbox.That'd good to know. Do you happen to know where I can get a copy of this "external script" you speak of? Will it simply be included in
the debian package (probably)?
http://wiki.dovecot.org/Migration/MailFormat -> mb2md.py
Also, given that i'm going to have to test this, i will obviously be running the conversion on a copy of the live data, and then i'll
have to run the conversion again during the migration outage - will i need to delete all the data and basically start again, or is it incremental?
I don't think it can do incremental, but I've never looked at the
script myself.
My guess is that two RAID-1s would be faster, but I haven't really
done any benchmarking. Anyway index files are 10-30% of the mailbox
size, so the index-disks would be using a lot less disk space.I assume you are talking about dovecot with maildir here, right?
The same applies to all mailbox formats Dovecot supports.
Also, what would we put on each array? Are the inboxes still stored separately to the IMAP folders when using dovecot and maildir?
Inboxes are stored inside Maildir like all other mailboxes.
Would it be best to put all data on one array, and the indices on the other? We're basically after the fastest way to distribute the
data! :-)
Last I heard it was faster to keep index files in a separate disk than
mailbox data. I've never verified this myself, but it sounds reasonable.
My colleague has mentioned something of interest... can dovecot keep
the index files in RAM? If so, the performance will obviously be *so* much better than running them off the hard disks.
Last I heard it didn't really help much. Assuming your OS works
properly it already keeps the necessary indexes in memory anyway. Also
I wrote a patch that tries to tell OS to do that by dropping message
files' data from cache after reading the messages:
http://dovecot.org/patches/1.1/fadvise.diff
But no one has told me if that helps or makes things worse..
This also raises questions about what happens if the machine is
powered off etc... but it's UPSd etc... so if it were to rebuild it's indexes every time it was booted up, that wouldn't be the end of the world.
Well, there are two parts of index rebuilding: dovecot.index files
which are quick to rebuild and dovecot.index.cache files that contain
the useful fields that clients want. The cache file is especially
useful with webmails and if it's gone it could mean opening user's all
messages and reading their headers and perhaps even bodies.
So depends on what clients your users use, but in some cases it could
be 10-100x slower to open the mailbox if the cache file is gone.