[Dovecot] Migration questions...

[Timo Sirainen]

On May 12, 2009, at 6:41 AM, Richard Hobbs wrote:

>>> Single-dbox is the highest performing, but note that it's not as  
>>> much
>>> tested as mbox and Maildir code. I think it should work ok, but  
>>> I'm not
>>> aware of any larger installations using dbox currently. So in case  
>>> you
>>> find a problem, you might have to upgrade/patch Dovecot to get it  
>>> fixed
>>> and that would require compiling from sources.
>>
>> In that case (and with a little further investigation which i've just
>> done) we've decided to go with maildir! That is still going to be
>> significantly better performing than mbox, right?

Depends on the usage, but it's significantly better performing than UW- 
IMAP. Dovecot+mbox is also significantly faster than UW-IMAP+mbox.

>> Also... do you know how uw-imapd & maildir compares to dovecot &  
>> maildir
>> in terms of performance?

Maildir is a patch on top of the official UW-IMAP distribution. I  
don't know how well it performs, but it doesn't use any indexes and  
indexes are what makes Dovecot fast.

>> Does dovecot still use indices with maildir?

Yes.

>>> mbox -> Maildir conversion can preserve both IMAP and POP3 UIDLs  
>>> using
>>> an external script. Maildir -> dbox conversion can also preserve  
>>> both,
>>> but that causes Dovecot to use this "hybrid Maildir-dbox format",  
>>> which
>>> is slower than the full native dbox.
>>
>> That'd good to know. Do you happen to know where I can get a copy of
>> this "external script" you speak of? Will it simply be included in  
>> the
>> debian package (probably)?

http://wiki.dovecot.org/Migration/MailFormat -> mb2md.py

>> Also, given that i'm going to have to test this, i will obviously be
>> running the conversion on a copy of the live data, and then i'll  
>> have to
>> run the conversion again during the migration outage - will i need to
>> delete all the data and basically start again, or is it incremental?

I don't think it can do incremental, but I've never looked at the  
script myself.

>>> My guess is that two RAID-1s would be faster, but I haven't really  
>>> done
>>> any benchmarking. Anyway index files are 10-30% of the mailbox  
>>> size, so
>>> the index-disks would be using a lot less disk space.
>>
>> I assume you are talking about dovecot with maildir here, right?

The same applies to all mailbox formats Dovecot supports.

>> Also, what would we put on each array? Are the inboxes still stored
>> separately to the IMAP folders when using dovecot and maildir?

Inboxes are stored inside Maildir like all other mailboxes.

>> Would it be best to put all data on one array, and the indices on the
>> other? We're basically after the fastest way to distribute the  
>> data! :-)

Last I heard it was faster to keep index files in a separate disk than  
mailbox data. I've never verified this myself, but it sounds reasonable.

> My colleague has mentioned something of interest... can dovecot keep  
> the
> index files in RAM? If so, the performance will obviously be *so* much
> better than running them off the hard disks.

Last I heard it didn't really help much. Assuming your OS works  
properly it already keeps the necessary indexes in memory anyway. Also  
I wrote a patch that tries to tell OS to do that by dropping message  
files' data from cache after reading the messages:

http://dovecot.org/patches/1.1/fadvise.diff

But no one has told me if that helps or makes things worse..

> This also raises questions about what happens if the machine is  
> powered
> off etc... but it's UPSd etc... so if it were to rebuild it's indexes
> every time it was booted up, that wouldn't be the end of the world.

Well, there are two parts of index rebuilding: dovecot.index files  
which are quick to rebuild and dovecot.index.cache files that contain  
the useful fields that clients want. The cache file is especially  
useful with webmails and if it's gone it could mean opening user's all  
messages and reading their headers and perhaps even bodies.

So depends on what clients your users use, but in some cases it could  
be 10-100x slower to open the mailbox if the cache file is gone.