[Dovecot] Migration questions...

[Richard Hobbs]

Hi All,

Replies to everyone below (to keep the number of emails down)...

Ed W wrote:
> Richard Hobbs wrote:
>> Hi All,
>>
>> We are soon to migrate our mail server from one piece of hardware
>> to another and we would like to take this opportunity to optimize
>> things.
>>
>
> Can I recommend you add virtualisation to your todo list.  I use
> linux-vserver, but there are plenty other ideas out there.
>
> It's just superb that later on you can migrate services between
> physical hardware with MUCH less hassle.  You can easily test
> upgrades in a sandbox first. etc
>
> I personally split all the tasks into different virtual servers.
> Right now I actually still have quite a few mail related services
> in a single vserver, but ideally you would split everything up and
> then later if you needed to upgrade a single service or move it to
> a new machine it would have minimal effect.
>
> Your general process sounds about right though - I think it may be
> possible for you to preserve pop uidls, but see the wiki for more
> notes on that

I appreciate that the world is going crazy for virtualization at the
moment and I agree that it is a truly great concept. However, in this
instance, where we want absolute ultimate performance and we know for a
fact that this machine will never be used for anything else, we are
shying away from virtualization. In some way, it's another layer to
potentially go wrong, and another layer to potentially slow things down.

I appreciate the advice though, it's just that i'm not sure it's for us
in this particular instance.

Regarding keeping msg UIDs, i know that can happen now, which is great :-)

Lou Duchez wrote:
>> I don't know whether this would help with the migration, but I
>> routinely solve a similar problem.  I have implemented mail failover
>> between two servers -- which are configured with identical sets of
>> mailboxes -- and every 10 minutes or so, a script grabs any E-Mails
>> from the other server and stores them locally in the proper
>> mailboxes.  This script relies on IMAP connectivity and passwords in
>> plaintext in a passwd-file.  Here it is:
>> <snip>
>
> Blast, forgot something: the "Simple.pm" referenced in the script is
> this thing:
>
> http://search.cpan.org/~jpaf/Net-IMAP-Simple-0.93/Simple.pm
>
> Download it, compile it, put it somewhere that the script can find it.

Thank you very much for that script - i will certainly take a good look
through that when the time comes. Does it rely on IMAP connectivity at
both ends? i.e. does it read from old IMAP and write to new IMAP, or
does it deal with the filesystem at one end or the other?

Timo Sirainen wrote:
> On May 12, 2009, at 6:41 AM, Richard Hobbs wrote:
> 
>>>> Single-dbox is the highest performing, but note that it's not as much
>>>> tested as mbox and Maildir code. I think it should work ok, but I'm not
>>>> aware of any larger installations using dbox currently. So in case you
>>>> find a problem, you might have to upgrade/patch Dovecot to get it fixed
>>>> and that would require compiling from sources.
>>>
>>> In that case (and with a little further investigation which i've just
>>> done) we've decided to go with maildir! That is still going to be
>>> significantly better performing than mbox, right?
> 
> Depends on the usage, but it's significantly better performing than
> UW-IMAP. Dovecot+mbox is also significantly faster than UW-IMAP+mbox.

OK... so Dovecot is certainly significantly faster that uw-imapd in both
cases, but is dovecot fastest with mbox or maildir? I would assume
maildir, but you never know...

>>> Also... do you know how uw-imapd & maildir compares to dovecot & maildir
>>> in terms of performance?
> 
> Maildir is a patch on top of the official UW-IMAP distribution. I don't
> know how well it performs, but it doesn't use any indexes and indexes
> are what makes Dovecot fast.
> 
>>> Does dovecot still use indices with maildir?
> 
> Yes.

Excellent :-)

>>>> mbox -> Maildir conversion can preserve both IMAP and POP3 UIDLs using
>>>> an external script. Maildir -> dbox conversion can also preserve both,
>>>> but that causes Dovecot to use this "hybrid Maildir-dbox format", which
>>>> is slower than the full native dbox.
>>>
>>> That'd good to know. Do you happen to know where I can get a copy of
>>> this "external script" you speak of? Will it simply be included in the
>>> debian package (probably)?
> 
> http://wiki.dovecot.org/Migration/MailFormat -> mb2md.py

how does this differ from the "convert-tool" script packaged with
dovecot itself?

>>> Also, given that i'm going to have to test this, i will obviously be
>>> running the conversion on a copy of the live data, and then i'll have to
>>> run the conversion again during the migration outage - will i need to
>>> delete all the data and basically start again, or is it incremental?
> 
> I don't think it can do incremental, but I've never looked at the script
> myself.
> 
>>>> My guess is that two RAID-1s would be faster, but I haven't really done
>>>> any benchmarking. Anyway index files are 10-30% of the mailbox size, so
>>>> the index-disks would be using a lot less disk space.
>>>
>>> I assume you are talking about dovecot with maildir here, right?
> 
> The same applies to all mailbox formats Dovecot supports.
> 
>>> Also, what would we put on each array? Are the inboxes still stored
>>> separately to the IMAP folders when using dovecot and maildir?
> 
> Inboxes are stored inside Maildir like all other mailboxes.
> 
>>> Would it be best to put all data on one array, and the indices on the
>>> other? We're basically after the fastest way to distribute the data! :-)
> 
> Last I heard it was faster to keep index files in a separate disk than
> mailbox data. I've never verified this myself, but it sounds reasonable.

Same here - i assume dovecot gives you a choice as to where it puts its
index files, right?

Also... where are the maildir data files kept? homedirs or elsewhere?

>> My colleague has mentioned something of interest... can dovecot keep the
>> index files in RAM? If so, the performance will obviously be *so* much
>> better than running them off the hard disks.
> 
> Last I heard it didn't really help much. Assuming your OS works properly
> it already keeps the necessary indexes in memory anyway. Also I wrote a
> patch that tries to tell OS to do that by dropping message files' data
> from cache after reading the messages:
> 
> http://dovecot.org/patches/1.1/fadvise.diff
> 
> But no one has told me if that helps or makes things worse..

Fair enough... i think we'll let the OS do it's thing, from what i've
heard in general.

>> This also raises questions about what happens if the machine is powered
>> off etc... but it's UPSd etc... so if it were to rebuild it's indexes
>> every time it was booted up, that wouldn't be the end of the world.
> 
> Well, there are two parts of index rebuilding: dovecot.index files which
> are quick to rebuild and dovecot.index.cache files that contain the
> useful fields that clients want. The cache file is especially useful
> with webmails and if it's gone it could mean opening user's all messages
> and reading their headers and perhaps even bodies.
> 
> So depends on what clients your users use, but in some cases it could be
> 10-100x slower to open the mailbox if the cache file is gone.

OK, this isn't too much of an issue now that we know we're just going to
let the OS cache whatever it wants to in RAM.

However, does dovecot allow configuration of where to store both the
index files *and* cache files?

Given that we have 2 RAID arrays, should we put index files *and* cache
files on one array, and the data on another, or should put either the
index files *or* cache files on the same array as the data?

Thanks again, people! :-)

Richard.

-- 
Richard Hobbs (IT Specialist)
Toshiba Research Europe Ltd. - Cambridge Research Laboratory
Email: richard.hobbs at crl.toshiba.co.uk
Web: http://www.toshiba-europe.com/research/
Tel: +44 1223 436999        Mobile: +44 7811 803377
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3306 bytes
Desc: S/MIME Cryptographic Signature
Url : http://dovecot.org/pipermail/dovecot/attachments/20090513/c28eb509/attachment.bin