[Dovecot] Please advise on very fast search

[Stan Hoeppner]

On 11/14/2011 3:16 PM, Alexander Chekalin wrote:
> Locking issues on mbox is the reason for my long-lasting love affair with maildir, 

Same reason most others fell in love with it.  Many now want to divorce
maildir, as the cost of the storage to maintain acceptable performance
is now too high.

> and it's lasts long years. Ok, the life's lessons are like this, learn something and move on with it ;) even if it's "new old thing". Thank you for pointing that!

Many old UNIX gurus still use mbox, not maildir, and never will.  If you
ask them why they'll likely say "you don't use a screwdriver to drive a
nail do you?"

> What I was doubt about is default rotate size of 2M, since I used to see pretty reasonable default settings in all Dovecot config. 32 or 64 are much close to the ones I'd personally prefer.

Given the fact that we're talking about an archive server, you'd be
better off using a very large mdbox file size, say 1GB.  You're never
deleting individual messages from this archive correct?  No expunges?

This is why I recommended mbox in the first place.  If your only writes
to these mailbox files are appends of new messages, mbox is the best
format by far.  It's faster at appending than any other format, and it's
faster for searching than any other.

> I also about to choose now is the OS and FS for the archive. I seriously think about ZFS with compression (in fact it will be stripes over couple of mirrors = software equivalent of RAID 10 on SATA drives, with compression on FS level) on FreeBSD, or XFS over LVM on Debian with compression in mdbox itself. I see pros and contras for both, so that's the question to answer!

It's an archive.  You're not going to use maildir so you don't need
random IOPS performance.  Thus RAID5/6 are a much better fit for an
archive as you get better read performance, with more than adequate
write performance, and you use less disks.  And as this is an archive,
you don't need real time automatic/transparent compression.  Thus I
recommend something like:

1.  Debian 6 w/linux-image-2.6.39-bpo.2-amd64 or a custom rolled
    2.6.39 or later kernel
2.  hardware RAID5 w/large (2TB) SATA disks, 512B native sectors
    e.g. MegaRAID SAS 9261-8i, 4 Seagate Constellation ES ST2000NM0011
    Specify a strip size of 256KB for the array
    Perma set /sys/block/sdX/read_ahead_kb to 512 so you're reading
    ahead 1024 sectors at a time instead of the default of 256.  This
    will speed up your searches quite a bit.
3.  XFS filesystem on the RAID device, created with mkfs.xfs defaults
4.  mbox w/zlib plugin.  Compress daily files each night with a script
5.  You don't need LVM with a good RAID card (or with mdraid).  This
    controller can expand the RAID5 up to 8 drives (up to 32 drives max
    using SAS expanders)

-- 
Stan