[Dovecot] Performance-Tuning

[Stan Hoeppner]

On 11/8/2011 7:50 AM, Peer Heinlein wrote:

> I have > 11 TB hard used Mailstorage, saved als maildir in ext3 on HP EVA.

That's a lot of mail (likely a large user base--not given), on a
filesystem not designed for such, on a decent SAN controller--LUN RAID
configuration not given.

> I always wanted to make some mesurements about several influences to the 
> performance (switch to ext4, switch to mdbox), but I never had enough time 
> to do that.

If you're going to switch filesystems, for this size dataset and
concurrent workload, you're moving in the wrong direction.

> At the moment I *need* more speed, we have too much waitI/O on the system 
> and I already used all other performance and tuning-tricks (separated cache, 
> noatime, fsync and all that stuff).

EXT3/4 are not designed, nor optimized, for high concurrency workloads.

> I have to change my setup, maybe somebody else here have hard facts:
> 
> *) Is ext4 faster? How much faster?

Simulated maildir workload test on 2.6.35-rc5, 128 threads
(No data published for newer kernels):

http://btrfs.boxacle.net/repository/raid/2.6.35-rc5/2.6.35-rc5/2.6.35-rc5_Mail_server_simulation._num_threads=128.html

As you can see EXT4 shows a small gain over EXT3, ~20%.  If you really
want high performance it's time to move to XFS, properly configured to
match the underlying RAID characteristics of the LUN(s) you're mounting.
 You'll prefer kernel 2.6.39+, 2.6.36 at minimum, so you get the delayed
logging feature (2.6.35 had delayed logging but had problems in other
areas).

I'll assume with a >10TB mail store that you're seeing greater than 128
concurrent user operations regularly.  As you can see from the graph,
XFS will give you ~50% greater ops/s than EXT4 and ~90% greater than
EXT3--yes, almost double that of EXT3.  As the concurrency increases, so
will this performance gap, as XFS was designed from day 1 for high
concurrency workloads.

This is a simulated mail server benchmark.  However you should see
similar gains with Dovecot.  The XFS delayed logging feature will
dramatically reduce the number of physical IOs required for journal
writes (i.e. metadata IO), as will delayed allocation, a feature of XFS
since its inception in 1994.  EXT4 was the first of its lineage to gain
delayed allocation, some 10+ years later, after Ted T'so studied the XFS
code.

In short, if you want an 'enterprise caliber' production Linux
filesystem tailor made for high IO concurrency, XFS is it.  JFS yields
similar performance, but hasn't been actively developed for 8 years or
so.  XFS has substantial ongoing feature and fix development.

> *) Is it faster because of the ext4 kernel-module (which can be used on ext3 
> to) or because of the ext4 filesystem layout?

AIUI, the bulk of the EXT4 performance advantage over EXT3 is the
delayed allocation logic.  The new EXT4 extent based on disk layout
yields little in the way of additional performance, but much in free
space management, fragmentation mitigation, etc.

> *) Is mdbox really faster? I'd like to have mdbox to have better performance 
> in running my backup-processes. But does it bring some performance boosts 
> to? 

mdbox will substantially decrease physical IOs to your storage back end
due to dramatically less metadata operations compared to maildir.
You've stated you currently have a storage IOPS bottleneck, so I'd have
to assume mdbox will seriously increase your overall performance.  Good
old mbox will do so as well, but everyone shuns it for various reasons,
some valid, some not so valid.

If you have an appropriate LUN available (sufficient size and spindle
speed/count of member disks), properly create an XFS filesystem on it
(read much before creating it), and moved to mdbox atop that, I think
you'll be really surprised by how much you gain from simply changing
filesystems and mailbox storage formats.  If you double the size of the
LUN you could potentially carry twice as many users with, fewer IOPS
than you're seeing now, on essentially the same hardware platform.

-- 
Stan