On 11/8/2011 7:50 AM, Peer Heinlein wrote:
I have > 11 TB hard used Mailstorage, saved als maildir in ext3 on HP EVA.
That's a lot of mail (likely a large user base--not given), on a filesystem not designed for such, on a decent SAN controller--LUN RAID configuration not given.
I always wanted to make some mesurements about several influences to the performance (switch to ext4, switch to mdbox), but I never had enough time to do that.
If you're going to switch filesystems, for this size dataset and concurrent workload, you're moving in the wrong direction.
At the moment I *need* more speed, we have too much waitI/O on the system and I already used all other performance and tuning-tricks (separated cache, noatime, fsync and all that stuff).
EXT3/4 are not designed, nor optimized, for high concurrency workloads.
I have to change my setup, maybe somebody else here have hard facts:
*) Is ext4 faster? How much faster?
Simulated maildir workload test on 2.6.35-rc5, 128 threads (No data published for newer kernels):
http://btrfs.boxacle.net/repository/raid/2.6.35-rc5/2.6.35-rc5/2.6.35-rc5_Ma...
As you can see EXT4 shows a small gain over EXT3, ~20%. If you really want high performance it's time to move to XFS, properly configured to match the underlying RAID characteristics of the LUN(s) you're mounting. You'll prefer kernel 2.6.39+, 2.6.36 at minimum, so you get the delayed logging feature (2.6.35 had delayed logging but had problems in other areas).
I'll assume with a >10TB mail store that you're seeing greater than 128 concurrent user operations regularly. As you can see from the graph, XFS will give you ~50% greater ops/s than EXT4 and ~90% greater than EXT3--yes, almost double that of EXT3. As the concurrency increases, so will this performance gap, as XFS was designed from day 1 for high concurrency workloads.
This is a simulated mail server benchmark. However you should see similar gains with Dovecot. The XFS delayed logging feature will dramatically reduce the number of physical IOs required for journal writes (i.e. metadata IO), as will delayed allocation, a feature of XFS since its inception in 1994. EXT4 was the first of its lineage to gain delayed allocation, some 10+ years later, after Ted T'so studied the XFS code.
In short, if you want an 'enterprise caliber' production Linux filesystem tailor made for high IO concurrency, XFS is it. JFS yields similar performance, but hasn't been actively developed for 8 years or so. XFS has substantial ongoing feature and fix development.
*) Is it faster because of the ext4 kernel-module (which can be used on ext3 to) or because of the ext4 filesystem layout?
AIUI, the bulk of the EXT4 performance advantage over EXT3 is the delayed allocation logic. The new EXT4 extent based on disk layout yields little in the way of additional performance, but much in free space management, fragmentation mitigation, etc.
*) Is mdbox really faster? I'd like to have mdbox to have better performance in running my backup-processes. But does it bring some performance boosts to?
mdbox will substantially decrease physical IOs to your storage back end due to dramatically less metadata operations compared to maildir. You've stated you currently have a storage IOPS bottleneck, so I'd have to assume mdbox will seriously increase your overall performance. Good old mbox will do so as well, but everyone shuns it for various reasons, some valid, some not so valid.
If you have an appropriate LUN available (sufficient size and spindle speed/count of member disks), properly create an XFS filesystem on it (read much before creating it), and moved to mdbox atop that, I think you'll be really surprised by how much you gain from simply changing filesystems and mailbox storage formats. If you double the size of the LUN you could potentially carry twice as many users with, fewer IOPS than you're seeing now, on essentially the same hardware platform.
-- Stan