On 3/26/2012 2:34 PM, Jeff Gustafson wrote:
Do you have any suggestions for a distributed replicated filesystem that works well with dovecot? I've looked into glusterfs, but the latency is way too high for lots of small files. They claim this problem is fixed in glusterfs 3.3. NFS too slow for my installation so I don't see how any of the distributed filesystems would help me. I've also tried out ZFS, but it appears to have issues with metadata look ups with directories that have tens or hundreds of thousands of files in them. For me, the best filesystem is straight up ext4 running on locally attached storage. I think a solid, fast dsync implementation would be very useful for a large installation.
It sounds like you're in need of a more robust and capable storage/backup solution, such as an FC/iSCSI SAN array with PIT and/or incremental snapshot capability.
Also, you speak of a very large maildir store, with hundreds of thousands of directories, obviously many millions of files, of 1TB total size. Thus I would assume you have many thousands of users, if not 10s of thousands.
It's a bit hard to believe you're not running XFS on your storage, given your level of parallelism. You'd get much better performance using XFS vs EXT4. Especially with kernel 2.6.39 or later which includes the delayed logging patch. This patch increases metadata write throughput by a factor of 2-50+ depending on thread count, and decreases IOPS and MB/s hitting the storage by about the same factor, depending on thread count.
Before this patch XFS sucked at the write portion of the maildir workload due to the extremely high IOPS and MB/s hitting just the log journal, not including the actual file writes. It's parallel maildir read performance was better than any other, but the write was so bad it bogged down the storage producing high latency for everything. With the delaylog patch, XFS now trounces every filesystem at medium to high parallelism levels. Delaylog was introduced in mid 2009, included in 2.6.35 as experimental, and is the default in 2.6.39 and later. If you're a Red Hat or CentOS user it's included in 6.2.
This one patch, which was 5+ years in development, dramatically changed the character of XFS with this class of metadata intensive parallel workloads. Many people with such a workload who ran from XFS in the past, as if it were the Fukushima reactor, are now adopting it in droves.
What a difference a few hundred lines of very creative code can make...
-- Stan