[Dovecot] dsync redesign

[Stan Hoeppner]

On 3/26/2012 2:34 PM, Jeff Gustafson wrote:

> 	Do you have any suggestions for a distributed replicated filesystem
> that works well with dovecot? I've looked into glusterfs, but the
> latency is way too high for lots of small files. They claim this problem
> is fixed in glusterfs 3.3. NFS too slow for my installation so I don't
> see how any of the distributed filesystems would help me. I've also
> tried out ZFS, but it appears to have issues with metadata look ups with
> directories that have tens or hundreds of thousands of files in them.
> For me, the best filesystem is straight up ext4 running on locally
> attached storage. 
> 	I think a solid, fast dsync implementation would be very useful for a
> large installation.

It sounds like you're in need of a more robust and capable
storage/backup solution, such as an FC/iSCSI SAN array with PIT and/or
incremental snapshot capability.

Also, you speak of a very large maildir store, with hundreds of
thousands of directories, obviously many millions of files, of 1TB total
size.  Thus I would assume you have many thousands of users, if not 10s
of thousands.

It's a bit hard to believe you're not running XFS on your storage, given
your level of parallelism.  You'd get much better performance using XFS
vs EXT4.  Especially with kernel 2.6.39 or later which includes the
delayed logging patch.  This patch increases metadata write throughput
by a factor of 2-50+ depending on thread count, and decreases IOPS and
MB/s hitting the storage by about the same factor, depending on thread
count.

Before this patch XFS sucked at the write portion of the maildir
workload due to the extremely high IOPS and MB/s hitting just the log
journal, not including the actual file writes.  It's parallel maildir
read performance was better than any other, but the write was so bad it
bogged down the storage producing high latency for everything.  With the
delaylog patch, XFS now trounces every filesystem at medium to high
parallelism levels.  Delaylog was introduced in mid 2009, included in
2.6.35 as experimental, and is the default in 2.6.39 and later.  If
you're a Red Hat or CentOS user it's included in 6.2.

This one patch, which was 5+ years in development, dramatically changed
the character of XFS with this class of metadata intensive parallel
workloads.  Many people with such a workload who ran from XFS in the
past, as if it were the Fukushima reactor, are now adopting it in droves.

What a difference a few hundred lines of very creative code can make...

-- 
Stan