[Dovecot] dsync redesign

[Jeff Gustafson]

On Tue, 2012-03-27 at 15:09 -0500, Stan Hoeppner wrote:
> On 3/26/2012 2:34 PM, Jeff Gustafson wrote:
> 
> > 	Do you have any suggestions for a distributed replicated filesystem
> > that works well with dovecot? I've looked into glusterfs, but the
> > latency is way too high for lots of small files. They claim this problem
> > is fixed in glusterfs 3.3. NFS too slow for my installation so I don't
> > see how any of the distributed filesystems would help me. I've also
> > tried out ZFS, but it appears to have issues with metadata look ups with
> > directories that have tens or hundreds of thousands of files in them.
> > For me, the best filesystem is straight up ext4 running on locally
> > attached storage. 
> 
> It sounds like you're in need of a more robust and capable
> storage/backup solution, such as an FC/iSCSI SAN array with PIT and/or
> incremental snapshot capability.

	We do have a FC system that another department is using. The company
dropped quite a bit of cash on it for a specific purpose. Our department
does not have access it to. People are somewhat afraid of iSCSI around
here because they believe it will add too much latency to the overall IO
performance. They're a big believer in locally attached disks. Less
features, but very good performance. 
	We thought ZFS would provide us with a nice snapshot and backup system
(with zfs send). We never got that far once we discovered that ZFS
doesn't work very well in this context. Running rsync on it gave us
terrible performance.

> Also, you speak of a very large maildir store, with hundreds of
> thousands of directories, obviously many millions of files, of 1TB total
> size.  Thus I would assume you have many thousands of users, if not 10s
> of thousands.
> 
> It's a bit hard to believe you're not running XFS on your storage, given
> your level of parallelism.  You'd get much better performance using XFS
> vs EXT4.  Especially with kernel 2.6.39 or later which includes the
> delayed logging patch.  This patch increases metadata write throughput
> by a factor of 2-50+ depending on thread count, and decreases IOPS and
> MB/s hitting the storage by about the same factor, depending on thread
> count.

	I've relatively new here, but I'll ask around about XFS and see if
anyone had tested it in the development environment. 

				...Jeff