Kelsey Cummings wrote:

On 06/28/12 05:56, Ed W wrote:

...

So given the statistics show us that 2 disk failures are much more common than we expect, and that "silent corruption" is likely occurring within (larger) real world file stores, there really aren't many battle tested options that can protect against this - really only RAID6 right now and that has significant limitations...

Has anyone tried or benchmarked ZFS, perhaps ZFS+NFS as backing store for spools? Sorry if I've missed it and this has already come up. We're using Netapp/NFS, and are likely to continue to do so but still curious.

Hi Kelsey,

We're running ZFS here, and have just started using dovecot on it. No stats yet to report, but you might be interested in this edge case. One of our server started behaving badly... the database would randomly crash and not restart due to corrupted indexed. It turns out that the memory had gone bad, and that it had been bad for a while. Disk blocks were getting corrupted on read, and some on write! Luckly because we were on ZFS, which checksums all data, we were able to detect and repair most of the data (some 80mb of bad blocks distributed evenly thoughout the entire file system!) automatically, and also know exactly which files were unrecoverable (in the end just two or three files!). Also, we have hourly snapshots of all the file systems, so we were able to recover older versions of those files with minimal loss.

I will never rely on a non-checksumming file system for production use again, for data that is existed to persist over time.

Joe