All, Thanks for all the information. I think I'm leaning towards locally attached fiber disk array. Couple of advantages I see, one it will be faster than NFS, second it will allow us to separate user home directory disk quotas and email disk quotas. Something we have been wanting to do for awhile.
Again thanks for all the view points and experiences with Maildir over NFS.
On 8/7/10 4:06 AM, Stan Hoeppner wrote:
Noel Butler put forth on 8/6/2010 4:29 PM:
Actually you will not notice any difference. How do you think all the big boys do it now :) Granted some opted for the SAN approach over NAS, but for mail, NAS is better way to go IMHO and plenty of large services, ISP, corporations, and universities etc, all use NAS.
The protocol overhead of the NFS stack is such that one way latency is in the 1-50 millisecond range, depending on specific implementations and server load. The one way latency of a fibre channel packet is in the sub 100 microsecond range and is fairly immune to system load. The performance of fibre channel is equal to local disk plus approximately one millisecond of additional effective head seek time due to switch latency, SAN array controller latency, and latency due to cable length. A filesystem block served out of SAN array controller cache returns to the kernel quicker than a block read from local disk that is not in cache because the former suffers no mechanical latency. Due to the complexity of the stack, NFS is far slower than either.
Those who would recommend NFS/NAS over fibre channel SAN have no experience with fibre channel SANs. I'm no fan of iSCSI SANs due to the reliance on TCP/IP for transport, and the low performance due to stck processing. However, using the same ethernet switches for both, iSCSI SAN arrays will also outperform NFS/NAS boxen by a decent margin.
Regarding the OP's case, given the low cost of new hardware, specifically locally attached RAID and the massive size and low cost of modern disks, I'd recommend storing user mail on the new mail host. It's faster and more cost effective than both NFS/SAN. Unless his current backup solution "requires" user mail dirs to be on that NFS server for nightly backup, local disk is definitely the way to go. Four 300GB 15k SAS drives on a good PCIe RAID card w/256-512MB cache in a RAID 10 configuration would yield ~350-400MB/s of real filesystem bandwidth, seek throughput equivalent to a 2 disk stripe--about 600 random seeks/s, 600GB of usable space, ability to sustain two simultaneous disk failures (assuming 1 failure per mirror pair), and cost effectiveness.
-- C. J. Keist Email: cj.keist@colostate.edu UNIX/Network Manager Phone: 970-491-0630 Engineering Network Services Fax: 970-491-5569 College of Engineering, CSU Ft. Collins, CO 80523-1301
All I want is a chance to prove 'Money can't buy happiness'