CJ Keist put forth on 8/7/2010 10:17 AM:
All, Thanks for all the information. I think I'm leaning towards locally attached fiber disk array. Couple of advantages I see, one it will be faster than NFS, second it will allow us to separate user home directory disk quotas and email disk quotas. Something we have been wanting to do for awhile.
If you're going to do locally attached storage, why spend the substantial additional treasure required for a fiber channel array and HBA solution? You're looking at a minimum of $10k-$20k USD for a 'low end' FC array solution. Don't get me wrong, I'm a huge fan of FC SANs, but only when it makes sense. And it only makes sense if you have multiple hosts and you're slicing capacity (and performance) to each host. So at least get an entry level Qlogic FC switch so you can attach others hosts in the future, or even right away, once you realize what you can do with this technology. If you're set on the FC path, I recommend these components, all of which I've used and are fantastic products when great performance and support:
http://www.qlogic.com/Products/SANandDataNetworking/FibreChannelSwitches/Pag... http://www.sandirect.com/product_info.php?products_id=1366
http://www.qlogic.com/Products/SANandDataNetworking/FibreChannelAdapters/Pag... http://www.sandirect.com/product_info.php?cPath=257_260_268&products_id=291
http://www.nexsan.com/sataboy.php http://www.sandirect.com/product_info.php?cPath=171_208_363&products_id=1434
Configure the first 12 of the 14 drives in the Nexsan as a RAID 1+0 array, the last two drives as hot spares. This will give you 6TB of usable array space sliceable to hosts as you see fit, 600 MB/s of sequential read throughput, ~1000 random seeks/sec to disk and 35k/s to cache, ultra fast rebuilds after drive failure (10x faster than a RAID 5 or 6 rebuild). RAID 1+0 does not suffer the mandatory RAID 5/6 read-modify-write cycle and thus the write throughput of RAID 1+0 has a 4:1 advantage over RAID 5 and an 8:1 advantage over RAID 6, if my math is correct.
The only downside of RAID 1+0 compared to RAID 5/6 is usable space after redundancy overhead. With our Nexsan unit above, RAID 5 with two hot spares will give us 11TB of usable space and RAID 6 will give us 10TB. Most people avoid RAID 5 these days because of the "write hole" silent data corruption issue, and go with RAID 6 instead, because they want to maximize their usable array space.
One of the nice things about the Nexsan is that you can mix & match RAID levels within the same chassis. Let's say you're wanting to consolidate the Postfix queues, INBOX and user maildir files, _and_ user home directories onto your new Nexsan array. You want faster performance for files that are often changing but you don't need as much total storage for these. You want more space for user home dirs but you don't need the fastest access times.
In this case you can create a RAID 1+0 array of the first 6 disks in the chassis giving you 3TB of fast usable space with highest redundancy for the mail queue, user INBOX and maildir files. Take the next 7 disks and create a RAID 5 array yielding 6TB of usable space, double that of the "fast" array. We now have one disk left for a hot space, and this is fine as long as you have another spare disk or 2 on the shelf. Speaking of spares, with one 14 drive RAID 1+0, you could actually gain 1TB of usable storage by using no hot spares and keeping spares on the shelf. The likelihood of a double drive failure is rare, and with RAID 1+0 is would be extremely rare for two failures to occur in the same mirror pair.
-- Stan