On 3/28/2013 3:34 PM, Ed W wrote:
I believe a variation on that theme is also to "double" each machine using DRBD so that machines are arranged in pairs. One can fail and the other will take over the load. ie each pair of machines mirrors the storage for the other. With this arrangement only warm failover is usually required and hence DRBD can run in async mode and performance impact is low
This is an active/passive setup, and doubles your hardware costs across the board, with no parallel performance gain. This is not financially feasible for 1M users. Going active/active would be better as you can cut in half the number of server nodes required. But here you must use a cluster filesystem, and you're still buying double the quantity of disks that are needed.
At this scale it is much more cost effective to acquire 4 midrange FC/iSCSI SAN heads with 120x 15K 600GB SAS drives each, 480 total. With RAID10 you get 144TB net capacity. An active/active DRBD solution would require 960 drives instead of 480 for the same net storage and IOPS. These drives run about $400 USD in such a bulk purchase depending on vendor. That's an extra ~$192,0000 wasted on drives. Not to mention all the extra JBOD chassis required, and more importantly the extra power/cooling cost. You can obtain 4 low frills high performance midrange SAN heads for quite a bit less than that $192,000. The Nexsan E60 comes to mind. Four FC SAN heads each with dual active/active controllers and four 8Gb FC ports plus four expansion chassis, w/480x 600GB 15K drives in 32U, leaving 8U at the bottom of the rack for the 10KVA UPS needed to power them.
-- Stan