Ed W put forth on 2/17/2010 12:25 PM:
I think Stan pretty much covered how to do this stuff *properly*,
At least for a VMware ESX + SAN environment, yes.
however, for those following along in the bedroom, there are a couple of interesting projects what might get you some of the ESX features (surely at the expense of far more support and likely reliability, but needs always vary...)
Surely. I hate the licensing cost of VMware ESX and the options. Also, the first time I was told about VMware ESX I was extremely skeptical. Once I started using it, and built out a SAN architecture under it, I was really, really impressed by what it can do, and its management capabilities. It will be a long time until a FOSS equivalent even comes close to its performance, reliability, capability, and ease of management. It really is a great solution. HA, Consolidated Backup, and a couple of other technologies are what really make this an enterprise solution, providing near 24x7x365 uptime and rapid redeployment of an infrastructure after catastrophic loss of the datacenter.
Note, I have no experience with any of these projects, they simply caught my eye for further research...
- Latest KVM+QEMU includes some of the desirable ESX features including hot migration
- Apparently Redhat have a nice management utility for this
I'll have to look into this.
- Or try ProxMox: http://pve.proxmox.com/wiki/Main_Page
(cheap) High availability storage seems to come down to:
- iSCSI
1Gbe iSCSI is great for targeted applications on moderate load SANs. With any kind of heavy lifting, you need either 10Gbe iSCSCI or Fiber Channel. Both of those are a bit more expensive, and 10Gbe iSCSI usually costing quite a bit more than FC because of the switch and HBA costs. Either is suitable for an HA SAN with live backup. 1Gbe iSCSI is not--simply too little bandwidth and too much latency.
- Add redundancy to the storage using DRDB (I believe a successful strategy with Dovecot is pairs of servers, replicated to each other - run each at 50% capacity and if one dies the other picks up the slack)
DRDB is alright for a couple of replicated hosts with moderate volume. If you run two load balanced hot hosts with DRDB, and your load increases to the point you need more capacity, a 3rd hot host, expanding with DRDB gets a bit messy. With an iSCSI or FC SAN you merely plug in a 3rd host, install and configure the cluster FS software, expose the shared LUN to the host, and basically you're up and running in little time. All 3 hosts share the exact same data on disk, so you have no replication issues, no matter how many systems you stick into the cluster. The only limitation is the throughput of your SAN array.
- Interesting developing ideas are: PVFS, GlusterFS (they have an interesting "appliance" which might get reliability to production levels?), CEPH (reviews suggest it's very easily days)
GlusterFS isn't designed as a primary storage system for servers or server clusters. A good description of it would be "cloud storage". It is designed to mask, or make irrelevant, the location of data storage devices and the distance to them. Server and datacenter architects need to know the latency characteristics and bandwidth of storage devices backing the servers. GlusterFS is the antithesis of this.
None of these solutions gets you an "enterprise" or proper high end solution as described by Stan, but may give some others some things to investigate
"Enterprise" capability, performance, and reliability don't necessarily have to come with an "Enterprise" price tag. ;)
Eric Rostetter is already using GFS2 over DRDB with two hot nodes. IIRC he didn't elaborate a lot on the performance or his hardware config. He seemed to think the performance was more than satisfactory.
Eric, can you tell us more about your setup, in detail? I promise I'll sit quiet and just listen. Everyone else may appreciate your information.
-- Stan