Am 13.01.2011 23:17, schrieb Jonathan Tripathy:
On 13/01/11 21:34, Stan Hoeppner wrote:
Jonathan Tripathy put forth on 1/13/2011 7:11 AM:
Would DRBD + GFS2 work better than NFS? While NFS is simple, I don't mind experimenting with DRBD and GFS2 is it means fewer problems? Depends on your definition of "better". If you do two dovecot+drbd nodes you have only two nodes. If you do NFS you have 3 including the NFS server. Performance would be very similar between the two.
Now, when you move to 3 dovecot nodes or more you're going to run into network scaling problems with the drbd traffic, because it increases logarithmically (or is it exponentially?) with node count. If using GFS2 atop drbd across all nodes, each time a node writes to GFS, the disk block gets encapsulated by the drbd driver and transmitted to all other drbd nodes. With each new mail that's written by each server, or each flag is updated, it gets written 4 times, once locally, and 3 times via drbd.
With NFS, each of these writes occurs over the network only once. With drbd it's always a good idea to dedicate a small high performance GbE switch to the cluster nodes just for drbd traffic. This may not be necessary in a low volume environment, but it's absolutely necessary in high traffic setups. Beyond a certain number of nodes even in a moderately busy mail network, drbd mirroring just doesn't work. The bandwidth requirements become too high, and nodes bog down from processing all of the drbd packets. Without actually using it myself, and just using some logical reasoning based on the technology, I'd say the ROI of drbd mirroring starts decreasing rapidly between 2 and 4 nodes, and beyond for nodes...
You'd be much better off with an NFS server, or GFS2 directly on a SAN LUN. CXFS would be far better, but it's not free. In fact it's rather expensive, and it requires a dedicated metadata server(s), which is one of the reasons it's so #@! damn fast compared to most clustered filesystems.
Another option is a hybrid setup, with dual NFS servers each running GFS2 accessing the shared SAN LUN(s). This eliminates the one NFS server as a potential single point of failure, but also increases costs significantly as you have to spend about $15K USD minimum for low end SAN array, and another NFS server box, although the latter need not be expensive.
Hi Stan,
The problem is, is that we do not have the budget at the minute to buy a SAN box, which is why I'm just looking to setup Linux environment to substitute for now.
Regarding the servers, I was thinking of having a 2 node drbd cluster (in active+standby), which would export a single iSCSI LUN. Then, I would have a 2 node dovecot+postfix cluster (in active-active), where each node would mount the same LUN (With GFS2 on top). This is 4 servers in total (Well, 4 VMs running on 4 physically separate servers).
I'm hearing different things on whether dovecot works well or not with GFS2. Of course, I could simply replace the iSCSI LUN above with an nfs server running on each DRBD node, if you feel NFS would work better than GFS2. Either way, I would probably use a crossover cable for the DRBD cluster. Could maybe even bond 2 cables together if I'm feeling adventurous!
The way I see it, is that there are 2 issues to deal with:
- Which "Shared Disk" technology is best (GFS2 over LUN or a simple NFS server) and
- What is the best method of HA for the storage system
Any advice is appreciated.
Hi Jonathan the explains from Stan were good enough to choose what fit to your needs ( thx Stan for explain drbd so deeply ) so what are the number of mailboxes and the traffic volume you wait for ? ) at minimum you should have 2 drbd nodes binding to a seperate gb interface via cross cable ( this might not be needed with virtual machines, but check before you setup, and dont forget for real ha you always need 2 vm master machines, so for your setup this may increase the budget), and 2 loadblancers with i.e keepalive, if this isnt enough for you, you should follow Stans advice using SAN or equal, after all this is the real world , budget must always be as high to solve your task, you cant press an elephant trough a mouse hole.... so there is no best solution, there is only a solution what fits best to your needs
-- Best Regards
MfG Robert Schetterer
Germany/Munich/Bavaria