[Dovecot] Best Cluster Storage

[Stan Hoeppner]

Jonathan Tripathy put forth on 1/13/2011 7:11 AM:

> Would DRBD + GFS2 work better than NFS? While NFS is simple, I don't mind
> experimenting with DRBD and GFS2 is it means fewer problems?

Depends on your definition of "better".  If you do two dovecot+drbd nodes you
have only two nodes.  If you do NFS you have 3 including the NFS server.
Performance would be very similar between the two.

Now, when you move to 3 dovecot nodes or more you're going to run into network
scaling problems with the drbd traffic, because it increases logarithmically (or
is it exponentially?) with node count.  If using GFS2 atop drbd across all
nodes, each time a node writes to GFS, the disk block gets encapsulated by the
drbd driver and transmitted to all other drbd nodes.  With each new mail that's
written by each server, or each flag is updated, it gets written 4 times, once
locally, and 3 times via drbd.

With NFS, each of these writes occurs over the network only once.  With drbd
it's always a good idea to dedicate a small high performance GbE switch to the
cluster nodes just for drbd traffic.  This may not be necessary in a low volume
environment, but it's absolutely necessary in high traffic setups.  Beyond a
certain number of nodes even in a moderately busy mail network, drbd mirroring
just doesn't work.  The bandwidth requirements become too high, and nodes bog
down from processing all of the drbd packets.  Without actually using it myself,
and just using some logical reasoning based on the technology, I'd say the ROI
of drbd mirroring starts decreasing rapidly between 2 and 4 nodes, and beyond
for nodes...

You'd be much better off with an NFS server, or GFS2 directly on a SAN LUN.
CXFS would be far better, but it's not free.  In fact it's rather expensive, and
it requires a dedicated metadata server(s), which is one of the reasons it's so
#@! damn fast compared to most clustered filesystems.

Another option is a hybrid setup, with dual NFS servers each running GFS2
accessing the shared SAN LUN(s).  This eliminates the one NFS server as a
potential single point of failure, but also increases costs significantly as you
have to spend about $15K USD minimum for low end SAN array, and another NFS
server box, although the latter need not be expensive.

-- 
Stan