[Dovecot] Best Cluster Storage

[Robert Schetterer]

Am 13.01.2011 23:17, schrieb Jonathan Tripathy:
> 
> On 13/01/11 21:34, Stan Hoeppner wrote:
>> Jonathan Tripathy put forth on 1/13/2011 7:11 AM:
>>
>>> Would DRBD + GFS2 work better than NFS? While NFS is simple, I don't
>>> mind
>>> experimenting with DRBD and GFS2 is it means fewer problems?
>> Depends on your definition of "better".  If you do two dovecot+drbd
>> nodes you
>> have only two nodes.  If you do NFS you have 3 including the NFS server.
>> Performance would be very similar between the two.
>>
>> Now, when you move to 3 dovecot nodes or more you're going to run into
>> network
>> scaling problems with the drbd traffic, because it increases
>> logarithmically (or
>> is it exponentially?) with node count.  If using GFS2 atop drbd across
>> all
>> nodes, each time a node writes to GFS, the disk block gets
>> encapsulated by the
>> drbd driver and transmitted to all other drbd nodes.  With each new
>> mail that's
>> written by each server, or each flag is updated, it gets written 4
>> times, once
>> locally, and 3 times via drbd.
>>
>> With NFS, each of these writes occurs over the network only once. 
>> With drbd
>> it's always a good idea to dedicate a small high performance GbE
>> switch to the
>> cluster nodes just for drbd traffic.  This may not be necessary in a
>> low volume
>> environment, but it's absolutely necessary in high traffic setups. 
>> Beyond a
>> certain number of nodes even in a moderately busy mail network, drbd
>> mirroring
>> just doesn't work.  The bandwidth requirements become too high, and
>> nodes bog
>> down from processing all of the drbd packets.  Without actually using
>> it myself,
>> and just using some logical reasoning based on the technology, I'd say
>> the ROI
>> of drbd mirroring starts decreasing rapidly between 2 and 4 nodes, and
>> beyond
>> for nodes...
>>
>> You'd be much better off with an NFS server, or GFS2 directly on a SAN
>> LUN.
>> CXFS would be far better, but it's not free.  In fact it's rather
>> expensive, and
>> it requires a dedicated metadata server(s), which is one of the
>> reasons it's so
>> #@! damn fast compared to most clustered filesystems.
>>
>> Another option is a hybrid setup, with dual NFS servers each running GFS2
>> accessing the shared SAN LUN(s).  This eliminates the one NFS server as a
>> potential single point of failure, but also increases costs
>> significantly as you
>> have to spend about $15K USD minimum for low end SAN array, and
>> another NFS
>> server box, although the latter need not be expensive.
>>
> Hi Stan,
> 
> The problem is, is that we do not have the budget at the minute to buy a
> SAN box, which is why I'm just looking to setup Linux environment to
> substitute for now.
> 
> Regarding the servers, I was thinking of having a 2 node drbd cluster
> (in active+standby), which would export a single iSCSI LUN. Then, I
> would have a 2 node dovecot+postfix cluster (in active-active), where
> each node would mount the same LUN (With GFS2 on top). This is 4 servers
> in total (Well, 4 VMs running on 4 physically separate servers).
> 
> I'm hearing different things on whether dovecot works well or not with
> GFS2. Of course, I could simply replace the iSCSI LUN above with an nfs
> server running on each DRBD node, if you feel NFS would work better than
> GFS2. Either way, I would probably use a crossover cable for the DRBD
> cluster. Could maybe even bond 2 cables together if I'm feeling
> adventurous!
> 
> The way I see it, is that there are 2 issues to deal with:
> 
> 1) Which "Shared Disk" technology is best (GFS2 over LUN or a simple NFS
> server)
> and
> 2) What is the best method of HA for the storage system
> 
> Any advice is appreciated.

Hi Jonathan
the explains from
Stan were good enough to choose what fit to your needs
( thx Stan for explain drbd so deeply )
so what are the number of mailboxes and the traffic volume you wait for ? )
at minimum you should have 2 drbd nodes binding to a seperate gb
interface via cross cable ( this might not be needed with virtual
machines, but check before you setup, and dont forget for real ha you
always need 2 vm master machines, so for your setup this may increase
the budget), and 2 loadblancers with i.e keepalive,  if this isnt enough
for you, you should follow Stans advice using SAN or equal,
after all this is the real world , budget must always be as high to
solve your task, you cant press an elephant trough a mouse hole....
so there is no best solution, there is only a solution what fits best to
your needs

-- 
Best Regards

MfG Robert Schetterer

Germany/Munich/Bavaria