Timo Sirainen put forth on 2/13/2010 11:40 AM:
On 13.2.2010, at 17.48, Stefan Foerster wrote:
- Use a HA shared storage, export either a cluster filesystem or NFS, and have your dovecot servers mount that file system. Load balance these servers (Cisco ACE, ldirectord, ...) and there you go.
With NFS if you want to avoid random errors, you still need to do the load balancing in a way that user's mails are never accessed simultaneously from two servers at the same time. See recent "quick question" thread about this. Cluster filesystems should be able to handle this better, although for performance it's probably still a good idea to do this.
Timo, are you saying cluster filesystems aren't suitable for clustered dovecot storage due to performance reasons?
I've managed a 500 user environment where *EVERYTHING* was hitting on two 2Gb/s FC SAN storage arrays, one IBM and one Nexsan, through a single Qlogic FC switch, and these were fairly low end FC arrays. We had 4 Citrix blades, 3 ESX blades, and an Exchange blade. On the ESX blades we were running two AD DCs plus windows CIFS serving, A LAMP Moodle database server, a dedicated Linux syslog server, a MS SQL server for Citrix licensing and the organization accounting application, a Citrix balancing director server, Novel Zen server, Novell iFolder server, a Linux print server, etc, etc.
All the ESX VM guest storage resided on the SAN and thus they all booted from SAN. All bare metal blades booted from SAN as well. The only blades with onboard disks were the ESX blades which booted the VMware kernel from mirrored local disks. Now, granted none of these systems were sharing the same filesystem on exported LUNs and sharing metadata over ethernet. But the overall load on the SAN controllers was fairly high.
I can't see how the metadata sharing of say GFS2 is going to create any serious performance impact on a cluster of dovecot servers using GFS2 and a shared SAN array especially if using maildir. If the load balancing is implemented correctly and a given user is only hitting one dovecot server at any one point in time, there should be few, if any, shared file locks. Thus, no negative impact due to shared locking.
A single one of these: http://www.nexsan.com/sataboy/tech.php
configured with 2GB of cache and 14 x 300GB 10K rpm SATA drives setup at as a single RAID 10 would yield ~2.1TB of fast redundant mail storage and ~370MB/s of (controller limited) read throughput using a single controller. Double that figure if you go with active-active dual controllers. Sustained IOPs with 14x 10K rpm drives is about 30K IOPs, which is not bad at all considering this is the lowest end array Nexsan sells. Double that to 60K IOPs for dual controllers.
I'd bet I could take this single low end Nexsan array, a low end 8 port Qlogic 4Gb FC switch, and 6 average dual socket servers with 8GB of RAM each and a single port 4Gb FC adapter, a decent gigabit ethernet switch, using Linux with GFS2 and Dovecot, and build a cluster that could easily handle a few hundred to a few thousand concurrent IMAP users and 30K+ total mailiboxen assuming a 500MB mailbox limit per user. This hardware configuration should be attainable for well less than $50K USD. This doesn't included the front end load balancer.
-- Stan