[Dovecot] Question about "slow" storage but fast cpus, plenty of ram and dovecot

[Stan Hoeppner]

Javier de Miguel Rodrí­guez put forth on 12/13/2010 3:26 AM:

>     Can you give me (off-list if you desire) more info about your setup?
> I am interested in the number and type of spindles you are using. We are
> using LeftHand because of their real time replication capabilities,
> something very interesting to us, and each node pair is relatively cheap
> (8x450 GB at 15K rpm sas disks per node, real time replication, 512 MB
> caché, about 25K € each node pair).
> 
>     We can throw more hardware to this, let's see if using memory-based
> indexes (via ramdisk) we get better results. Zlib compression on indexes
> should be great for this.

Javier, your number and type of spindles aren't the source of your I/O
bottleneck.  Your array controller is.  Or, more specifically, its lack
of enough fast network ports is the problem.  The P4300 BK718A model of
which you speak only has two 1GbE iSCSI ports.  That's only 200 MB/s
full duplex.  That's less link bandwidth than a single 2 Gbit fiber
channel interface.  2 Gbit FC is 2 generations and over 8 years old now.
 It was superseded by 4 Gbit FC and then 8 Gbit FC, which is the current
standard.  8 Gbit FC provides 800 MB/s full duplex bandwidth per link.

Merely two of the 8 SAS drives in the P4300 unit can saturate the iSCSI
interfaces on the controller.  This has always been the downside of low
end iSCSI arrays.  The same eight 450GB 15k spindles backed by a decent
controller with 512MB to 1GB cache and dual 4 Gbit FC ports or 10 GbE
ports would give you well over double the bandwidth using the same RAID
level you do now, and would increase IOPS substantially.

You don't need more or faster spindles to increase your performance and
decrease interface load Javier.  You need more interface throughput than
your two 1 GbE ports currently provide.  You either need more 1 GbE
ports per P4300, say 4 of them, link bonded to your ethernet switch as a
single 400 Mb/s full duplex channel (assuming the P4300 supports link
bonding), or you need two 10 GbE ports per P4300, if it is upgradeable
in this manner.  If it is not, you need to move up to another HP model
with much greater interface throughput, or a competing vendor's unit
with 10 GbE or 8 Gbit FC ports.  1 Gbe iSCSI is a budget solution, not a
performance solution, and I think that's what you're beginning to realize.

Nexsan sells a low end 14 drive SAS array with dual 2 GB cache
active/active controllers for about the same price you're paying for the
P4300, maybe a little less.  It has dual 4 Gbit FC ports _and_ dual 1
GbE iSCSI ports.  It stomps the P4300 in performance when using the FC
ports, offering a sustained 800+ MB/s and around 4,300 IOPS to disk
(50K+ to cache), but it doesn't have the advanced management features of
the HP unit such as built-in snapshots, etc.  IMHO VMware Consolidated
Backup is a superior SAN backup solution than node-node replication on
the P4000 series, which is just one more load sucking the life out of
the two tiny 1 GbE ports.  Using VCB with an FC SAN is a much more
flexible and higher performance disaster recover solution than vendor
specific hardware mirroring of two SAN arrays.

IMHO, you should look into a phased migration to an FC SAN.  Qlogic's FC
switches and HBAs are excellent, and affordable on a per port and per
chassis basis.  They are easy to setup.  I taught myself to program the
switches in the CLI in a day.  The web gui may be even easier.

For about $50-60K USD you could have the big brother to the Nexsan unit
I mentioned above.  It's called the SASBeast.  It contains dual
active/active controllers, each with 2 GB mirrored cache, 2 8Gbit FC
ports, and 2 1GbE iSCSI ports, for a total of 4 8Gb FC ports and 4 1GbE
iSCSI ports.  It contains 42 x 300, 450, or 600GB SAS 15k drives.  It
can be expanded with one additional chassis containing another 60 SAS or
SATA drives for a total of 104 drives.

It offers sustained throughput of over 1.2 GB/s (about 4 times that of
the P4300) and over 12,000+ IOPS to disk with 42 SAS drives, 50K+ to
cache, increasing to approximately 30k IOPS to disk with a fully
populated expansion chassis w/60 SAS 15K drives, 104 drives total in the
array.  This device's performance is controller limited, similar to the
P4300, but due to the processor, not the interfaces, simply for the fact
you can expand it to 104 SAS drives for less than $100K USD.  If one
needs maximum IOPS and bandwidth rather than total storage, one would be
better off buying two SASBeasts than one unit with the expansion
chassis.  IIRC it's also a little cheaper.

There are many other fine SAN arrays on the market.  I mention Nexsan
merely because they are very affordable, fast as hell for the price,
easy to manage, and I've used them.  Due to the low price, they lack the
more advanced features of higher priced units, such as snapshots, remote
replication, etc.  As I mentioned, I'm not a big fan of relying on SAN
controllers to perform all my disaster recovery functions, so I do that
with software, such as VCB, and a dedicated backup server with a SAN
attached robotic library.  Thus, the lack of this functionality in the
Nexsan units is not an issue here.

-- 
Stan