[Dovecot] Indexes to MLC-SSD

[Stan Hoeppner]

On 11/3/2011 10:21 AM, Ed W wrote:
> 
>> I'm using NexentaStor (Solaris, ZFS) to export iSCSI-LUNs and I was
>> thinking about a SSD based LUN for the indexes. As I'm using multiple
>> servers this LUN will use OCFS2.
> 
> Given that the SAN always has the network latency behind it, might you
> be better to look at putting the SSDs in the frontend machines?

The latency of the GbE IP network, iSCSI HBAs, GbE switches, etc, is but
a fraction of the overhead of the out of band OCFS metadata exchange
between cluster members, and the general overhead of OCFS, or of cluster
filesystems in general.

> Obviously this then needs some way to make users "sticky" to one machine
> (or some few machines) where the indexes are stored?
> 
> This seems theoretically likely to give you higher IOPs to the index
> than having them on the OCFS2 storage? (At a trade off with more
> complexity for the load balancer front end...)

Following this logic, simply using local mechanical disk would yield
improvement without the cost of SSDs.  Depending on the number of nodes,
putting a couple of SSDs in the SAN controller may likely be cheaper
overall than adding mech disks to each node, let alone SSDs to each
node.  The random IO latency of SSD is so considerably lower than
mechanical disk, even with the OCFS and iSCSI SAN overhead, overall
read/write latency will likely be lower than using local mech disk in
the nodes.  And you get to retain centralized storage of the indexes,
eliminating stickiness complexity issues.

Something else to consider is the read/write caching performance of
NexentaStor (I've never used it, know nothing about it).  If it's very
good, and the NexentaStor host has gobs of RAM (think 64-128GB), then
adding SSDs for indexes may not improve performance much, if any,
depending on the concurrent user load.

I've read cases where adding SLC-SSD to high end FC SAN controllers with
gobs of writeback cache RAM yielded little benefit with similar random
IO workloads, simply because the cache was never taxed enough to force
regular flushing.  If your cache is large and fast enough to buffer most
of your IOPS, then your current spindle speed is already irrelevant.  In
such a case adding SSD will yield little, or no, advantage.

-- 
Stan