On 11/3/2011 10:21 AM, Ed W wrote:
I'm using NexentaStor (Solaris, ZFS) to export iSCSI-LUNs and I was thinking about a SSD based LUN for the indexes. As I'm using multiple servers this LUN will use OCFS2.
Given that the SAN always has the network latency behind it, might you be better to look at putting the SSDs in the frontend machines?
The latency of the GbE IP network, iSCSI HBAs, GbE switches, etc, is but a fraction of the overhead of the out of band OCFS metadata exchange between cluster members, and the general overhead of OCFS, or of cluster filesystems in general.
Obviously this then needs some way to make users "sticky" to one machine (or some few machines) where the indexes are stored?
This seems theoretically likely to give you higher IOPs to the index than having them on the OCFS2 storage? (At a trade off with more complexity for the load balancer front end...)
Following this logic, simply using local mechanical disk would yield improvement without the cost of SSDs. Depending on the number of nodes, putting a couple of SSDs in the SAN controller may likely be cheaper overall than adding mech disks to each node, let alone SSDs to each node. The random IO latency of SSD is so considerably lower than mechanical disk, even with the OCFS and iSCSI SAN overhead, overall read/write latency will likely be lower than using local mech disk in the nodes. And you get to retain centralized storage of the indexes, eliminating stickiness complexity issues.
Something else to consider is the read/write caching performance of NexentaStor (I've never used it, know nothing about it). If it's very good, and the NexentaStor host has gobs of RAM (think 64-128GB), then adding SSDs for indexes may not improve performance much, if any, depending on the concurrent user load.
I've read cases where adding SLC-SSD to high end FC SAN controllers with gobs of writeback cache RAM yielded little benefit with similar random IO workloads, simply because the cache was never taxed enough to force regular flushing. If your cache is large and fast enough to buffer most of your IOPS, then your current spindle speed is already irrelevant. In such a case adding SSD will yield little, or no, advantage.
-- Stan