Thank you for your responses Stan, I reply you belowFor that many users I'm guessing you can't physically stuff enough RAM into the machines in your ESX cluster to use a ramdisk for the index files, and if you could, you probably couldn't, or wouldn't want to, afford the DIMMs required to meet the need.
Yes, I have a cluster of 4 ESX servers. I am going to do some scriptting to see how much space we are allocating to indexes.
- In my setup I have 25.000+ users, almost 7.000.000messages in my maildir. How much memory should I need in a ramdisk to hold that?
- What happens if something fails? I think that if Ilose the indexes (ej: kernel crash) the next time I boot the system the ramdisk will be empty, so the indexes should be recreated. Am I right?
Given the size of your mail user base, I'd probably avoid the ramdisk option, and go with a couple of striped (RAID 0) 100+ GB SSDs connected on the iSCSI SAN. This is an ESX cluster of more than one machine correct? You never confirmed this, but it seems a logical assumption based on what you've stated. If it's a single machine you should obviously go with locally attached SATA II SSDs as it's far cheaper with much greater real bandwidth by a factor of 100:1 vs iSCSI connection.
My SAN(s) (HP LeftHand Networks) do not support SSD, though. But I have several LeftHand nodes, some of them with raid5, others with raid 1+0. Maildirs+indexes are now in raid5, maybe I can separate the indexes to raid 1+0 iscsi target in a different san
- If I buy a SSD system and export that little and faststorage via iSCSI, does zlib compression applies to indexes?
Timo will have to answer this regarding zlib on indexes.
That would be rather interesting.- Any additional filesystem info? I am using ext3 on RHEL 5.5, inRHEL 5.6 ext4 will be supported. Any performance hint/tuning (I already use noatime, 4k blocksize)?
I'm shocked you're running 25K mailboxen with 7 million messages on maildir atop EXT3! On your fast iSCSI SAN array, I assume with at least 14 spindles in the RAID group LUN where the mail is stored, you should be using XFS.
I have two raid5 (7 disks+1 spare) and I have joined them via LVM stripping. Each disk is SAS 15k rpm 450GB, and the SANs have 512 MB-battery-backed-cache. In our real workload (imapsync), each raid5 gives around 1700-1800 IOPS, combined 3.500 IOPS.
Formatted with the correct parameters, and mounted with the correct options, XFS will give you _at minimum_ a factor of 2 performance gain over EXT3 with 128 concurrent users. As you add more concurrent users, this ratio will grow even greater in XFS' favor.
Sadly, Red Hat Enterprise Linux 5 does not support natively XFS. I can install it via CentosPlus, but we need Red Hat support if somethings goes VERY wrong. Red Hat Enterprise Linux 6 supports XFS (and gives me dovecot 2.0), but maybe it is "too early" for a RHEL6 deployment for so many users (sigh).
I will continue investigating about indexes. Any additional hint?
Regards
Javier