[Dovecot] Question about "slow" storage but fast cpus, plenty of ram and dovecot

[Javier de Miguel Rodrí­guez]

     Thank you for your responses Stan, I reply you below
> For that many users I'm guessing you can't physically stuff enough RAM
> into the machines in your ESX cluster to use a ramdisk for the index
> files, and if you could, you probably couldn't, or wouldn't want to,
> afford the DIMMs required to meet the need.
>
     Yes, I have a cluster of 4 ESX servers. I am going to do some 
scriptting to see how much space we are allocating to indexes.



>>                  - In my setup I have 25.000+ users, almost 7.000.000
>> messages in my maildir. How much memory                 should I need in
>> a ramdisk to hold that?
>>
>>                   - What happens if something fails? I think that if I
>> lose the indexes (ej: kernel crash) the next time I                 boot
>> the system the ramdisk will be empty, so the indexes should be
>> recreated. Am I right?
> Given the size of your mail user base, I'd probably avoid the ramdisk
> option, and go with a couple of striped (RAID 0) 100+ GB SSDs connected
> on the iSCSI SAN.  This is an ESX cluster of more than one machine
> correct?  You never confirmed this, but it seems a logical assumption
> based on what you've stated.  If it's a single machine you should
> obviously go with locally attached SATA II SSDs as it's far cheaper with
> much greater real bandwidth by a factor of 100:1 vs iSCSI connection.
>

     My SAN(s) (HP LeftHand Networks) do not support SSD, though. But I 
have several LeftHand nodes, some of them with raid5, others with raid 
1+0. Maildirs+indexes are now in raid5, maybe I can separate the indexes 
to raid 1+0 iscsi target in a different san


>>                  - If I buy a SSD system and export that little and fast
>> storage via iSCSI, does zlib compression                         applies
>> to indexes?
> Timo will have to answer this regarding zlib on indexes.
>

     That would be rather interesting.


>>      - Any additional filesystem info? I am using ext3 on RHEL 5.5, in
>> RHEL 5.6 ext4 will be supported. Any performance hint/tuning (I already
>> use noatime, 4k blocksize)?
> I'm shocked you're running 25K mailboxen with 7 million messages on
> maildir atop EXT3!  On your  fast iSCSI SAN array, I assume with at
> least 14 spindles in the RAID group LUN where the mail is stored, you
> should be using XFS.


     I have two raid5 (7 disks+1 spare) and I have joined them via LVM 
stripping. Each disk is SAS 15k rpm 450GB, and the SANs have 512 
MB-battery-backed-cache. In our real workload (imapsync), each raid5 
gives around 1700-1800 IOPS, combined 3.500 IOPS.

> Formatted with the correct parameters, and mounted with the correct
> options, XFS will give you _at minimum_ a factor of 2 performance gain
> over EXT3 with 128 concurrent users.  As you add more concurrent users,
> this ratio will grow even greater in XFS' favor.

     Sadly, Red Hat Enterprise Linux 5 does not support natively XFS. I 
can install it via CentosPlus, but we need Red Hat support if somethings 
goes VERY wrong. Red Hat Enterprise Linux 6 supports XFS (and gives me 
dovecot 2.0), but maybe it is "too early" for a RHEL6 deployment for so 
many users (sigh).

     I will continue investigating about indexes. Any additional hint?

     Regards

     Javier