Javier de Miguel RodrÃguez put forth on 12/12/2010 1:26 PM:
Thank you very much for all the responses in this thread. Now I have more questions:
- I have "slow" I/O (about 3.5000-4.000 IOPS, measured viaimapsync), if I enable zlib compression in my maildirs, that should lower the number the IOPS (less to read, less to write, less IOPS, more CPU). Dovecot 2.0 is better for zlib (lda support) than dovecot 1.2.X..
- I understand that indexes should go to the fastest storage I own.Somebody talked about storing them in a ramdisk and then backup them to disk on shutdown. I have several questions about that:
For that many users I'm guessing you can't physically stuff enough RAM into the machines in your ESX cluster to use a ramdisk for the index files, and if you could, you probably couldn't, or wouldn't want to, afford the DIMMs required to meet the need.
- In my setup I have 25.000+ users, almost 7.000.000messages in my maildir. How much memory should I need in a ramdisk to hold that?
- What happens if something fails? I think that if Ilose the indexes (ej: kernel crash) the next time I boot the system the ramdisk will be empty, so the indexes should be recreated. Am I right?
Given the size of your mail user base, I'd probably avoid the ramdisk option, and go with a couple of striped (RAID 0) 100+ GB SSDs connected on the iSCSI SAN. This is an ESX cluster of more than one machine correct? You never confirmed this, but it seems a logical assumption based on what you've stated. If it's a single machine you should obviously go with locally attached SATA II SSDs as it's far cheaper with much greater real bandwidth by a factor of 100:1 vs iSCSI connection.
- If I buy a SSD system and export that little and faststorage via iSCSI, does zlib compression applies to indexes?
Timo will have to answer this regarding zlib on indexes.
- Any additional filesystem info? I am using ext3 on RHEL 5.5, inRHEL 5.6 ext4 will be supported. Any performance hint/tuning (I already use noatime, 4k blocksize)?
I'm shocked you're running 25K mailboxen with 7 million messages on maildir atop EXT3! On your fast iSCSI SAN array, I assume with at least 14 spindles in the RAID group LUN where the mail is stored, you should be using XFS.
Formatted with the correct parameters, and mounted with the correct options, XFS will give you _at minimum_ a factor of 2 performance gain over EXT3 with 128 concurrent users. As you add more concurrent users, this ratio will grow even greater in XFS' favor.
XFS was designed specifically, and has been optimized since 1994, for large parallel workloads. EXT3 has traditionally been optimized for use as a single user desktop filesystem. Its performance with highly parallel workloads pales in comparison to XFS:
http://btrfs.boxacle.net/repository/raid/2.6.35-rc5/2.6.35-rc5/2.6.35-rc5_Ma...
If you do add the two ~100 GB SSDs and stripe them via the RAID hardware or via mdadm or LVM, format the resulting striped device with XFS and give it 16 allocation groups. If using kernel 2.6.36 or grater, mount the resulting filesystem with the delaylog option. You will be doing yourself a huge favor, and will be amazed at the index performance.
-- Stan