Henrique Fernandes put forth on 4/10/2011 5:29 PM:

Thanks, but we did ran teste in the disks, and the problem was find file, not actuly writing or reading them, so we guess is a ofs2 problem, thats why

Hi Henrique,

Finding a file is a filesystem metadata operation, which entails walking the directory tree(s). Thus, finding a file generates IO to/from the disk array where the filesystem metadata resides. The IO requests are small, but there are many of them, and typically generate more head seeks than normal file read/write operations. Currently you don't have enough disk spindles in your array to keep up with the IOPS demands of both metadata and file operations.

now we are thinking in mdbox, to reduce the "find files" problem one thing

From a total IOPS perspective, there is not a huge difference between mdbox and maildir. And mdbox ties your indexes to the mail files, eliminating the flexibility of putting indexes on dedicated storage such as local SSD. If you really want to eliminate the massive IOPS load you currently have, switch to mbox. mbox storage generates almost no metadata IOPS at all.

that realy helped the performance was give lots of ram in the virtual machines where dovecot is running, with lots of ram, the kernel makes cache

Yes, a filesystem with 5 million maildir files is going to have a large directory tree to walk. If OCFS has a tweakable setting to force the amount of metadata that is cached, setting this as high as you possibly can (without destroying other things that need RAM) is a smart move. If not configurable, giving as much RAM as possible to each VM is also worthwhile, as you've noticed, as more more buffer space is allocated to filesystem metadata.

Worth noting: if you have multiple VM guest OSes on a single host, which all access the same OCFS filesystem, you will be unnecessarily caching OCFS data multiple times, wasting memory. If your hypervisor supports memory deduplication, as VMware ESX does, you should enable it, if you haven't already. If that is not an option, it's best to only run one OCFS guest per host. It may be possible to run OCFS in the hypervisor, and map a virtual filesystem to each guest on that host. Read your virtual platform documentation to find the right solution for your environment.

of "location" and it finds much faster the files. As matter fact the space provide to us, we might have to give back some of it. Buying hardware is not an option yet. So we keep thinking how to imprive performance tunning everything up.

Losing some space is fine as long as you don't lose spindles. Losing spindles decreases IOPS. Be sure that OCFS supports filesystem shrinking before the EMC OPs decrease the size of your LUN. And, make sure you've shutdown all of your OCFS hosts/guests _before_ they resize your LUN. If you don't, you'll possibly sustain corruption that will cause the loss of your entire OCFS filesystem. After they resize the LUN, bring one OCFS machine back online but without mounting the OCFS filesystem. Use the appropriate tool to shrink the filesystem, again, if OCFS can even do it. If not, do NOT let them resize your LUN. Make other arrangements.

I am thinking of chancge io schelduler also. But have not research a lot to try this yet, but it is in the plans.

Changing the Linux elevator has little effect in VM guests. Use the "noop" elevator in a VM environment, especially when using SAN storage. The Linux elevator cannot make head positioning decisions with SAN arrays--use noop.

The backup problem is because the machine that backups ( it joins the ocfs2 cluester just to backup files ) we are not able to make cache. Cause as much as we give RAM the bacula just eat up all ram we gave. I guess it is because of accurate option, still find a way to limit bacule ram use, so the kernel became able to cache some inodes.

Considering the nature of backup, I don't think the caching of metadata or files will decrease the IOPS load on the disk array. Backup is fairly sequential in nature, so caching anything would simply waste memory. I'm guessing what you really want is something like read ahead caching of metadata, so after finding one file, the next few directory table reads are from memory instead of disk. I don't know of a way to optimize for this.

Thanks for your reply and i am glad you remembered that old posts.

It's an interesting problem/scenario, and a couple of my interests/specialties are SAN storage arrays and filesystems.

But i still looking for some info about mdbox.

It's a hybrid between maildir and mbox, but with inbuilt indexes. maildir is one mail per file. mbox is one file with lots of mail. mdbox stores multiple mails per file across many files. Using mdbox will drop your IOPS load. Exactly how much I can't say, but it won't be anything near as large a drop as converting to mbox, which again, has almost zero metadata overhead.

Before we thought of using mdbox but we did not want to stick with dovecot, i mean, we like the ideia we could be able to change the imap server and etc. But as someone said, what are the other choises we have for opensource imap server ? Even with cyrus we still need to come acroos a big convertion, so it does not make much diference.

My only concern would be integrated indexes. Once you convert to mdbox you no longer have the flexibility of moving index files to fast dedicated storage.

Right now your best option for a quick decisive solution to your SAN/OCFS bottleneck is to move the maildir indexes off the SAN to fast local disks in the cluster hosts, either an SSD or 15k SAS drive per host. It's cheap, it will work, and you can implement and test it quickly at zero risk. It will be even better if you can use kernel 2.6.36 or later and XFS w/delaylog mount option on the local disks. Delaylog can reduce metadata write disk IOPS by an order of magnitude.

About the index files you said, SSD disks are not an possibility, but i though of using another partition as place for index files. Does it will make lot of diference ? As i said, we are using ldiretord with lbrlc that keeps track of ip address to some servers, is not ALWASY send it to the same server, but i tries to do it.

It will likely make a big difference in OCFS load. But...

The whole point of moving the index files to a local disk is to *dedicate* the entire performance of the disk to index file IOPS and remove that load from the SAN. An average decent quality SATA3 SSD today can perform about 50k random IOPS--far more than you actually need, but the best bang for the buck IOPS wise. A single 15k SAS drive can perform about 300 random IOPS, a 10k SAS drive about 200 random IOPS, and a 7.2k SATA drive performs about 150 random IOPS.

If you have 3 physical Dovecot hosts in your cluster, and can dedicate a 15k SAS drive on each for index file use only, you can potentially decrease the IOPS load on the EMC by 900. That should solve your OCFS performance problems pretty handily. That's a big win for less than $500 USD outlay on 3 15k SAS drives.

If you simply use leftover space on a currently installed local disk in each host, the amount of benefit you receive is completely dependent on the current seek load on that disk. I would do some investigation with iostat over a period of a few days to make sure said disk in each host has plenty of spare IOPS capacity. It's possible that you'll actually decrease overall Dovecot performance substantially if the disks you mention don't have enough spare performance.

Make sure you size the local disks to accommodate the current index files, with 50% headroom for load balancer stickiness overhead inefficiency and index growth. If you have 300GB total of index files currently on the EMC, a new 147GB 15k SAS drive in each of your 3 Dovecot hosts should suffice. 3.5" 15k 147GB Fujitsu Enterprise SAS drives can be obtained for as little as $155 USD. If your servers are HP/IBM/Dell and require their hot swap caged drives, you'll obviously pay quite a bit more.

-- Stan