Dovecot 2.2.18 on CentOS 6
I have a pair of servers setup with MySQL, Postfix, and Dovecot. Replication is setup and working between the two dovecot instances.
The problem I'm running into is that a single mailbox receives a lot of messages, at times the rate is multiple messages per second. Delivery from Postfix to Dovecot is via tcp based LMTP. When I do 'ps -aef|grep lmtp|wc -l' I get 62 and does not appear to go higher than that. At the moment I have 4500 and 8300 messages queued on two Postfix instances waiting to deliver via LMTP to the same dovecot instance. Deliveries only happen via LMTP and only one of the two nodes actually gets the deliveries.
What I'm seeing is very high load on the system (40) and queues building on the Postfix side. Replication is keeping up. Looking at the logs now I see anywhere from 4-7 messages per second delivered to this single mailbox. I would like to increase that rate a lot.
These machines are VMs hosted on Xenserver 6.x. I have them setup with 8 vCPUs (2 sockets with 4 cores per socket), the dom0 machines have dual HBA connectors back to a SAN and have 128 CPUs and 256GB of RAM and are not taxed. I added a 2nd virtual disk that is used for storing mailbox data. It is ext4 and has noatime set during mount. /var is also mounted with noatime.
The performance graphs in XenCenter show nearly all 8 vCPUs at about 50%, and the writes on the mailbox data disk are about 20%. iostat is showing mostly <5 for await times for the disks, though I do see a 10 now and again.
I'm guessing that maybe I'm hitting a mailbox locking issue and not sure how to reduce the contention and thereby increase the delivery rate to this mailbox.
-Chad