[Dovecot] Performance, dovecot vs tpop3d

[Nate]

Hello, I am new to the list and have spent extensive time researching 
performance suggestions through the list and the 
documentation.  Particularly interesting is the recent performance 
thread related to the upcoming release.

I just upgraded one of our email servers from tpop3d to dovecot 
1.0rc19.  tpop3d has served us well for many years; however, it 
basically has lost all it's support and developers.  My main liking 
of tpop3d was it's speed.  After the upgrade to dovecot, the server 
began to crawl.  This server runs postfix for the delivery agent and 
now dovecot for pop3 and imap.  The box serves roughly 9000 virtual 
maildir mailboxes with roughly 20 pop3 processes active at any given time.

I ran some tests this morning to determine an exact difference 
between running tpop3d and dovecot.
dovecot:  I/O load averages 81%, System load averages 12.00
tpop3d:   I/O load averages 20%, System load averages  0.50

To ensure it wasn't just a fluke of logins, I let each app run for 
several minutes, and did the test multiple times.  The results were 
not any abnormal user usage and definitely related to application performance.

My current settings in dovecot related to performance are:
mmap_diable = yes
mmap_no_write = yes
Modifying these to no seems to decrease performance slightly but only 
a few percent.

I have also played with various other settings, which didn't give me 
any significant performance boost.

I'm eager for the next release which will have the fsync_disable 
option to see if that helps, it certainly looks promising.

One observation where the majority of the load seems to be generated 
from is when a pop3 connection is loading up a very large 
maildir.  The pop3 process seems to remain around using I/O for a 
considerable time, even when the user has no new email.  Perhaps it 
is rebuilding or verifying it's index files?  Here is an example of a 
lsof of the pop3 process with a user who has roughly 17000 
messages.  They check their email every 10 minutes which seems to add 
significant load.  Note the final line, it's reading an individual 
message file, yet the user has no new email since the last login.  If 
i keep lsofing the process, the final file seems to change and keep 
going through many of their mailbox files.  The process takes many 
minutes to complete.  Is it rebuilding the index on every login?

COMMAND   PID    USER   FD   TYPE    DEVICE    SIZE      NODE NAME
pop3    15148 postfix  cwd    DIR       9,2     152    850160 
/home/domain.net/brock
pop3    15148 postfix  rtd    DIR       9,2     816         2 /
pop3    15148 postfix  txt    REG       9,2 1888690   7096106 
/usr/local/libexec/dovecot/pop3
pop3    15148 postfix  mem    REG       9,2  107724    182934 /lib/ld-2.3.2.so
pop3    15148 postfix  mem    REG       9,2 1578228    182936 
/lib/tls/libc-2.3.2.so
pop3    15148 postfix  mem    REG       9,2   16312     88977 
/lib/libdl-2.3.2.so
pop3    15148 postfix    0u  IPv4 873280915               TCP 
domain.net:pop3->dsl.dyn.06-99.uslogin.net:40950 (CLOSE_WAIT)
pop3    15148 postfix    1u  IPv4 873280915               TCP 
domain.net:pop3->dsl.dyn.06-99.uslogin.net:40950 (CLOSE_WAIT)
pop3    15148 postfix    2w  FIFO       0,5         873280916 pipe
pop3    15148 postfix    3r   CHR       1,9             27963 /dev/urandom
pop3    15148 postfix    4r  FIFO       0,5         873281456 pipe
pop3    15148 postfix    5w  FIFO       0,5         873281456 pipe
pop3    15148 postfix    6r  FIFO       0,5         873281457 pipe
pop3    15148 postfix    7w  FIFO       0,5         873281457 pipe
pop3    15148 postfix    8u   REG       9,2  205740   7272682 
/var/spool/mail/domain.net/brock/dovecot.index
pop3    15148 postfix    9u   REG       9,2   15020   7225137 
/var/spool/mail/domain.net/brock/dovecot.index.log
pop3    15148 postfix   10u   REG       9,2 1585152   7290906 
/var/spool/mail/domain.net/brock/dovecot.index.cache
pop3    15148 postfix   11r   REG       9,2    1992   5040734 
/var/spool/mail/domain.net/brock/cur/1165164651.V902I4cea5e.s3.visp.net:2,S

I'm eager to solve this new I/O issue which has cropped up after 
upgrading to dovecot.  Despite the severe performance degradation, I 
feel dovecot has awesome potential and is feature rich, and most 
importantly it's being maintained, so I want to stick with it!

Thank you for any help offered!  I'm hopeful for any suggestions to 
solve this performance issue or diagnose it further to determine the cause.

- Nate