[Dovecot] Thunderbird very slow startup, 1.2.11, mbox, postfix local delivery to /var/mail

[Stan Hoeppner]

Charles Marcus put forth on 5/7/2010 5:29 AM:
> On 2010-05-07 1:02 AM, Stan Hoeppner wrote:
>> I know TBird isn't the greatest IMAP client around,
> 
> Actually, its better than most (at least those with a decent GUI)...
> 
>> but I think taking over
>> 60 seconds just to download ~300 messages is way too damn long given the
>> hardware resources, network, and load on the client and server machines.
> 
> There is definitely something wrong.

I agree.

> Do you store your profile on a remote filesystem? There is a known major
> TB bug that causes it to be dog slow if your profile is not on a local
> hard drive. It is apparently fixed for 3.1, and I think it even made it
> into the 3.1b2 that was just released, so you might give it a try if
> your profile is on a remote filesystem.

The profiles are on the local machine, which is W2K Pro SP4 all M$ patches
via auto updates, Win32 TB 3.0.4, Athlon XP 2GHz, 1GB dual channel RAM, 7.2K
rpm 120GB Seagate UDMA100.  The server is an old dual CPU 500MHz Intel box
with 384MB PC100, a new 500GB 7.2K rpm single platter WD Blue SATA drive on
a new SiI 3512 card, Intel Pro 100 NIC, Debian Lenny 5.0.4 with custom
rolled 2.6.32.9 from kernel.org source, Dovecot 1.2.11 from Lenny backports.
 For practical purposes, this is a personal server with only a single IMAP
client, load average: 0.01, 0.06, 0.03.  The only real load it gets is an
occasional kernel make, or processing a batch of digital camera photos with
imagemagick and curator.

I tested out Outlook Express 6.0, which I've never used before, but was
already on the machine as part of W2K.  There were only a couple of new
messages to grab so I couldn't test new message retrieval speed.  However,
when I clicked on a couple of IMAP folders containing over 11,000 messages
each, they transferred in about 15 seconds per folder.  It was freak'n fast.
 I was pleasantly surprised.  Granted this wasn't an apples to apples test.

>> TB basically seems to be pulling, or dovecot serving, only about 5
>> messages/sec over 100Mb ethernet, which is abysmal performance given neither
>> the server nor client have any load.  The messages are mostly list mail
>> which are at max a few kilobytes each.
>>
>> I'm leaning toward a problem with TBird but I've been unable to find a bug
>> report that covers this, nor a forum post anywhere, etc.  The closest I've
>> found for "slow startup" are recommendations to compact folders.  I have no
>> local folders to compact.  I delete immediately and expunge on exit.
> 
> Ummm... compacting has nothing to do with 'Local Folders', it has to do
> with the local mbox files that are used to store the message headers
> (and bodies of downloaded messages) - and simply expunging is *not*
> enough. You need to either manually compact them every now and then, or
> set it to automatically compact regularly.

There are no local mbox files.  Those are only created if one sets TB to
synchronize IMAP folders to the local drive for offline use, which I do
_NOT_ do.  That defeats the whole purpose of having a nearby (network
latency and b/w wise) fast IMAP server.  If I wanted copies of all my mail
on my workstation I'd run POP.  But I don't.  Thus, I don't synchronize.

The only noteworthy TB files I have locally are .msf files in the
~\Application Data\~\ImapMail directory, one per IMAP folder on the server.
 AFAIK these are the index files TB creates of the message headers it d/l's
from the IMAP server.  I also have a couple of cache files in the other TB
profile directory ~\Local Settings\~\Cache that are rather large, one being
~50MB, the other being ~30MB, both with a current timestamp, meaning both
are actively being used.  AIUI, compacting folders in TB only affects local
mbox files, removing deleted messages, and rewriting the file to eliminate
whitespace.  In absence of this, I defrag both partitions on my workstation
disk frequently.  Even after a fresh thorough defrag, this TB startup
performance problem still exists.  AFAIK, Dovecot does something similar to
TB compacting automatically on its mbox files upon expunge.

Regardless of all the mbox and compacting talk, why would this ever affect
new message headers being served up to TB by dovecot from the /var/mail/stan
file?  Every time I exit TB /var/mail/stan gets automatically compacted by
dovecot.  When I open TB the next time, and there are 300 messages, dovecot
reads the partial headers and funnels them to TB.  Correct?  It seems TB
then spins at 100% CPU for 60+ seconds saying "Downloading header x of 300".
 When it hits ~300, then there is finally network activity as TB seems to
sort the messages into the proper IMAP folders, which is lightning quick
compared to "downloading message headers".

I don't recall having this performance issue with dovecot 1.0.15.  Just in
case it's something I nurfed in my dovecot config, here's my dovecot -n
output.  Keep in mind I've made modifications appropriate for serving a
single or just a couple of clients while attempting to cut resources
consumed to the bare minimum while maintaining performance.

Thanks for the interest and help Charles.  The cause is probably something
stupid I did in an effort to optimize.  Or something I did that the TB
developers never actually expected someone to do, but didn't bother
documenting that one shouldn't do it.

# 1.2.11: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32.9 i686 Debian 5.0.4
log_timestamp: %Y-%m-%d %H:%M:%S
protocols: imap
disable_plaintext_auth: no
login_dir: /var/run/dovecot/login
login_executable: /usr/lib/dovecot/imap-login
login_process_per_connection: no
login_process_size: 16
login_processes_count: 1
login_max_processes_count: 1
login_max_connections: 8
max_mail_processes: 4
mail_privileged_group: mail
mailbox_idle_check_interval: 15
mbox_write_locks: fcntl
mbox_min_index_size: 2048
mbox_lazy_writes: no
mail_process_size: 192
mail_plugins: fts fts_squat
imap_client_workarounds: tb-extra-mailbox-sep
auth default:
  worker_max_count: 1
  process_size: 16
  passdb:
    driver: pam
    args: max_requests=1
  userdb:
    driver: passwd
plugin:
  fts: squat
  fts_squat: partial=4 full=10


-- 
Stan