[Dovecot] Thunderbird very slow startup, 1.2.11, mbox, postfix local delivery to /var/mail

[Ken A]

On 5/7/2010 7:00 AM, Stan Hoeppner wrote:
> Charles Marcus put forth on 5/7/2010 5:29 AM:
>> On 2010-05-07 1:02 AM, Stan Hoeppner wrote:
>>> I know TBird isn't the greatest IMAP client around,
>>
>> Actually, its better than most (at least those with a decent GUI)...
>>
>>> but I think taking over
>>> 60 seconds just to download ~300 messages is way too damn long given the
>>> hardware resources, network, and load on the client and server machines.
>>
>> There is definitely something wrong.
>
> I agree.
>
>> Do you store your profile on a remote filesystem? There is a known major
>> TB bug that causes it to be dog slow if your profile is not on a local
>> hard drive. It is apparently fixed for 3.1, and I think it even made it
>> into the 3.1b2 that was just released, so you might give it a try if
>> your profile is on a remote filesystem.
>
> The profiles are on the local machine, which is W2K Pro SP4 all M$ patches
> via auto updates, Win32 TB 3.0.4, Athlon XP 2GHz, 1GB dual channel RAM, 7.2K
> rpm 120GB Seagate UDMA100.

Maybe some a/v scanning your mail?
If so, try turning it off, or switching TB to port 993 (and enable imaps 
in dovecot).

Ken


  The server is an old dual CPU 500MHz Intel box
> with 384MB PC100, a new 500GB 7.2K rpm single platter WD Blue SATA drive on
> a new SiI 3512 card, Intel Pro 100 NIC, Debian Lenny 5.0.4 with custom
> rolled 2.6.32.9 from kernel.org source, Dovecot 1.2.11 from Lenny backports.
>   For practical purposes, this is a personal server with only a single IMAP
> client, load average: 0.01, 0.06, 0.03.  The only real load it gets is an
> occasional kernel make, or processing a batch of digital camera photos with
> imagemagick and curator.
>
> I tested out Outlook Express 6.0, which I've never used before, but was
> already on the machine as part of W2K.  There were only a couple of new
> messages to grab so I couldn't test new message retrieval speed.  However,
> when I clicked on a couple of IMAP folders containing over 11,000 messages
> each, they transferred in about 15 seconds per folder.  It was freak'n fast.
>   I was pleasantly surprised.  Granted this wasn't an apples to apples test.
>
>>> TB basically seems to be pulling, or dovecot serving, only about 5
>>> messages/sec over 100Mb ethernet, which is abysmal performance given neither
>>> the server nor client have any load.  The messages are mostly list mail
>>> which are at max a few kilobytes each.
>>>
>>> I'm leaning toward a problem with TBird but I've been unable to find a bug
>>> report that covers this, nor a forum post anywhere, etc.  The closest I've
>>> found for "slow startup" are recommendations to compact folders.  I have no
>>> local folders to compact.  I delete immediately and expunge on exit.
>>
>> Ummm... compacting has nothing to do with 'Local Folders', it has to do
>> with the local mbox files that are used to store the message headers
>> (and bodies of downloaded messages) - and simply expunging is *not*
>> enough. You need to either manually compact them every now and then, or
>> set it to automatically compact regularly.
>
> There are no local mbox files.  Those are only created if one sets TB to
> synchronize IMAP folders to the local drive for offline use, which I do
> _NOT_ do.  That defeats the whole purpose of having a nearby (network
> latency and b/w wise) fast IMAP server.  If I wanted copies of all my mail
> on my workstation I'd run POP.  But I don't.  Thus, I don't synchronize.
>
> The only noteworthy TB files I have locally are .msf files in the
> ~\Application Data\~\ImapMail directory, one per IMAP folder on the server.
>   AFAIK these are the index files TB creates of the message headers it d/l's
> from the IMAP server.  I also have a couple of cache files in the other TB
> profile directory ~\Local Settings\~\Cache that are rather large, one being
> ~50MB, the other being ~30MB, both with a current timestamp, meaning both
> are actively being used.  AIUI, compacting folders in TB only affects local
> mbox files, removing deleted messages, and rewriting the file to eliminate
> whitespace.  In absence of this, I defrag both partitions on my workstation
> disk frequently.  Even after a fresh thorough defrag, this TB startup
> performance problem still exists.  AFAIK, Dovecot does something similar to
> TB compacting automatically on its mbox files upon expunge.
>
> Regardless of all the mbox and compacting talk, why would this ever affect
> new message headers being served up to TB by dovecot from the /var/mail/stan
> file?  Every time I exit TB /var/mail/stan gets automatically compacted by
> dovecot.  When I open TB the next time, and there are 300 messages, dovecot
> reads the partial headers and funnels them to TB.  Correct?  It seems TB
> then spins at 100% CPU for 60+ seconds saying "Downloading header x of 300".
>   When it hits ~300, then there is finally network activity as TB seems to
> sort the messages into the proper IMAP folders, which is lightning quick
> compared to "downloading message headers".
>
> I don't recall having this performance issue with dovecot 1.0.15.  Just in
> case it's something I nurfed in my dovecot config, here's my dovecot -n
> output.  Keep in mind I've made modifications appropriate for serving a
> single or just a couple of clients while attempting to cut resources
> consumed to the bare minimum while maintaining performance.
>
> Thanks for the interest and help Charles.  The cause is probably something
> stupid I did in an effort to optimize.  Or something I did that the TB
> developers never actually expected someone to do, but didn't bother
> documenting that one shouldn't do it.
>
> # 1.2.11: /etc/dovecot/dovecot.conf
> # OS: Linux 2.6.32.9 i686 Debian 5.0.4
> log_timestamp: %Y-%m-%d %H:%M:%S
> protocols: imap
> disable_plaintext_auth: no
> login_dir: /var/run/dovecot/login
> login_executable: /usr/lib/dovecot/imap-login
> login_process_per_connection: no
> login_process_size: 16
> login_processes_count: 1
> login_max_processes_count: 1
> login_max_connections: 8
> max_mail_processes: 4
> mail_privileged_group: mail
> mailbox_idle_check_interval: 15
> mbox_write_locks: fcntl
> mbox_min_index_size: 2048
> mbox_lazy_writes: no
> mail_process_size: 192
> mail_plugins: fts fts_squat
> imap_client_workarounds: tb-extra-mailbox-sep
> auth default:
>    worker_max_count: 1
>    process_size: 16
>    passdb:
>      driver: pam
>      args: max_requests=1
>    userdb:
>      driver: passwd
> plugin:
>    fts: squat
>    fts_squat: partial=4 full=10
>
>

-- 
Ken Anderson
Pacific Internet - http://www.pacific.net