Charles Marcus put forth on 5/7/2010 5:29 AM:
On 2010-05-07 1:02 AM, Stan Hoeppner wrote:
I know TBird isn't the greatest IMAP client around,
Actually, its better than most (at least those with a decent GUI)...
but I think taking over 60 seconds just to download ~300 messages is way too damn long given the hardware resources, network, and load on the client and server machines.
There is definitely something wrong.
I agree.
Do you store your profile on a remote filesystem? There is a known major TB bug that causes it to be dog slow if your profile is not on a local hard drive. It is apparently fixed for 3.1, and I think it even made it into the 3.1b2 that was just released, so you might give it a try if your profile is on a remote filesystem.
The profiles are on the local machine, which is W2K Pro SP4 all M$ patches via auto updates, Win32 TB 3.0.4, Athlon XP 2GHz, 1GB dual channel RAM, 7.2K rpm 120GB Seagate UDMA100. The server is an old dual CPU 500MHz Intel box with 384MB PC100, a new 500GB 7.2K rpm single platter WD Blue SATA drive on a new SiI 3512 card, Intel Pro 100 NIC, Debian Lenny 5.0.4 with custom rolled 2.6.32.9 from kernel.org source, Dovecot 1.2.11 from Lenny backports. For practical purposes, this is a personal server with only a single IMAP client, load average: 0.01, 0.06, 0.03. The only real load it gets is an occasional kernel make, or processing a batch of digital camera photos with imagemagick and curator.
I tested out Outlook Express 6.0, which I've never used before, but was already on the machine as part of W2K. There were only a couple of new messages to grab so I couldn't test new message retrieval speed. However, when I clicked on a couple of IMAP folders containing over 11,000 messages each, they transferred in about 15 seconds per folder. It was freak'n fast. I was pleasantly surprised. Granted this wasn't an apples to apples test.
TB basically seems to be pulling, or dovecot serving, only about 5 messages/sec over 100Mb ethernet, which is abysmal performance given neither the server nor client have any load. The messages are mostly list mail which are at max a few kilobytes each.
I'm leaning toward a problem with TBird but I've been unable to find a bug report that covers this, nor a forum post anywhere, etc. The closest I've found for "slow startup" are recommendations to compact folders. I have no local folders to compact. I delete immediately and expunge on exit.
Ummm... compacting has nothing to do with 'Local Folders', it has to do with the local mbox files that are used to store the message headers (and bodies of downloaded messages) - and simply expunging is *not* enough. You need to either manually compact them every now and then, or set it to automatically compact regularly.
There are no local mbox files. Those are only created if one sets TB to synchronize IMAP folders to the local drive for offline use, which I do _NOT_ do. That defeats the whole purpose of having a nearby (network latency and b/w wise) fast IMAP server. If I wanted copies of all my mail on my workstation I'd run POP. But I don't. Thus, I don't synchronize.
The only noteworthy TB files I have locally are .msf files in the ~\Application Data\~\ImapMail directory, one per IMAP folder on the server. AFAIK these are the index files TB creates of the message headers it d/l's from the IMAP server. I also have a couple of cache files in the other TB profile directory ~\Local Settings\~\Cache that are rather large, one being ~50MB, the other being ~30MB, both with a current timestamp, meaning both are actively being used. AIUI, compacting folders in TB only affects local mbox files, removing deleted messages, and rewriting the file to eliminate whitespace. In absence of this, I defrag both partitions on my workstation disk frequently. Even after a fresh thorough defrag, this TB startup performance problem still exists. AFAIK, Dovecot does something similar to TB compacting automatically on its mbox files upon expunge.
Regardless of all the mbox and compacting talk, why would this ever affect new message headers being served up to TB by dovecot from the /var/mail/stan file? Every time I exit TB /var/mail/stan gets automatically compacted by dovecot. When I open TB the next time, and there are 300 messages, dovecot reads the partial headers and funnels them to TB. Correct? It seems TB then spins at 100% CPU for 60+ seconds saying "Downloading header x of 300". When it hits ~300, then there is finally network activity as TB seems to sort the messages into the proper IMAP folders, which is lightning quick compared to "downloading message headers".
I don't recall having this performance issue with dovecot 1.0.15. Just in case it's something I nurfed in my dovecot config, here's my dovecot -n output. Keep in mind I've made modifications appropriate for serving a single or just a couple of clients while attempting to cut resources consumed to the bare minimum while maintaining performance.
Thanks for the interest and help Charles. The cause is probably something stupid I did in an effort to optimize. Or something I did that the TB developers never actually expected someone to do, but didn't bother documenting that one shouldn't do it.
# 1.2.11: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32.9 i686 Debian 5.0.4 log_timestamp: %Y-%m-%d %H:%M:%S protocols: imap disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable: /usr/lib/dovecot/imap-login login_process_per_connection: no login_process_size: 16 login_processes_count: 1 login_max_processes_count: 1 login_max_connections: 8 max_mail_processes: 4 mail_privileged_group: mail mailbox_idle_check_interval: 15 mbox_write_locks: fcntl mbox_min_index_size: 2048 mbox_lazy_writes: no mail_process_size: 192 mail_plugins: fts fts_squat imap_client_workarounds: tb-extra-mailbox-sep auth default: worker_max_count: 1 process_size: 16 passdb: driver: pam args: max_requests=1 userdb: driver: passwd plugin: fts: squat fts_squat: partial=4 full=10
-- Stan