[Dovecot] Thunderbird very slow startup, 1.2.11, mbox, postfix local delivery to /var/mail

[Stan Hoeppner]

Timo Sirainen put forth on 5/22/2010 3:47 AM:
> On 9.5.2010, at 9.54, Stan Hoeppner wrote:
> 
>> It's frustrating trying to solve this, to say the least. :(  I don't have
>> enough in depth knowledge of either application to figure this out on my
>> own.  It seems the only way to communicate with the Tbird devs is via bug
>> reports.  I can't really file a Tbird bug report as I just don't have enough
>> information available to file one that they can do anything with.
> 
> If you didn't solve it yet .. the way I normally try to figure out why something is taking 100% CPU load is attaching gdb to the process and getting a backtrace. I don't know if there's an easy way to do that with Windows, but if you can manage to do that a few times for a build with debug symbols enabled, it could be enough for developers to figure out what the problem is.

I turned every related knob I could find in about:config and couldn't fix it.
 And since I use Roundcube now and then I've been wanting to do server side
sorting anyway.  So, instead of attempting any kind of tracing of TBird, I
switched to LDA, wrote a sieve script, and set TBird to check for new mail in
IMAP folders.  Afterward, when I'd click on an IMAP folder containing new
mail, small folders would respond instantly displaying the new mail and the
old already cached mails.  On large folders (around 3000+ messages and up) it
would display the old cached mails instantly, but it would take a few seconds
to show the new messages, up to 10 seconds for folders containing 10k+
messages.  This didn't really make sense to me as I thought Dovecot's index
architecture would work faster.

Upon investigating this issue, I found that the imap process was spinning at
100% CPU until TBird displayed the new messages.  As a test I deleted all the
messages in my debian-users list mail folder which was at 13k+ messages.  I
rarely if ever searched it and there is a public archive, so I just whacked
it.  After doing so, upon opening the folder which would have say, 80 new
messages in it, the imap process ran 100% CPU for a second or less, and all
the new messages displayed pretty much instantly.

This led me to do more testing, and it basically appears that on my server,
there is a linear relationship between mailbox (and thus index.cache) size and
response time, regardless of how many new messages are in the imap folder
(mbox file).  Given that TBird caches all message headers, even when not using
gloda, when it asks Dovecot for new mail in a folder, Dovecot would simply
respond by sending new mail headers.  Maybe this is exactly what Dovecot does.
 But apparently it's reading a lot more from the cache file or the mailbox
file, as it takes forever to respond if the mbox file (imap folder) and
respective index.cache file is large.

I've since thinned out most of my list mail folders that were 3000-10000+
messages.  All of my folders are at less than 1,000 messages now except for my
spam-l folder which is 13,200 messages.  All folders respond instantly now
when I select them regardless of the number of new messages, _except_ the
spam-l folder.  It can have one new message in it or 100 new messages, either
way, it takes almost exactly 10 seconds for TBird to display the folder
contents and the Dovecot imap process runs 100% CPU for that 10 seconds.  If
there are no new messages, the folder displays instantly.  The
dovecot.index.cache file is 30MB.  The mbox file is 57MB.  Apparently Dovecot
is reading more than just the new message headers?

-- 
Stan