Timo Sirainen put forth on 5/22/2010 3:47 AM:
On 9.5.2010, at 9.54, Stan Hoeppner wrote:
It's frustrating trying to solve this, to say the least. :( I don't have enough in depth knowledge of either application to figure this out on my own. It seems the only way to communicate with the Tbird devs is via bug reports. I can't really file a Tbird bug report as I just don't have enough information available to file one that they can do anything with.
If you didn't solve it yet .. the way I normally try to figure out why something is taking 100% CPU load is attaching gdb to the process and getting a backtrace. I don't know if there's an easy way to do that with Windows, but if you can manage to do that a few times for a build with debug symbols enabled, it could be enough for developers to figure out what the problem is.
I turned every related knob I could find in about:config and couldn't fix it. And since I use Roundcube now and then I've been wanting to do server side sorting anyway. So, instead of attempting any kind of tracing of TBird, I switched to LDA, wrote a sieve script, and set TBird to check for new mail in IMAP folders. Afterward, when I'd click on an IMAP folder containing new mail, small folders would respond instantly displaying the new mail and the old already cached mails. On large folders (around 3000+ messages and up) it would display the old cached mails instantly, but it would take a few seconds to show the new messages, up to 10 seconds for folders containing 10k+ messages. This didn't really make sense to me as I thought Dovecot's index architecture would work faster.
Upon investigating this issue, I found that the imap process was spinning at 100% CPU until TBird displayed the new messages. As a test I deleted all the messages in my debian-users list mail folder which was at 13k+ messages. I rarely if ever searched it and there is a public archive, so I just whacked it. After doing so, upon opening the folder which would have say, 80 new messages in it, the imap process ran 100% CPU for a second or less, and all the new messages displayed pretty much instantly.
This led me to do more testing, and it basically appears that on my server, there is a linear relationship between mailbox (and thus index.cache) size and response time, regardless of how many new messages are in the imap folder (mbox file). Given that TBird caches all message headers, even when not using gloda, when it asks Dovecot for new mail in a folder, Dovecot would simply respond by sending new mail headers. Maybe this is exactly what Dovecot does. But apparently it's reading a lot more from the cache file or the mailbox file, as it takes forever to respond if the mbox file (imap folder) and respective index.cache file is large.
I've since thinned out most of my list mail folders that were 3000-10000+ messages. All of my folders are at less than 1,000 messages now except for my spam-l folder which is 13,200 messages. All folders respond instantly now when I select them regardless of the number of new messages, _except_ the spam-l folder. It can have one new message in it or 100 new messages, either way, it takes almost exactly 10 seconds for TBird to display the folder contents and the Dovecot imap process runs 100% CPU for that 10 seconds. If there are no new messages, the folder displays instantly. The dovecot.index.cache file is 30MB. The mbox file is 57MB. Apparently Dovecot is reading more than just the new message headers?
-- Stan