[Dovecot] Thunderbird very slow startup, 1.2.11, mbox, postfix local delivery to /var/mail

[Stan Hoeppner]

Greetings Patrick, fellow Postfix user, and Postfix author.

Patrick Ben Koetter put forth on 5/8/2010 5:04 PM:
> * Stan Hoeppner <stan at hardwarefreak.com>:
>> If this is the case, if I switch from having Postfix do local delivery to
>> /var/mail to having Postfix use dovecot LDA, what other changes would I need
> 
> <http://wiki.dovecot.org/LDA/Postfix>
> 
>> to make?  Would I still be able to sort new messages with Tbird filter
> 
> Sure. Tbird filter rules are local, client rules run on your computer and not
> on the mailserver.

I should have worded that question better.  What I was trying to ask is
that, if using dovecot LDA, if new messages would still be presented to
Tbird in the INBOX imap folder.  If LDA dropped new messages into a
different imap folder then Tbird wouldn't see them in the INBOX, and that's
where all my filter rules apply.  Looking back this was probably a dumb
question.  I asked it because I'm totally unfamiliar with LDA.  I'll be
reading up on it.

>> rules?  What directory would dovecot LDA drop the new messages into?  Would
> 
> Dovecot deliver will put messages into the same place where Postfix local puts
> them now.

Got it.  That's good to know.

>> I need to make name space changes?  I've never actually done anything
> 
> Nope, deliver replaces Postfix local LDA transparently.

Good.  Fantastic, actually.

>> manually with name spaces.  I let dovecot figure it out automagically.
>>
>> Thanks for the interest in my topic.  I know it's not a very interesting one
>> since apparently I'm the only one on the planet experiencing this.
> 
> Putting 40k+ messages - IIRC you wrote that - into a single mailbox is not
> unusual allthough I hardly now any postmaster who recommends that.

The total is somewhere around there, and growing daily due to list mail.
I'm not sure of your definition of "single mailbox" here.  Yes, the messages
are all in the same UNIX user account, but the messages are spread across a
number of mbox files, not a single file.  I have ~12,000 messages in each of
two mbox files, ~5,000 in another, a couple with ~3,000, and a number with
less than 1,000.  Dovecot handles all of these quickly and easily, with low
memory consumption.  Searching the large mbox files is quick when the squat
indexes aren't stale.  Mailbox size, individual mbox size, don't seem to be
a performance factor.  My only performance problem is related to the INBOX,
which usually has less than 150 messages in it.  Tbird loads those 150-300
headers from the INBOX folder slower than it loads 12,000+ headers from the
debian-users folder or spam-l folder.

At this point I'm assuming the problem has to lie with Tbird, as dovecot
should be performing the same regardless of folder size or location on disk,
*UNLESS* dovecot hasn't already indexed /var/mail/%user when the client
connects.  I don't know how dovecot handles indexing of /var/mail/%user
files, or more importantly, I don't know *when* dovecot indexes those files.
 If they aren't indexed before the client connects and requests the contents
of INBOX, I can see there being a performance drop, although I wouldn't
think it would be great enough to cause a ~20 second delay for 125 messages
or 60 seconds for ~300 messages.

> Having no index on 40k messages means the mail server needs to scan the
> directory any time a mail client accesses the mailbox. This takes time and
> might explain the delay you experience.

Whoa.  We've had a major disconnect somewhere in the conversation.  I'm not
lacking indexes on 40k messages.  They're all well indexed by dovecot.  Some
have multiple indexes due to squat.  Performance while accessing any of
those 40k is fast as lightning.  Again, the performance problem I have is
_strictly with new messages in the INBOX folder_.  More precisely, Tbird
pulling the headers for new messages.  IFAIK, Tbird only pulls the headers,
not the entire message.  It only pulls the entire message when you
open/access it.

> Dovecots ability always to provide an up to date if you use its LDA deliver
> was one of the major reasons to use Dovecot for me. We had a non-Dovecot IMAP
> server with a load of 60. It went down to 2 the day we began to use Dovecot
> with deliver AND access to messages became faster.

I agree.  Dovecot rocks.  And it's probably not at fault WRT my current
performance issue.  I'm just asking the question in the off chance that my
performance issue WRT INBOX is due to something I may have screwed up in my
dovecot config or setup, or maybe...

Timo identified a bug related to processing of mbox files that was to have
been fixed in 1.2.11.  I don't think my current issue is related.  But, I
get the feeling that mbox code paths have taken a back seat to maildir and
other code paths through the last few release cycles due to the popularity
of the latter and the declining popularity of the former.  I am one of the
users who helped Timo identify that mbox bug merely by observation here.
It's always within the realm of possibility that I've run into another less
than optimal code path.  Like I sad, it's probably a Tbird issue, but it
can't hurt to ask here just in case there might be a dovecot issue in play
as well.  It could be a problem with both code sets, Tbird and Dovecot, that
only surfaces when using both together.

It's an odd obscure "mild" performance problem that just shouldn't exist.  I
can't find it documented anywhere, nothing like it shows up searching
Google.  I thought it was related to Tbird's GLODA, but after disabled that
and scrubbing all the Tbird cache and index files, the problem still exists.
 Again, only for INBOX, but not for any other imap folders.

It's frustrating trying to solve this, to say the least. :(  I don't have
enough in depth knowledge of either application to figure this out on my
own.  It seems the only way to communicate with the Tbird devs is via bug
reports.  I can't really file a Tbird bug report as I just don't have enough
information available to file one that they can do anything with.

-- 
Stan