[Dovecot] Re: New mail sometimes fails to appear

[Tom Alsberg]

On Tue, Feb 14, 2006 at 10:59:49AM +0200, Timo Sirainen wrote:
> >    Attached is an example of one such last message saved from
> >    Thunderbird after new mail arrived that was not seen.  The filler
> >    characters are bytes of 0x80, and then the next message from
> >    "n-Path: " is present.
> 
> They're actually 0x00 inside the mbox file, but Dovecot just translates
> them to 0x80 since IMAP doesn't allow sending 0x00 characters.

Oh, I see...

BTW, why 0x80, of all characters?

> Now, these 0x00 characters are added to mbox file when it's rewriting
> the headers and it figures out that it needs more space. So it first
> grows the file by writing 0x00 characters to the end of the file. Then
> it starts moving message data towards the end of file.

I'll look for the code that does that, and put some debugging prints
there to see if I can verify it.

> So you mean that this corruption is only seen by Thunderbird, and it
> won't stay in the mbox file itself?

Indeed.  The corruption is as seen by Thunderbird through IMAP.
Accessing the mbox file directly, there is no corruption in it.

> In that case it means that mbox locking isn't working properly,
> ie. a Dovecot process is reading the mbox while another Dovecot
> process is modifying it.

May very well be the case.  Especially considering that Thunderbird
has multiple connections to the server open.  I'm not sure what causes
Dovecot to rewrite headers at each such point (given no client
initiated write or even read), but I can think of some...

> I don't remember having tested mbox_read_locks=dotlock, but I'd have
> thought it would have worked..

I'll try to investigate more soon.  If you can provide some ideas on
how to check the locking it could help (given that Dovecot locks for a
short period of time only, I can't easily fabricate concurrent locking
from outside, and it appears just holding a read lock all the time
will not do much).

Perhaps some of the other details of the problem I have described can
shed some more light (i.e. the tracing of mbox_sync, the fact that
opening a new IMAP connection corrects the problem also for the old
one, etc.)?

I will look some more and probably follow up to you with some further
details.

  Cheers,
  -- Tom

-- 
  Tom Alsberg - hacker (being the best description fitting this space)
  Web page:	http://www.cs.huji.ac.il/~alsbergt/
DISCLAIMER:  The above message does not even necessarily represent what
my fingers have typed on the keyboard, save anything further.