[Dovecot] Re: New mail sometimes fails to appear

[Tom Alsberg]

Hi, Timo.

Have you been looking into what I described in my last few messages
about this problem?  Some more information about the problem in them,
but I could not yet see it exactly, and I got no replies from you...

Some assistance in overcoming this problem would be greatly
appreciated.

Here's some more information I gathered in the meanwhile:

* It appears this does not necessarily have to do with the IDLE
  extension.  Even when Thunderbird is set not to use it, but poll
  once in a while, still at some point it does not see new messages on
  SELECT.  Pressing "Get Mail" sends a SELECT but that doesn't do much
  either.

* The abovementioned point combined with the corrupted last message
  seen in new mail that arrives (point 4 in my quoted message) makes
  me believe this is probably a bug in the code syncing between
  mbox and index.

* This is interesting:  Having Thunderbird open, I wait until the
  problem occurs (speeding it up by sending a test message every
  minute - after in average about 10 messages suddenly a new message
  arriving will not be seen), then send a few more test messages that
  Thunderbird does not see arriving either.  Now I open another IMAP
  client (say, mutt on another computer) to the same account on
  Dovecot.  On connection of the second IMAP client, suddenly it will
  see all mail and also the old open Thunderbird will suddenly get all
  the messages it lost.

Any ideas?

  Thanks, help appreciated,
  -- Tom

On Tue, Feb 07, 2006 at 10:13:03AM +0200, Tom Alsberg wrote:
> Some more new information...
> 
> 1. First, I was wrong in the last noted observation (probably my eyes
>    got confused between the lines in the log).  When the first call to
>    mbox_sync_has_changed returns 1 and the next ones return 0,
>    Thunderbird is not notified of the new mail that has arrived (no
>    EXISTS is sent), even the first time.  So I can't tell in advance what
>    will be with the next message that arrives.
> 
> 2. Also, I put some more debugging into mbox_sync_has_changed, and can
>    say the following:  When everything is right, mbox_sync_has_changed
>    returns 1 (the second and third times too) because
>    mbox->mbox_dirty_stamp is different from st->st_mtime, in the last
>    statement of mbox_sync_has_changed:
> 
> 	return st->st_mtime != mbox->mbox_dirty_stamp ||
> 		st->st_size != mbox->mbox_dirty_size;
> 
>    While when it does not work, mbox->mbox_dirty_stamp equals
>    st->st_mtime, and it returns 0.  In both cases, hdr->sync_stamp is
>    different from st->st_mtime.
> 
>    (The same applies to the sizes as well, that is, they are different
>    where the timestamps are different and equal where the timestamps are
>    equal).
> 
> 3. A more interesting revalation:  It is not only that EXISTS is not
>    sent, and not only to do with IDLE.  When new mail arrives, even when
>    Thunderbird issues a SELECT (by doing a check for new mail), it does
>    not see the new messages.
> 
> 4. Furthermore:  If the last message seen from within Thunderbird was
>    not yet read, and the problem occurs (new mail arrives that is not
>    seen), this last message will then seem corrupted from Thunderbird
>    - it will contain at the end some garbage filling, and then part of
>    the next message in the mailbox.
> 
>    This is while the mailbox itself is completely valid, has no such
>    garbage in it, and the messages are validly separated by an empty
>    line and a valid "From " line.  Perhaps this means there's some
>    race-condition bug in the mbox reading/index syncing code?  That's
>    the code I least understand as of now...
> 
>    Attached is an example of one such last message saved from
>    Thunderbird after new mail arrived that was not seen.  The filler
>    characters are bytes of 0x80, and then the next message from
>    "n-Path: " is present.
> 
> Any hints?
> 
>   Cheers,
>   -- Tom

-- 
  Tom Alsberg - hacker (being the best description fitting this space)
  Web page:	http://www.cs.huji.ac.il/~alsbergt/
DISCLAIMER:  The above message does not even necessarily represent what
my fingers have typed on the keyboard, save anything further.