[Dovecot] phantom new mail (mailboxes keep on being reported as having new mail)

[Grobian]

Hi,

I'm using dovecot 1.0_beta3 on Gentoo Linux, and am experiencing a
problem of what I call 'phantom mail'.  This problem I have for a long
time (pre 1.0), but at the moment I'm using 1.0_beta3 and the problem
still is in there.

My setup is as follows:
- regular x86 box
- running Exim and Dovecot
- Exim saves new mail in maildir format in a local filesystem
  /mail/user/blabla
- Dovecot reads and serves IMAP from local filesystem /mail/user/blabla
- client is Mutt and/or Thunderbird from any OS (Mac, Linux, Windows)
  via IMAP

Situation that happens:
I get new mail in a folder, say =news (in Mutt style, it's .news dir in
the maildir directory).  I read the item, Mutt (or Thunderbird) marks
the item as read (moving it over to cur aswell I think), save the
folder, then move on to another folder.  At this point dovecot IMAP
server indicates to the client that there is new mail in the folder
=news.  If I visit the folder with the client, I see no new mail, and
sometimes I have two or three of those folders, and I can keep on
cycling with Mutt till I get bored :).
Before upgrading Dovecot, I knew of no way to resolve this problem;
eventually, it seemed that at some point the folder seemed to figure out
itself that there was indeed no new mail.  Sometimes I could trick it by
letting Mutt mark the mail as new again, and then save the folder.
Didn't always work.
Now, I think a week or two ago I upgraded Dovecot and had to manually
move all the .subscription files, and I also in one go decided to remove
the dovecot* and .imap* files.  That cleaned up, and at first that
looked like it solved my problem.  Not.  So I figured out throwing away
dovecot's index stuff (rm dovecot* in the directory that shows phantom
mail) resolves the problem.  However, this method starts to get annoying
so to say.

So, I did some investigation and found out that the dovecot-uidlist file
seems to be not up-to-date on a folder that shows phantom mails:

# awk '{ print $2 }' dovecot-uidlist | sort > tmp.uids
# ls cur | sort | diff -u - tmp.uids
--- -   2006-03-08 17:45:26.428168000 +0100
+++ tmp.uids    2006-03-08 17:45:19.000000000 +0100
@@ -47,4 +47,5 @@
 1141340181.H111425P8973.hermes.orakel.ods.org:2,S
 1141387968.P13010Q0M35974.hermes:2,S
 1141528391.H769531P12886.hermes.orakel.ods.org:2,S
-1141830301.H349821P23427.hermes.orakel.ods.org:2,S
+1141544380
+1141830301.H349821P23427.hermes.orakel.ods.org

It seems the dovecot uidlist doesn't have the right name for the file on
disk, probably therefore "seeing" a new file all the time and reporting
the folder as having new mail.  The 1141544380 entry comes from the
first row of the dovecot-uidlist file:
1 1141544380 51
I guess it's just some sort of line to keep track of the last timestamp
and the number of messages or something.  If my theory is right here
then it looks as if the 'bad entry' is actually more recent than this
most recent timestamp reports.  But this is only speculation of course.

Enough for this lenghty mail I guess.  Is this a known issue, or can it
be caused by the way I set up my system?  I must note that this only
happens for some folders, not all of them.  I have folders which get a
lot of mail, but don't have this happening on.  Any clues go resolve
this are welcome.


Regards,

-- 
Fabian Groffen