[Dovecot] phantom new mail (mailboxes keep on being reported as having new mail)
Hi, I'm using dovecot 1.0_beta3 on Gentoo Linux, and am experiencing a problem of what I call 'phantom mail'. This problem I have for a long time (pre 1.0), but at the moment I'm using 1.0_beta3 and the problem still is in there. My setup is as follows: - regular x86 box - running Exim and Dovecot - Exim saves new mail in maildir format in a local filesystem /mail/user/blabla - Dovecot reads and serves IMAP from local filesystem /mail/user/blabla - client is Mutt and/or Thunderbird from any OS (Mac, Linux, Windows) via IMAP Situation that happens: I get new mail in a folder, say =news (in Mutt style, it's .news dir in the maildir directory). I read the item, Mutt (or Thunderbird) marks the item as read (moving it over to cur aswell I think), save the folder, then move on to another folder. At this point dovecot IMAP server indicates to the client that there is new mail in the folder =news. If I visit the folder with the client, I see no new mail, and sometimes I have two or three of those folders, and I can keep on cycling with Mutt till I get bored :). Before upgrading Dovecot, I knew of no way to resolve this problem; eventually, it seemed that at some point the folder seemed to figure out itself that there was indeed no new mail. Sometimes I could trick it by letting Mutt mark the mail as new again, and then save the folder. Didn't always work. Now, I think a week or two ago I upgraded Dovecot and had to manually move all the .subscription files, and I also in one go decided to remove the dovecot* and .imap* files. That cleaned up, and at first that looked like it solved my problem. Not. So I figured out throwing away dovecot's index stuff (rm dovecot* in the directory that shows phantom mail) resolves the problem. However, this method starts to get annoying so to say. So, I did some investigation and found out that the dovecot-uidlist file seems to be not up-to-date on a folder that shows phantom mails: # awk '{ print $2 }' dovecot-uidlist | sort > tmp.uids # ls cur | sort | diff -u - tmp.uids --- - 2006-03-08 17:45:26.428168000 +0100 +++ tmp.uids 2006-03-08 17:45:19.000000000 +0100 @@ -47,4 +47,5 @@ 1141340181.H111425P8973.hermes.orakel.ods.org:2,S 1141387968.P13010Q0M35974.hermes:2,S 1141528391.H769531P12886.hermes.orakel.ods.org:2,S -1141830301.H349821P23427.hermes.orakel.ods.org:2,S +1141544380 +1141830301.H349821P23427.hermes.orakel.ods.org It seems the dovecot uidlist doesn't have the right name for the file on disk, probably therefore "seeing" a new file all the time and reporting the folder as having new mail. The 1141544380 entry comes from the first row of the dovecot-uidlist file: 1 1141544380 51 I guess it's just some sort of line to keep track of the last timestamp and the number of messages or something. If my theory is right here then it looks as if the 'bad entry' is actually more recent than this most recent timestamp reports. But this is only speculation of course. Enough for this lenghty mail I guess. Is this a known issue, or can it be caused by the way I set up my system? I must note that this only happens for some folders, not all of them. I have folders which get a lot of mail, but don't have this happening on. Any clues go resolve this are welcome. Regards, -- Fabian Groffen
This problem gets rather annoying. Isn't there anything that I can try to resolve this? It dovecot can't work so well with its indices, is there an option to disable them? I keep on deleting those files to get rid of phantom new mail. On 09-03-2006 20:21:26 +0100, Grobian wrote:
Hi,
I'm using dovecot 1.0_beta3 on Gentoo Linux, and am experiencing a problem of what I call 'phantom mail'. This problem I have for a long time (pre 1.0), but at the moment I'm using 1.0_beta3 and the problem still is in there.
My setup is as follows: - regular x86 box - running Exim and Dovecot - Exim saves new mail in maildir format in a local filesystem /mail/user/blabla - Dovecot reads and serves IMAP from local filesystem /mail/user/blabla - client is Mutt and/or Thunderbird from any OS (Mac, Linux, Windows) via IMAP
Situation that happens: I get new mail in a folder, say =news (in Mutt style, it's .news dir in the maildir directory). I read the item, Mutt (or Thunderbird) marks the item as read (moving it over to cur aswell I think), save the folder, then move on to another folder. At this point dovecot IMAP server indicates to the client that there is new mail in the folder =news. If I visit the folder with the client, I see no new mail, and sometimes I have two or three of those folders, and I can keep on cycling with Mutt till I get bored :). Before upgrading Dovecot, I knew of no way to resolve this problem; eventually, it seemed that at some point the folder seemed to figure out itself that there was indeed no new mail. Sometimes I could trick it by letting Mutt mark the mail as new again, and then save the folder. Didn't always work. Now, I think a week or two ago I upgraded Dovecot and had to manually move all the .subscription files, and I also in one go decided to remove the dovecot* and .imap* files. That cleaned up, and at first that looked like it solved my problem. Not. So I figured out throwing away dovecot's index stuff (rm dovecot* in the directory that shows phantom mail) resolves the problem. However, this method starts to get annoying so to say.
So, I did some investigation and found out that the dovecot-uidlist file seems to be not up-to-date on a folder that shows phantom mails:
# awk '{ print $2 }' dovecot-uidlist | sort > tmp.uids # ls cur | sort | diff -u - tmp.uids --- - 2006-03-08 17:45:26.428168000 +0100 +++ tmp.uids 2006-03-08 17:45:19.000000000 +0100 @@ -47,4 +47,5 @@ 1141340181.H111425P8973.hermes.orakel.ods.org:2,S 1141387968.P13010Q0M35974.hermes:2,S 1141528391.H769531P12886.hermes.orakel.ods.org:2,S -1141830301.H349821P23427.hermes.orakel.ods.org:2,S +1141544380 +1141830301.H349821P23427.hermes.orakel.ods.org
It seems the dovecot uidlist doesn't have the right name for the file on disk, probably therefore "seeing" a new file all the time and reporting the folder as having new mail. The 1141544380 entry comes from the first row of the dovecot-uidlist file: 1 1141544380 51 I guess it's just some sort of line to keep track of the last timestamp and the number of messages or something. If my theory is right here then it looks as if the 'bad entry' is actually more recent than this most recent timestamp reports. But this is only speculation of course.
Enough for this lenghty mail I guess. Is this a known issue, or can it be caused by the way I set up my system? I must note that this only happens for some folders, not all of them. I have folders which get a lot of mail, but don't have this happening on. Any clues go resolve this are welcome.
Regards,
-- Fabian Groffen Gentoo for Mac OS X Project
On Wed, 2006-03-29 at 20:41 +0200, Grobian wrote:
This problem gets rather annoying. Isn't there anything that I can try to resolve this? It dovecot can't work so well with its indices, is there an option to disable them? I keep on deleting those files to get rid of phantom new mail.
On 09-03-2006 20:21:26 +0100, Grobian wrote:
Hi,
I'm using dovecot 1.0_beta3 on Gentoo Linux, and am experiencing a problem of what I call 'phantom mail'. This problem I have for a long time (pre 1.0), but at the moment I'm using 1.0_beta3 and the problem still is in there.
My setup is as follows:
- regular x86 box
- running Exim and Dovecot
- Exim saves new mail in maildir format in a local filesystem /mail/user/blabla
- Dovecot reads and serves IMAP from local filesystem /mail/user/blabla
- client is Mutt and/or Thunderbird from any OS (Mac, Linux, Windows) via IMAP
FTR, I can confirm this with Fedora Core 4/i386 and 5/i386, sendmail, mbox, and Evolution 2.2.3 and 2.6.0. Backing down to 1.0 beta 2 seemed to stop it, but we'll see for how long...
-- Ignacio Vazquez-Abrams ivazquez@ivazquez.net http://fedora.ivazquez.net/
gpg --keyserver hkp://subkeys.pgp.net --recv-key 38028b72
On Wed, Mar 29, 2006 at 14:02:53 -0500, Ignacio Vazquez-Abrams wrote:
On Wed, 2006-03-29 at 20:41 +0200, Grobian wrote:
This problem gets rather annoying. Isn't there anything that I can try to resolve this? It dovecot can't work so well with its indices, is there an option to disable them? I keep on deleting those files to get rid of phantom new mail.
On 09-03-2006 20:21:26 +0100, Grobian wrote:
Hi,
I'm using dovecot 1.0_beta3 on Gentoo Linux, and am experiencing a problem of what I call 'phantom mail'. This problem I have for a long time (pre 1.0), but at the moment I'm using 1.0_beta3 and the problem still is in there.
My setup is as follows:
- regular x86 box
- running Exim and Dovecot
- Exim saves new mail in maildir format in a local filesystem /mail/user/blabla
- Dovecot reads and serves IMAP from local filesystem /mail/user/blabla
- client is Mutt and/or Thunderbird from any OS (Mac, Linux, Windows) via IMAP
FTR, I can confirm this with Fedora Core 4/i386 and 5/i386, sendmail, mbox, and Evolution 2.2.3 and 2.6.0. Backing down to 1.0 beta 2 seemed to stop it, but we'll see for how long...
Hello, I can also confirm this Problem with Dovecot1.0b3, Postfix, procmail and maildir on a local filesystem (XFS) using Gentoo Linux. Clients are Thunderbird, mutt and Apple Mail.
Thanks, Chris Borsbach
On Thu, 2006-03-30 at 08:03 +0200, Christoph Borsbach wrote:
On Wed, Mar 29, 2006 at 14:02:53 -0500, Ignacio Vazquez-Abrams wrote:
On Wed, 2006-03-29 at 20:41 +0200, Grobian wrote:
This problem gets rather annoying. Isn't there anything that I can try to resolve this? It dovecot can't work so well with its indices, is there an option to disable them? I keep on deleting those files to get rid of phantom new mail. .. I can also confirm this Problem with Dovecot1.0b3, Postfix, procmail and maildir on a local filesystem (XFS) using Gentoo Linux. Clients are Thunderbird, mutt and Apple Mail.
What exactly does this mean with Thunderbird and Apple Mail? As far as I know they have no "there's no mail" announcements like mutt, but only show the number of mails in mailboxes?
On 02-04-2006 20:16:06 +0300, Timo Sirainen wrote:
On Thu, 2006-03-30 at 08:03 +0200, Christoph Borsbach wrote:
I can also confirm this Problem with Dovecot1.0b3, Postfix, procmail and maildir on a local filesystem (XFS) using Gentoo Linux. Clients are Thunderbird, mutt and Apple Mail.
What exactly does this mean with Thunderbird and Apple Mail? As far as I know they have no "there's no mail" announcements like mutt, but only show the number of mails in mailboxes?
In Thunderbird, a folder shows up in blue when there is new mail in it. Then when you click it, new mail shows up as bold. When this happens with Thunderbird, the blue folder name just goes black again once you click it and no new mail is in the folder. Upon Thunderbird's next automatic mail check, it makes the folder blue again because it thinks new mail is in there, like Mutt says "new mail in =folder".
Hope that helps.
-- Fabian Groffen Gentoo for Mac OS X Project
On Thu, 2006-03-09 at 20:21 +0100, Grobian wrote:
I'm using dovecot 1.0_beta3 on Gentoo Linux, and am experiencing a problem of what I call 'phantom mail'. This problem I have for a long time (pre 1.0), but at the moment I'm using 1.0_beta3 and the problem still is in there.
The problem was that STATUS command synchronized the index and maildir, so that the next time mailbox was opened Dovecot thought the new/ directory was already synchronized so it didn't bother looking into it. But that also meant that it didn't move the mails to cur/ and their recent flag state was kept.. Fixed now in CVS, and upcoming beta4.
# awk '{ print $2 }' dovecot-uidlist | sort > tmp.uids # ls cur | sort | diff -u - tmp.uids --- - 2006-03-08 17:45:26.428168000 +0100 +++ tmp.uids 2006-03-08 17:45:19.000000000 +0100 @@ -47,4 +47,5 @@ 1141340181.H111425P8973.hermes.orakel.ods.org:2,S 1141387968.P13010Q0M35974.hermes:2,S 1141528391.H769531P12886.hermes.orakel.ods.org:2,S -1141830301.H349821P23427.hermes.orakel.ods.org:2,S +1141544380 +1141830301.H349821P23427.hermes.orakel.ods.org
It seems the dovecot uidlist doesn't have the right name for the file on disk,
This is done on purpose. The filenames keep changing and only the base filename is important. Dovecot just tries to optimize file opening a bit by keeping the last known full filename in the dovecot-uidlist file.
On 02-04-2006 20:36:08 +0300, Timo Sirainen wrote:
On Thu, 2006-03-09 at 20:21 +0100, Grobian wrote:
I'm using dovecot 1.0_beta3 on Gentoo Linux, and am experiencing a problem of what I call 'phantom mail'. This problem I have for a long time (pre 1.0), but at the moment I'm using 1.0_beta3 and the problem still is in there.
The problem was that STATUS command synchronized the index and maildir, so that the next time mailbox was opened Dovecot thought the new/ directory was already synchronized so it didn't bother looking into it. But that also meant that it didn't move the mails to cur/ and their recent flag state was kept..
Fixed now in CVS, and upcoming beta4.
Ok, cool and thanks! Is beta4 long away? Otherwise I might try a nightly/CVS build to see if it really fixes the problem for me.
[snip]
-- Fabian Groffen Gentoo for Mac OS X Project
On Sun, Apr 02, 2006 at 19:36:08 +0300, Timo Sirainen wrote:
On Thu, 2006-03-09 at 20:21 +0100, Grobian wrote:
I'm using dovecot 1.0_beta3 on Gentoo Linux, and am experiencing a problem of what I call 'phantom mail'. This problem I have for a long time (pre 1.0), but at the moment I'm using 1.0_beta3 and the problem still is in there.
The problem was that STATUS command synchronized the index and maildir, so that the next time mailbox was opened Dovecot thought the new/ directory was already synchronized so it didn't bother looking into it. But that also meant that it didn't move the mails to cur/ and their recent flag state was kept..
Fixed now in CVS, and upcoming beta4.
Thanks a lot, now (..that i've managed to upgrade to beta5) it works very well!
participants (4)
-
Christoph Borsbach
-
Grobian
-
Ignacio Vazquez-Abrams
-
Timo Sirainen