[Dovecot] Multiple copies of a message
I have a problem and I don't know which part of the mail chain could be causing it, although dovecot does seem to be the most logical source.
I get multiple copies of messages to my Inbox. I have just deleted 24 copies of a newsletter from Computer Shopper. The worst example I have seen of this had more than 90 copies. This doesn't happen to any subfolder of Inbox, just Inbox itself.
The problem started about 10-14 days ago, on the old mail server (running Mandriva 2007.1). The new server runs CentOS and exhibits the same problem.
I'd be interested in any comments, as my efforts to track it down have got nowhere.
Anne
On Tue, 29 Jan 2008, Anne Wilson wrote:
I have a problem and I don't know which part of the mail chain could be causing it, although dovecot does seem to be the most logical source.
I get multiple copies of messages to my Inbox. I have just deleted 24 copies of a newsletter from Computer Shopper. The worst example I have seen of this had more than 90 copies. This doesn't happen to any subfolder of Inbox, just Inbox itself.
Does your SMTP server's logs indicate receiving it multiple times?
Me, I use a procmail recipe that removes duplicates, so I don't see duplicate messages if they have the same Message ID. Also, what do the headers of these "duplicates" suggest - are they fully identical?.
-- Asheesh.
-- I've run DOOM more in the last few days than I have the last few months. I just love debugging ;-) (Linus Torvalds)
On Tuesday 29 Jan 2008, you wrote:
On Tue, 29 Jan 2008, Anne Wilson wrote:
I have a problem and I don't know which part of the mail chain could be causing it, although dovecot does seem to be the most logical source.
I get multiple copies of messages to my Inbox. I have just deleted 24 copies of a newsletter from Computer Shopper. The worst example I have seen of this had more than 90 copies. This doesn't happen to any subfolder of Inbox, just Inbox itself.
Does your SMTP server's logs indicate receiving it multiple times?
I don't have one of the big ones still on my disk, but I have a couple where I have allowed two copies to remain. /var/log/maillog shows just one copy of each of them.
Me, I use a procmail recipe that removes duplicates, so I don't see duplicate messages if they have the same Message ID. Also, what do the headers of these "duplicates" suggest - are they fully identical?.
They are identical until the last lines, which read
X-UID: 2039 X-KMail-Filtered: 65014 Status: RO X-Status: OC
and
X-UID: 2040 X-KMail-Filtered: 65015 Status: RO X-Status: OC
Similarly,
X-UID: 2041 X-KMail-Filtered: 65016 Status: RO X-Status: OAC
and
X-UID: 2042 X-KMail-Filtered: 65017 Status: RO X-Status: OAC
Anne
On Wed, 30 Jan 2008, Anne Wilson wrote:
On Tuesday 29 Jan 2008, Asheesh Laroia wrote:
On Tue, 29 Jan 2008, Anne Wilson wrote:
I have a problem and I don't know which part of the mail chain could be causing it, although dovecot does seem to be the most logical source.
I get multiple copies of messages to my Inbox. I have just deleted 24 copies of a newsletter from Computer Shopper. The worst example I have seen of this had more than 90 copies. This doesn't happen to any subfolder of Inbox, just Inbox itself.
Does your SMTP server's logs indicate receiving it multiple times?
I don't have one of the big ones still on my disk, but I have a couple where I have allowed two copies to remain. /var/log/maillog shows just one copy of each of them.
Me, I use a procmail recipe that removes duplicates, so I don't see duplicate messages if they have the same Message ID. Also, what do the headers of these "duplicates" suggest - are they fully identical?.
They are identical until the last lines, which read
X-UID: 2039 X-KMail-Filtered: 65014 Status: RO X-Status: OC
and
X-UID: 2040 X-KMail-Filtered: 65015 Status: RO X-Status: OC
Similarly,
X-UID: 2041 X-KMail-Filtered: 65016 Status: RO X-Status: OAC
and
X-UID: 2042 X-KMail-Filtered: 65017 Status: RO X-Status: OAC
Have you changed anything recently with your kmail filters? Are you running more than one mailserver? Dovecot doesn't add the X-UID field, so I suspect it wouldn't be changing it, either.
Best, Ben
On Wednesday 30 January 2008 23:16, Benjamin R. Haskell wrote:
X-UID: 2041 X-KMail-Filtered: 65016 Status: RO X-Status: OAC
and
X-UID: 2042 X-KMail-Filtered: 65017 Status: RO X-Status: OAC
Have you changed anything recently with your kmail filters?
KMail filters generally don't work on imap. All my filtering is done in procmail.
Are you running more than one mailserver? Dovecot doesn't add the X-UID field, so I suspect it wouldn't be changing it, either.
Temporarily, yes. I have just set up a new server, which is now carrying all the traffic. I have been leaving the old server readable while I check everything out. I will probably turn dovecot off on it this weekend.
As for the headers, yes, I would suspect that the X-UID could well be added by kmail. However, multiple copies of the messages are appearing in ~/Maildir/cur, and surely only dovecot could be putting them there?
Anne
At 12:07 PM +0000 1/31/08, Anne Wilson wrote:
On Wednesday 30 January 2008 23:16, Benjamin R. Haskell wrote:
X-UID: 2041 X-KMail-Filtered: 65016 Status: RO X-Status: OAC
and
X-UID: 2042 X-KMail-Filtered: 65017 Status: RO X-Status: OAC
Have you changed anything recently with your kmail filters?
KMail filters generally don't work on imap.
That's a tellingly vague statement. Do you mean that your KMail filters don't do what you expect with IMAP, that you believe you have filtering turned off for IMAP, that they are widely acknowledged to be broken for IMAP, that KMail claims to not be able to filter IMAP mail, or something else?
All my filtering is done in procmail.
Do you have procmail adding a X-KMail-Filtered header? That seems unlikely...
Nothing other than KMail will add that header to a message unless told to do so explicitly. Of course you can make procmail do anything you want it to do to a piece of mail, but having any other program add a header that is only meaningful to KMail seems like a bad idea.
Are you running more than one mailserver? Dovecot doesn't add the X-UID field, so I suspect it wouldn't be changing it, either.
Temporarily, yes. I have just set up a new server, which is now carrying all the traffic. I have been leaving the old server readable while I check everything out. I will probably turn dovecot off on it this weekend.
I think the question might have been aimed at the concept of KMail (or whatever is adding the X-UID header) playing ping-pong with a message between mailboxes on different machines, but maybe I'm misunderstanding that...
As for the headers, yes, I would suspect that the X-UID could well be added by kmail. However, multiple copies of the messages are appearing in ~/Maildir/cur, and surely only dovecot could be putting them there?
Only Dovecot should be doing the specific task of moving message files between the tmp/new/cur subdirectories, i.e. of actually creating new files in ~/Maildir/cur, but Dovecot will do that because of what an IMAP client tells it to do. For a client to add a header to an existing message for its own use, it constructs a new version of the message with the added header and tell Dovecot to delete the old message after adding the new version as a new message. Implementing this incorrectly in an IMAP client could easily result in multiple copies of a message in a mailbox, and even implementing it correctly can leave multiple versions of the same message in the maildir, depending on how the client and server handle the concepts of deleting and expunging.
My guess is that you have something broken in KMail which is filtering messages in a loop. Whether that's an inherent KMail problem or a configuration problem (i.e. a pathological filter) I can't say. You may find it useful to set up raw IMAP logging on Dovecot to see what exactly is happening, or if you are using unencrypted connections you could use tcpdump or wireshark to capture the IMAP sessions. The goal of either would be to figure out what KMail is actually doing, since it seems certain that an IMAP client is responsible for the existence of messages whose only differences are in headers that only an IMAP client would manipulate, including one that is clearly KMail's doing.
Incidentally, a very small amount of Google-work turns up multiple descriptions of KMail's involvement with the creation of duplicate messages on IMAP servers. It seems from a fast skim that it may be problematic to have anything other than KMail filtering delivered mail....
-- Bill Cole bill@scconsult.com
On Thursday 31 January 2008 15:31, Bill Cole wrote:
At 12:07 PM +0000 1/31/08, Anne Wilson wrote:
On Wednesday 30 January 2008 23:16, Benjamin R. Haskell wrote:
X-UID: 2041 X-KMail-Filtered: 65016 Status: RO X-Status: OAC
and
X-UID: 2042 X-KMail-Filtered: 65017 Status: RO X-Status: OAC
Have you changed anything recently with your kmail filters?
KMail filters generally don't work on imap.
That's a tellingly vague statement. Do you mean that your KMail filters don't do what you expect with IMAP, that you believe you have filtering turned off for IMAP, that they are widely acknowledged to be broken for IMAP, that KMail claims to not be able to filter IMAP mail, or something else?
KMail for 3.5.x has not had a lot of development for some time, as effort goes
into kde 4.x. KMail specifically warns that filtering does not work on imap
directories. An older version used to work on anything that came into the
imap Inbox, but presumably that caused some problems, as it was taken out.
KMail 1.9.7 in the version I have doesn't work at all on any imap directory.
All my filtering is done in procmail.
Do you have procmail adding a X-KMail-Filtered header? That seems unlikely...
Agreed
Nothing other than KMail will add that header to a message unless told to do so explicitly. Of course you can make procmail do anything you want it to do to a piece of mail, but having any other program add a header that is only meaningful to KMail seems like a bad idea.
Are you running more than one mailserver? Dovecot doesn't add the X-UID field, so I suspect it wouldn't be changing it, either.
Temporarily, yes. I have just set up a new server, which is now carrying all the traffic. I have been leaving the old server readable while I check everything out. I will probably turn dovecot off on it this weekend.
I think the question might have been aimed at the concept of KMail (or whatever is adding the X-UID header) playing ping-pong with a message between mailboxes on different machines, but maybe I'm misunderstanding that...
I assumed also that that was the sort of thing implied. I've turned off the other server, but I don't think that was the problem. Of course it could be some time before another message is delivered direct to Inbox, so I can't be certain.
As for the headers, yes, I would suspect that the X-UID could well be added by kmail. However, multiple copies of the messages are appearing in ~/Maildir/cur, and surely only dovecot could be putting them there?
Only Dovecot should be doing the specific task of moving message files between the tmp/new/cur subdirectories, i.e. of actually creating new files in ~/Maildir/cur, but Dovecot will do that because of what an IMAP client tells it to do. For a client to add a header to an existing message for its own use, it constructs a new version of the message with the added header and tell Dovecot to delete the old message after adding the new version as a new message. Implementing this incorrectly in an IMAP client could easily result in multiple copies of a message in a mailbox, and even implementing it correctly can leave multiple versions of the same message in the maildir, depending on how the client and server handle the concepts of deleting and expunging.
That explains a lot. FWIW, I'm currently working on the CentOS box that is the new dovecot server, and which runs kmail 1.9.4. That recognised the multiple copies as duplicates, and happily removed them with the 'Remove Duplicate' option from the menus.
My guess is that you have something broken in KMail which is filtering messages in a loop. Whether that's an inherent KMail problem or a configuration problem (i.e. a pathological filter) I can't say. You may find it useful to set up raw IMAP logging on Dovecot to see what exactly is happening, or if you are using unencrypted connections you could use tcpdump or wireshark to capture the IMAP sessions. The goal of either would be to figure out what KMail is actually doing, since it seems certain that an IMAP client is responsible for the existence of messages whose only differences are in headers that only an IMAP client would manipulate, including one that is clearly KMail's doing.
Incidentally, a very small amount of Google-work turns up multiple descriptions of KMail's involvement with the creation of duplicate messages on IMAP servers. It seems from a fast skim that it may be problematic to have anything other than KMail filtering delivered mail....
It's probably never a good idea to have two applications doing the same thing.
However, your explanations made me look again at my various installations. I
had been concentrating on this, the server, and the old server box. I hadn't
looked closely at the laptop. Now I realise that the problem started at
around the time that I brought that laptop into the system. On there, kmail
had some old filters set up that I had completely forgotten about. Knowing
that filtering was not supposed to work on imap directories, I simply hadn't
checked that. I have removed them.
I'll clean out any remaining duplicates, then see what happens over the next few days. I'll let you know.
Thanks again for the detailed reply. I suspect that we may have hit gold this time.
Anne
On Wednesday 30 January 2008 23:16, Benjamin R. Haskell wrote:
Dovecot doesn't add the X-UID field, so I suspect it wouldn't be changing it, either.
It may also be relevant that while I was checking a couple of these message by opening in kwrite I got the message that one of the messages had been deleted by another program. This had not been by any action of mine. Seven more copies of the same message have just arrived.
I'll switch dovecot off on the old server and see if that makes any difference.
Anne
On Wed, 2008-01-30 at 20:20 +0000, Anne Wilson wrote:
X-UID: 2039 X-KMail-Filtered: 65014 Status: RO X-Status: OC
and
X-UID: 2040 X-KMail-Filtered: 65015 Status: RO X-Status: OC
I think this shows pretty clearly that kmail adds those duplicates. I doubt anything else would add unique X-KMail-Filtered headers.
On Tue, 29 Jan 2008, Anne Wilson wrote:
I have a problem and I don't know which part of the mail chain could be causing it, although dovecot does seem to be the most logical source.
I get multiple copies of messages to my Inbox. I have just deleted 24 copies of a newsletter from Computer Shopper. The worst example I have seen of this had more than 90 copies. This doesn't happen to any subfolder of Inbox, just Inbox itself.
The problem started about 10-14 days ago, on the old mail server (running Mandriva 2007.1). The new server runs CentOS and exhibits the same problem.
I'd be interested in any comments, as my efforts to track it down have got nowhere.
Are you using Dovecot's 'deliver' as your LDA (local delivery agent)? Otherwise I'd suspect the problem lies in whatever you're using for your local deliveries, creating some kind of mail loop. Check your .forward/.qmail/.procmail/whatever files. I haven't heard of/seen dovecot duplicating messages unless there are actually multiple copies in your mailbox.
If your LDA isn't already doing so, add a 'Delivered-To' or 'X-Mail-Loop' header at each stage, so you can more easily tell if this is what's happening. And then (I assume you've done this, but it should be said) look at the headers of the duplicated messages to see if they're 1) actually distinct messages [external problem] 2) being delivered multiple times
Best, Ben
participants (5)
-
Anne Wilson
-
Asheesh Laroia
-
Benjamin R. Haskell
-
Bill Cole
-
Timo Sirainen