On Thursday 31 January 2008 15:31, Bill Cole wrote:
At 12:07 PM +0000 1/31/08, Anne Wilson wrote:
On Wednesday 30 January 2008 23:16, Benjamin R. Haskell wrote:
X-UID: 2041 X-KMail-Filtered: 65016 Status: RO X-Status: OAC
and
X-UID: 2042 X-KMail-Filtered: 65017 Status: RO X-Status: OAC
Have you changed anything recently with your kmail filters?
KMail filters generally don't work on imap.
That's a tellingly vague statement. Do you mean that your KMail filters don't do what you expect with IMAP, that you believe you have filtering turned off for IMAP, that they are widely acknowledged to be broken for IMAP, that KMail claims to not be able to filter IMAP mail, or something else?
KMail for 3.5.x has not had a lot of development for some time, as effort goes
into kde 4.x. KMail specifically warns that filtering does not work on imap
directories. An older version used to work on anything that came into the
imap Inbox, but presumably that caused some problems, as it was taken out.
KMail 1.9.7 in the version I have doesn't work at all on any imap directory.
All my filtering is done in procmail.
Do you have procmail adding a X-KMail-Filtered header? That seems unlikely...
Agreed
Nothing other than KMail will add that header to a message unless told to do so explicitly. Of course you can make procmail do anything you want it to do to a piece of mail, but having any other program add a header that is only meaningful to KMail seems like a bad idea.
Are you running more than one mailserver? Dovecot doesn't add the X-UID field, so I suspect it wouldn't be changing it, either.
Temporarily, yes. I have just set up a new server, which is now carrying all the traffic. I have been leaving the old server readable while I check everything out. I will probably turn dovecot off on it this weekend.
I think the question might have been aimed at the concept of KMail (or whatever is adding the X-UID header) playing ping-pong with a message between mailboxes on different machines, but maybe I'm misunderstanding that...
I assumed also that that was the sort of thing implied. I've turned off the other server, but I don't think that was the problem. Of course it could be some time before another message is delivered direct to Inbox, so I can't be certain.
As for the headers, yes, I would suspect that the X-UID could well be added by kmail. However, multiple copies of the messages are appearing in ~/Maildir/cur, and surely only dovecot could be putting them there?
Only Dovecot should be doing the specific task of moving message files between the tmp/new/cur subdirectories, i.e. of actually creating new files in ~/Maildir/cur, but Dovecot will do that because of what an IMAP client tells it to do. For a client to add a header to an existing message for its own use, it constructs a new version of the message with the added header and tell Dovecot to delete the old message after adding the new version as a new message. Implementing this incorrectly in an IMAP client could easily result in multiple copies of a message in a mailbox, and even implementing it correctly can leave multiple versions of the same message in the maildir, depending on how the client and server handle the concepts of deleting and expunging.
That explains a lot. FWIW, I'm currently working on the CentOS box that is the new dovecot server, and which runs kmail 1.9.4. That recognised the multiple copies as duplicates, and happily removed them with the 'Remove Duplicate' option from the menus.
My guess is that you have something broken in KMail which is filtering messages in a loop. Whether that's an inherent KMail problem or a configuration problem (i.e. a pathological filter) I can't say. You may find it useful to set up raw IMAP logging on Dovecot to see what exactly is happening, or if you are using unencrypted connections you could use tcpdump or wireshark to capture the IMAP sessions. The goal of either would be to figure out what KMail is actually doing, since it seems certain that an IMAP client is responsible for the existence of messages whose only differences are in headers that only an IMAP client would manipulate, including one that is clearly KMail's doing.
Incidentally, a very small amount of Google-work turns up multiple descriptions of KMail's involvement with the creation of duplicate messages on IMAP servers. It seems from a fast skim that it may be problematic to have anything other than KMail filtering delivered mail....
It's probably never a good idea to have two applications doing the same thing.
However, your explanations made me look again at my various installations. I
had been concentrating on this, the server, and the old server box. I hadn't
looked closely at the laptop. Now I realise that the problem started at
around the time that I brought that laptop into the system. On there, kmail
had some old filters set up that I had completely forgotten about. Knowing
that filtering was not supposed to work on imap directories, I simply hadn't
checked that. I have removed them.
I'll clean out any remaining duplicates, then see what happens over the next few days. I'll let you know.
Thanks again for the detailed reply. I suspect that we may have hit gold this time.
Anne