[Dovecot] Multiple copies of a message

[Anne Wilson]

On Thursday 31 January 2008 15:31, Bill Cole wrote:
> At 12:07 PM +0000 1/31/08, Anne Wilson wrote:
> >On Wednesday 30 January 2008 23:16, Benjamin R. Haskell wrote:
> >>  > X-UID: 2041
> >>  > X-KMail-Filtered: 65016
> >>  > Status: RO
> >>  > X-Status: OAC
> >>  >
> >>  > and
> >>  >
> >>  > X-UID: 2042
> >>  > X-KMail-Filtered: 65017
> >>  > Status: RO
> >>  > X-Status: OAC
> >>
> >>  Have you changed anything recently with your kmail filters?
> >
> >KMail filters generally don't work on imap.
>
> That's a tellingly vague statement. Do you mean that your KMail
> filters don't do what you expect with IMAP, that you believe you have
> filtering turned off for IMAP, that they are widely acknowledged to
> be broken for IMAP, that KMail claims to not be able to filter IMAP
> mail, or something else?
>
KMail for 3.5.x has not had a lot of development for some time, as effort goes 
into kde 4.x.  KMail specifically warns that filtering does not work on imap 
directories.  An older version used to work on anything that came into the 
imap Inbox, but presumably that caused some problems, as it was taken out.  
KMail 1.9.7 in the version I have doesn't work at all on any imap directory.

> >  All my filtering is done in
> >procmail.
>
> Do you have procmail adding a X-KMail-Filtered header? That seems
> unlikely...
>
Agreed

> Nothing other than KMail will add that header to a message unless
> told to do so explicitly. Of course you can make procmail do anything
> you want it to do to a piece of mail, but having any other program
> add a header that is only meaningful to KMail seems like a bad idea.
>
> >>  Are you
> >>  running more than one mailserver? Dovecot doesn't add the X-UID field,
> >> so I suspect it wouldn't be changing it, either.
> >
> >Temporarily, yes.  I have just set up a new server, which is now carrying
> > all the traffic.  I have been leaving the old server readable while I
> > check everything out.  I will probably turn dovecot off on it this
> > weekend.
>
> I think the question might have been aimed at the concept of KMail
> (or whatever is adding the X-UID header) playing ping-pong with a
> message between mailboxes on different machines, but maybe I'm
> misunderstanding that...
>
I assumed also that that was the sort of thing implied.  I've turned off the 
other server, but I don't think that was the problem.  Of course it could be 
some time before another message is delivered direct to Inbox, so I can't be 
certain.

> >As for the headers, yes, I would suspect that the X-UID could well be
> > added by kmail.  However, multiple copies of the messages are appearing
> > in ~/Maildir/cur, and surely only dovecot could be putting them there?
>
> Only Dovecot should be doing the specific task of moving message
> files between the tmp/new/cur subdirectories, i.e. of actually
> creating new files in ~/Maildir/cur, but Dovecot will do that because
> of what an IMAP client tells it to do. For a client to add a header
> to an existing message for its own use, it constructs a new version
> of the message with the added header and tell Dovecot to delete the
> old message after adding the new version as a new message.
> Implementing this incorrectly in an IMAP client could easily result
> in  multiple copies of a message in a mailbox, and even implementing
> it correctly can leave multiple versions of the same message in the
> maildir, depending on how the client and server handle the concepts
> of deleting and expunging.
>
That explains a lot.  FWIW, I'm currently working on the CentOS box that is 
the new dovecot server, and which runs kmail 1.9.4.  That recognised the 
multiple copies as duplicates, and happily removed them with the 'Remove 
Duplicate' option from the menus.

> My guess is that you have something broken in KMail which is
> filtering messages in a loop. Whether that's an inherent  KMail
> problem or a configuration problem (i.e. a pathological filter) I
> can't say.  You may find it useful to set up raw IMAP logging on
> Dovecot to see what exactly is happening, or if you are using
> unencrypted connections you could use tcpdump or wireshark to capture
> the IMAP sessions. The goal of either would be to figure out what
> KMail is actually doing, since it seems certain that an IMAP client
> is responsible for the existence of messages whose only differences
> are in headers that only an IMAP client would manipulate, including
> one that is clearly KMail's doing.
>
> Incidentally, a very small amount of Google-work turns up multiple
> descriptions of KMail's involvement with the creation of duplicate
> messages on IMAP servers. It seems from a fast skim that it may be
> problematic to have anything other than KMail filtering delivered
> mail....

It's probably never a good idea to have two applications doing the same thing.  
However, your explanations made me look again at my various installations.  I 
had been concentrating on this, the server, and the old server box.  I hadn't 
looked closely at the laptop.  Now I realise that the problem started at 
around the time that I brought that laptop into the system.  On there, kmail 
had some old filters set up that I had completely forgotten about.  Knowing 
that filtering was not supposed to work on imap directories, I simply hadn't 
checked that.  I have removed them.

I'll clean out any remaining duplicates, then see what happens over the next 
few days.  I'll let you know.

Thanks again for the detailed reply.  I suspect that we may have hit gold this 
time.

Anne