dovecot sieve duplicates detection

[James Cassell]

On Wed, Apr 25, 2018, at 3:20 PM, Stephan Bosch wrote:
> 
> 
> Op 23/04/2018 om 22:03 schreef André Rodier:
> > On 23/04/18 14:18, Stephan Bosch wrote:
> >>
> >>
> >> Op 11-4-2018 om 23:58 schreef André Rodier:
> >>> Hello,
> >>>
> >>> I have tested the sieve duplicate script with success so far, but I 
> >>> have
> >>> a question.
> >>
> >> Sieve duplicate script? You mean the Sieve duplicate extension (RFC 
> >> 7352)?
> >>
> >>> I would like to know if the "duplicate" sieve flag in Dovecot is global
> >>> to all folders, or specific to one folder only.
> >>
> >> It uses the lda-dupes file in the user's home directory. So, it is 
> >> not normally related to folders, although the identifier used for 
> >> duplicate matching could be composed of the mailbox name if you want.
> >>
> >>> For instance, if I copy an email from one folder to another, and I have
> >>> a discard action on duplicate email, is this action will be applied (in
> >>> this case, discard) or not.
> >>
> >> Are you talking about IMAPSieve now? I am not sure "duplicate" is 
> >> currently even allowed in that context.
> >>
> >>> If the duplicate is global to all folders, is there a way to restrict
> >>> the search in one folder only.
> >>
> >> You can set the :uniqueid parameter accordingly.
> >>
> >> Regards,
> >>
> >> Stephan.
> >
> > Thank you, Stephan.
> >
> > Yes, I meant the Sieve duplicate extension.
> >
> > I am using a program to import email (mbsync), which use the IMAP 
> > append function. Sometimes, the import fail and I have to restart the 
> > program. Unfortunately, the same emails are imported again.
> >
> > I found a fix by using a dovecot IMAP sieve script executed on the 
> > APPEND action 
> > (https://wiki.dovecot.org/Pigeonhole/Sieve/Plugins/IMAPSieve). I wrote 
> > a custom sieve script that "discard" the ones that are detected as 
> > "duplicate". It worked very well and the emails were not any more 
> > imported twice.
> >
> > However, there was a huge side effect: archiving an email with 
> > Thunderbird is not working any more, and even lost! I have been able 
> > to understand the error as this:
> >
> > 1. When archiving an email with Thunderbird, it is first copied 
> > (APPEND) into the archive folder, but the original folder is not 
> > expunged.
> > 2. The sieve script detect the email as duplicate, and discard it.
> > 3. When the original folder is expunged, the source email is lost...
> >
> > My conclusion was the duplicate detection function is global to all 
> > folders.
> >
> > If I could restrict the detection of duplicates in the current folder 
> > only, this would let me run the import program again without error.
> 
> Specify the ID used for duplicate checking explicitly using the 
> :uniqueid argument (https://tools.ietf.org/html/rfc7352#section-3.1). 
> Using the variables extenion, compose the uniqueid from the message-id 
> and the mailbox name.
> 

In my experience with dovecot's implementation, you can set the ID only once in a script.  If you try to filter duplicates based on multiple IDs, only the first (or last, I don't remember) takes effect.

V/r,
James Cassell