[Dovecot] duplicates with multiple To/CC and sieve redirect copy
Hi, In one installation we are using sieve "redirect :copy" to create copy of the incoming email (local boss gets copy of its employees mails). There is a problem when the incoming email has multiple To/CC entries with local users. Multiple copy/redirects are created (one for each employee) and there are multiple copies in "boss" account. Each of this emails has the same Message-ID.
Is there a way to detect those duplicates and discard them when saving copies into boss account?
There was a discussion on the list a while ago (Feb 10/11, 2011) mentioning this problem, but with no clear solution.
We are using Dovecot 2.0.15. Mail is delivered via LMTP (MDA/postfix is on separate machine then MAA/dovecot).
-- Adam Szpakowski
On 22.2.2012, at 0.46, Adam Szpakowski wrote:
In one installation we are using sieve "redirect :copy" to create copy of the incoming email (local boss gets copy of its employees mails). There is a problem when the incoming email has multiple To/CC entries with local users. Multiple copy/redirects are created (one for each employee) and there are multiple copies in "boss" account.
Could this instead be implemented by MTA as automatic bcc: boss of all incoming mails?
On 21.02.2012 23:57, Timo Sirainen wrote:
On 22.2.2012, at 0.46, Adam Szpakowski wrote:
In one installation we are using sieve "redirect :copy" to create copy of the incoming email (local boss gets copy of its employees mails). There is a problem when the incoming email has multiple To/CC entries with local users. Multiple copy/redirects are created (one for each employee) and there are multiple copies in "boss" account. Could this instead be implemented by MTA as automatic bcc: boss of all incoming mails? Unfortunately no. The employee/boss structure is quite complicated. Some employees have multiple "bosses", there are also some "project" emails when the structure is even more complicated. We are using dedicated sieve_before script for each user based on %u variable.
Adam Szpakowski
On 22.2.2012, at 1.02, Adam Szpakowski wrote:
On 21.02.2012 23:57, Timo Sirainen wrote:
On 22.2.2012, at 0.46, Adam Szpakowski wrote:
In one installation we are using sieve "redirect :copy" to create copy of the incoming email (local boss gets copy of its employees mails). There is a problem when the incoming email has multiple To/CC entries with local users. Multiple copy/redirects are created (one for each employee) and there are multiple copies in "boss" account. Could this instead be implemented by MTA as automatic bcc: boss of all incoming mails? Unfortunately no. The employee/boss structure is quite complicated. Some employees have multiple "bosses", there are also some "project" emails when the structure is even more complicated. We are using dedicated sieve_before script for each user based on %u variable.
Well, it would be possible to build a doveadm script that deletes the duplicates after delivery, but currently there's no implementation to avoid delivering duplicate Message-IDs in the first place.
I don't really like such a Message-ID-based deduplication feature enabled by default, but something like this could be nice:
fileinto :copy :x-deduplicate "boss";
Anyway, probably not going to be implemented anytime soon.
On 22.02.2012 00:09, Timo Sirainen wrote:
On 22.2.2012, at 1.02, Adam Szpakowski wrote:
On 21.02.2012 23:57, Timo Sirainen wrote:
On 22.2.2012, at 0.46, Adam Szpakowski wrote:
In one installation we are using sieve "redirect :copy" to create copy of the incoming email (local boss gets copy of its employees mails). There is a problem when the incoming email has multiple To/CC entries with local users. Multiple copy/redirects are created (one for each employee) and there are multiple copies in "boss" account. Could this instead be implemented by MTA as automatic bcc: boss of all incoming mails? Unfortunately no. The employee/boss structure is quite complicated. Some employees have multiple "bosses", there are also some "project" emails when the structure is even more complicated. We are using dedicated sieve_before script for each user based on %u variable. Well, it would be possible to build a doveadm script that deletes the duplicates after delivery, but currently there's no implementation to avoid delivering duplicate Message-IDs in the first place.
I don't really like such a Message-ID-based deduplication feature enabled by default, but something like this could be nice:
fileinto :copy :x-deduplicate "boss";
Anyway, probably not going to be implemented anytime soon. Maybe there is a way to use a procmail with something like this:
:0 Wh: msgid.lock | formail -D 8192 .msgid.cache
But is there a safe way to use it together with sieve? Using Pigeonhole Sieve Pipe Plugin?
-- Adam Szpakowski
On 2/22/2012 12:15 AM, Adam Szpakowski wrote:
On 22.02.2012 00:09, Timo Sirainen wrote:
Well, it would be possible to build a doveadm script that deletes the duplicates after delivery, but currently there's no implementation to avoid delivering duplicate Message-IDs in the first place.
I don't really like such a Message-ID-based deduplication feature enabled by default, but something like this could be nice:
fileinto :copy :x-deduplicate "boss";
Anyway, probably not going to be implemented anytime soon. Maybe there is a way to use a procmail with something like this:
:0 Wh: msgid.lock | formail -D 8192 .msgid.cache
But is there a safe way to use it together with sieve? Using Pigeonhole Sieve Pipe Plugin?
There are a few options:
You can use Procmail as primary delivery agent and invoke dovecot-lda/sieve from within Procmail once Procmail has determined that it is not a duplicate.
Invoke procmail from Sieve using the pipe extension (i.e. the other way around). This has the disadvantage that Procmail will have to take care of final delivery, meaning the Dovecot indexes are not updated.
For Pigeonhole v0.3 there is the possibility to "filter" the message through Procmail using the sieve_extprograms plugin, but I haven't actually tested something like that.
I've just created an alternative that implements something similar to the Procmail code you posted above, but from within Sieve itself. It is a custom language extension called vnd.dovecot.duplicate and it adds the "duplicate" test. This test keeps track of which Message-IDs it has seen before in earlier deliveries and yields a true result if the message was seen before, e.g.:
require "vnd.dovecot.duplicate";
if duplicate { discard; }
Read the specification for details ("name" argument is not yet implemented):
http://hg.rename-it.nl/pigeonhole-0.3-sieve-duplicate/raw-file/4b1dbda4d3fc/...
The repository is at: http://hg.rename-it.nl/pigeonhole-0.3-sieve-duplicate
This plugin is only a few hours old, experimental, and largely untested, so test it thoroughly before considering to use this. Read the INSTALL file for compile and installation instructions.
Comments are welcome.
Regards,
Stephan.
On 23.02.2012 02:15, Stephan Bosch wrote:
On 2/22/2012 12:15 AM, Adam Szpakowski wrote:
Maybe there is a way to use a procmail with something like this:
:0 Wh: msgid.lock | formail -D 8192 .msgid.cache
But is there a safe way to use it together with sieve? Using Pigeonhole Sieve Pipe Plugin?
There are a few options:
- You can use Procmail as primary delivery agent and invoke dovecot-lda/sieve from within Procmail once Procmail has determined that it is not a duplicate.
This is great option but if I understand correctly not when Postfix is using LMTP to talk to Dovecot. In this client of mine, the Postfix (MDA) and Dovecot (MAA) are in different physical locations and LMTP is tunnelled via SSH.
- I've just created an alternative that implements something similar to the Procmail code you posted above, but from within Sieve itself. [cut]
Great extension. I'll try and use it on one of the servers as soon as possible. Thanks for the work.
BTW... There is another solution to my problem, which ends up to be implemented in this case. I'm using Postfix to BCC all mails on the domain level to the "archive" account. In this account I've set up all the logic which ends calling "redirect :copy". This solution eliminates copies from multiple To/CC.
-- Adam Szpakowski
Op 2/23/2012 4:37 PM, Adam Szpakowski schreef:
On 23.02.2012 02:15, Stephan Bosch wrote:
- I've just created an alternative that implements something similar to the Procmail code you posted above, but from within Sieve itself. [cut]
Great extension. I'll try and use it on one of the servers as soon as possible. Thanks for the work.
BTW... There is another solution to my problem, which ends up to be implemented in this case. I'm using Postfix to BCC all mails on the domain level to the "archive" account. In this account I've set up all the logic which ends calling "redirect :copy". This solution eliminates copies from multiple To/CC.
Ok, please keep me uprised. Once it is tested a bit more I'll add it to the main Pigeonhole package. This plugin does not add any dependencies, so this is no problem whatsoever.
Regards,
Stephan.
On 2012-02-22 8:15 PM, Stephan Bosch <stephan@rename-it.nl> wrote:
- I've just created an alternative that implements something similar to the Procmail code you posted above, but from within Sieve itself. It is a custom language extension called vnd.dovecot.duplicate and it adds the "duplicate" test. This test keeps track of which Message-IDs it has seen before in earlier deliveries and yields a true result if the message was seen before, e.g.:
require "vnd.dovecot.duplicate";
if duplicate { discard; }
Interesting... I'll have to look into this later to see how it might be leveraged only for deduplicating mailing list messages (suppress duplicate messages when someone is a member of multiple lists that are members of other lists - I use mailman currently).
--
Best regards,
Charles
Op 23/02/2012 om 02:15:48 +0100, schreef Stephan Bosch:
On 2/22/2012 12:15 AM, Adam Szpakowski wrote:
On 22.02.2012 00:09, Timo Sirainen wrote:
Well, it would be possible to build a doveadm script that deletes the duplicates after delivery, but currently there's no implementation to avoid delivering duplicate Message-IDs in the first place.
I don't really like such a Message-ID-based deduplication feature enabled by default, but something like this could be nice:
fileinto :copy :x-deduplicate "boss";
Anyway, probably not going to be implemented anytime soon. Maybe there is a way to use a procmail with something like this:
:0 Wh: msgid.lock | formail -D 8192 .msgid.cache
But is there a safe way to use it together with sieve? Using Pigeonhole Sieve Pipe Plugin?
There are a few options:
You can use Procmail as primary delivery agent and invoke dovecot-lda/sieve from within Procmail once Procmail has determined that it is not a duplicate.
Invoke procmail from Sieve using the pipe extension (i.e. the other way around). This has the disadvantage that Procmail will have to take care of final delivery, meaning the Dovecot indexes are not updated.
For Pigeonhole v0.3 there is the possibility to "filter" the message through Procmail using the sieve_extprograms plugin, but I haven't actually tested something like that.
I've just created an alternative that implements something similar to the Procmail code you posted above, but from within Sieve itself. It is a custom language extension called vnd.dovecot.duplicate and it adds the "duplicate" test. This test keeps track of which Message-IDs it has seen before in earlier deliveries and yields a true result if the message was seen before, e.g.:
require "vnd.dovecot.duplicate";
if duplicate { discard; }
Read the specification for details ("name" argument is not yet implemented):
http://hg.rename-it.nl/pigeonhole-0.3-sieve-duplicate/raw-file/4b1dbda4d3fc/...
The repository is at: http://hg.rename-it.nl/pigeonhole-0.3-sieve-duplicate
This plugin is only a few hours old, experimental, and largely untested, so test it thoroughly before considering to use this. Read the INSTALL file for compile and installation instructions.
Comments are welcome.
I did some very basic testing and it seems to work fine.
The example in spec-bosch-sieve-duplicate.txt however says:
if duplicate { fileinto :create "Trash/Duplicate"; }
This assumes the hierarchy separator is '/', but in Maildir this defaults to '.'
So this leads to: failed to store into mailbox 'Trash/Duplicate': Invalid mailbox name
I am not sure if this a bug or not, I suppose you know the rfc's better than I do, is the sieve language supposed to be agnostic of the internals of the storage-engine (dovecot)?
--
Leo Baltus, internetbeheerder /
NPO ICT Internet Services /NPO/
Sumatralaan 45, 1217 GP Hilversum, Filmcentrum, west \ /\/
beheer@omroep.nl, 035-6773555 \/
On 3/8/2012 12:56 PM, Leo Baltus wrote:
Op 23/02/2012 om 02:15:48 +0100, schreef Stephan Bosch:
The repository is at: http://hg.rename-it.nl/pigeonhole-0.3-sieve-duplicate
This plugin is only a few hours old, experimental, and largely untested, so test it thoroughly before considering to use this. Read the INSTALL file for compile and installation instructions.
Comments are welcome. I did some very basic testing and it seems to work fine.
The example in spec-bosch-sieve-duplicate.txt however says:
if duplicate { fileinto :create "Trash/Duplicate"; }This assumes the hierarchy separator is '/', but in Maildir this defaults to '.'
So this leads to: failed to store into mailbox 'Trash/Duplicate': Invalid mailbox name
I am not sure if this a bug or not, I suppose you know the rfc's better than I do, is the sieve language supposed to be agnostic of the internals of the storage-engine (dovecot)?
For Sieve, the mailbox name is pretty much opaque. Usually, it matches what is used through IMAP.
http://tools.ietf.org/html/rfc5228#section-4.1
So, in your case, just use "Trash.Duplicate" instead.
Regards,
Stephan.
participants (5)
-
Adam Szpakowski
-
Charles Marcus
-
Leo Baltus
-
Stephan Bosch
-
Timo Sirainen