Pigeonhole and duplicate checking

Tue Feb 16 02:47:07 UTC 2016

Perfect!

Using the "auth" trick and appropriate "discard" statements allowed me 
to get the scripts setup exactly as I had originally intended. Thanks!

Dan

On 2/15/2016 8:29 PM, Stephan Bosch wrote:
> Op 2/16/2016 om 1:32 AM schreef Dan Ragle:
>> In my prior procmail setup, I auto-forwarded all my SPAM E-mail to a
>> separate user on the system with their own account (called spamuser).
>> In the Pigeonhole setup it looks like I have to redirect those
>> messages (i.e., redirect "spamuser at mydomain.com";). That works, but
>> I'm having issues with "duplicate" messages getting delivered directly
>> to my inbox instead of getting redirected. IOW, my regular user's
>> .dovecot.sieve has:
>>
>>      if header :contains "X-Spam-Flag" "YES" {
>>          redirect "spamuser at mydomain.com";
>>          stop;
>>      }
>>
>> and the spamuser's .dovecot.sieve has:
>>
>>      require ["imap4flags"];
>>      setflag "\\seen";
>>
>> (this system is entirely for personal use, no concerns with a user's
>> Spam being seen via the separate spamuser account).
>>
>> Now, the problem I'm having is that sometimes a message arrives with a
>> duplicate message-ID. Here's an abbreviated example from my maillog:
>>
>>    Feb 15 00:30:46 myhost sendmail[26844]: u1F5UeBP026844:
>> from=<ciuehazt at autoonecareers.com>, size=613, class=0, nrcpts=1,
>> msgid=<68895654496651-HLTYGGRLPDJDVTZRHFIKGAON at hverovvxzb.arkansas-email.com>,
>> proto=SMTP, daemon=MTA,
>> relay=117.27.191.61.broad.static.hf.ah.cndata.com [61.191.27.117] (may
>> be forged)
>>    Feb 15 00:31:05 myhost dovecot: lmtp(26856): Connect from local
>>    Feb 15 00:31:07 myhost sendmail[26867]: u1F5UvFi026867:
>> from=<ciuehazt at autoonecareers.com>, size=613, class=0, nrcpts=1,
>> msgid=<68895654496651-HLTYGGRLPDJDVTZRHFIKGAON at hverovvxzb.arkansas-email.com>,
>> proto=SMTP, daemon=MTA,
>> relay=117.27.191.61.broad.static.hf.ah.cndata.com [61.191.27.117] (may
>> be forged)
>>    Feb 15 00:31:10 myhost dovecot: lmtp(26856, dmr):
>> 7Hj/LoBiwVboaAAACXJZQA: sieve:
>> msgid=<68895654496651-HLTYGGRLPDJDVTZRHFIKGAON at hverovvxzb.arkansas-email.com>:
>> forwarded to <spamuser at mydomain.com>
>>    Feb 15 00:31:10 myhost dovecot: lmtp(26856): Disconnect from local:
>> Client quit
>>    Feb 15 00:31:10 myhost dovecot: lmtp(26893): Connect from local
>>    Feb 15 00:31:10 myhost dovecot: lmtp(26893, spamuser):
>> 9TPnN55iwVYNaQAACXJZQA: sieve:
>> msgid=<68895654496651-HLTYGGRLPDJDVTZRHFIKGAON at hverovvxzb.arkansas-email.com>:
>> stored mail into mailbox 'INBOX'
>>    Feb 15 00:31:11 myhost dovecot: lmtp(26893): Disconnect from local:
>> Client quit
>>    Feb 15 00:31:18 myhost dovecot: lmtp(26893): Connect from local
>>    Feb 15 00:31:18 myhost dovecot: lmtp(26893, dmr):
>> +TPnN55iwVYNaQAACXJZQA: sieve:
>> msgid=<68895654496651-HLTYGGRLPDJDVTZRHFIKGAON at hverovvxzb.arkansas-email.com>:
>> discarded duplicate forward to <spamuser at biblestuph.com>
>>    Feb 15 00:31:18 myhost dovecot: lmtp(26893, dmr):
>> +TPnN55iwVYNaQAACXJZQA: sieve:
>> msgid=<68895654496651-HLTYGGRLPDJDVTZRHFIKGAON at hverovvxzb.arkansas-email.com>:
>> stored mail into mailbox 'INBOX'
>>
>> So the first message comes in and is redirected properly to my
>> spamuser and stored in the inbox. The second one comes in, pigeonhole
>> sees it's a duplicate, refuses to redirect it, and stores it in my
>> inbox instead.
>
> This expected behavior.
>
>> I tried forcing the .dovecot.lda-dupes file to be a symlink to
>> /dev/null just to see if it would work, but unfortunately dovecot just
>> recreates it as a normal file the next time it delivers to that user.
>
> Exactly.
>
>> Ideally, I'd like to just discard the duplicates. It looks like there
>> is duplicate testing functionality available, but not until later
>> versions of Dovecot/Pigeonhole.
>
> Yes, but you will not need that.
>
>> Is there anyway I can either just discard the duplicates, or get them
>> to be redirected to the spamuser?
>
> What happens is that the redirect action is ignored the second time,
> which means that the implicit keep is not canceled
> (https://tools.ietf.org/html/rfc5228#section-2.10.2). Upon executing
> "stop;", the script ends and the implicit keep is executed, hence the
> message is stored in "INBOX".
>
> So, what you need to do is cancel the implicit keep, no matter what
> redirect does. This can be achieved as follows:
>
>      if header :contains "X-Spam-Flag" "YES" {
>          redirect "spamuser at mydomain.com";
>          discard;
>          stop;
>      }
>
> The discard action will cancel the implicit keep. It will not affect the
> redirect action in any way. If you're a bit scared of the discard
> action, you can also replace the it with some other action that cancels
> the implicit keep, such as "fileinto" to put duplicates in their own
> little black hole folder.
>
>> Also, some other questions I came up with along the way:
>>
>> Is there any way I can force a message to fileinto a different user's
>> Mailbox? I'm guessing no since it appears that the lmtp drops root
>> privileges before the global sieve script is interpreted, but thought
>> I'd ask anyway. I did try:
>>
>>      fileinto "/var/mail/spamuser";
>>
>> But Dovecot complained, something about the mailbox pattern being
>> invalid. If it were possible, I would think it would want something like
>>
>>      fileinto "spamuser:INBOX";
>>
>> instead, but I don't know if that is even possible.
>
> Your guess is right on the money.
>
>> In a global sieve script, is there anyway to know/test which system
>> user is the targeted user for delivery? So in a global sieve_before
>> script I could test the target user, and if it's spamuser just file it
>> immediately and stop with no further testing?
>
> The latest versions support this:
>
> https://raw.githubusercontent.com/dovecot/pigeonhole/master/doc/rfc/spec-bosch-sieve-dovecot-environment.txt
>
> For older versions, you can use the non-standard well-hidden "auth"
> field for the envelope test. This is some deprecated heritage from the
> old CMU implementation.
>
> require ["fileinto", "envelope"];
> if envelope "auth" "spamuser" {
>      fileinto "INBOX";
>      stop;
> }
>
> Regards,
>
> Stephan.
>
>
>
>