Pigeonhole and duplicate checking

Tue Feb 16 01:29:51 UTC 2016

Op 2/16/2016 om 1:32 AM schreef Dan Ragle:
> In my prior procmail setup, I auto-forwarded all my SPAM E-mail to a
> separate user on the system with their own account (called spamuser).
> In the Pigeonhole setup it looks like I have to redirect those
> messages (i.e., redirect "spamuser at mydomain.com";). That works, but
> I'm having issues with "duplicate" messages getting delivered directly
> to my inbox instead of getting redirected. IOW, my regular user's
> .dovecot.sieve has:
>
>     if header :contains "X-Spam-Flag" "YES" {
>         redirect "spamuser at mydomain.com";
>         stop;
>     }
>
> and the spamuser's .dovecot.sieve has:
>
>     require ["imap4flags"];
>     setflag "\\seen";
>
> (this system is entirely for personal use, no concerns with a user's
> Spam being seen via the separate spamuser account).
>
> Now, the problem I'm having is that sometimes a message arrives with a
> duplicate message-ID. Here's an abbreviated example from my maillog:
>
>   Feb 15 00:30:46 myhost sendmail[26844]: u1F5UeBP026844:
> from=<ciuehazt at autoonecareers.com>, size=613, class=0, nrcpts=1,
> msgid=<68895654496651-HLTYGGRLPDJDVTZRHFIKGAON at hverovvxzb.arkansas-email.com>,
> proto=SMTP, daemon=MTA,
> relay=117.27.191.61.broad.static.hf.ah.cndata.com [61.191.27.117] (may
> be forged)
>   Feb 15 00:31:05 myhost dovecot: lmtp(26856): Connect from local
>   Feb 15 00:31:07 myhost sendmail[26867]: u1F5UvFi026867:
> from=<ciuehazt at autoonecareers.com>, size=613, class=0, nrcpts=1,
> msgid=<68895654496651-HLTYGGRLPDJDVTZRHFIKGAON at hverovvxzb.arkansas-email.com>,
> proto=SMTP, daemon=MTA,
> relay=117.27.191.61.broad.static.hf.ah.cndata.com [61.191.27.117] (may
> be forged)
>   Feb 15 00:31:10 myhost dovecot: lmtp(26856, dmr):
> 7Hj/LoBiwVboaAAACXJZQA: sieve:
> msgid=<68895654496651-HLTYGGRLPDJDVTZRHFIKGAON at hverovvxzb.arkansas-email.com>:
> forwarded to <spamuser at mydomain.com>
>   Feb 15 00:31:10 myhost dovecot: lmtp(26856): Disconnect from local:
> Client quit
>   Feb 15 00:31:10 myhost dovecot: lmtp(26893): Connect from local
>   Feb 15 00:31:10 myhost dovecot: lmtp(26893, spamuser):
> 9TPnN55iwVYNaQAACXJZQA: sieve:
> msgid=<68895654496651-HLTYGGRLPDJDVTZRHFIKGAON at hverovvxzb.arkansas-email.com>:
> stored mail into mailbox 'INBOX'
>   Feb 15 00:31:11 myhost dovecot: lmtp(26893): Disconnect from local:
> Client quit
>   Feb 15 00:31:18 myhost dovecot: lmtp(26893): Connect from local
>   Feb 15 00:31:18 myhost dovecot: lmtp(26893, dmr):
> +TPnN55iwVYNaQAACXJZQA: sieve:
> msgid=<68895654496651-HLTYGGRLPDJDVTZRHFIKGAON at hverovvxzb.arkansas-email.com>:
> discarded duplicate forward to <spamuser at biblestuph.com>
>   Feb 15 00:31:18 myhost dovecot: lmtp(26893, dmr):
> +TPnN55iwVYNaQAACXJZQA: sieve:
> msgid=<68895654496651-HLTYGGRLPDJDVTZRHFIKGAON at hverovvxzb.arkansas-email.com>:
> stored mail into mailbox 'INBOX'
>
> So the first message comes in and is redirected properly to my
> spamuser and stored in the inbox. The second one comes in, pigeonhole
> sees it's a duplicate, refuses to redirect it, and stores it in my
> inbox instead.

This expected behavior.

> I tried forcing the .dovecot.lda-dupes file to be a symlink to
> /dev/null just to see if it would work, but unfortunately dovecot just
> recreates it as a normal file the next time it delivers to that user.

Exactly.

> Ideally, I'd like to just discard the duplicates. It looks like there
> is duplicate testing functionality available, but not until later
> versions of Dovecot/Pigeonhole.

Yes, but you will not need that.

> Is there anyway I can either just discard the duplicates, or get them
> to be redirected to the spamuser?

What happens is that the redirect action is ignored the second time,
which means that the implicit keep is not canceled
(https://tools.ietf.org/html/rfc5228#section-2.10.2). Upon executing
"stop;", the script ends and the implicit keep is executed, hence the
message is stored in "INBOX".

So, what you need to do is cancel the implicit keep, no matter what
redirect does. This can be achieved as follows:

    if header :contains "X-Spam-Flag" "YES" {
        redirect "spamuser at mydomain.com";
        discard;
        stop;
    }

The discard action will cancel the implicit keep. It will not affect the
redirect action in any way. If you're a bit scared of the discard
action, you can also replace the it with some other action that cancels
the implicit keep, such as "fileinto" to put duplicates in their own
little black hole folder.

> Also, some other questions I came up with along the way:
>
> Is there any way I can force a message to fileinto a different user's
> Mailbox? I'm guessing no since it appears that the lmtp drops root
> privileges before the global sieve script is interpreted, but thought
> I'd ask anyway. I did try:
>
>     fileinto "/var/mail/spamuser";
>
> But Dovecot complained, something about the mailbox pattern being
> invalid. If it were possible, I would think it would want something like
>
>     fileinto "spamuser:INBOX";
>
> instead, but I don't know if that is even possible.

Your guess is right on the money.

> In a global sieve script, is there anyway to know/test which system
> user is the targeted user for delivery? So in a global sieve_before
> script I could test the target user, and if it's spamuser just file it
> immediately and stop with no further testing?

The latest versions support this:

https://raw.githubusercontent.com/dovecot/pigeonhole/master/doc/rfc/spec-bosch-sieve-dovecot-environment.txt

For older versions, you can use the non-standard well-hidden "auth"
field for the envelope test. This is some deprecated heritage from the
old CMU implementation.

require ["fileinto", "envelope"];
if envelope "auth" "spamuser" {
    fileinto "INBOX";
    stop;
}

Regards,

Stephan.