Pigeonhole and duplicate checking
I spent some time setting up Dovecot LMTP with Pigeonhole this weekend. I got most of the things I wanted to working but have a few issues that I'm hoping that someone might be able to help with.
First, I'm running the stock version of the tools from the core repos for CentOS/6, which means Dovecot 2.0.9 and Pigeonhole 0.2.2. Unfortunately, I don't expect to be able to upgrade those anytime soon, so I'm hoping I can find workarounds for my existing versions.
My mailserver is sendmail, I linked to Dovecot LMTP by adding:
FEATURE(`local_lmtp', `[IPC]', `FILE /var/run/dovecot/lmtp')dnlto my sendmail.mc.
In my prior procmail setup, I auto-forwarded all my SPAM E-mail to a separate user on the system with their own account (called spamuser). In the Pigeonhole setup it looks like I have to redirect those messages (i.e., redirect "spamuser@mydomain.com";). That works, but I'm having issues with "duplicate" messages getting delivered directly to my inbox instead of getting redirected. IOW, my regular user's .dovecot.sieve has:
if header :contains "X-Spam-Flag" "YES" {
redirect "spamuser@mydomain.com";
stop;
}and the spamuser's .dovecot.sieve has:
require ["imap4flags"];
setflag "\\seen";(this system is entirely for personal use, no concerns with a user's Spam being seen via the separate spamuser account).
Now, the problem I'm having is that sometimes a message arrives with a duplicate message-ID. Here's an abbreviated example from my maillog:
Feb 15 00:30:46 myhost sendmail[26844]: u1F5UeBP026844: from=ciuehazt@autoonecareers.com, size=613, class=0, nrcpts=1, msgid=68895654496651-HLTYGGRLPDJDVTZRHFIKGAON@hverovvxzb.arkansas-email.com, proto=SMTP, daemon=MTA, relay=117.27.191.61.broad.static.hf.ah.cndata.com [61.191.27.117] (may be forged) Feb 15 00:31:05 myhost dovecot: lmtp(26856): Connect from local Feb 15 00:31:07 myhost sendmail[26867]: u1F5UvFi026867: from=ciuehazt@autoonecareers.com, size=613, class=0, nrcpts=1, msgid=68895654496651-HLTYGGRLPDJDVTZRHFIKGAON@hverovvxzb.arkansas-email.com, proto=SMTP, daemon=MTA, relay=117.27.191.61.broad.static.hf.ah.cndata.com [61.191.27.117] (may be forged) Feb 15 00:31:10 myhost dovecot: lmtp(26856, dmr): 7Hj/LoBiwVboaAAACXJZQA: sieve: msgid=68895654496651-HLTYGGRLPDJDVTZRHFIKGAON@hverovvxzb.arkansas-email.com: forwarded to spamuser@mydomain.com Feb 15 00:31:10 myhost dovecot: lmtp(26856): Disconnect from local: Client quit Feb 15 00:31:10 myhost dovecot: lmtp(26893): Connect from local Feb 15 00:31:10 myhost dovecot: lmtp(26893, spamuser): 9TPnN55iwVYNaQAACXJZQA: sieve: msgid=68895654496651-HLTYGGRLPDJDVTZRHFIKGAON@hverovvxzb.arkansas-email.com: stored mail into mailbox 'INBOX' Feb 15 00:31:11 myhost dovecot: lmtp(26893): Disconnect from local: Client quit Feb 15 00:31:18 myhost dovecot: lmtp(26893): Connect from local Feb 15 00:31:18 myhost dovecot: lmtp(26893, dmr): +TPnN55iwVYNaQAACXJZQA: sieve: msgid=68895654496651-HLTYGGRLPDJDVTZRHFIKGAON@hverovvxzb.arkansas-email.com: discarded duplicate forward to spamuser@biblestuph.com Feb 15 00:31:18 myhost dovecot: lmtp(26893, dmr): +TPnN55iwVYNaQAACXJZQA: sieve: msgid=68895654496651-HLTYGGRLPDJDVTZRHFIKGAON@hverovvxzb.arkansas-email.com: stored mail into mailbox 'INBOX'
So the first message comes in and is redirected properly to my spamuser and stored in the inbox. The second one comes in, pigeonhole sees it's a duplicate, refuses to redirect it, and stores it in my inbox instead.
I tried forcing the .dovecot.lda-dupes file to be a symlink to /dev/null just to see if it would work, but unfortunately dovecot just recreates it as a normal file the next time it delivers to that user.
Ideally, I'd like to just discard the duplicates. It looks like there is duplicate testing functionality available, but not until later versions of Dovecot/Pigeonhole.
Is there anyway I can either just discard the duplicates, or get them to be redirected to the spamuser?
Also, some other questions I came up with along the way:
Is there any way I can force a message to fileinto a different user's Mailbox? I'm guessing no since it appears that the lmtp drops root privileges before the global sieve script is interpreted, but thought I'd ask anyway. I did try:
fileinto "/var/mail/spamuser";But Dovecot complained, something about the mailbox pattern being invalid. If it were possible, I would think it would want something like
fileinto "spamuser:INBOX";instead, but I don't know if that is even possible.
In a global sieve script, is there anyway to know/test which system user is the targeted user for delivery? So in a global sieve_before script I could test the target user, and if it's spamuser just file it immediately and stop with no further testing?
Thanks for any assistance!
Dan
Op 2/16/2016 om 1:32 AM schreef Dan Ragle:
In my prior procmail setup, I auto-forwarded all my SPAM E-mail to a separate user on the system with their own account (called spamuser). In the Pigeonhole setup it looks like I have to redirect those messages (i.e., redirect "spamuser@mydomain.com";). That works, but I'm having issues with "duplicate" messages getting delivered directly to my inbox instead of getting redirected. IOW, my regular user's .dovecot.sieve has:
if header :contains "X-Spam-Flag" "YES" { redirect "spamuser@mydomain.com"; stop; }and the spamuser's .dovecot.sieve has:
require ["imap4flags"]; setflag "\\seen";(this system is entirely for personal use, no concerns with a user's Spam being seen via the separate spamuser account).
Now, the problem I'm having is that sometimes a message arrives with a duplicate message-ID. Here's an abbreviated example from my maillog:
Feb 15 00:30:46 myhost sendmail[26844]: u1F5UeBP026844: from=ciuehazt@autoonecareers.com, size=613, class=0, nrcpts=1, msgid=68895654496651-HLTYGGRLPDJDVTZRHFIKGAON@hverovvxzb.arkansas-email.com, proto=SMTP, daemon=MTA, relay=117.27.191.61.broad.static.hf.ah.cndata.com [61.191.27.117] (may be forged) Feb 15 00:31:05 myhost dovecot: lmtp(26856): Connect from local Feb 15 00:31:07 myhost sendmail[26867]: u1F5UvFi026867: from=ciuehazt@autoonecareers.com, size=613, class=0, nrcpts=1, msgid=68895654496651-HLTYGGRLPDJDVTZRHFIKGAON@hverovvxzb.arkansas-email.com, proto=SMTP, daemon=MTA, relay=117.27.191.61.broad.static.hf.ah.cndata.com [61.191.27.117] (may be forged) Feb 15 00:31:10 myhost dovecot: lmtp(26856, dmr): 7Hj/LoBiwVboaAAACXJZQA: sieve: msgid=68895654496651-HLTYGGRLPDJDVTZRHFIKGAON@hverovvxzb.arkansas-email.com: forwarded to spamuser@mydomain.com Feb 15 00:31:10 myhost dovecot: lmtp(26856): Disconnect from local: Client quit Feb 15 00:31:10 myhost dovecot: lmtp(26893): Connect from local Feb 15 00:31:10 myhost dovecot: lmtp(26893, spamuser): 9TPnN55iwVYNaQAACXJZQA: sieve: msgid=68895654496651-HLTYGGRLPDJDVTZRHFIKGAON@hverovvxzb.arkansas-email.com: stored mail into mailbox 'INBOX' Feb 15 00:31:11 myhost dovecot: lmtp(26893): Disconnect from local: Client quit Feb 15 00:31:18 myhost dovecot: lmtp(26893): Connect from local Feb 15 00:31:18 myhost dovecot: lmtp(26893, dmr): +TPnN55iwVYNaQAACXJZQA: sieve: msgid=68895654496651-HLTYGGRLPDJDVTZRHFIKGAON@hverovvxzb.arkansas-email.com: discarded duplicate forward to spamuser@biblestuph.com Feb 15 00:31:18 myhost dovecot: lmtp(26893, dmr): +TPnN55iwVYNaQAACXJZQA: sieve: msgid=68895654496651-HLTYGGRLPDJDVTZRHFIKGAON@hverovvxzb.arkansas-email.com: stored mail into mailbox 'INBOX'
So the first message comes in and is redirected properly to my spamuser and stored in the inbox. The second one comes in, pigeonhole sees it's a duplicate, refuses to redirect it, and stores it in my inbox instead.
This expected behavior.
I tried forcing the .dovecot.lda-dupes file to be a symlink to /dev/null just to see if it would work, but unfortunately dovecot just recreates it as a normal file the next time it delivers to that user.
Exactly.
Ideally, I'd like to just discard the duplicates. It looks like there is duplicate testing functionality available, but not until later versions of Dovecot/Pigeonhole.
Yes, but you will not need that.
Is there anyway I can either just discard the duplicates, or get them to be redirected to the spamuser?
What happens is that the redirect action is ignored the second time, which means that the implicit keep is not canceled (https://tools.ietf.org/html/rfc5228#section-2.10.2). Upon executing "stop;", the script ends and the implicit keep is executed, hence the message is stored in "INBOX".
So, what you need to do is cancel the implicit keep, no matter what redirect does. This can be achieved as follows:
if header :contains "X-Spam-Flag" "YES" {
redirect "spamuser@mydomain.com";
discard;
stop;
}The discard action will cancel the implicit keep. It will not affect the redirect action in any way. If you're a bit scared of the discard action, you can also replace the it with some other action that cancels the implicit keep, such as "fileinto" to put duplicates in their own little black hole folder.
Also, some other questions I came up with along the way:
Is there any way I can force a message to fileinto a different user's Mailbox? I'm guessing no since it appears that the lmtp drops root privileges before the global sieve script is interpreted, but thought I'd ask anyway. I did try:
fileinto "/var/mail/spamuser";But Dovecot complained, something about the mailbox pattern being invalid. If it were possible, I would think it would want something like
fileinto "spamuser:INBOX";instead, but I don't know if that is even possible.
Your guess is right on the money.
In a global sieve script, is there anyway to know/test which system user is the targeted user for delivery? So in a global sieve_before script I could test the target user, and if it's spamuser just file it immediately and stop with no further testing?
The latest versions support this:
https://raw.githubusercontent.com/dovecot/pigeonhole/master/doc/rfc/spec-bos...
For older versions, you can use the non-standard well-hidden "auth" field for the envelope test. This is some deprecated heritage from the old CMU implementation.
require ["fileinto", "envelope"]; if envelope "auth" "spamuser" { fileinto "INBOX"; stop; }
Regards,
Stephan.
Perfect!
Using the "auth" trick and appropriate "discard" statements allowed me to get the scripts setup exactly as I had originally intended. Thanks!
Dan
On 2/15/2016 8:29 PM, Stephan Bosch wrote:
Op 2/16/2016 om 1:32 AM schreef Dan Ragle:
In my prior procmail setup, I auto-forwarded all my SPAM E-mail to a separate user on the system with their own account (called spamuser). In the Pigeonhole setup it looks like I have to redirect those messages (i.e., redirect "spamuser@mydomain.com";). That works, but I'm having issues with "duplicate" messages getting delivered directly to my inbox instead of getting redirected. IOW, my regular user's .dovecot.sieve has:
if header :contains "X-Spam-Flag" "YES" { redirect "spamuser@mydomain.com"; stop; }and the spamuser's .dovecot.sieve has:
require ["imap4flags"]; setflag "\\seen";(this system is entirely for personal use, no concerns with a user's Spam being seen via the separate spamuser account).
Now, the problem I'm having is that sometimes a message arrives with a duplicate message-ID. Here's an abbreviated example from my maillog:
Feb 15 00:30:46 myhost sendmail[26844]: u1F5UeBP026844: from=ciuehazt@autoonecareers.com, size=613, class=0, nrcpts=1, msgid=68895654496651-HLTYGGRLPDJDVTZRHFIKGAON@hverovvxzb.arkansas-email.com, proto=SMTP, daemon=MTA, relay=117.27.191.61.broad.static.hf.ah.cndata.com [61.191.27.117] (may be forged) Feb 15 00:31:05 myhost dovecot: lmtp(26856): Connect from local Feb 15 00:31:07 myhost sendmail[26867]: u1F5UvFi026867: from=ciuehazt@autoonecareers.com, size=613, class=0, nrcpts=1, msgid=68895654496651-HLTYGGRLPDJDVTZRHFIKGAON@hverovvxzb.arkansas-email.com, proto=SMTP, daemon=MTA, relay=117.27.191.61.broad.static.hf.ah.cndata.com [61.191.27.117] (may be forged) Feb 15 00:31:10 myhost dovecot: lmtp(26856, dmr): 7Hj/LoBiwVboaAAACXJZQA: sieve: msgid=68895654496651-HLTYGGRLPDJDVTZRHFIKGAON@hverovvxzb.arkansas-email.com: forwarded to spamuser@mydomain.com Feb 15 00:31:10 myhost dovecot: lmtp(26856): Disconnect from local: Client quit Feb 15 00:31:10 myhost dovecot: lmtp(26893): Connect from local Feb 15 00:31:10 myhost dovecot: lmtp(26893, spamuser): 9TPnN55iwVYNaQAACXJZQA: sieve: msgid=68895654496651-HLTYGGRLPDJDVTZRHFIKGAON@hverovvxzb.arkansas-email.com: stored mail into mailbox 'INBOX' Feb 15 00:31:11 myhost dovecot: lmtp(26893): Disconnect from local: Client quit Feb 15 00:31:18 myhost dovecot: lmtp(26893): Connect from local Feb 15 00:31:18 myhost dovecot: lmtp(26893, dmr): +TPnN55iwVYNaQAACXJZQA: sieve: msgid=68895654496651-HLTYGGRLPDJDVTZRHFIKGAON@hverovvxzb.arkansas-email.com: discarded duplicate forward to spamuser@biblestuph.com Feb 15 00:31:18 myhost dovecot: lmtp(26893, dmr): +TPnN55iwVYNaQAACXJZQA: sieve: msgid=68895654496651-HLTYGGRLPDJDVTZRHFIKGAON@hverovvxzb.arkansas-email.com: stored mail into mailbox 'INBOX'
So the first message comes in and is redirected properly to my spamuser and stored in the inbox. The second one comes in, pigeonhole sees it's a duplicate, refuses to redirect it, and stores it in my inbox instead.
This expected behavior.
I tried forcing the .dovecot.lda-dupes file to be a symlink to /dev/null just to see if it would work, but unfortunately dovecot just recreates it as a normal file the next time it delivers to that user.
Exactly.
Ideally, I'd like to just discard the duplicates. It looks like there is duplicate testing functionality available, but not until later versions of Dovecot/Pigeonhole.
Yes, but you will not need that.
Is there anyway I can either just discard the duplicates, or get them to be redirected to the spamuser?
What happens is that the redirect action is ignored the second time, which means that the implicit keep is not canceled (https://tools.ietf.org/html/rfc5228#section-2.10.2). Upon executing "stop;", the script ends and the implicit keep is executed, hence the message is stored in "INBOX".
So, what you need to do is cancel the implicit keep, no matter what redirect does. This can be achieved as follows:
if header :contains "X-Spam-Flag" "YES" { redirect "spamuser@mydomain.com"; discard; stop; }The discard action will cancel the implicit keep. It will not affect the redirect action in any way. If you're a bit scared of the discard action, you can also replace the it with some other action that cancels the implicit keep, such as "fileinto" to put duplicates in their own little black hole folder.
Also, some other questions I came up with along the way:
Is there any way I can force a message to fileinto a different user's Mailbox? I'm guessing no since it appears that the lmtp drops root privileges before the global sieve script is interpreted, but thought I'd ask anyway. I did try:
fileinto "/var/mail/spamuser";But Dovecot complained, something about the mailbox pattern being invalid. If it were possible, I would think it would want something like
fileinto "spamuser:INBOX";instead, but I don't know if that is even possible.
Your guess is right on the money.
In a global sieve script, is there anyway to know/test which system user is the targeted user for delivery? So in a global sieve_before script I could test the target user, and if it's spamuser just file it immediately and stop with no further testing?
The latest versions support this:
https://raw.githubusercontent.com/dovecot/pigeonhole/master/doc/rfc/spec-bos...
For older versions, you can use the non-standard well-hidden "auth" field for the envelope test. This is some deprecated heritage from the old CMU implementation.
require ["fileinto", "envelope"]; if envelope "auth" "spamuser" { fileinto "INBOX"; stop; }
Regards,
Stephan.
participants (2)
-
Dan Ragle
-
Stephan Bosch