Dovecot Sieve and Postfix header_checks Issue
Hello List,
I tried to subscribe but it's taking forever for the confirmation email to arrive so I thought I would ask away by emailing directly. My apologies in advance should this question appear twice.
It may seem real simple to experts but I cannot really figure it out. I'll try to be concise:
Dovecot version is 2.1.7. Its dovecot –n is real short one and follows right after my question.
I have my Postfix 2.9.6 properties set like this: header_checks = /etc/path/to/myfile. Then I have Dovecot Sieve also configured and working fine.
Now the trouble is that these two cannot be combined together can they? Dovecot Sieve and Postfix's header_checks? If I turn off header_checks in Postfix then Dovecot Sieve is working fine. If I vise versa turn off Dovecot Sieve then Postfix's header-checks are also working fine. However, when I keep them both turned on it also works but only partially and in this case Dovecot Sieve never bounces back any rejected massages remotely, the bounce happens only locally. That is I can see it in my logs that it was rejected locally but the message is never sent back to the sender saying something like, "Spam is rejected here". Thus the reject turns into a silent discard without telling the sender anything at all. Any ideas, help, advices how do I fix that? Would be really grateful for any suggestions / assistance at all. Many thanks in advance!
P.S. I would like to use both because Dovecot Sieve is a very powerful and great plugin that uses Spamassassin's added tags and headers based on which I create my rules. Postfix in its turn has a very neat feature of checking the headers via regexp but Postfix does not see / detect Spamassassin's added headers at all because I passed delivery and authentication (which is a way better than Postfix's) to dovecot deliver.
# 2.1.7: /etc/dovecot/dovecot.conf # OS: Linux 3.2.0-4-amd64 x86_64 Debian 7.6 ext4 auth_mechanisms = plain login cram-md5 scram-sha-1 auth_verbose = yes hostname = WindTalker info_log_path = /var/log/dovecot-sieve.log log_path = /var/log/dovecot-sieve-errors.log mail_location = maildir:/home/mvail/%d/%n managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave passdb { args = /etc/dovecot/passwd driver = passwd-file } plugin { sieve = ~/.dovecot.sieve sieve_default = /etc/dovecot/default.sieve sieve_dir = ~/sieve } pop3_uidl_format = %g postmaster_address = postmaster@example.com protocols = imap pop3 sieve service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } } ssl_ca =
Regards, Dennis.
That would most likely be something in your header_checks that is causing the bounce from Sieve to be rejected.
There is no reason why you cannot use both.
On 26/09/14 12:35, Klaipedaville on Google wrote:
Hello List,
I tried to subscribe but it's taking forever for the confirmation email to arrive so I thought I would ask away by emailing directly. My apologies in advance should this question appear twice.
It may seem real simple to experts but I cannot really figure it out. I'll try to be concise:
Dovecot version is 2.1.7. Its dovecot –n is real short one and follows right after my question.
I have my Postfix 2.9.6 properties set like this: header_checks = /etc/path/to/myfile. Then I have Dovecot Sieve also configured and working fine.
Now the trouble is that these two cannot be combined together can they? Dovecot Sieve and Postfix's header_checks? If I turn off header_checks in Postfix then Dovecot Sieve is working fine. If I vise versa turn off Dovecot Sieve then Postfix's header-checks are also working fine. However, when I keep them both turned on it also works but only partially and in this case Dovecot Sieve never bounces back any rejected massages remotely, the bounce happens only locally. That is I can see it in my logs that it was rejected locally but the message is never sent back to the sender saying something like, "Spam is rejected here". Thus the reject turns into a silent discard without telling the sender anything at all. Any ideas, help, advices how do I fix that? Would be really grateful for any suggestions / assistance at all. Many thanks in advance!
P.S. I would like to use both because Dovecot Sieve is a very powerful and great plugin that uses Spamassassin's added tags and headers based on which I create my rules. Postfix in its turn has a very neat feature of checking the headers via regexp but Postfix does not see / detect Spamassassin's added headers at all because I passed delivery and authentication (which is a way better than Postfix's) to dovecot deliver.
# 2.1.7: /etc/dovecot/dovecot.conf # OS: Linux 3.2.0-4-amd64 x86_64 Debian 7.6 ext4 auth_mechanisms = plain login cram-md5 scram-sha-1 auth_verbose = yes hostname = WindTalker info_log_path = /var/log/dovecot-sieve.log log_path = /var/log/dovecot-sieve-errors.log mail_location = maildir:/home/mvail/%d/%n managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave passdb { args = /etc/dovecot/passwd driver = passwd-file } plugin { sieve = ~/.dovecot.sieve sieve_default = /etc/dovecot/default.sieve sieve_dir = ~/sieve } pop3_uidl_format = %g postmaster_address = postmaster@example.com protocols = imap pop3 sieve service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } } ssl_ca =
Regards, Dennis.
-- This message is intended only for the addressee and may contain confidential information. Unless you are that person, you may not disclose its contents or use it in any way and are requested to delete the message along with any attachments and notify us immediately. "Transact" is operated by Integrated Financial Arrangements plc. 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300. (Registered office: as above; Registered in England and Wales under number: 3727592). Authorised and regulated by the Financial Conduct Authority (entered on the Financial Services Register; no. 190856).
Thank you for your suggestion, Alex.
However, my header_checks file has just 5 lines of regexp as follows:
/^Subject:.**{5}SPAM*{5}/ REJECT No spammers allowed here. /^Subject:.*\*\*\*\*\*SPAM\*\*\*\*\*/ REJECT No spammers allowed. /\s**{5}SPAM*{5}/ REJECT No spamming hullababballos allowed. /^Subject:(.*)SPAM/ REJECT Spam is not allowed. DISCARD. /^From:.*\@.*\.tw/ REJECT Sorry, Taiwanese mail is not allowed.
All the regexp are correct. The first and the third lines actually do the same thing and they are there for testing purposes only. Execution stops at the very first rule matched so there is no problem in having any rules repeated. I can't see how a few the most simplest regular expressions can cause the bounce from Sieve not to go through.. Do you see any problems in these lines above that could possibly be the cause of your suggestion?
From: Alex Crow Sent: Friday, September 26, 2014 14:47 To: dovecot@dovecot.org Subject: Re: Dovecot Sieve and Postfix header_checks Issue
That would most likely be something in your header_checks that is causing the bounce from Sieve to be rejected.
There is no reason why you cannot use both.
On 26/09/14 12:35, Klaipedaville on Google wrote:
Hello List,
I tried to subscribe but it's taking forever for the confirmation email to arrive so I thought I would ask away by emailing directly. My apologies in advance should this question appear twice.
It may seem real simple to experts but I cannot really figure it out. I'll try to be concise:
Dovecot version is 2.1.7. Its dovecot –n is real short one and follows right after my question.
I have my Postfix 2.9.6 properties set like this: header_checks = /etc/path/to/myfile. Then I have Dovecot Sieve also configured and working fine.
Now the trouble is that these two cannot be combined together can they? Dovecot Sieve and Postfix's header_checks? If I turn off header_checks in Postfix then Dovecot Sieve is working fine. If I vise versa turn off Dovecot Sieve then Postfix's header-checks are also working fine. However, when I keep them both turned on it also works but only partially and in this case Dovecot Sieve never bounces back any rejected massages remotely, the bounce happens only locally. That is I can see it in my logs that it was rejected locally but the message is never sent back to the sender saying something like, "Spam is rejected here". Thus the reject turns into a silent discard without telling the sender anything at all. Any ideas, help, advices how do I fix that? Would be really grateful for any suggestions / assistance at all. Many thanks in advance!
P.S. I would like to use both because Dovecot Sieve is a very powerful and great plugin that uses Spamassassin's added tags and headers based on which I create my rules. Postfix in its turn has a very neat feature of checking the headers via regexp but Postfix does not see / detect Spamassassin's added headers at all because I passed delivery and authentication (which is a way better than Postfix's) to dovecot deliver.
# 2.1.7: /etc/dovecot/dovecot.conf # OS: Linux 3.2.0-4-amd64 x86_64 Debian 7.6 ext4 auth_mechanisms = plain login cram-md5 scram-sha-1 auth_verbose = yes hostname = WindTalker info_log_path = /var/log/dovecot-sieve.log log_path = /var/log/dovecot-sieve-errors.log mail_location = maildir:/home/mvail/%d/%n managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave passdb { args = /etc/dovecot/passwd driver = passwd-file } plugin { sieve = ~/.dovecot.sieve sieve_default = /etc/dovecot/default.sieve sieve_dir = ~/sieve } pop3_uidl_format = %g postmaster_address = postmaster@example.com protocols = imap pop3 sieve service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } } ssl_ca =
Regards, Dennis.
-- This message is intended only for the addressee and may contain confidential information. Unless you are that person, you may not disclose its contents or use it in any way and are requested to delete the message along with any attachments and notify us immediately. "Transact" is operated by Integrated Financial Arrangements plc. 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300. (Registered office: as above; Registered in England and Wales under number: 3727592). Authorised and regulated by the Financial Conduct Authority (entered on the Financial Services Register; no. 190856).
Hey! You are right Alex! Many thanks for pointing me to head over to the right direction!
It was a clash on rules for some reason. Now, I was also right that these two systems could not be used together because the rules declared in different systems to perform the same action (REJECT) cause the error I was having!
The following rule in default.sieve:
require ["reject"]; # rule: Reject on "x-spam-flag" header if header :contains "X-Spam-Flag" "YES" { reject "No spamming allowed here."; stop; }
and the following Postfix's regexp header_check rules on the subject field:
/^Subject:.**{5}SPAM*{5}/ REJECT No spammers allowed here. /^Subject:.*\*\*\*\*\*SPAM\*\*\*\*\*/ REJECT No spammers allowed. /\s**{5}SPAM*{5}/ REJECT No spamming hullababballos allowed. /^Subject:(.*)SPAM/ REJECT Spam is not allowed. DISCARD.
were causing the Dovecot Sieve rejection bounce not to go through. The rules blocked the spam all right but rejection was turned into discard for some reason.
Now the question is how do I find out which regular expressions will be in conflict with default.sieve scripting rules? Default.sieve is set to block spam on the X-Spam-Flag header and header_checks is set to block spam on the subject field. I am still clueless why didn't these two "cooperate"? Was it just because they were "told" to perform the same action as per my previous guess? But the target to perform this same action on was different... Any more ideas anyone? Alex? Many thanks in advance for any input!
From: Klaipedaville on Google Sent: Friday, September 26, 2014 15:00 To: Alex Crow ; dovecot@dovecot.org Subject: Re: Dovecot Sieve and Postfix header_checks Issue
Thank you for your suggestion, Alex.
However, my header_checks file has just 5 lines of regexp as follows:
/^Subject:.**{5}SPAM*{5}/ REJECT No spammers allowed here. /^Subject:.*\*\*\*\*\*SPAM\*\*\*\*\*/ REJECT No spammers allowed. /\s**{5}SPAM*{5}/ REJECT No spamming hullababballos allowed. /^Subject:(.*)SPAM/ REJECT Spam is not allowed. DISCARD. /^From:.*\@.*\.tw/ REJECT Sorry, Taiwanese mail is not allowed.
All the regexp are correct. The first and the third lines actually do the same thing and they are there for testing purposes only. Execution stops at the very first rule matched so there is no problem in having any rules repeated. I can't see how a few the most simplest regular expressions can cause the bounce from Sieve not to go through.. Do you see any problems in these lines above that could possibly be the cause of your suggestion?
From: Alex Crow Sent: Friday, September 26, 2014 14:47 To: dovecot@dovecot.org Subject: Re: Dovecot Sieve and Postfix header_checks Issue
That would most likely be something in your header_checks that is causing the bounce from Sieve to be rejected.
There is no reason why you cannot use both.
On 26/09/14 12:35, Klaipedaville on Google wrote:
Hello List,
I tried to subscribe but it's taking forever for the confirmation email to arrive so I thought I would ask away by emailing directly. My apologies in advance should this question appear twice.
It may seem real simple to experts but I cannot really figure it out. I'll try to be concise:
Dovecot version is 2.1.7. Its dovecot –n is real short one and follows right after my question.
I have my Postfix 2.9.6 properties set like this: header_checks = /etc/path/to/myfile. Then I have Dovecot Sieve also configured and working fine.
Now the trouble is that these two cannot be combined together can they? Dovecot Sieve and Postfix's header_checks? If I turn off header_checks in Postfix then Dovecot Sieve is working fine. If I vise versa turn off Dovecot Sieve then Postfix's header-checks are also working fine. However, when I keep them both turned on it also works but only partially and in this case Dovecot Sieve never bounces back any rejected massages remotely, the bounce happens only locally. That is I can see it in my logs that it was rejected locally but the message is never sent back to the sender saying something like, "Spam is rejected here". Thus the reject turns into a silent discard without telling the sender anything at all. Any ideas, help, advices how do I fix that? Would be really grateful for any suggestions / assistance at all. Many thanks in advance!
P.S. I would like to use both because Dovecot Sieve is a very powerful and great plugin that uses Spamassassin's added tags and headers based on which I create my rules. Postfix in its turn has a very neat feature of checking the headers via regexp but Postfix does not see / detect Spamassassin's added headers at all because I passed delivery and authentication (which is a way better than Postfix's) to dovecot deliver.
# 2.1.7: /etc/dovecot/dovecot.conf # OS: Linux 3.2.0-4-amd64 x86_64 Debian 7.6 ext4 auth_mechanisms = plain login cram-md5 scram-sha-1 auth_verbose = yes hostname = WindTalker info_log_path = /var/log/dovecot-sieve.log log_path = /var/log/dovecot-sieve-errors.log mail_location = maildir:/home/mvail/%d/%n managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave passdb { args = /etc/dovecot/passwd driver = passwd-file } plugin { sieve = ~/.dovecot.sieve sieve_default = /etc/dovecot/default.sieve sieve_dir = ~/sieve } pop3_uidl_format = %g postmaster_address = postmaster@example.com protocols = imap pop3 sieve service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } } ssl_ca =
Regards, Dennis.
-- This message is intended only for the addressee and may contain confidential information. Unless you are that person, you may not disclose its contents or use it in any way and are requested to delete the message along with any attachments and notify us immediately. "Transact" is operated by Integrated Financial Arrangements plc. 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300. (Registered office: as above; Registered in England and Wales under number: 3727592). Authorised and regulated by the Financial Conduct Authority (entered on the Financial Services Register; no. 190856).
On 26/09/14 14:10, Klaipedaville on Google wrote:
Hey! You are right Alex! Many thanks for pointing me to head over to the right direction! It was a clash on rules for some reason. Now, I was also right that these two systems could not be used together because the rules declared in different systems to perform the same action (REJECT) cause the error I was having! The following rule in default.sieve: require ["reject"]; # rule: Reject on "x-spam-flag" header if header :contains "X-Spam-Flag" "YES" { reject "No spamming allowed here."; stop; } and the following Postfix's regexp header_check rules on the subject field: /^Subject:.**{5}SPAM*{5}/ REJECT No spammers allowed here. /^Subject:.*\*\*\*\*\*SPAM\*\*\*\*\*/ REJECT No spammers allowed. /\s**{5}SPAM*{5}/ REJECT No spamming hullababballos allowed.
I think it may be this one above. From the postfix manuals:
"By default, matching is case-insensitive, and newlines are not treated as special characters. The behavior is controlled by flags, which are toggled by appending one or more of the following characters after the pattern: *i* (default: on) Toggles the case sensitivity flag. By default, matching is case insensitive."
And it looks like * needs escaping there too (if you're trying to match exactly 5 asterisks, you should probably do "\*{5}" not just *{5}.
/^Subject:(.*)SPAM/ REJECT Spam is not allowed. DISCARD. were causing the Dovecot Sieve rejection bounce not to go through. The rules blocked the spam all right but rejection was turned into discard for some reason. Now the question is how do I find out which regular expressions will be in conflict with default.sieve scripting rules?
That's just about learning Posix Regex syntax.
Default.sieve is set to block spam on the X-Spam-Flag header and header_checks is set to block spam on the subject field. I am still clueless why didn't these two "cooperate"? Was it just because they were "told" to perform the same action as per my previous guess?
I'm almost 100% sure that that regex also matched the bounce from your sieve rules. There is no mysterious interaction between header_checks and sieve rules, it's just your pattern was too liberal in the former.
But the target to perform this same action on was different... Any more ideas anyone? Alex? Many thanks in advance for any input!
I think if you tune that header_checks file correctly you should have no more issues.
Thanks
Alex
-- This message is intended only for the addressee and may contain confidential information. Unless you are that person, you may not disclose its contents or use it in any way and are requested to delete the message along with any attachments and notify us immediately. "Transact" is operated by Integrated Financial Arrangements plc. 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300. (Registered office: as above; Registered in England and Wales under number: 3727592). Authorised and regulated by the Financial Conduct Authority (entered on the Financial Services Register; no. 190856).
/^Subject:.**{5}SPAM*{5}/ REJECT No spammers allowed here. /^Subject:.*\*\*\*\*\*SPAM\*\*\*\*\*/ REJECT No spammers allowed. /\s**{5}SPAM*{5}/ REJECT No spamming hullababballos allowed.
I think it may be this one above. From the postfix manuals"By default, matching is case-insensitive, and newlines are not treated as special characters. The behavior is controlled by flags, which are toggled by appending one or more of the following characters after the pattern: *i* (default: on) Toggles the case sensitivity flag. By default, matching is case insensitive."
Case insensitive is declared by putting this /i at the end of a rule. Postfix has nothing to do with regular expressions (regexp) and regexp is not controlled by postfix. There should be a regexp library available on the server where you are using regexp. It’s like PHP, or tml, or js, or css, it cannot be controlled by postfix.
And it looks like * needs escaping there too (if you're trying to match exactly 5 asterisks, you should probably do "\*{5}" not just *{5}. Yes, the escape character in front \*{5} to match 5 asterisks is the correct one. You are right. I am an expert on regexp and this (incorrect one) was there just for testing purposes because there was a problem with the library on the server so I had this bad rule over there to follow up on error in logs. The library has been fixed by now and as I said earlier execution stops on the first rule matched but does not really do any harm if there is a mistake in the rule, in this 'mistake' case the rule is simply skipped.
/^Subject:(.*)SPAM/ REJECT Spam is not allowed. DISCARD. were causing the Dovecot Sieve rejection bounce not to go through. The rules blocked the spam all right but rejection was turned into discard for some reason. Now the question is how do I find out which regular >expressions will be in conflict with default.sieve scripting rules?
That's just about learning Posix Regex syntax.
All the rules are 100% correct as there is a very simple and useful tool in postfix to check if regexp is correct. The tool can be used even by people who don't have a foggiest idea how regexp work. All you have to do is to type on a command line this postmap -q "Subject: *****SPAM***** blablablabla" regexp:/etc/postfix/header_checks or this postmap -q "X-Spam-Flag: YES" regexp:/etc/postfix/header_checks and it will tell you if your rule is correct or not. It is bullet and fool proof system with 100% guarantee. All these rules have been checked like that despite the fact that I know for a fact that they are correct by my own knowledge and experience.
I'm almost 100% sure that that regex also matched the bounce from your sieve rules. There is no mysterious interaction between header_checks and sieve rules, it's just your pattern was too liberal in the former.
No, no. The regex could not have matched the bounce from my own rules because it would be silly to send a test message from the same server that would loop back and block myself by my own rules. I sent test messages from another server's accounts. Plus, there is a difference. Header_checks in Postfix use only customized rules that do not involve any Spamassassin's added headers. Now in my case only Dovecot Sieve goes through Spamassassin headers because as mentioned earlier I passed delivery from Postfix to dovecot LDA in my configuration. That's why everything that has Spamassassin's headers and tags added has to be configured via default.sieve scripting and everything else (that do not get Spamassassin's headers added) may use header_checks of Postfix. I have just figured that out by runnning quite a few different and simple tests.
I think if you tune that header_checks file correctly you should have no more issues.
The header_check rules are fine tuned to their best.
Anyway, I am thankful for your suggestion as it pointed me to the right direction. Then I picked it up and simply followed onwards by elaborating and building on top which led me to a solved problem Thank you.
/^Subject:.**{5}SPAM*{5}/ REJECT No spammers allowed here. /^Subject:.*\*\*\*\*\*SPAM\*\*\*\*\*/ REJECT No spammers allowed. /\s**{5}SPAM*{5}/ REJECT No spamming hullababballos allowed. I think it may be this one above. From the postfix manuals"By default, matching is case-insensitive, and newlines are not treated as special characters. The behavior is controlled by flags, which are toggled by appending one or more of the following characters after the pattern: *i* (default: on) Toggles the case sensitivity flag. By default, matching is case insensitive." Case insensitive is declared by putting this /i at the end of a rule. Postfix has nothing to do with regular expressions (regexp) and regexp is not controlled by postfix. There should be a regexp library available on the server where you are using regexp. It’s like PHP, or tml, or js, or css, it cannot be controlled by postfix. So why does it state in man 5 regexp_table that such tables are *case insensitive* by default and the /i actually toggles that? Are you saying
On 26/09/14 15:27, Klaipedaville on Google wrote: that man page is wrong? I'd be surprised as I don't think I've yet come across an occasion where postfix man pages are incorrect!
And it looks like * needs escaping there too (if you're trying to match exactly 5 asterisks, you should probably do "\*{5}" not just *{5}. Yes, the escape character in front \*{5} to match 5 asterisks is the correct one. You are right. I am an expert on regexp and this (incorrect one) was there just for testing purposes because there was a problem with the library on the server so I had this bad rule over there to follow up on error in logs. The library has been fixed by now and as I said earlier execution stops on the first rule matched but does not really do any harm if there is a mistake in the rule, in this 'mistake' case the rule is simply skipped.
/^Subject:(.*)SPAM/ REJECT Spam is not allowed. DISCARD. were causing the Dovecot Sieve rejection bounce not to go through. The rules blocked the spam all right but rejection was turned into discard for some reason. Now the question is how do I find out which regular >expressions will be in conflict with default.sieve scripting rules? That's just about learning Posix Regex syntax. All the rules are 100% correct as there is a very simple and useful tool in postfix to check if regexp is correct. The tool can be used even by people who don't have a foggiest idea how regexp work. All you have to do is to type on a command line this postmap -q "Subject: *****SPAM***** blablablabla" regexp:/etc/postfix/header_checks or this postmap -q "X-Spam-Flag: YES" regexp:/etc/postfix/header_checks and it will tell you if your rule is correct or not. It is bullet and fool proof system with 100% guarantee. All these rules have been checked like that despite the fact that I know for a fact that they are correct by my own knowledge and experience.
I'm almost 100% sure that that regex also matched the bounce from your sieve rules. There is no mysterious interaction between header_checks and sieve rules, it's just your pattern was too liberal in the former. No, no. The regex could not have matched the bounce from my own rules because it would be silly to send a test message from the same server that would loop back and block myself by my own rules. I sent test messages from another server's accounts. Plus, there is a difference. Header_checks in Postfix use only customized rules that do not involve any Spamassassin's added headers. Now in my case only Dovecot Sieve goes through Spamassassin headers because as mentioned earlier I passed delivery from Postfix to dovecot LDA in my configuration. That's why everything that has Spamassassin's headers and tags added has to be configured via default.sieve scripting and everything else (that do not get Spamassassin's headers added) may use header_checks of Postfix. I have just figured that out by runnning quite a few different and simple tests.
I think if you tune that header_checks file correctly you should have no more issues. The header_check rules are fine tuned to their best.
Anyway, I am thankful for your suggestion as it pointed me to the right direction. Then I picked it up and simply followed onwards by elaborating and building on top which led me to a solved problem Thank you.
So if the regexes were all correct, then:
a) what was your actual problem once you identified it
and
b) for the benefit of the list, how did you actually solve it?
Alex
-- This message is intended only for the addressee and may contain confidential information. Unless you are that person, you may not disclose its contents or use it in any way and are requested to delete the message along with any attachments and notify us immediately. "Transact" is operated by Integrated Financial Arrangements plc. 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300. (Registered office: as above; Registered in England and Wales under number: 3727592). Authorised and regulated by the Financial Conduct Authority (entered on the Financial Services Register; no. 190856).
So why does it state in man 5 regexp_table that such tables are *case insensitive* by default and the /i actually toggles that? Are you saying that man page is wrong? I'd be surprised as I don't think I've yet come across an occasion where postfix man pages are incorrect! I am not saying that the Postfix man is incorrect. It is 100% case insensitive and SPAM (upper case) and spam (lower case) would mean absolutely the same. Therefore my SPAM (upper case) regexp was correct.
So if the regexes were all correct, then:
a) what was your actual problem once you identified it
and
b) for the benefit of the list, how did you actually solve it?
Alex
I explained it all in my previous post.
On 26/09/14 16:00, Klaipedaville on Google wrote:
So why does it state in man 5 regexp_table that such tables are *case insensitive* by default and the /i actually toggles that? Are you saying that man page is wrong? I'd be surprised as I don't think I've yet come across an occasion where postfix man pages are incorrect! I am not saying that the Postfix man is incorrect. It is 100% case insensitive and SPAM (upper case) and spam (lower case) would mean absolutely the same. Therefore my SPAM (upper case) regexp was correct.
So if the regexes were all correct, then:
a) what was your actual problem once you identified it
and
b) for the benefit of the list, how did you actually solve it?
Alex
I explained it all in my previous post.
I realise it's probably because of the use of the reject action, which presumably inserts the text "No spamming allowed here." into the subject of the bounce.
However what also concerns me is that sending MDN's back to the envelope sender of SPAM messages is very likely to cause your server to be the origin of "Joe-Job" spam. The sieve RFCs state that if using reject (therefore sending MDNs) you should also be able to check for forged envelope senders and in that case do not send bounces. In real life it's considered best to just not bother sending anything back at all, and either discard, or block at the SMTP level using RBLs.
Cheers
Alex
-- This message has been scanned for viruses and dangerous content by *MailScanner* http://www.mailscanner.info/, and is believed to be clean.
-- This message is intended only for the addressee and may contain confidential information. Unless you are that person, you may not disclose its contents or use it in any way and are requested to delete the message along with any attachments and notify us immediately. "Transact" is operated by Integrated Financial Arrangements plc. 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300. (Registered office: as above; Registered in England and Wales under number: 3727592). Authorised and regulated by the Financial Conduct Authority (entered on the Financial Services Register; no. 190856).
I realise it's probably because of the use of the reject action, which presumably inserts the text "No spamming allowed here." into the subject of the bounce.
However what also concerns me is that sending MDN's back to the envelope sender of SPAM messages is very likely to cause your server to be the origin of "Joe-Job" spam. The sieve RFCs state that if using reject (therefore >sending MDNs) you should also be able to check for forged envelope senders and in that case do not send bounces. In real life it's considered best to just not bother sending anything back at all, and either discard, or block at >the SMTP level using RBLs.
Cheers
Alex
Whatever's the case the backscatter you're talking about has its own ways and methods to be fought with. There are countries for example Germany where it is prohibited by law to discard any email messages silently. You must reject them so that the senders would be aware what is going on. I was told that by one German admin. I am not sure if this is really true but it has some logic on one hand and it is completely groundless on the other hand because you cannot substitute a live person by a machine. Let's say if the machine (computer / server) confirmed reception of the email it does not necessarily mean that any person on the other end also received and read that email. But this is just demagogy that has very vogue legal / law aspects involved.
Am 26.09.2014 um 17:44 schrieb Klaipedaville on Google:
There are countries for example Germany where it is prohibited by law to discard any email messages silently. You must reject them so that the senders would be aware what is going on. I was told that by one German admin. I am not sure if this is really true but it has some logic on one hand
it is true and besides the german legal letter below you violate a second law at the same time - that is why you have to run a spamfilter *before queue* and sa-milter exists - in case you reject a message the sending server is responsible for a bounce
in case you accept and silently drop it you have a unacceptable configuration - independent of laws - in case of a important and time critical mail i need to know it was rejected and so can call the person by phone or try to remove something which triggered a false positive
there are two important rules for mail:
a) if you accept it you have to deliver it b) if you can't deliver it you must not accept it
if you follow b) you don't become a backscatter
http://www.postfix.org/MILTER_README.html
§ 303a StGB - Datenveränderung (1) Wer rechtswidrig Daten (§ 202a Abs. 2) löscht, unterdrückt, unbrauchbar macht oder verändert, wird mit Freiheitsstrafe bis zu zwei Jahren oder mit Geldstrafe bestraft
it is true and besides the german legal letter below you violate a second law at the same time - that is why you have to run a spamfilter *before queue* and sa-milter exists - in case you reject a message the sending server is responsible for a bounce
in case you accept and silently drop it you have a unacceptable configuration - independent of laws - in case of a important and time critical mail i need to know it was rejected and so can call the person by phone or try to >remove something which triggered a false positive
there are two important rules for mail:
a) if you accept it you have to deliver it b) if you can't deliver it you must not accept it
if you follow b) you don't become a backscatter
Yep, that's correct about point b). Plus, I also personally agree that rejecting is better, than having it silently discarded. Good thing is that I do not reside in Germany so I am safe on that
In fact, postfix before-queue content filter has limits in the amount of mail that a site can handle: http://www.postfix.org/SMTPD_PROXY_README.html therefore legal part of this in Germany limits all German people on resources as every single person is forced to run before-queue filters which in my opinion has its own disadvantages on top of everything else and despite the laws as well.
Am 26.09.2014 um 18:18 schrieb Klaipedaville on Google:
it is true and besides the german legal letter below you violate a second law at the same time - that is why you have to run a spamfilter *before queue* and sa-milter exists - in case you reject a message the sending server is responsible for a bounce
in case you accept and silently drop it you have a unacceptable configuration - independent of laws - in case of a important and time critical mail i need to know it was rejected and so can call the person by phone or try to >remove something which triggered a false positive
there are two important rules for mail:
a) if you accept it you have to deliver it b) if you can't deliver it you must not accept it
if you follow b) you don't become a backscatter
Yep, that's correct about point b). Plus, I also personally agree that rejecting is better, than having it silently discarded. Good thing is that I do not reside in Germany so I am safe on that
In fact, postfix before-queue content filter has limits in the amount of mail that a site can handle: http://www.postfix.org/SMTPD_PROXY_README.html therefore legal part of this in Germany limits all German people on resources as every single person is forced to run before-queue filters which in my opinion has its own disadvantages on top of everything else and despite the laws as well.
that's no problem because with RBL weighting and postscreen you reject 95% of the crap before it ever touchs smtpd or even the contentfilter
that stats below are about a maillog starting with Sep 18 19:50:39 for some hundrest domains and currently 2000 valid RCPT, if the contentfilter has to handle most of your incoming flow you made a mistake by not reject earlier with "cheaper" methods
that part never touched the contentfilter
Invalid User: 3074 Disallowed User: 2 Reject Postscreen: 148451 Reject Postfix: 6225 Blacklist: 144880 Pregreet: 5143 Protocol Error: 1607 Helo: 190 Sender Blocked: 200 Sender Invalid: 229 Sender Spoofed: 847 PTR Missing: 1347 PTR Generic: 319
Connections: 185848 Delivered: 20293 Invalid User: 3074 Disallowed User: 2 Reject Postscreen: 148451 Reject Postfix: 6225 Reject Temporary: 507 Blacklist: 144880 Pregreet: 5143 Protocol Error: 1607 Spamfilter: 2297 Virus: 115 Helo: 190 Subject: 42 Attachment: 0 Sender Blocked: 200 Sender Invalid: 229 Sender Spoofed: 847 PTR Missing: 1347 PTR Generic: 319
Reindl: that's no problem because with RBL weighting and postscreen you reject 95% of the crap before it ever touchs smtpd or even the contentfilter that stats below are about a maillog starting with Sep 18 19:50:39 for some hundrest domains and currently 2000 valid RCPT, if the contentfilter has to handle most of your incoming flow you made a mistake by not reject earlier with "cheaper" methods
that part never touched the contentfilter
Invalid User: 3074 Disallowed User: 2 Reject Postscreen: 148451 Reject Postfix: 6225 Blacklist: 144880 Pregreet: 5143 Protocol Error: 1607 Helo: 190 Sender Blocked: 200 Sender Invalid: 229 Sender Spoofed: 847 PTR Missing: 1347 PTR Generic: 319
Connections: 185848 Delivered: 20293 Invalid User: 3074 Disallowed User: 2 Reject Postscreen: 148451 Reject Postfix: 6225 Reject Temporary: 507 Blacklist: 144880 Pregreet: 5143 Protocol Error: 1607 Spamfilter: 2297 Virus: 115 Helo: 190 Subject: 42 Attachment: 0 Sender Blocked: 200 Sender Invalid: 229 Sender Spoofed: 847 PTR Missing: 1347 PTR Generic: 319
That sounds interesting. Since you are German and most probably are an expert on before-queue filtering could you advise if it is actually possible to use both before-queue and after-queue filtering? Since it is the Dovecot list I would be curious to know how to do it Dovecot Sieve way. That is my Postfix passes it over to Dovecot for delivery but all the milters work with Postfix directly and my passing delivery over to Dovecot should theoretically simply disable any before-queue filters. I ran a few tests and it did not work for the reason I have just mentioned, because it is piped to dovecot.
Am 27.09.2014 um 12:01 schrieb Klaipedaville on Google:
Reindl: that's no problem because with RBL weighting and postscreen you reject 95% of the crap before it ever touchs smtpd or even the contentfilter that stats below are about a maillog starting with Sep 18 19:50:39 for some hundrest domains and currently 2000 valid RCPT, if the contentfilter has to handle most of your incoming flow you made a mistake by not reject earlier with "cheaper" methods
That sounds interesting. Since you are German and most probably are an expert on before-queue filtering could you advise if it is actually possible to use both before-queue and after-queue filtering?
surely but how does that make sense?
Since it is the Dovecot list I would be curious to know how to do it Dovecot Sieve way.
doing *what*?
That is my Postfix passes it over to Dovecot for delivery but all the milters work with Postfix directly and my passing delivery over to Dovecot should theoretically simply disable any before-queue filters
that's just impossible
you can't control a before-queue filter that way because, well, he is before-queue and has no idea what later happens with that message
...could you advise if it is actually possible to use both before-queue and after-queue filtering? Reindl surely but how does that make sense?
It makes because it will use two filters, not just one. It will filter before queue first and then anything that may be missed or let through on purpose will be filtered after queue by the second filter.
Since it is the Dovecot list I would be curious to know how to do it Dovecot Sieve way.
doing *what*?
Making two filters work together (dovecot after-queue and postfix before-queue).
That is my Postfix passes it over to Dovecot for delivery but all the milters work with Postfix directly and my passing delivery over to Dovecot should theoretically simply disable any before-queue filters
that's just impossible
It is technically and theoretically impossible as far as I understand but I thought that I also could be wrong at some point and there were ways to arrange it somehow... that’s why I asked away.
you can't control a before-queue filter that way because, well, he is before-queue and has no idea what later happens with that message.
I wouldn’t care what the before-queue would be up to as long as they would "cooperate" together with the after-queue one. That is I would create different rules for "before" and for "after" filters making sure that the rules do not clash, conflict, or contradict with each other in any way. But then again my delivery is done by either Postfix with filters or Dovecot with filters and I am unable to use both. That's actually how my current settings on this server I was setting up work. Postfix uses its regexp header_checks without any filters and it works just fine, then it is passed over to Dovecot for delivery where Spamassassin filters kick in and do their job on top. Thus I have two "spam stoppers" so to speak. The advantage is that I can use both regexp / pcre in Postfix and sieve scripting in Dovecot Sieve.
Am 27.09.2014 um 15:04 schrieb Klaipedaville on Google:
...could you advise if it is actually possible to use both before-queue and after-queue filtering? Reindl surely but how does that make sense?
It makes because it will use two filters, not just one. It will filter before queue first and then anything that may be missed
it don't
having sa-milter, clamav-milter, postscreen, PTR/HELO checks is all before queue there is not much left....
or let through on purpose will be filtered after queue by the second filter
that's not possible except the milter itself has whitelists which are triggered
you need to realize that if a reject before queue was triggered there is nothing happing after because there just exists no queue at all
Since it is the Dovecot list I would be curious to know how to do it Dovecot Sieve way.
doing *what*?
Making two filters work together (dovecot after-queue and postfix before-queue).
it's not the job of dovecot at all
That is my Postfix passes it over to Dovecot for delivery but all the milters work with Postfix directly and my passing delivery over to Dovecot should theoretically simply disable any before-queue filters
that's just impossible
It is technically and theoretically impossible
technically
you can't even have a DNSWL before a milter
you only can have other thins rejecting on meta-data and stop the milter overhead because a decision is already made
you can't control a before-queue filter that way because, well, he is before-queue and has no idea what later happens with that message.
I wouldn’t care what the before-queue would be up to as long as they would "cooperate" together with the after-queue
there is no "cooperate" - if the milter rejects it's rejected if a PTR check rejectes it is rejected no layer after that will get in touch with the message
that's the idea of having several layers and try to get the cheaper ones in front - frankly after postscreen if correctly used the layers after only face 5-10% of all incoming mail
On 26/09/14 16:44, Klaipedaville on Google wrote:
Whatever's the case the backscatter you're talking about has its own ways and methods to be fought with. There are countries for example Germany where it is prohibited by law to discard any email messages silently. You must reject them so that the senders would be aware what is going on. I was told that by one German admin. I am not sure if this is really true but it has some logic on one hand and it is completely groundless on the other hand because you cannot substitute a live person by a machine. Let's say if the machine (computer / server) confirmed reception of the email it does not necessarily mean that any person on the other end also received and read that email. But this is just demagogy that has very vogue legal / law aspects involved.
One *very* convincing argument not to send an *email* response (reject at SMTP is fine) is that it is very likely indeed you'll end up on an RBL yourself for doing this. It happened to us when we were still bouncing (probably about 8-10 years ago). It was the main reason we stopped.
Reindl, I respecfully disagree with (a) at least for the UK. It may be the case in Germany but I'll be damned if I'm going to give up on my Mailscanner
- tuned over the years enough that we've never had a legit mail get canned.
Anyway, that's enough for me otherwise this is going to turn into a flamewar rather than informational.
Am 26.09.2014 um 18:29 schrieb Alex Crow:
Reindl, I respecfully disagree with (a) at least for the UK. It may be the case in Germany but I'll be damned if I'm going to give up on my Mailscanner - tuned over the years enough that we've never had a legit mail get canned.
I respecfully disagree - why can't it run before-queue as spamassassin does with sa-milter and in combination with a wise postscreen/postfix setup faces only 5% of all mail sicne the rest got washed away before?
Anyway, that's enough for me otherwise this is going to turn into a flamewar rather than informational
well, i would not want to be in danger one of my mails got silently dropped and was a false positive
frankly, i was in that situation because using a "Barracuda Spamfirewall" and expecting a sane behavior (reject in case of spam) and it took years to find out it was not the case and needed to be replaced because *in fact* it silently dropped way too much of my own messages as well as from other users too
you just don't know about false positives because not every mail is missed by somebody which don't imply it was junk
frankly while replace that commercial mails i had mail-discussions where i personally did not miss a few messages until the other side talked about a message in the same thread i never saw
Alex: One *very* convincing argument not to send an *email* response (reject at SMTP is fine) is that it is very likely indeed you'll end up on an RBL yourself for doing this. It happened to us when we were still bouncing (probably >about 8-10 years ago). It was the main reason we stopped.
Reindl, I respecfully disagree with (a) at least for the UK. It may be the case in Germany but I'll be damned if I'm going to give up on my Mailscanner - tuned over the years enough that we've never had a legit mail get canned.
Anyway, that's enough for me otherwise this is going to turn into a flamewar rather than informational. It is really convincing but if you do it right I guess you should be safe .
Well, bouncing it all bluntly like shooting from a machine gun absolutely everywhere at random will certainly put you on RBL. How about checking it first if the sender's email address really exists and only then either bounce back (it the address exists) or do not bounce it back if it doesn't. You were the one to tell me in your previous messages to comply with RFC first that is to check for forged envelope senders first. That’s why I have been bouncing it back (rejecting) for the past 4 years and have never ever hit any RBLs at all.
Am 26.09.2014 um 16:49 schrieb Alex Crow:
On 26/09/14 15:27, Klaipedaville on Google wrote:
/^Subject:.**{5}SPAM*{5}/ REJECT No spammers allowed here. /^Subject:.*\*\*\*\*\*SPAM\*\*\*\*\*/ REJECT No spammers allowed. /\s**{5}SPAM*{5}/ REJECT No spamming hullababballos allowed. I think it may be this one above. From the postfix manuals"By default, matching is case-insensitive, and newlines are not treated as special characters. The behavior is controlled by flags, which are toggled by appending one or more of the following characters after the pattern: *i* (default: on) Toggles the case sensitivity flag. By default, matching is case insensitive." Case insensitive is declared by putting this /i at the end of a rule. Postfix has nothing to do with regular expressions (regexp) and regexp is not controlled by postfix. There should be a regexp library available on the server where you are using regexp. It’s like PHP, or tml, or js, or css, it cannot be controlled by postfix
this is just unqualified junk - surely it *has a lot* to do with it because it can and do add the flag as default
frankly every script can add i after / as default
So why does it state in man 5 regexp_table that such tables are *case insensitive* by default and the /i actually toggles that? Are you saying that man page is wrong? I'd be surprised as I don't think I've yet come across an occasion where postfix man pages are incorrect!
they are *not* case-insensitive and it takes 5 seconds to verify that
On 9/26/2014 9:27 AM, Klaipedaville on Google wrote:
/^Subject:.**{5}SPAM*{5}/ REJECT No spammers allowed here. /^Subject:.*\*\*\*\*\*SPAM\*\*\*\*\*/ REJECT No spammers allowed. /\s**{5}SPAM*{5}/ REJECT No spamming hullababballos allowed. I think it may be this one above. From the postfix manuals"By default, matching is case-insensitive, and newlines are not treated as special characters. The behavior is controlled by flags, which are toggled by appending one or more of the following characters after the pattern: *i* (default: on) Toggles the case sensitivity flag. By default, matching is case insensitive." Case insensitive is declared by putting this /i at the end of a rule. Postfix has nothing to do with regular expressions (regexp) and regexp is not controlled by postfix.
Not true. Postfix regexp (and pcre) matches are case insensitive by default, adding the /i flag makes them case sensitive. This should be quite clear in the postfix docs quoted above. This documented default behavior may be different from other software you're familiar with.
You're welcome to discuss postfix header checks further on the postfix-users list.
-- Noel Jones
Not true. Postfix regexp (and pcre) matches are case insensitive by default, adding the /i flag makes them case sensitive. This should be quite clear in the postfix docs quoted above. This documented default behavior may be different from other software you're familiar with.
You're welcome to discuss postfix header checks further on the postfix-users list.
-- Noel Jones
I am not here to argue on details. It's just getting side tracked. I meant in general. A couple of 100 tests that I ran trying to solve this issue included both case sensitive and case insensitive and all the other possible and impossible options I could think of. Perhaps I missed that little detail, but postfix-user list members and Postfix creator Mr. Wetsie Venema in person said that the regexp were correct. I have discussed it there before as well.
participants (4)
-
Alex Crow
-
Klaipedaville on Google
-
Noel
-
Reindl Harald