[Dovecot] sieve problem email silently discard
hello folks hello Timo hello all the guru
I met a few times the problem or an email that passes through a sieve script is silently discard after delivery but never returned to the inbox
all testimonials are welcome
my dovecot -n ~]# /usr/sbin/dovecot -n # 2.0.13: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32.2-xxxx-grs-ipv6-32 i686 CentOS release 5.6 (Final) auth_mechanisms = plain login base_dir = /var/run/dovecot/ lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes listen = [::] log_path = /var/log/maillog log_timestamp = %Y-%m-%d %H:%M:%S login_log_format_elements = user=<%u> method=%m rip=%r lip=%l %c mail_debug = yes mail_location = maildir:~/Maildir mail_max_userip_connections = 20 managesieve_notify_capability = mailto managesieve_sieve_capability = comparator-i;octet comparator-i;ascii-casemap fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date spamtest spamtestplus virustest namespace { inbox = yes location = prefix = separator = . } passdb { driver = pam } plugin { autocreate = Trash autocreate2 = Junk autocreate3 = Sent autocreate4 = Drafts autosubscribe = Trash autosubscribe2 = Junk autosubscribe3 = Sent autosubscribe4 = Drafts plugin = $mail_plugins autocreate managesieve sieve sieve = ~/.dovecot.sieve sieve_before = /var/sieve-scripts/roundcube.sieve sieve_dir = ~/sieve sieve_global_path = whatever } protocols = sieve imap pop3 service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0600 user = postfix } unix_listener auth-master { mode = 0666 } unix_listener auth-userdb { mode = 0666 } vsz_limit = 64 M } service imap-login { inet_listener imap { port = 0 } inet_listener imaps { address = * , [::] port = 993 } process_limit = 128 vsz_limit = 64 M } service imap { client_limit = 1 service_count = 0 } service managesieve-login { inet_listener managesieve-login { address = * , [::] port = 2000 } process_limit = 128 vsz_limit = 64 M } service pop3-login { inet_listener pop3 { port = 0 } inet_listener pop3s { address = * , [::] port = 995 } process_limit = 128 vsz_limit = 64 M } ssl_ca =
egrep lda /var/Log/maillog 2011-07-03 19:47:15lda: Debug: Loading modules from directory: /usr/lib/dovecot 2011-07-03 19:47:15lda: Debug: Module loaded: /usr/lib/dovecot/lib20_autocreate_plugin.so 2011-07-03 19:47:15lda: Debug: Module loaded: /usr/lib/dovecot/lib90_sieve_plugin.so 2011-07-03 19:47:15lda(fakessh): Debug: Effective uid=500, gid=100, home=/home/fakessh 2011-07-03 19:47:15lda(fakessh): Debug: Namespace : type=private, prefix=, sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:~/Maildir 2011-07-03 19:47:15lda(fakessh): Debug: maildir++: root=/home/fakessh/Maildir, index=, control=, inbox=/home/fakessh/Maildir 2011-07-03 19:47:15lda(fakessh): Debug: Namespace : Using permissions from /home/fakessh/Maildir: mode=0775 gid=-1 2011-07-03 19:47:15lda(fakessh): Debug: userdb lookup skipped, username taken from USER environment 2011-07-03 19:47:15lda(fakessh): Debug: none: root=, index=, control=, inbox= 2011-07-03 19:47:15lda(fakessh): Debug: Destination address: fakessh@r13151.ovh.net (source: user@hostname) 2011-07-03 19:47:15lda(fakessh): Debug: sieve: using sieve path for user's script: /home/fakessh/.dovecot.sieve 2011-07-03 19:47:15lda(fakessh): Debug: sieve: executed before user's script(1): /var/sieve-scripts/roundcube.sieve 2011-07-03 19:47:15lda(fakessh): Debug: sieve: opening script /var/sieve-scripts/roundcube.sieve 2011-07-03 19:47:15lda(fakessh): Debug: sieve: script binary /var/sieve-scripts/roundcube.svbin successfully loaded 2011-07-03 19:47:15lda(fakessh): Debug: sieve: binary save: not saving binary /var/sieve-scripts/roundcube.svbin, because it is already stored 2011-07-03 19:47:15lda(fakessh): Debug: sieve: executing script from /var/sieve-scripts/roundcube.svbin 2011-07-03 19:47:15lda(fakessh): Info: sieve: msgid=6EBEE5FC-62B1-4C73-B26E-DEBFD6E26DB6@shorewall.net: marked message to be discarded if not explicitly delivered (discard action)
-- http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x092164A7 gpg --keyserver pgp.mit.edu --recv-key 092164A7
On 07/03/2011 10:22 PM, ml@smtp.fakessh.eu wrote:
I met a few times the problem or an email that passes through a sieve script is silently discard after delivery but never returned to the inbox
all testimonials are welcome
[...]
plugin { [...] plugin = $mail_plugins autocreate managesieve sieve sieve = ~/.dovecot.sieve sieve_before = /var/sieve-scripts/roundcube.sieve sieve_dir = ~/sieve sieve_global_path = whatever }
[...]
egrep lda /var/Log/maillog [...] 2011-07-03 19:47:15lda(fakessh): Debug: sieve: executed before user's script(1): /var/sieve-scripts/roundcube.sieve [...] 2011-07-03 19:47:15lda(fakessh): Debug: sieve: executing script from /var/sieve-scripts/roundcube.svbin 2011-07-03 19:47:15lda(fakessh): Info: sieve: msgid=6EBEE5FC-62B1-4C73-B26E-DEBFD6E26DB6@shorewall.net: marked message to be discarded if not explicitly delivered (discard action)
I would not call this a silent discard: it is reported in the log above. Also, the contents of roundcube.sieve that you showed in an earlier thread contained a single discard action, which is therefore the only likely culprit. Now the question boils down to: why is this discard action triggered? The interesting part of the script is the following:
require ["comparator-i;ascii-numeric","relational"]; if header :value "ge" :comparator "i;ascii-numeric" ["X-Spam-score"]["500"] { discard; stop; }
Since the message is discarded, it will most likely be impossible to retrieve that message and check why it fires this rule. The relatively innocent situation would be that your Spam filter truly produces this interesting score of > 500. A less innocent case would be a bug in the Sieve interpreter. Either way, we need to have access to one of those messages that triggers this rule and gets discarded without apparent reason.
My suggestion is to replace that discard;' action with a fileinto
:create "Debug";' action (:create creates the folder implicitly; depends
on mailbox extension require) to file messages that would normally be
discarded into a special folder for later evaluation. Alternatively, you
can redirect such messages to a special mail account.
Only then can we trace this problem any deeper.
Regards,
Stephan.
Le dimanche 3 juillet 2011 23:57, Stephan Bosch a écrit :
On 07/03/2011 10:22 PM, ml@smtp.fakessh.eu wrote:
I met a few times the problem or an email that passes through a sieve script is silently discard after delivery but never returned to the inbox
all testimonials are welcome
[...]
plugin {
[...]
plugin = $mail_plugins autocreate managesieve sieve sieve = ~/.dovecot.sieve sieve_before = /var/sieve-scripts/roundcube.sieve sieve_dir = ~/sieve sieve_global_path = whatever }
[...]
egrep lda /var/Log/maillog
[...]
2011-07-03 19:47:15lda(fakessh): Debug: sieve: executed before user's script(1): /var/sieve-scripts/roundcube.sieve
[...]
2011-07-03 19:47:15lda(fakessh): Debug: sieve: executing script from /var/sieve-scripts/roundcube.svbin 2011-07-03 19:47:15lda(fakessh): Info: sieve: msgid=6EBEE5FC-62B1-4C73-B26E-DEBFD6E26DB6@shorewall.net: marked message to be discarded if not explicitly delivered (discard action)
I would not call this a silent discard: it is reported in the log above. Also, the contents of roundcube.sieve that you showed in an earlier thread contained a single discard action, which is therefore the only likely culprit. Now the question boils down to: why is this discard action triggered? The interesting part of the script is the following:
require ["comparator-i;ascii-numeric","relational"]; if header :value "ge" :comparator "i;ascii-numeric" ["X-Spam-score"]["500"] { discard; stop; }
Since the message is discarded, it will most likely be impossible to retrieve that message and check why it fires this rule. The relatively innocent situation would be that your Spam filter truly produces this interesting score of > 500. A less innocent case would be a bug in the Sieve interpreter. Either way, we need to have access to one of those messages that triggers this rule and gets discarded without apparent reason.
My suggestion is to replace that
discard;' action with afileinto:create "Debug";' action (:create creates the folder implicitly; depends
on mailbox extension require) to file messages that would normally be discarded into a special folder for later evaluation. Alternatively, you can redirect such messages to a special mail account.
Only then can we trace this problem any deeper.
Regards,
Stephan.
thanks Stephan
I just change my sieve script by removing the implicit discard a fileinto :create "Junk.spam.spam"
until next time
-- http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x092164A7 gpg --keyserver pgp.mit.edu --recv-key 092164A7
Le lundi 4 juillet 2011 00:40, ml@smtp.fakessh.eu a écrit :
Le dimanche 3 juillet 2011 23:57, Stephan Bosch a écrit :
On 07/03/2011 10:22 PM, ml@smtp.fakessh.eu wrote:
I met a few times the problem or an email that passes through a sieve script is silently discard after delivery but never returned to the inbox
all testimonials are welcome
[...]
plugin {
[...]
plugin = $mail_plugins autocreate managesieve sieve sieve = ~/.dovecot.sieve sieve_before = /var/sieve-scripts/roundcube.sieve sieve_dir = ~/sieve sieve_global_path = whatever }
[...]
egrep lda /var/Log/maillog
[...]
2011-07-03 19:47:15lda(fakessh): Debug: sieve: executed before user's script(1): /var/sieve-scripts/roundcube.sieve
[...]
2011-07-03 19:47:15lda(fakessh): Debug: sieve: executing script from /var/sieve-scripts/roundcube.svbin 2011-07-03 19:47:15lda(fakessh): Info: sieve: msgid=<6EBEE5FC-62B1-4C73-B26E-DEBFD6E26DB6@shorewall.net>: marked message to be discarded if not explicitly delivered (discard action)
I would not call this a silent discard: it is reported in the log above. Also, the contents of roundcube.sieve that you showed in an earlier thread contained a single discard action, which is therefore the only likely culprit. Now the question boils down to: why is this discard action triggered? The interesting part of the script is the following:
require ["comparator-i;ascii-numeric","relational"]; if header :value "ge" :comparator "i;ascii-numeric" ["X-Spam-score"]["500"] { discard; stop; }
Since the message is discarded, it will most likely be impossible to retrieve that message and check why it fires this rule. The relatively innocent situation would be that your Spam filter truly produces this interesting score of > 500. A less innocent case would be a bug in the Sieve interpreter. Either way, we need to have access to one of those messages that triggers this rule and gets discarded without apparent reason.
My suggestion is to replace that `discard;' action with a `fileinto
:create "Debug";' action (:create creates the folder implicitly; depends
on mailbox extension require) to file messages that would normally be discarded into a special folder for later evaluation. Alternatively, you can redirect such messages to a special mail account.
Only then can we trace this problem any deeper.
Regards,
Stephan.
thanks Stephan
I just change my sieve script by removing the implicit discard a fileinto :create "Junk.spam.spam"
until next time
Hi Stephan
it just happened a mail that was issued in INBOX.spam.spam
supposedly a hit with spam than 500
which does not appear in the body of the mail here
Return-Path:
Op 4-7-2011 14:19, ml@smtp.fakessh.eu schreef:
Le lundi 4 juillet 2011 00:40, ml@smtp.fakessh.eu a écrit :
I just change my sieve script by removing the implicit discard a fileinto :create "Junk.spam.spam" [...] it just happened a mail that was issued in INBOX.spam.spam supposedly a hit with spam than 500 which does not appear in the body of the mail here [...] I do not see why this email was issued in this box
I've executed sieve-test with your script and message, which reproduces the problem at my end:
===
$ sieve-test -t - -T level=matching ~/fakessh.sieve ~/fakessh.eml
## Started executing script 'frop'
2: header test
2: starting :value-ge' match with i;ascii-numeric' comparator:
2: extracting X-Spam-score' headers from message 2: matching value -1.9'
2: with key `500' => 1
2: finishing match with result: matched
3: jump if result is false
3: not jumping
5: discard action; cancel implicit keep
6: stop command; end all script execution
## Finished executing script 'frop'
Performed actions:
- discard
Implicit keep:
(none)
sieve-test(stephan): Info: final result: success
This turns out to be a classic mistake actually (which I didn't think of either). It is related to the (admittedly counter-intuitive) nature of the i;ascii-numeric comparator.
From RFC4790, Section 9.1.1 (http://tools.ietf.org/html/rfc4790#section-9.1.1): `The "i;ascii-numeric" collation is a simple collation intended for use with arbitrarily-sized, unsigned decimal integer numbers stored as octet strings. US-ASCII digits (0x30 to 0x39) represent digits of the numbers. Before converting from string to integer, the input string is truncated at the first non-digit character. All input is valid; strings that do not start with a digit represent positive infinity.'
This comparator thus works on UNSIGNED integers only. Even worse, negative numbers are mapped to positive infinity, which is obviously > 500! There is your problem. I remember that issue was reported some time ago by someone else too.
To solve your problem, you need to check for the negative sign first. E.g.:
require ["comparator-i;ascii-numeric","relational"]; if allof( not header :matches "x-spam-score" "-*", header :value "ge" :comparator "i;ascii-numeric" "x-spam-score" "500") { discard; stop; }
Or, even better: start using the spamtest(plus) extension.
Regards,
Stephan.
On Mon, 04 Jul 2011 15:01:55 +0200 Stephan Bosch stephan@rename-it.nl wrote:
Op 4-7-2011 14:19, ml@smtp.fakessh.eu schreef:
Le lundi 4 juillet 2011 00:40, ml@smtp.fakessh.eu a écrit :
I just change my sieve script by removing the implicit discard a fileinto :create "Junk.spam.spam" [...] it just happened a mail that was issued in INBOX.spam.spam supposedly a hit with spam than 500 which does not appear in the body of the mail here [...] I do not see why this email was issued in this box
I've executed sieve-test with your script and message, which reproduces the problem at my end:
=== $ sieve-test -t - -T level=matching ~/fakessh.sieve ~/fakessh.eml ## Started executing script 'frop' 2: header test 2: starting
:value-ge' match withi;ascii-numeric' comparator: 2: extractingX-Spam-score' headers from message 2: matching value-1.9' 2: with key `500' => 1 2: finishing match with result: matched 3: jump if result is false 3: not jumping 5: discard action; cancel implicit keep 6: stop command; end all script execution ## Finished executing script 'frop'Performed actions:
- discard
Implicit keep:
(none)
sieve-test(stephan): Info: final result: success
This turns out to be a classic mistake actually (which I didn't think of either). It is related to the (admittedly counter-intuitive) nature of the i;ascii-numeric comparator.
From RFC4790, Section 9.1.1 (http://tools.ietf.org/html/rfc4790#section-9.1.1): `The "i;ascii-numeric" collation is a simple collation intended for use with arbitrarily-sized, unsigned decimal integer numbers stored as octet strings. US-ASCII digits (0x30 to 0x39) represent digits of the numbers. Before converting from string to integer, the input string is truncated at the first non-digit character. All input is valid; strings that do not start with a digit represent positive infinity.'
This comparator thus works on UNSIGNED integers only. Even worse, negative numbers are mapped to positive infinity, which is obviously > 500! There is your problem. I remember that issue was reported some time ago by someone else too.
To solve your problem, you need to check for the negative sign first. E.g.:
require ["comparator-i;ascii-numeric","relational"]; if allof( not header :matches "x-spam-score" "-*", header :value "ge" :comparator "i;ascii-numeric" "x-spam-score" "500") { discard; stop; }
a simple script are one syntax proximate to a sample exemple to stephan
how to make a complex script that deals with both spam spam hist flag suspicious address diverse
i try this ~]# cat /var/sieve-scripts/roundcube.sieve
require ["fileinto","regex","comparator-i;ascii-numeric","reject","relational","mailbox","reject","variables","envelope","subaddress"]; # rule:[spammanage] #if anyof (header :contains "X-Spam-Flag" "YES") #{ # fileinto "Junk"; #} if allof( not header :matches "x-spam-score" "-*", header :value "ge" :comparator "i;ascii-numeric" "x-spam-score" "500") { discard; stop; } if anyof ( # puremsg scores (30% or higher) header :matches ["X-Spam-Flag"] ["Yes"] ) { fileinto "Junk"; stop; }
elsif anyof ( header :contains "Received" [ "[4.63.221.224",
]) { fileinto :create "Junk"; } elsif anyof ( header :contains ["SPAM", "X-Spam-Status"] ["ADDRESSES_ON_CD","ACT_NOW", ] ) { fileinto :create "Junk"; }
or better much approch is the succession a if anyof elsif anyof not work for the discard action
Or, even better: start using the spamtest(plus) extension.
Regards,
Stephan.
-- http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x092164A7 gpg --keyserver pgp.mit.edu --recv-key 092164A7
On Wed, 6 Jul 2011 18:17:07 +0200 fakessh @ ml@smtp.fakessh.eu wrote:
On Mon, 04 Jul 2011 15:01:55 +0200 Stephan Bosch stephan@rename-it.nl wrote:
Op 4-7-2011 14:19, ml@smtp.fakessh.eu schreef:
Le lundi 4 juillet 2011 00:40, ml@smtp.fakessh.eu a écrit :
I just change my sieve script by removing the implicit discard a fileinto :create "Junk.spam.spam" [...] it just happened a mail that was issued in INBOX.spam.spam supposedly a hit with spam than 500 which does not appear in the body of the mail here [...] I do not see why this email was issued in this box
I've executed sieve-test with your script and message, which reproduces the problem at my end:
=== $ sieve-test -t - -T level=matching ~/fakessh.sieve ~/fakessh.eml ## Started executing script 'frop' 2: header test 2: starting
:value-ge' match withi;ascii-numeric' comparator: 2: extractingX-Spam-score' headers from message 2: matching value-1.9' 2: with key `500' => 1 2: finishing match with result: matched 3: jump if result is false 3: not jumping 5: discard action; cancel implicit keep 6: stop command; end all script execution ## Finished executing script 'frop'Performed actions:
- discard
Implicit keep:
(none)
sieve-test(stephan): Info: final result: success
This turns out to be a classic mistake actually (which I didn't think of either). It is related to the (admittedly counter-intuitive) nature of the i;ascii-numeric comparator.
From RFC4790, Section 9.1.1 (http://tools.ietf.org/html/rfc4790#section-9.1.1): `The "i;ascii-numeric" collation is a simple collation intended for use with arbitrarily-sized, unsigned decimal integer numbers stored as octet strings. US-ASCII digits (0x30 to 0x39) represent digits of the numbers. Before converting from string to integer, the input string is truncated at the first non-digit character. All input is valid; strings that do not start with a digit represent positive infinity.'
This comparator thus works on UNSIGNED integers only. Even worse, negative numbers are mapped to positive infinity, which is obviously > 500! There is your problem. I remember that issue was reported some time ago by someone else too.
To solve your problem, you need to check for the negative sign first. E.g.:
require ["comparator-i;ascii-numeric","relational"]; if allof( not header :matches "x-spam-score" "-*", header :value "ge" :comparator "i;ascii-numeric" "x-spam-score" "500") { discard; stop; }
a simple script are one syntax proximate to a sample exemple to stephan
how to make a complex script that deals with both spam spam hist flag suspicious address diverse
i try this ~]# cat /var/sieve-scripts/roundcube.sieve
require ["fileinto","regex","comparator-i;ascii-numeric","reject","relational","mailbox","reject","variables","envelope","subaddress"]; # rule:[spammanage] #if anyof (header :contains "X-Spam-Flag" "YES") #{ # fileinto "Junk"; #} if allof( not header :matches "x-spam-score" "-*", header :value "ge" :comparator "i;ascii-numeric" "x-spam-score" "500") { discard; stop; } if anyof ( # puremsg scores (30% or higher) header :matches ["X-Spam-Flag"] ["Yes"] ) { fileinto "Junk"; stop; }
elsif anyof ( header :contains "Received" [ "[4.63.221.224",
]) { fileinto :create "Junk"; } elsif anyof ( header :contains ["SPAM", "X-Spam-Status"] ["ADDRESSES_ON_CD","ACT_NOW", ] ) { fileinto :create "Junk"; }
or better much approch is the succession a if anyof elsif anyof not work for the discard action
Or, even better: start using the spamtest(plus) extension.
Regards,
Stephan.
work well transform allof in anyof
-- http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x092164A7 gpg --keyserver pgp.mit.edu --recv-key 092164A7
participants (3)
-
unknown＠example.com
-
ml＠smtp.fakessh.eu
-
Stephan Bosch