New subject: Sieve filter extracting wrong header value on certain mail messages

6 Aug 2014

      # 2.2.12: /etc/dovecot/dovecot.conf
# OS: Linux 3.15.4-x86_64-linode45 x86_64 openSUSE 12.3 (x86_64)
Short version:

I filter out spam by using a dovecot sieve filter to act based on a
header inserted by bogofilter
In some cases, sieve fails to correctly file based on this header
Testing with sieve-test indicates that it's testing the wrong header
Suggestions?  Am I doing something wrong?  Is there a problem with
these headers that bends test-sieve out of shape?
Test files from below are also available via
http://www.swynwyr.com/sieve-test/

Any help is appreciated!!!
Long version with full details:
Here is the simple sieve filter I'm using to test, and two different
variants of an email which triggers this problem.  I basically deleted
things one by one from an original email (1) to get minimal testcases
that triggered (2) or did not trigger (3) the problem.
The filter:
=== begin test.sieve ===
require ["fileinto", "body", "regex"];
if header :contains "X-Bogosity" "Spam," {
fileinto "SPAM.SPAM";
stop;
} elsif header :contains "X-Bogosity" "Unsure," {
fileinto "SPAM.UNSURE";
stop;
}
keep;
stop;
=== end test.sieve ===
Here is an example email that will cause this filter to not function as
expected:
=== begin message.2 ===
Return-Path: alora@promosforusio.org
Subject:   =?utf-8?B?QUhTIEtpdGNoZW4gTWFrZW92ZXIgR2l2ZWF3YXkg?=
Subject:                                          ÎHS KitÑhen ÎakÐµovÐµr
GivÐµaway
Subject:      =?ISO-8859-1?Q?AHS Kitchen Makeover Giveaway =A0=A0=A0?=
Subject:
Subject:                AHS Kitchen Makeover Giveaway
X-Bogosity: Spam, tests=bogofilter, spamicity=1.000000, version=1.2.3
Home Warranty Summer Giveaway
=== end message.2 ===
If I run test-sieve on this, we can clearly see in the trace log that
when it's supposed to test for header "X-Bogosity" it in fact looks at
one of the multiple "Subject" headers.  Note that the result is storing
in INBOX rather than SPAM.SPAM, and that the "matching value" for
"X-Bogosity headers" is actually the contents of the final "Subject"
header in the test on line 3.  In the second test, on line 6, it uses a
different and correct value for the same header!  But too late to help
as the first test was the one it should have had a match on.
=== begin session ===
$ sieve-test -t trace2.log -Tlevel=matching test.sieve message.2
Performed actions:

store message in folder: INBOX

Implicit keep:
(none)
sieve-test(gowen): Info: final result: success
$ cat trace2.log
   ## Started executing script 'test'
3: header test
3:   starting `:contains' match with `i;ascii-casemap' comparator:
3:   extracting `X-Bogosity' headers from message
3:   matching value `AHS Kitchen Makeover Giveaway'
3:     with key `Spam,' => 0
3:   finishing match with result: not matched
3: jump if result is false
3:   jumping to line 6
6: header test
6:   starting `:contains' match with `i;ascii-casemap' comparator:
6:   extracting `X-Bogosity' headers from message
6:   matching value `Spam, tests=bogofilter, spamicity=1.000000, 
version=1.2.3'
6:     with key `Unsure,' => 0
6:   finishing match with result: not matched
6: jump if result is false
6:   jumping to line 11
11: keep action; store message in default mailbox
12: stop command; end all script execution
## Finished executing script 'test'
$
=== end session ===
Now lets remove the "Return-Path" from that message.  (Of the original
26 headers, all but the 7 shown in message.2 could be removed and still
have this message break the filter).  This alters thing enough that
sieve starts working as expected (in fact, removing any one of the 6
non-X-Bogosity headers will cause this to start working).
=== begin message.3 ===
Subject:   =?utf-8?B?QUhTIEtpdGNoZW4gTWFrZW92ZXIgR2l2ZWF3YXkg?=
Subject:                                          ÎHS KitÑhen ÎakÐµovÐµr
GivÐµaway
Subject:      =?ISO-8859-1?Q?AHS Kitchen Makeover Giveaway =A0=A0=A0?=
Subject:
Subject:                AHS Kitchen Makeover Giveaway
X-Bogosity: Spam, tests=bogofilter, spamicity=1.000000, version=1.2.3
Home Warranty Summer Giveaway
=== end message.3 ===
Now when we run test-sieve it works as expected, filing into SPAM.SPAM
and showing the correct matching value for line 3:
=== begin session ===
$ sieve-test -t trace3.log -Tlevel=matching test.sieve message.3
Performed actions:

store message in folder: SPAM.SPAM

Implicit keep:
(none)
sieve-test(gowen): Info: final result: success
$ cat trace3.log
   ## Started executing script 'test'
3: header test
3:   starting `:contains' match with `i;ascii-casemap' comparator:
3:   extracting `X-Bogosity' headers from message
3:   matching value `Spam, tests=bogofilter, spamicity=1.000000, 
version=1.2.3'
3:     with key Spam,' => 1     3:   finishing match with result: matched     3: jump if result is false     3:   not jumping     4: fileinto action     4:   store message in mailbox SPAM.SPAM'
5: stop command; end all script execution
## Finished executing script 'test'
$
=== end session ===

Sieve filter extracting wrong header value on certain mail messages

gowen (dovecot)

Stephan Bosch

gowen (dovecot)

Timo Sirainen

tags

participants (3)