Sieve filter extracting wrong header value on certain mail messages

gowen (dovecot) gowen-dovecot at swynwyr.com
Wed Aug 6 15:56:10 UTC 2014


# 2.2.12: /etc/dovecot/dovecot.conf
# OS: Linux 3.15.4-x86_64-linode45 x86_64 openSUSE 12.3 (x86_64)

Short version:

1) I filter out spam by using a dovecot sieve filter to act based on a 
header inserted by bogofilter
2) In some cases, sieve fails to correctly file based on this header
3) Testing with sieve-test indicates that it's testing the wrong header
4) Suggestions?  Am I doing something wrong?  Is there a problem with 
these headers that bends test-sieve out of shape?
5) Test files from below are also available via 
http://www.swynwyr.com/sieve-test/

Any help is appreciated!!!


Long version with full details:

Here is the simple sieve filter I'm using to test, and two different 
variants of an email which triggers this problem.  I basically deleted 
things one by one from an original email (1) to get minimal testcases 
that triggered (2) or did not trigger (3) the problem.

The filter:

=== begin test.sieve ===
require ["fileinto", "body", "regex"];

if header :contains "X-Bogosity" "Spam," {
     fileinto "SPAM.SPAM";
     stop;
} elsif header :contains "X-Bogosity" "Unsure," {
     fileinto "SPAM.UNSURE";
     stop;
}

keep;
stop;
=== end test.sieve ===

Here is an example email that will cause this filter to not function as 
expected:

=== begin message.2 ===
Return-Path: <alora at promosforusio.org>
Subject:   =?utf-8?B?QUhTIEtpdGNoZW4gTWFrZW92ZXIgR2l2ZWF3YXkg?=
Subject:                                          ÎHS KitÑhen Îakеovеr 
Givеaway
Subject:      =?ISO-8859-1?Q?AHS Kitchen Makeover Giveaway =A0=A0=A0?=
Subject:
Subject:                AHS Kitchen Makeover Giveaway
X-Bogosity: Spam, tests=bogofilter, spamicity=1.000000, version=1.2.3


Home Warranty Summer Giveaway
=== end message.2 ===

If I run test-sieve on this, we can clearly see in the trace log that 
when it's supposed to test for header "X-Bogosity" it in fact looks at 
one of the multiple "Subject" headers.  Note that the result is storing 
in INBOX rather than SPAM.SPAM, and that the "matching value" for 
"X-Bogosity headers" is actually the contents of the final "Subject" 
header in the test on line 3.  In the second test, on line 6, it uses a 
different and correct value for the same header!  But too late to help 
as the first test was the one it should have had a match on.

=== begin session ===
$ sieve-test -t trace2.log -Tlevel=matching test.sieve message.2

Performed actions:

  * store message in folder: INBOX

Implicit keep:

   (none)

sieve-test(gowen): Info: final result: success
$ cat trace2.log

       ## Started executing script 'test'
    3: header test
    3:   starting `:contains' match with `i;ascii-casemap' comparator:
    3:   extracting `X-Bogosity' headers from message
    3:   matching value `AHS Kitchen Makeover Giveaway'
    3:     with key `Spam,' => 0
    3:   finishing match with result: not matched
    3: jump if result is false
    3:   jumping to line 6
    6: header test
    6:   starting `:contains' match with `i;ascii-casemap' comparator:
    6:   extracting `X-Bogosity' headers from message
    6:   matching value `Spam, tests=bogofilter, spamicity=1.000000, 
version=1.2.3'
    6:     with key `Unsure,' => 0
    6:   finishing match with result: not matched
    6: jump if result is false
    6:   jumping to line 11
   11: keep action; store message in default mailbox
   12: stop command; end all script execution
       ## Finished executing script 'test'

$
=== end session ===

Now lets remove the "Return-Path" from that message.  (Of the original 
26 headers, all but the 7 shown in message.2 could be removed and still 
have this message break the filter).  This alters thing enough that 
sieve starts working as expected (in fact, removing any one of the 6 
non-X-Bogosity headers will cause this to start working).

=== begin message.3 ===
Subject:   =?utf-8?B?QUhTIEtpdGNoZW4gTWFrZW92ZXIgR2l2ZWF3YXkg?=
Subject:                                          ÎHS KitÑhen Îakеovеr 
Givеaway
Subject:      =?ISO-8859-1?Q?AHS Kitchen Makeover Giveaway =A0=A0=A0?=
Subject:
Subject:                AHS Kitchen Makeover Giveaway
X-Bogosity: Spam, tests=bogofilter, spamicity=1.000000, version=1.2.3


Home Warranty Summer Giveaway
=== end message.3 ===

Now when we run test-sieve it works as expected, filing into SPAM.SPAM 
and showing the correct matching value for line 3:

=== begin session ===
$ sieve-test -t trace3.log -Tlevel=matching test.sieve message.3

Performed actions:

  * store message in folder: SPAM.SPAM

Implicit keep:

   (none)

sieve-test(gowen): Info: final result: success
$ cat trace3.log

       ## Started executing script 'test'
    3: header test
    3:   starting `:contains' match with `i;ascii-casemap' comparator:
    3:   extracting `X-Bogosity' headers from message
    3:   matching value `Spam, tests=bogofilter, spamicity=1.000000, 
version=1.2.3'
    3:     with key `Spam,' => 1
    3:   finishing match with result: matched
    3: jump if result is false
    3:   not jumping
    4: fileinto action
    4:   store message in mailbox `SPAM.SPAM'
    5: stop command; end all script execution
       ## Finished executing script 'test'

$
=== end session ===



More information about the dovecot mailing list