Sieve filter extracting wrong header value on certain mail messages

[Timo Sirainen]

Finally fixed: http://hg.dovecot.org/dovecot-2.2/rev/0e1a3c909a13

On 06 Aug 2014, at 18:56, gowen (dovecot) <gowen-dovecot at swynwyr.com> wrote:

> 
> # 2.2.12: /etc/dovecot/dovecot.conf
> # OS: Linux 3.15.4-x86_64-linode45 x86_64 openSUSE 12.3 (x86_64)
> 
> Short version:
> 
> 1) I filter out spam by using a dovecot sieve filter to act based on a header inserted by bogofilter
> 2) In some cases, sieve fails to correctly file based on this header
> 3) Testing with sieve-test indicates that it's testing the wrong header
> 4) Suggestions?  Am I doing something wrong?  Is there a problem with these headers that bends test-sieve out of shape?
> 5) Test files from below are also available via http://www.swynwyr.com/sieve-test/
> 
> Any help is appreciated!!!
> 
> 
> Long version with full details:
> 
> Here is the simple sieve filter I'm using to test, and two different variants of an email which triggers this problem.  I basically deleted things one by one from an original email (1) to get minimal testcases that triggered (2) or did not trigger (3) the problem.
> 
> The filter:
> 
> === begin test.sieve ===
> require ["fileinto", "body", "regex"];
> 
> if header :contains "X-Bogosity" "Spam," {
>    fileinto "SPAM.SPAM";
>    stop;
> } elsif header :contains "X-Bogosity" "Unsure," {
>    fileinto "SPAM.UNSURE";
>    stop;
> }
> 
> keep;
> stop;
> === end test.sieve ===
> 
> Here is an example email that will cause this filter to not function as expected:
> 
> === begin message.2 ===
> Return-Path: <alora at promosforusio.org>
> Subject:   =?utf-8?B?QUhTIEtpdGNoZW4gTWFrZW92ZXIgR2l2ZWF3YXkg?=
> Subject:                                          ÎHS KitÑhen Îakеovеr Givеaway
> Subject:      =?ISO-8859-1?Q?AHS Kitchen Makeover Giveaway =A0=A0=A0?=
> Subject:
> Subject:                AHS Kitchen Makeover Giveaway
> X-Bogosity: Spam, tests=bogofilter, spamicity=1.000000, version=1.2.3
> 
> 
> Home Warranty Summer Giveaway
> === end message.2 ===
> 
> If I run test-sieve on this, we can clearly see in the trace log that when it's supposed to test for header "X-Bogosity" it in fact looks at one of the multiple "Subject" headers.  Note that the result is storing in INBOX rather than SPAM.SPAM, and that the "matching value" for "X-Bogosity headers" is actually the contents of the final "Subject" header in the test on line 3.  In the second test, on line 6, it uses a different and correct value for the same header!  But too late to help as the first test was the one it should have had a match on.
> 
> === begin session ===
> $ sieve-test -t trace2.log -Tlevel=matching test.sieve message.2
> 
> Performed actions:
> 
> * store message in folder: INBOX
> 
> Implicit keep:
> 
>  (none)
> 
> sieve-test(gowen): Info: final result: success
> $ cat trace2.log
> 
>      ## Started executing script 'test'
>   3: header test
>   3:   starting `:contains' match with `i;ascii-casemap' comparator:
>   3:   extracting `X-Bogosity' headers from message
>   3:   matching value `AHS Kitchen Makeover Giveaway'
>   3:     with key `Spam,' => 0
>   3:   finishing match with result: not matched
>   3: jump if result is false
>   3:   jumping to line 6
>   6: header test
>   6:   starting `:contains' match with `i;ascii-casemap' comparator:
>   6:   extracting `X-Bogosity' headers from message
>   6:   matching value `Spam, tests=bogofilter, spamicity=1.000000, version=1.2.3'
>   6:     with key `Unsure,' => 0
>   6:   finishing match with result: not matched
>   6: jump if result is false
>   6:   jumping to line 11
>  11: keep action; store message in default mailbox
>  12: stop command; end all script execution
>      ## Finished executing script 'test'
> 
> $
> === end session ===
> 
> Now lets remove the "Return-Path" from that message.  (Of the original 26 headers, all but the 7 shown in message.2 could be removed and still have this message break the filter).  This alters thing enough that sieve starts working as expected (in fact, removing any one of the 6 non-X-Bogosity headers will cause this to start working).
> 
> === begin message.3 ===
> Subject:   =?utf-8?B?QUhTIEtpdGNoZW4gTWFrZW92ZXIgR2l2ZWF3YXkg?=
> Subject:                                          ÎHS KitÑhen Îakеovеr Givеaway
> Subject:      =?ISO-8859-1?Q?AHS Kitchen Makeover Giveaway =A0=A0=A0?=
> Subject:
> Subject:                AHS Kitchen Makeover Giveaway
> X-Bogosity: Spam, tests=bogofilter, spamicity=1.000000, version=1.2.3
> 
> 
> Home Warranty Summer Giveaway
> === end message.3 ===
> 
> Now when we run test-sieve it works as expected, filing into SPAM.SPAM and showing the correct matching value for line 3:
> 
> === begin session ===
> $ sieve-test -t trace3.log -Tlevel=matching test.sieve message.3
> 
> Performed actions:
> 
> * store message in folder: SPAM.SPAM
> 
> Implicit keep:
> 
>  (none)
> 
> sieve-test(gowen): Info: final result: success
> $ cat trace3.log
> 
>      ## Started executing script 'test'
>   3: header test
>   3:   starting `:contains' match with `i;ascii-casemap' comparator:
>   3:   extracting `X-Bogosity' headers from message
>   3:   matching value `Spam, tests=bogofilter, spamicity=1.000000, version=1.2.3'
>   3:     with key `Spam,' => 1
>   3:   finishing match with result: matched
>   3: jump if result is false
>   3:   not jumping
>   4: fileinto action
>   4:   store message in mailbox `SPAM.SPAM'
>   5: stop command; end all script execution
>      ## Finished executing script 'test'
> 
> $
> === end session ===