Once upon a time, Stephan Bosch stephan@rename-it.nl said:

You can check the handling of a particular message yourself using the sieve-test tool (there is a man page for it). By specifying the -t - -Tlevel=matching options, you'll get detailed information on why a particular decision is made.

Hmm, that's weird. sieve-test says it would store the message into the Spam folder (as expected), but it was definitely delivered to INBOX.

Without insight in your configuration (output from dovecot -n) and an example of a message that is mishandled, I cannot tell you more.

Here you go. I changed the local part of the user's email address in the spam message, and the doveadm password in the doveconf output, but that's all.

http://www.cmadams.net/misc/dove/doveconf-n.txt http://www.cmadams.net/misc/dove/spam-message.txt

I have three Dovecot VMs behind a pair of Dovecot director VMs; this is from the actual host that handled the delivery (all three backend hosts are showing the same issue). Users are in MySQL and mail is on NFS. There are some things configured, like managesieve, that aren't currently used.

This is a new setup, migrated from Courier; I made the switch a couple of weeks ago (not my first time using Dovecot, just finally got a chance to migrate this setup).

Thanks.

Chris Adams cma@cmadams.net

Once upon a time, Stephan Bosch stephan@rename-it.nl said:

On 5/15/2015 5:56 PM, Chris Adams wrote:

...

Once upon a time, Stephan Bosch stephan@rename-it.nl said:

...

You can check the handling of a particular message yourself using the sieve-test tool (there is a man page for it). By specifying the -t - -Tlevel=matching options, you'll get detailed information on why a particular decision is made. Hmm, that's weird. sieve-test says it would store the message into the Spam folder (as expected), but it was definitely delivered to INBOX.

BTW, did you consult your logs for any errors? If an error occurs during Sieve processing, the default behavior is to file the message into INBOX (e.g. when the Spam folder doesn't exist).

Yes, I didn't find any errors, just the lmtp log entry for storing into INBOX. In the several cases I looked at, the Spam folder exists, is getting most spam-flagged messages filed into it, it just seems to be something about some messages (for example, got a bunch of copies of this particular spam to different users and they all went to INBOX instead of Spam).

May 15 09:44:04 dovecot2 dovecot: lmtp(10157): Connect from 10.0.9.71 May 15 09:44:05 dovecot2 dovecot: lmtp(10157, localuser9@foothills.net): MikHD8/YVVWtJwAAIYJ+iw: sieve: msgid=438088200.476329351.0075518260474.JavaMail.root@sjmas02.lococandles.co: stored mail into mailbox 'INBOX' May 15 09:44:05 dovecot2 dovecot: lmtp(10157): Disconnect from 10.0.9.57: Successful quit

Also, if the Spam folder didn't exist, I have it set to be autocreated in the Dovecot config.

I also know that Dovecot can write to the Spam folder okay, because I did a "doveadm move" to move several "X-Spam-Flag: YES" messages to the Spam folder, and that worked.

-- Chris Adams cma@cmadams.net

Once upon a time, Chris Adams cma@cmadams.net said:

I can confirm that a message with multiple Subject: and multiple From: headers does not get filed correctly into the Spam folder. The sieve-test tools shows the correct action, but when the message comes in via LMTP, it goes into INBOX.

Okay, digging some more, it looks like something in sieve is overwriting the wrong thing when it gets messages with some headers (at least From: and Subject:) repeated. I enabled the vnd.dovecot.debug sieve plugin, and used this sieve script:

require [ "fileinto", "variables", "vnd.dovecot.debug" ];
if header :matches "from" "*" { set "from" "${1}"; }
if header :matches "subject" "*" { set "subject" "${1}"; }
if header :matches "x-spam-flag" "*" { set "spam" "${1}"; }
debug_log "From='${from}' Subject='${subject}' Spam='${spam}'";
if header :contains "X-Spam-Flag" "YES" {
        fileinto "Spam";
}

When I feed a message to dovecot-lda with single From: and Subject: headers and X-Spam-Flag: YES set, I get this logged:

May 19 14:25:25 hosting dovecot: lda(local@example.com): sieve: default: line 6: DEBUG: From='test@cmadams.net' Subject='Test' Spam='YES'

If I duplicate the From: and Subject: headers, I get:

May 19 14:25:29 hosting dovecot: lda(local@example.com): sieve: default: line 6: DEBUG: From='test@cmadams.net' Subject='Test' Spam='local@example.com'

So, X-Spam-Flag: is somehow getting the To: address in it?

Oddly, with this debugging enabled, even though the message with the duplicate headers appears to have the wrong value for X-Spam-Flag:, it then procedes to actually match the YES somehow and file the message in the Spam folder.

-- Chris Adams cma@cmadams.net

On 5/19/2015 10:00 PM, Chris Adams wrote:

Once upon a time, Chris Adams cma@cmadams.net said:

...

Okay, digging some more, it looks like something in sieve is overwriting the wrong thing when it gets messages with some headers (at least From: and Subject:) repeated. I enabled the vnd.dovecot.debug sieve plugin, and used this sieve script: And I guess something is re-parsing them at some point? The following sieve script has the problem:

# compile with "sievec /etc/dovecot/default.sieve" require "fileinto"; if header :contains "X-Spam-Flag" "YES" { fileinto "Spam"; }

Doubling up the test makes it work however:

# compile with "sievec /etc/dovecot/default.sieve" require "fileinto"; if header :contains "X-Spam-Flag" "YES" { fileinto "Spam"; } if header :contains "X-Spam-Flag" "YES" { fileinto "Spam"; }

I will probably have time to investigate this more this weekend.

BTW, you're using a pretty old version of Dovecot and Pigeonhole. Do you have the possibility to try a newer version?

Regards,

Stephan.

Once upon a time, Stephan Bosch stephan@rename-it.nl said:

You're using Dovecot 2.2.10, which is quite old. I remembered a bug like this, but I had to look it up. This is the original bug report:

Ah, sorry for taking your time on an old bug. For most of my servers, I try to use the RHEL/CentOS-provided packages where practical (because with over 100 servers running a wide variety of services, building everything from source takes more hours than I have in a day). The hazzard of course is chasing already-fixed bugs from time to time.

I also understand Red Hat's "stability" approach, where they tend to stick with a given upstream version and just patch it; few Open Source software developers have the time to maintain bugfix-only branches for the lifetime of a RHEL major version, and new features tend to mean new bugs as well.

Thanks for taking the time to point out the fix. I was hoping to have time this weekend to try the latest Dovecot release, but hadn't yet done that. I have opened a Red Hat bug to try to get this patch applied.

https://bugzilla.redhat.com/show_bug.cgi?id=1224496

-- Chris Adams cma@cmadams.net