Hello,
I've recently converted from using procmail as an lda to using lmtp+sieve (and it's overall great).
My setup is currently doing post-queue spamassassin to pickup user rules (and to be substantially similar to how it worked before). It mostly works, but I get some mail forwarded from another host that also runs spamassassin, so the mail comes with X-Spam headers already set.
I can see that my sieve_before script successfully sends the message off to spamc/spamd - however the message ends up in my mailbox with only the pre-existing X-Spam headers (I had expected them to be replaced or to see duplicates). I thought I'd try to just strip them from incoming mail so I used deleteheader, I got a trace saying it matches and deletes the headers, then sends the mail off to spamc/spamd - but the message that ends up in my mailbox has the headers that were presumably deleted.
My before.sieve looks like this:
require ["vnd.dovecot.filter","editheader"];
# rule:[spamfilter]
if size :under 10485760
{
deleteheader :matches "X-Spam-Flag" "*";
deleteheader :matches "X-Spam-Status" "*";
deleteheader :matches "X-Spam-Level" "*";
deleteheader :matches "X-Spam-Checker-Version" "*";
filter "spamc" ["-s","10485760","-U","/var/run/spamd.sock"];
}
Trace file says:
## Started executing script 'before'6: size :under test
6: comparing message size 4197
6: with lower limit 10485760
6: jump if result is false
6: not jumping
11: deleteheader command
11: deleting matching occurrences of header X-Spam-Flag' 11: header X-Spam-Flag' not found
12: deleteheader command
12: deleting matching occurrences of header X-Spam-Status' 12: starting :matches' match with i;ascii-casemap' comparator: 12: matching value No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_..
.'
12: with key *' => 1 12: deleting header with value No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALI
D_AU,DKIM_VALID_EF,FREEMAIL_FROM,HTML_MESSAGE, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham au
tolearn_force=no version=3.4.2'
12: finishing match with result: matched
13: deleteheader command
13: deleting matching occurrences of header X-Spam-Level' 13: header X-Spam-Level' not found
14: deleteheader command
14: deleting matching occurrences of header X-Spam-Checker-Version' 14: starting :matches' match with i;ascii-casemap' comparator: 14: matching value SpamAssassin 3.4.2 (2018-09-13) on braeburn.macports.org'
14: with key *' => 1 14: deleting header with value SpamAssassin 3.4.2 (2018-09-13) on braeburn.macports.org'
14: finishing match with result: matched
16: filter action
16: execute program `spamc'
16: executed program successfully
16: changed message
## Finished executing script 'before'
It then falls through my personal sieve filter and hits the implicit keep at the end and ends up in my inbox - with the original headers.
So, two questions:
- What am I missing?
- Is there a way to match any X-Spam-* header? (when I tried deleteheader :matches "X-Spam*" it didn't match anything).
Thanks.
-- Daniel J. Luke
Hi!
It then falls through my personal sieve filter and hits the implicit keep at the end and ends up in my inbox - with the original headers.
I had a similar issue recently (with addheader, not deleteheader) and I was pointed to the RFC: https://datatracker.ietf.org/doc/html/rfc5293#section-5
The deleteheader action does not affect Sieve's implicit keep.
My addheader case was solved by adding an explicit keep or a fileinto.
I hope that will also help you, Vincent
On Jun 11, 2021, at 3:51 AM, Vincent Brillault <vincent.brillault@cern.ch> wrote:
It then falls through my personal sieve filter and hits the implicit keep at the end and ends up in my inbox - with the original headers.
I had a similar issue recently (with addheader, not deleteheader) and I was pointed to the RFC: https://datatracker.ietf.org/doc/html/rfc5293#section-5
The deleteheader action does not affect Sieve's implicit keep.
I haven't gone over the RFCs with a fine toothed comb, but having implicit and explicit keep with different behaviors is surprising.
My addheader case was solved by adding an explicit keep or a fileinto.
I added a keep; to then end of my before.sieve and it doesn't change the behavior (with or without the deleteheaders I always see the other hosts X-Spam headers in the delivered mail).
Daniel J. Luke
On Jun 11, 2021, at 9:54 AM, Daniel J. Luke <dluke@geeklair.net> wrote:
On Jun 11, 2021, at 3:51 AM, Vincent Brillault <vincent.brillault@cern.ch> wrote:
It then falls through my personal sieve filter and hits the implicit keep at the end and ends up in my inbox - with the original headers.
I had a similar issue recently (with addheader, not deleteheader) and I was pointed to the RFC: https://datatracker.ietf.org/doc/html/rfc5293#section-5
The deleteheader action does not affect Sieve's implicit keep.I haven't gone over the RFCs with a fine toothed comb, but having implicit and explicit keep with different behaviors is surprising.
My addheader case was solved by adding an explicit keep or a fileinto.
I added a keep; to then end of my before.sieve and it doesn't change the behavior (with or without the deleteheaders I always see the other hosts X-Spam headers in the delivered mail).
While that didn't work, adding a keep to the end of the user rules works - and even better, I don't need the deleteheaders (as it preserves the headers as set by the spamc filter).
This behavior is very surprising to me (I wouldn't expect rules ending with an explicit keep to be any different from rules ending with an implicit one), but at least it's working now - and hopefully being in the list archive will save someone else some time in the future.
-- Daniel J. Luke
On 11 Jun 2021, at 07:54, Daniel J. Luke <dluke@geeklair.net> wrote:
On Jun 11, 2021, at 3:51 AM, Vincent Brillault <vincent.brillault@cern.ch> wrote:
It then falls through my personal sieve filter and hits the implicit keep at the end and ends up in my inbox - with the original headers.
I had a similar issue recently (with addheader, not deleteheader) and I was pointed to the RFC: https://datatracker.ietf.org/doc/html/rfc5293#section-5
The deleteheader action does not affect Sieve's implicit keep.```I haven't gone over the RFCs with a fine toothed comb, but having implicit and explicit keep with different behaviors is surprising.
Agreed. And I think we all hit this.
My addheader case was solved by adding an explicit keep or a fileinto.
I added a keep; to then end of my before.sieve and it doesn't change the behavior (with or without the deleteheaders I always see the other hosts X-Spam headers in the delivered mail).
Are you sure the message you are seeing is being processed by your before.sieve? Os there any evidence at all in the mail that it was hit sieve?
For example, one of my sieve_before
# cat /usr/lib/dovecot/sieve/filespam.sieve require ["fileinto", "imap4flags"];
if header :contains "X-spam-flag" "YES" { fileinto :flags "\Seen" "Junk"; }
I know that sieve fires because messages marked as spam are tied into the user's Junk mail.
You know that the sieve_before scripts are run AS THE USER, not as dovecot or root, right? I mention this because it was something that caught me out.
-- Growing up leads to growing old, and then to dying/And dying to me don't sound like all that much fun.
On 11/06/2021 04:25, Daniel J. Luke wrote:
Hello,
I've recently converted from using procmail as an lda to using lmtp+sieve (and it's overall great).
My setup is currently doing post-queue spamassassin to pickup user rules (and to be substantially similar to how it worked before). It mostly works, but I get some mail forwarded from another host that also runs spamassassin, so the mail comes with X-Spam headers already set.
I can see that my sieve_before script successfully sends the message off to spamc/spamd - however the message ends up in my mailbox with only the pre-existing X-Spam headers (I had expected them to be replaced or to see duplicates). I thought I'd try to just strip them from incoming mail so I used deleteheader, I got a trace saying it matches and deletes the headers, then sends the mail off to spamc/spamd - but the message that ends up in my mailbox has the headers that were presumably deleted.
I cannot reproduce this with master. Keep and implicit keep act the same. My filter program successfully changes the message and the deleteheader commands properly drop the indicated headers.
What version is this? What is your configuration (output from dovecot -n)
Regards,
Stephan.
My before.sieve looks like this:
require ["vnd.dovecot.filter","editheader"]; # rule:[spamfilter] if size :under 10485760 { deleteheader :matches "X-Spam-Flag" "*"; deleteheader :matches "X-Spam-Status" "*"; deleteheader :matches "X-Spam-Level" "*"; deleteheader :matches "X-Spam-Checker-Version" "*"; filter "spamc" ["-s","10485760","-U","/var/run/spamd.sock"]; }
Trace file says:
## Started executing script 'before' 6: size :under test 6: comparing message size 4197 6: with lower limit 10485760 6: jump if result is false 6: not jumping11: deleteheader command 11: deleting matching occurrences of header
X-Spam-Flag' 11: headerX-Spam-Flag' not found 12: deleteheader command 12: deleting matching occurrences of headerX-Spam-Status' 12: starting:matches' match withi;ascii-casemap' comparator: 12: matching valueNo, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_.. .' 12: with key*' => 1 12: deleting header with valueNo, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALI D_AU,DKIM_VALID_EF,FREEMAIL_FROM,HTML_MESSAGE, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham au tolearn_force=no version=3.4.2' 12: finishing match with result: matched 13: deleteheader command 13: deleting matching occurrences of headerX-Spam-Level' 13: headerX-Spam-Level' not found 14: deleteheader command 14: deleting matching occurrences of headerX-Spam-Checker-Version' 14: starting:matches' match withi;ascii-casemap' comparator: 14: matching valueSpamAssassin 3.4.2 (2018-09-13) on braeburn.macports.org' 14: with key*' => 1 14: deleting header with valueSpamAssassin 3.4.2 (2018-09-13) on braeburn.macports.org' 14: finishing match with result: matched 16: filter action 16: execute program `spamc' 16: executed program successfully 16: changed message ## Finished executing script 'before'It then falls through my personal sieve filter and hits the implicit keep at the end and ends up in my inbox - with the original headers.
So, two questions:
- What am I missing?
- Is there a way to match any X-Spam-* header? (when I tried deleteheader :matches "X-Spam*" it didn't match anything).
Thanks.
On Jun 12, 2021, at 5:16 AM, Stephan Bosch <stephan@rename-it.nl> wrote:
I can see that my sieve_before script successfully sends the message off to spamc/spamd - however the message ends up in my mailbox with only the pre-existing X-Spam headers (I had expected them to be replaced or to see duplicates). I thought I'd try to just strip them from incoming mail so I used deleteheader, I got a trace saying it matches and deletes the headers, then sends the mail off to spamc/spamd - but the message that ends up in my mailbox has the headers that were presumably deleted.
I cannot reproduce this with master. Keep and implicit keep act the same. My filter program successfully changes the message and the deleteheader commands properly drop the indicated headers.
What version is this? What is your configuration (output from
dovecot -n)
My before script sends to spamc or does deleteheader (as noted) but user script does:
# rule:[mailman lists with mailboxes] if allof( header :matches "List-Id" "*<*.*", mailboxexists "${2}" ) { fileinto "${2}"; stop; }
# rule:[discard duplicates] if duplicate { discard; }
keep;
I don't see header problems in mail that ends up ina fileinto'd mailbox, just my INBOX - I suspect it's interaction with the 'duplicate' implementation (which the RFC says operates on the original un-modified message).
dovecot -n output below -
# 2.3.14 (cee3cbc0d): /usr/local/etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.14 (1b5c82b2)
# OS: Darwin 20.5.0 x86_64
# Hostname: vroomfondel.geeklair.net
auth_username_format = %Ln
auth_verbose = yes
default_internal_group = mail
default_internal_user = _dovecot
default_login_user = _dovenull
first_valid_gid = 500
first_valid_uid = 501
last_valid_gid = 599
last_valid_uid = 599
login_greeting = geeklair.net mail ready.
mail_location = mbox:~/Mail/:INBOX=~/.mbox:INDEX=~/.dovecot-indexes
mail_plugins = fts fts_lucene zlib
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext
namespace {
inbox = yes
location =
mailbox Drafts {
auto = no
special_use = \Drafts
}
mailbox Sent {
auto = subscribe
special_use = \Sent
}
mailbox Trash {
auto = no
special_use = \Trash
}
mailbox spam {
auto = create
special_use = \Junk
}
prefix =
separator = /
}
namespace archive {
inbox = no
list = no
location = mbox:~/MailArchive/
prefix = "#Archive/"
separator = /
}
passdb {
args = dovecot
driver = pam
}
plugin {
fts = lucene
fts_autoindex = yes
fts_autoindex_exclude = "#Archive/*"
fts_lucene = whitespace_chars=@.
recipient_delimiter = +
sieve = file:~/.sieve;active=~/.dovecot.sieve
sieve_before = /usr/local/etc/dovecot/sieve/before.sieve
sieve_default = /usr/local/etc/dovecot/sieve/default.sieve
sieve_default_name = file_spam
sieve_filter_bin_dir = /usr/local/lib/dovecot/sieve-filter
sieve_filter_exec_timeout = 720s
sieve_global_extensions = +vnd.dovecot.filter +editheader
sieve_plugins = sieve_extprograms
}
protocols = imap pop3 lmtp sieve
service auth {
unix_listener /usr/local/var/spool/postfix/private/auth {
group = _postfix
mode = 0660
user = _postfix
}
user = root
}
service imap-login {
inet_listener imaps {
port = 993
ssl = yes
}
process_min_avail = 1
service_count = 0
}
service imap {
vsz_limit = 512 M
}
service lmtp {
unix_listener /usr/local/var/spool/postfix/private/dovecot-lmtp {
group = postfix
mode = 0600
user = postfix
}
}
service managesieve-login {
inet_listener sieve {
port = 4190
}
process_min_avail = 0
service_count = 0
vsz_limit = 64 M
}
service managesieve {
process_limit = 12
}
service pop3-login {
inet_listener pop3 {
port = 0
}
inet_listener pop3s {
port = 995
ssl = yes
}
process_min_avail = 1
service_count = 0
}
service pop3 {
vsz_limit = 512 M
}
ssl_cert = </opt/local/etc/letsencrypt/live/geeklair.net/fullchain.pem
ssl_cipher_list = ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
ssl_dh = # hidden, use -P to show it
ssl_key = # hidden, use -P to show it
ssl_prefer_server_ciphers = yes
syslog_facility = local1
userdb {
driver = passwd
}
verbose_proctitle = yes
version_ignore = yes
protocol imap {
mail_max_userip_connections = 32
mail_plugins = fts fts_lucene zlib imap_zlib
}
protocol pop3 {
pop3_uidl_format = %08Xu%08Xv
}
protocol lda {
mail_plugins = fts fts_lucene zlib sieve
postmaster_address = postmaster@geeklair.net
}
protocol lmtp {
mail_plugins = fts fts_lucene zlib sieve
postmaster_address = postmaster@geeklair.net
}
-- Daniel J. Luke
participants (4)
-
＠lbutlr
-
Daniel J. Luke
-
Stephan Bosch
-
Vincent Brillault