FW: imapsieve rules not matching at all?
Greetings!
I feel like this has been beaten to death, but my searches on the web (and about 10 hours spent over the last two days) haven't revealed what's going on.
Basically, it's the usual "I'd like to auto-learn spam/ham based on moves to/from a folder" problem. But in my debugging, I don't see any evidence that the static rules are matching, so the scripts aren't running, which makes me think I'm missing something obvious.
Below I list doveconf -n output, and some log snippets. In my experiments, I've tried removing the report-spam.sieve and report-ham.sieve scripts, and absolutely nothing changed, which is why I think the rules aren't being tried.
The requisite doveconf -n output:
- -8< - - - -
2.3.13 (89f716dc2): /usr/local/etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.13 (cdd19fe3)
# OS: FreeBSD 12.2-RELEASE-p1 amd64
# Hostname: mhv3
lock_method = dotlock
mail_attribute_dict = file:%h/Maildir/dovecot-attributes
mail_debug = yes
mail_location = maildir:~/Maildir
mail_privileged_group = mail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext vnd.dovecot.debug vnd.dovecot.environment imapsieve vnd.dovecot.imapsieve vnd.dovecot.pipe vnd.dovecot.execute
namespace inbox {
inbox = yes
location =
mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Trash {
special_use = \Trash
}
prefix =
}
passdb {
driver = pam
}
plugin {
imapsieve_mailbox1_before = file:/usr/local/etc/dovecot/sieve/report-spam.sieve
imapsieve_mailbox1_causes = COPY APPEND
imapsieve_mailbox1_name = Spam
imapsieve_mailbox2_before = file:/usr/local/etc/dovecot/sieve/report-ham.sieve
imapsieve_mailbox2_causes = COPY
imapsieve_mailbox2_from = Spam
imapsieve_mailbox2_name = *
imapsieve_url = sieve://127.0.0.1:4190
sieve = file:~/sieve;active=~/.dovecot.sieve
sieve_before = /usr/local/etc/dovecot/sieve/common.sieve
sieve_default = /usr/local/etc/dovecot/sieve/default.sieve
sieve_default_name = SystemDefault
sieve_execute_bin_dir = /usr/local/etc/dovecot/sieve/bin
sieve_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment +vnd.dovecot.execute +vnd.dovecot.debug
sieve_filter_bin_dir = /usr/local/etc/dovecot/sieve/bin
sieve_global = /usr/local/etc/dovecot/sieve
sieve_pipe_bin_dir = /usr/local/etc/dovecot/sieve/bin
sieve_plugins = sieve_imapsieve sieve_extprograms
sieve_trace_addresses = no
sieve_trace_debug = yes
sieve_trace_dir = /var/log/dovecot/trace
sieve_trace_level = matching
}
protocols = imap lmtp sieve
service imap-login {
inet_listener imap {
address = 127.0.0.1
}
}
service managesieve-login {
inet_listener sieve {
port = 4190
}
}
ssl_cert =
ssl_dh = # hidden, use -P to show it
ssl_key = # hidden, use -P to show it
userdb {
driver = passwd
}
protocol lda {
mail_plugins = " sieve"
}
protocol imap {
imap_metadata = yes
mail_max_userip_connections = 30
mail_plugins = " imap_sieve"
}
- ->8 - - - -
I see that the static rule ought to have matched, but it didn't?
- 8<- - - -
Mar 19 16:21:47 mhv3 dovecot[47532]: imap(steve)<47539>
Mar 19 16:21:47 mhv3 dovecot[47532]: imap(steve)<47539>
Mar 19 16:21:47 mhv3 dovecot[47532]: imap(steve)<47539>
Mar 19 16:21:47 mhv3 dovecot[47532]: imap(steve)<47539>Spam' from=
*'
causes=(COPY APPEND) =>
before=`file:/usr/local/etc/dovecot/sieve/report-spam.sieve' after=(none)
Mar 19 16:21:47 mhv3 dovecot[47532]: imap(steve)<47539>*' from=
Spam'
causes=(COPY) => before=`file:/usr/local/etc/dovecot/sieve/report-ham.sieve'
after=(none)
Mar 19 16:21:47 mhv3 dovecot[47532]: imap(steve)<47539>
Mar 19 16:21:48 mhv3 dovecot[47532]: imap(steve)<47541>
Mar 19 16:21:48 mhv3 dovecot[47532]: imap(steve)<47541>
Mar 19 16:21:48 mhv3 dovecot[47532]: imap(steve)<47541>
Mar 19 16:21:48 mhv3 dovecot[47532]: imap(steve)<47541>
Mar 19 16:21:48 mhv3 dovecot[47532]: imap(steve)<47541>
Mar 19 16:21:48 mhv3 dovecot[47532]: imap(steve)<47541>
Mar 19 16:21:48 mhv3 dovecot[47532]: imap(steve)<47541>
Mar 19 16:21:48 mhv3 dovecot[47532]: imap(steve)<47541>
Mar 19 16:21:48 mhv3 dovecot[47532]: imap(steve)<47541>
Mar 19 16:21:48 mhv3 dovecot[47532]: imap(steve)<47541>
Mar 19 16:21:48 mhv3 dovecot[47532]: imap(steve)<47541>
Mar 19 16:21:48 mhv3 dovecot[47532]: imap(steve)<47541>
Mar 19 16:21:48 mhv3 dovecot[47532]: imap(steve)<47541>Spam' from=
*'
causes=(COPY APPEND) =>
before=`file:/usr/local/etc/dovecot/sieve/report-spam.sieve' after=(none)
Mar 19 16:21:48 mhv3 dovecot[47532]: imap(steve)<47541>*' from=
Spam'
causes=(COPY) => before=`file:/usr/local/etc/dovecot/sieve/report-ham.sieve'
after=(none)
Mar 19 16:21:49 mhv3 dovecot[47532]: imap(steve)<47541>
Mar 19 16:21:54 mhv3 dovecot[47532]: imap(steve)<47541>
Mar 19 16:21:54 mhv3 dovecot[47532]: imap(steve)<47541>
Mar 19 16:21:54 mhv3 dovecot[47532]: imap(steve)<47541>
- ->8 - - - -
Thanks for clues,
--
Steve Watt KD6GGD PP-ASEL-IA factories.words.yappy
Don't let your schooling get in the way of your education.
On 3/20/21 7:37 AM, dovecot@steve.wattlink.net wrote:
Greetings!
I feel like this has been beaten to death, but my searches on the web (and about 10 hours spent over the last two days) haven’t revealed what’s going on.
Basically, it’s the usual “I’d like to auto-learn spam/ham based on moves to/from a folder” problem. But in my debugging, I don’t see any evidence that the static rules are matching, so the scripts aren’t running, which makes me think I’m missing something obvious.
plugin {
imapsieve_url = sieve://127.0.0.1:4190
}
Mar 19 16:21:48 mhv3 dovecot[47532]: imap(steve)<47541>
: Debug: imapsieve: mailbox INBOX: Mailbox attribute /shared/imapsieve/script not found Mar 19 16:21:48 mhv3 dovecot[47532]: imap(steve)<47541>
: Debug: imapsieve: mailbox INBOX: Server attribute /shared/imapsieve/script not found
Try to fix or remove that.
https://www.mail-archive.com/dovecot@dovecot.org/msg82002.html
From: Gedalya gedalya@gedalya.net Subject: Re: FW: imapsieve rules not matching at all?
On 3/20/21 7:37 AM, dovecot@steve.wattlink.net mailto:dovecot@steve.wattlink.net wrote:
Greetings!
I feel like this has been beaten to death, but my searches on the web (and about 10 hours spent over the last two days) haven't revealed what's going on.
Basically, it's the usual "I'd like to auto-learn spam/ham based on moves to/from a folder" problem. But in my debugging, I don't see any evidence that the static rules are matching, so the scripts aren't running, which makes me think I'm missing something obvious.
plugin {
imapsieve_url = sieve://127.0.0.1:4190
}
Mar 19 16:21:48 mhv3 dovecot[47532]: imap(steve)<47541>
Mar 19 16:21:48 mhv3 dovecot[47532]: imap(steve)<47541>
Try to fix or remove that.
https://www.mail-archive.com/dovecot@dovecot.org/msg82002.html
I thought I had enabled that - check out the doveconf -n listing. Did I miss something?
--
Steve Watt KD6GGD PP-ASEL-IA factories.words.yappy
Don't let your schooling get in the way of your education.
On 3/20/21 10:54 AM, Steve Watt wrote:
I thought I had enabled that – check out the doveconf -n listing. Did I miss something?
IMAP METADATA for user-defined imapsieve scripts would be useful to you if you have clients that support that. If you know of any, please do share.
Mar 19 16:21:48 mhv3 dovecot[47532]: imap(steve)<47541>
: Debug: imapsieve: mailbox INBOX: Mailbox attribute /shared/imapsieve/script not found Mar 19 16:21:48 mhv3 dovecot[47532]: imap(steve)<47541>
: Debug: imapsieve: mailbox INBOX: Server attribute /shared/imapsieve/script not found
This is saying that imapsieve scripts are not present in your defined mail_attribute_dict.
If you're not going to use that, you might as well disable it. However, I looked at the code and this is indeed not an error and it's not causing the imapsieve processing to stop as it would without METADATA enabled.
On 3/20/21 7:37 AM, dovecot@steve.wattlink.net wrote:
plugin {
imapsieve_mailbox1_before = file:/usr/local/etc/dovecot/sieve/report-spam.sieve
imapsieve_mailbox1_causes = COPY APPEND
imapsieve_mailbox1_name = Spam
imapsieve_mailbox2_before = file:/usr/local/etc/dovecot/sieve/report-ham.sieve
imapsieve_mailbox2_causes = COPY
imapsieve_mailbox2_from = Spam
imapsieve_mailbox2_name = *
}
- ->8 - - - -
I see that the static rule ought to have matched,
no!
- 8<- - - -
Mar 19 16:21:48 mhv3 dovecot[47532]: imap(steve)<47541>
: Debug: imapsieve: mailbox INBOX: APPEND event
- ->8 - - - -
For INBOX (or * in your case) you only have COPY from Spam configured, not APPEND.
APPENDing to Spam should trigger the relevant script though.
If you want to enable ham training by uploading a message to INBOX you could add a third rule mentioning INBOX by name with APPEND as cause.
On 20/03/2021 05:55 Gedalya gedalya@gedalya.net wrote:
On 3/20/21 7:37 AM, dovecot@steve.wattlink.net wrote:
plugin { imapsieve_mailbox1_before = file:/usr/local/etc/dovecot/sieve/report-spam.sieve imapsieve_mailbox1_causes = COPY APPEND imapsieve_mailbox1_name = Spam imapsieve_mailbox2_before = file:/usr/local/etc/dovecot/sieve/report-ham.sieve imapsieve_mailbox2_causes = COPY imapsieve_mailbox2_from = Spam imapsieve_mailbox2_name = * }
- ->8 - - - -
I see that the static rule ought to have matched, no!
- 8<- - - - Mar 19 16:21:48 mhv3 dovecot[47532]: imap(steve)<47541>
: Debug: imapsieve: mailbox INBOX: APPEND event
- ->8 - - - -
For INBOX (or * in your case) you only have COPY from Spam configured, not APPEND. APPENDing to Spam should trigger the relevant script though. If you want to enable ham training by uploading a message to INBOX you could add a third rule mentioning INBOX by name with APPEND as cause.
We provide this handy guide for teaching spam filters, see https://doc.dovecot.org/configuration_manual/howto/antispam_with_sieve/
Aki
I have not yet enabled imapsieve -- so far I have had fairly good luck avoiding spam simply by using SPF+DKIM+DMARC and enabling basic verification of incoming mail with opendkim and opendmarc.
Lately I have been reading some books on "fuzzy logic" and "fuzzy sets" with quite serious applications to artificial intelligence and neural networks that might be useful to classify "ham versus spam" based on actual content and context.
Spam versus ham is not the only sort of classification I would want to do on large volumes of email -- I might want to have separate folders to automatically classify incoming messages into separate categories for, say, friends-and-family, legal-related email, specific business interests, open-source-software or technical related email, mathematics, arts or crafts or literature or hobbies, etc.
This kind of stuff must be easily configurable -- per user -- by individual end users who are not experts in editing configuration files.
On March 19, 2021 11:38:19 PM AKDT, Aki Tuomi aki.tuomi@open-xchange.com wrote:
On 20/03/2021 05:55 Gedalya gedalya@gedalya.net wrote:
On 3/20/21 7:37 AM, dovecot@steve.wattlink.net wrote:
plugin { imapsieve_mailbox1_before = file:/usr/local/etc/dovecot/sieve/report-spam.sieve imapsieve_mailbox1_causes = COPY APPEND imapsieve_mailbox1_name = Spam imapsieve_mailbox2_before = file:/usr/local/etc/dovecot/sieve/report-ham.sieve imapsieve_mailbox2_causes = COPY imapsieve_mailbox2_from = Spam imapsieve_mailbox2_name = * }
- ->8 - - - -
I see that the static rule ought to have matched, no!
- - 8<- - - - Mar 19 16:21:48 mhv3 dovecot[47532]:
imap(steve)<47541>
: Debug: imapsieve: mailbox INBOX: APPEND event
- ->8 - - - - For INBOX (or * in your case) you only have COPY from Spam configured, not APPEND. APPENDing to Spam should trigger the relevant script though. If you want to enable ham training by uploading a message to INBOX you could add a third rule mentioning INBOX by name with APPEND as cause.
We provide this handy guide for teaching spam filters, see https://doc.dovecot.org/configuration_manual/howto/antispam_with_sieve/
Aki
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
On 2021-03-20 7:14 a.m., Justina Colmena ~biz wrote:
Spam versus ham is not the only sort of classification I would want to do on large volumes of email -- I might want to have separate folders to automatically classify incoming messages into separate categories for, say, friends-and-family, legal-related email, specific business interests, open-source-software or technical related email, mathematics, arts or crafts or literature or hobbies, etc.
This kind of stuff must be easily configurable -- per user -- by individual end users who are not experts in editing configuration files.
Many commercial platforms specialize in this, but what you are talking about requires co-ordination between email clients and the email server backends, or ONLY in the webmail, as logic can be built on current IMAP (eg implementing logic on moving messages external to the IMAP server) but is even better implemented as server side filters during delivery.
However, that discussion goes beyond dovecot, and probably not the right place to talk about this on this list.
There are many email client technologies focused on what you are discussing.
However, we 'could' use an industry standard for email clients to be able to pass a 'Mark as Spam/NotSpam' instruction over IMAP.
-- "Catch the Magic of Linux..."
Michael Peddemors, President/CEO LinuxMagic Inc. Visit us at http://www.linuxmagic.com @linuxmagic A Wizard IT Company - For More Info http://www.wizard.ca "LinuxMagic" a Registered TradeMark of Wizard Tower TechnoServices Ltd.
604-682-0300 Beautiful British Columbia, Canada
This email and any electronic data contained are confidential and intended solely for the use of the individual or entity to which they are addressed. Please note that any views or opinions presented in this email are solely those of the author and are not intended to represent those of the company.
Aki Tuomi wrote:
We provide this handy guide for teaching spam filters, see https://doc.dovecot.org/configuration_manual/howto/antispam_with_sieve/
Yes, I followed that and am having trouble. I think (haven't yet confirmed through a protocol trace) that the problem is Outlook using APPEND when it moves messages...
-- Steve Watt KD6GGD PP-ASEL-IA factories.words.yappy Don't let your schooling get in the way of your education.
On 20/03/2021 20:30 dovecot@steve.wattlink.net wrote:
Aki Tuomi wrote:
We provide this handy guide for teaching spam filters, see https://doc.dovecot.org/configuration_manual/howto/antispam_with_sieve/
Yes, I followed that and am having trouble. I think (haven't yet confirmed through a protocol trace) that the problem is Outlook using APPEND when it moves messages...
-- Steve Watt KD6GGD PP-ASEL-IA factories.words.yappy Don't let your schooling get in the way of your education.
Yeah. Outlook does APPEND+EXPUNGE instead of MOVE.
Aki
Aki Tuomi wrote:
Yeah. Outlook does APPEND+EXPUNGE instead of MOVE.
That rather critical item should probably be mentioned in the configuration manual; in my searching about on the problem, I didn't find anything that indicated this Microsoft-caused limitation. On the upside, I'm glad I found it in my testing since I often use Roundcube and Thunderbird, which (now that I've looked) seem to do it correctly.
Besides, I can't think of a safe way to reliably detect that other than declaring moves to any folder other than spam to be ham. Guess I continue having Outlook users copy false positives from the spam folder into a separate reporting folder for cron to pick up.
Thanks!
Steve Watt KD6GGD PP-ASEL-IA factories.words.yappy Don't let your schooling get in the way of your education.
participants (6)
-
Aki Tuomi
-
dovecot@steve.wattlink.net
-
Gedalya
-
Justina Colmena ~biz
-
Michael Peddemors
-
Steve Watt