antispam plugin pipe script seems not to be called when it "should be"
Hello,
(Full "doveconf -n" output is at the end of this message.)
I'm attempting to recreate a "recipe" I had used successfully for quite some time (in Ubuntu 12.04 LTS), which allowed me to train SpamAssassin's Bayes database when someone dragged an email message from one sub-folder of the account to another.
I later tried to re-implement it (in Ubuntu 14.04 LTS), and core-dumps resulted. I posted to this list about it several times, but nobody was ever able to help me solve the problem.
I just tried the same (in Ubuntu 16.04 LTS), hoping that the source had evolved since then, and while I don't see a core-dump, the pipe script doesn't seem to be called.
I've booby-trapped the pipe script with quite a bit of debugging output, which should be written to a file, but I don't see any indication that the pipe script is ever executed.
When logged-in as an IMAP user, and I drag a message from the user's Inbox into the "Spam" folder, I do see the following activity in the syslog:
Aug 18 22:12:15 example.com imap: antispam: mailbox_is_unsure(Junk): 0 Aug 18 22:12:15 example.com imap: antispam: mailbox_is_trash(INBOX): 0 Aug 18 22:12:15 example.com imap: antispam: mailbox_is_trash(Junk): 0 Aug 18 22:12:15 example.com imap: antispam: mail copy: from trash: 0, to trash: 0 Aug 18 22:12:15 example.com imap: antispam: mailbox_is_spam(INBOX): 0 Aug 18 22:12:15 example.com imap: antispam: mailbox_is_spam(Junk): 1 Aug 18 22:12:15 example.com imap: antispam: mailbox_is_unsure(INBOX): 0 Aug 18 22:12:15 example.com imap: antispam: mail copy: src spam: 0, dst spam: 1, src unsure: 0 Aug 18 22:12:15 example.com imap: antispam: running mailtrain backend program /bin/bash Aug 18 22:12:15 example.com imap: antispam: running mailtrain backend program /bin/bash Aug 18 22:12:15 example.com imap: antispam: running mailtrain backend program parameter 1 /usr/local/bin/sa-learn-pipe.sh Aug 18 22:12:15 example.com imap: antispam: running mailtrain backend program parameter 2 --spam
But the piped message is never delivered to the target mailbox. The mail log at /var/log/mail.log does not register any activity when I move messages (and maybe that's expected).
Here are the contents of the /usr/local/bin/sa-learn-pipe.sh script:
The permissions and mode on the script seem appropriate:
$ stat /usr/local/bin/sa-learn-pipe.sh File: '/usr/local/bin/sa-learn-pipe.sh' Size: 1765 Blocks: 8 IO Block: 4096 regular file Device: fd01h/64769d Inode: 669345 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 5000/ vmail) Gid: ( 5000/ vmail) Access: 2016-08-18 19:09:47.307392919 -0400 Modify: 2016-08-18 19:09:31.231391749 -0400 Change: 2016-08-18 19:09:31.231391749 -0400 Birth: -
What might I be doing incorrectly here?
Please let me know if I can provide any other information.
Thank you in advance,
-Ben
Full "doveconf -n" output:
$ doveconf -n
2.2.22 (fe789d2): /etc/dovecot/dovecot.conf
Pigeonhole version 0.4.13 (7b14904)
OS: Linux 4.4.0-34-generic x86_64 Ubuntu 16.04.1 LTS
auth_mechanisms = plain login disable_plaintext_auth = no listen = *,[::] log_timestamp = "%Y-%m-%d %H:%M:%S " mail_max_userip_connections = 100 mail_plugins = " quota" mail_privileged_group = vmail passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } plugin { antispam_backend = pipe antispam_debug_target = syslog antispam_pipe_program = /bin/bash antispam_pipe_program_args = /usr/local/bin/sa-learn-pipe.sh antispam_pipe_program_notspam_arg = --ham antispam_pipe_program_spam_arg = --spam antispam_pipe_tmpdir = /tmp antispam_spam_pattern_ignorecase = SPAM;JUNK antispam_trash_pattern_ignorecase = trash;Deleted * antispam_verbose_debug = 1 quota = dict:user::file:/var/vmail/%d/%n/.quotausage quota_rule2 = Trash:storage=+100M quota_rule3 = Junk:ignore quota_rule4 = INBOX:storage=+100M quota_warning = storage=100%% quota-reached 100 %u %d quota_warning2 = storage=95%% quota-warning 95 %u %d quota_warning3 = storage=80%% quota-warning 80 %u %d quota_warning4 = -storage=100%% quota-below below %u %d sieve = /var/vmail/%d/%n/.sieve sieve_max_redirects = 25 } postmaster_address = postmaster@example.com protocols = imap pop3 service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } user = root } service imap-login { client_limit = 1000 process_limit = 512 } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } } service quota-below { executable = script /usr/local/bin/quota-below.sh unix_listener quota-below { group = vmail mode = 0666 user = vmail } user = vmail } service quota-reached { executable = script /usr/local/bin/quota-reached.sh unix_listener quota-reached { group = vmail mode = 0666 user = vmail } user = vmail } service quota-warning { executable = script /usr/local/bin/quota-warning.sh unix_listener quota-warning { group = vmail mode = 0666 user = vmail } user = vmail } ssl_cert = </etc/postfix/smtpd.cert ssl_key = </etc/postfix/smtpd.key ssl_protocols = !SSLv2 !SSLv3 userdb { driver = prefetch } userdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } protocol imap { mail_plugins = quota imap_quota antispam } protocol pop3 { mail_plugins = quota pop3_uidl_format = %08Xu%08Xv } protocol lda { mail_plugins = sieve quota } protocol lmtp { mail_plugins = quota sieve postmaster_address = webmaster@localhost }
On 8/18/2016 10:26 PM, Ben Johnson wrote:
If it's helpful, I've tried replacing the pipe script's contents with the following:
##################################################################### #!/bin/bash
/usr/lib/dovecot/deliver -d "sa-training@example.com" -m "Training.$mode" -p /var/vmail/gtube.txt
Exit with "success" status code.
exit 0 #####################################################################
And still, the message never arrives when I drag a message from "Inbox" to "Spam" in the IMAP client, despite the messages logged to syslog, which "all seem to be correct".
Yet, if I do this using the above content for the script, the "gtube" message is delivered successfully:
su vmail
$ /usr/local/bin/sa-learn-pipe-test.sh
The message arrives in the Inbox, presumably because -m "Training.$mode" does not "resolve" to anything, and per the documentation, the delivery destination will default to the Inbox in such cases.
What's wrong with my the custom pipe script to which I posted a link in my previous message, I wonder?
Thanks again for any assistance,
-Ben
On 16-08-18 22:43:32, Ben Johnson wrote:
I don't know how to solve your problem, but I have an alternate method.
Create a Junk folder and a ham folder then add the following cron jobs:
30 5 * * * /usr/local/bin/sa-learn --spam /var/vmail/Maildir/.Junk 30 6 * * 6 /usr/local/bin/sa-learn --backup > /etc/mail/spamassassin/spam.db 30 7 * * * /usr/local/bin/sa-learn --ham /var/vmail/Maildir/.Ham
-- Edgar Pettijohn
On 2016-08-18 22:58, Edgar Pettijohn wrote:
Thank you for taking a look, Edgar!
I already have the very setup that you describe.
The challenge at hand, however, is with regard to *how* messages end-up in the "ham" and "spam" folders in the first place.
This is a multi-user system and each user must be able to contribute to ham/spam training. To make this as easy as possible, and transparent to the end-users, I need for the simple act of dragging a message from Inbox -> Spam (or vice versa) to submit the sample on the user's behalf, automatically. This has all worked swimmingly in the past.
In any case, I've made some progress on this go-around.
I tested my backend pipe script by doing this, as root, and it works:
/bin/bash /usr/local/bin/sa-learn-pipe.sh --spam <
/var/vmail/gtube.txt
The message is delivered to the appropriate mailbox and all is well in the world.
Of course, in practice, Dovecot does not run as root. So, let's try the same as the vmail user:
su vmail
$ /bin/bash /usr/local/bin/sa-learn-pipe.sh --spam < /var/vmail/gtube.txt /usr/local/bin/sa-learn-pipe.sh: line 8: /tmp/sa-learn-pipe.log: Permission denied /usr/local/bin/sa-learn-pipe.sh: line 10: /tmp/sa-learn-pipe.log: Permission denied /usr/local/bin/sa-learn-pipe.sh: line 12: /tmp/sa-learn-pipe.log: Permission denied /usr/local/bin/sa-learn-pipe.sh: line 16: /tmp/sendmail-parms.txt: Permission denied /usr/local/bin/sa-learn-pipe.sh: line 22: /tmp/sa-learn-pipe.log: Permission denied /usr/local/bin/sa-learn-pipe.sh: line 37: /tmp/sa-learn-pipe.log: Permission denied /usr/local/bin/sa-learn-pipe.sh: line 40: /tmp/sa-learn-pipe.log: Permission denied /usr/local/bin/sa-learn-pipe.sh: line 43: /tmp/strace.txt: Permission denied /usr/local/bin/sa-learn-pipe.sh: line 47: /tmp/sa-learn-pipe.log: Permission denied /usr/local/bin/sa-learn-pipe.sh: line 54: /tmp/sa-learn-pipe.log: Permission denied
Aha! Clearly, the vmail user cannot read from nor write to /tmp. (Why that is, I have no idea, as the /tmp directory's permissions certainly allow for both; maybe Dovecot implements this as a security measure.)
This prompted me to change all references to /tmp in the pipe script to ~/tmp, and create this directory:
$ whoami vmail $ mkdir ~/tmp && chmod 770 ~/tmp $ /bin/bash /usr/local/bin/sa-learn-pipe.sh --ham < /var/vmail/gtube.txt
No errors this time (at least not on the console).
But I do get this in /var/log/mail.err:
Aug 19 12:04:24 example.com dovecot: lda(sa-training@example.com): Fatal: Can't open delivery mail as raw: Permission denied
I'm not sure how to interpret this message. Where is permission being denied? More importantly, what's the fix?
Thanks for any hints!
-Ben
On 2016-08-19 12:17, ben@indietorrent.org wrote:
Apologies for the rapid-fire replies here.
The strace output that I'm capturing in the pipe script pinpointed the problem:
open("/root/~/tmp/sendmail-msg-26272.txt", O_RDONLY) = -1 EACCES (Permission denied)
There seems to be some expansion occurring that assumes the root user, despite executing the pipe script as the vmail user, so I changed all references to ~/tmp in the pipe script to /var/vmail/tmp and permission is no longer denied.
But, now dovecot-lda is core-dumping. Here is the strace output:
So, I'm back to where I was with this problem two years ago.
At that time, I gave-up, because I couldn't invest the time required to compile the latest versions of Dovecot and all plugins from scratch in an effort to prove that the bug exists in the latest source.
"Dovecot always logs a detailed error message if something goes wrong. If it doesn't, it's considered a bug and will be fixed." - http://wiki2.dovecot.org/Logging
I'm happy to help identify the root-cause, but I need some guidance here.
Thank you,
-Ben
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Fri, 19 Aug 2016, ben@indietorrent.org wrote:
Do you have SELinux active? See almost at the end of http://wiki2.dovecot.org/WhyDoesItNotWork?highlight=%28selinux%29
allow for both; maybe Dovecot implements this as a security measure.)
No. Dovecot does not implement anything like that. Do you chroot ?
Er, '/root/~/tmp/' ??
First: check the SELinux thing. Second: Do you run in a chrooted environment? Third: Enclose all your script with logging, e.g.:
#!/bin/bash ( date echo "$@" id id -a echo environment env set
check for chroot
echo stat / stat / echo /proc/1/mountinfo awk '$5=="/" {print}' </proc/1/mountinfo echo /proc/$$/mountinfo awk '$5=="/" {print}' </proc/$$/mountinfo
enable bash tracing
set -vx
... # old script ) >> /var/tmp/antispam.$$.log 2>&1
Make sure /var/tmp/antispam.$$.log is writeable, maybe create a new directory with owner vmail. Make sure you have 2>&1 at the end. Your log misses all the error messages. Also, you will now have a log file for each run of the script.
To check for chroot: stat / should print inode 2, but any mountpoint has inode 2. /proc/$$/mountinfo displays the physical information of a mount, if both differ, the current process is chrooted. "1" should be the init process.
In your script:
for opt; do if [[ "$*" =~ .*ham.* ]]
This makes no sense, either use for loop and test "$opt" here, or do not use for, but use "$*"; .*ham.* should be quoted anyway.
cat<&0 >> /tmp/sendmail-msg-$$.txt Well, if for any reason this file exists, .. cat - >/tmp/sendmail-msg-$$.txt
/usr/lib/dovecot/deliver -d "sa-training@example.com" -m "Training.$mode" You've already scraped the message from stdin into a file, so add: < /tmp/sendmail-msg-$$.txt
About the '-p' switch present in the strace-variant: Please scan the mailing list for the status of it, IMHO, there had been lots of trouble in certain cases.
The strace variant should use -oLogfile.strace.$$.log in order to separate the output of the command and strace logging.
Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1
iQEVAwUBV7qnd3z1H7kL/d9rAQJXWQf9E/ucaEXMy10IE5f7JY3tbZVlROGrz+wk 5rA0/Xe/aFwgNvCzyTX+MV7BblHH//aDwlNs3L4P+bZatCjAVCmoDdQ/WDZ7wr51 mBq/vOjcullnzz8NHv2+gQgRCKhGGd8M+mVjGUlyK6jXEFjwAaivEnRA86AudZi4 ybK0CZKw+Pg+VzDcfGjvO4PHZWAxvbqktqVOUhQwEL/+A/CZ7FNSsBuuZug42TGK tmghQmAKuwY96djSV/vFax8J8WyVnGKBVLpONP9iMllGkZ7MHGacpfm0MSgsIgPv DTTdjdk1P6FIQ615rp6BRg0JKaTn7COC6YxMnuaNtlXJ2t/M5zoCNA== =/xgA -----END PGP SIGNATURE-----
On 2016-08-22 03:19, Steffen Kaiser wrote:
Thank you very much for taking the time to investigate my use-case and help me work through this, Steffen. I really appreciate it.
No.
apparmor_status
The program 'apparmor_status' is currently not installed. You can install it by typing: apt install apparmor
sestatus
The program 'sestatus' is currently not installed. You can install it by typing: apt install policycoreutils
No. Certainly not intentionally, anyway.
I know. It's weird. Presumably, Bash is responsible for this "unusual" expansion. The raw script source has ~/tmp, so why would Bash prepend it with "/root/", especially when the script is executed as the "vmail" user? Perhaps it's academic at this point, because I've changed all paths to be absolute and they are now resolved correctly.
Nice catch. I am by no means a Bash wizard, and I cobbled this together 3-4 years ago. I have no idea what I was thinking at the time.
I removed the "for" loop, leaving only the regex check, and it seems to work as intended. (FWIW, if I add quotes around the test expression, e.g., ".*ham.*", a match is never found and the check fails.)
Yes, but I uncomment that line only for debugging purposes. Normally, I do not take the interim step of saving the message contents to disk, in which case I want dovecot-lda to read from stdin.
Good to know; I have made that adjustment, too.
Bash issues aside, I've taken a step back and attempted to make the reproducible test-case as simple as possible, whittling-down the script to only the following, and dovecot-lda still segfaults:
I've added a couple of things to Dovecot's configuration, but they don't make any difference:
Required for "vmail" user to be able to call dovecot-lda/deliver.
See: http://wiki.dovecot.org/LDA ("Logging" section)
service config { unix_listener config { mode = 0600 user = vmail group = vmail } }
protocol lda {
Enable logging for dovecot-lda.
info_log_path = /var/log/dovecot-lda.log log_path = /var/log/dovecot-lda-errors.log mail_plugins = sieve quota }
Yet, nothing is logged to either of these files when the pipe script is called.
The permissions on these files look reasonable to me:
-rw-rw---- 1 vmail vmail 0 Aug 23 12:02 dovecot-lda-errors.log -rw-rw---- 1 vmail vmail 0 Aug 23 12:01 dovecot-lda.log
Here is my current "doveconf -n" output:
At this point, this seems obvious, but the problem is definitely with using dovecot-lda to send the mail.
If I simply write the contents of the spam/ham message to disk in the pipe script, it works fine. But that approach feels "hackish" to me, as it doesn't account for sieve, quota, etc. And I want the delivery to be logged.
Again, this works fine:
su vmail
$ /usr/lib/dovecot/deliver -d "sa-training@example.org" -m "Training.HAM" -p /var/vmail/gtube.txt
Here's the output from the logging that we added:
Does anything jump-out?
Thanks again for all your help with this!
--Ben
On 19/08/16 17:35, ben@indietorrent.org wrote:
So, I'm back to where I was with this problem two years ago.
Maybe this will help you. I've been using antispam plugin in the same way you intend to do it for years now. (script modification date Feb 2014).
All the Maildirs on my system are under /var/vmail/%d/%u/ and chmod'ed as vmail:vmail user. This is the script that is working for sure. You can test it by changinf the output path, but anyway it has been with me since dovecot 1.x as far as I remember, no problems at all!
#!/bin/bash
T=date +%s%N
cat<&0 >> /var/vmail/learn/$1/$T-$$.txt
exit 0
Good luck!
Karol
-- Karol Augustin karol@augustin.pl http://karolaugustin.pl/ +353 85 775 5312
participants (5)
-
Ben Johnson
-
ben＠indietorrent.org
-
Edgar Pettijohn
-
Karol Augustin
-
Steffen Kaiser