On 2016-08-18 22:58, Edgar Pettijohn wrote:
On 16-08-18 22:43:32, Ben Johnson wrote:
On 8/18/2016 10:26 PM, Ben Johnson wrote:
Hello,
(Full "doveconf -n" output is at the end of this message.)
I'm attempting to recreate a "recipe" I had used successfully for quite some time (in Ubuntu 12.04 LTS), which allowed me to train SpamAssassin's Bayes database when someone dragged an email message from one sub-folder of the account to another.
I later tried to re-implement it (in Ubuntu 14.04 LTS), and core-dumps resulted. I posted to this list about it several times, but nobody was ever able to help me solve the problem.
I just tried the same (in Ubuntu 16.04 LTS), hoping that the source had evolved since then, and while I don't see a core-dump, the pipe script doesn't seem to be called.
I've booby-trapped the pipe script with quite a bit of debugging output, which should be written to a file, but I don't see any indication that the pipe script is ever executed.
When logged-in as an IMAP user, and I drag a message from the user's Inbox into the "Spam" folder, I do see the following activity in the syslog:
Aug 18 22:12:15 example.com imap: antispam: mailbox_is_unsure(Junk): 0 Aug 18 22:12:15 example.com imap: antispam: mailbox_is_trash(INBOX): 0 Aug 18 22:12:15 example.com imap: antispam: mailbox_is_trash(Junk): 0 Aug 18 22:12:15 example.com imap: antispam: mail copy: from trash: 0, to trash: 0 Aug 18 22:12:15 example.com imap: antispam: mailbox_is_spam(INBOX): 0 Aug 18 22:12:15 example.com imap: antispam: mailbox_is_spam(Junk): 1 Aug 18 22:12:15 example.com imap: antispam: mailbox_is_unsure(INBOX): 0 Aug 18 22:12:15 example.com imap: antispam: mail copy: src spam: 0, dst spam: 1, src unsure: 0 Aug 18 22:12:15 example.com imap: antispam: running mailtrain backend program /bin/bash Aug 18 22:12:15 example.com imap: antispam: running mailtrain backend program /bin/bash Aug 18 22:12:15 example.com imap: antispam: running mailtrain backend program parameter 1 /usr/local/bin/sa-learn-pipe.sh Aug 18 22:12:15 example.com imap: antispam: running mailtrain backend program parameter 2 --spam
But the piped message is never delivered to the target mailbox. The mail log at /var/log/mail.log does not register any activity when I move messages (and maybe that's expected).
Here are the contents of the /usr/local/bin/sa-learn-pipe.sh script:
The permissions and mode on the script seem appropriate:
$ stat /usr/local/bin/sa-learn-pipe.sh File: '/usr/local/bin/sa-learn-pipe.sh' Size: 1765 Blocks: 8 IO Block: 4096 regular file Device: fd01h/64769d Inode: 669345 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 5000/ vmail) Gid: ( 5000/ vmail) Access: 2016-08-18 19:09:47.307392919 -0400 Modify: 2016-08-18 19:09:31.231391749 -0400 Change: 2016-08-18 19:09:31.231391749 -0400 Birth: -
What might I be doing incorrectly here?
Please let me know if I can provide any other information.
Thank you in advance,
-Ben
Full "doveconf -n" output:
$ doveconf -n # 2.2.22 (fe789d2): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.13 (7b14904) # OS: Linux 4.4.0-34-generic x86_64 Ubuntu 16.04.1 LTS auth_mechanisms = plain login disable_plaintext_auth = no listen = *,[::] log_timestamp = "%Y-%m-%d %H:%M:%S " mail_max_userip_connections = 100 mail_plugins = " quota" mail_privileged_group = vmail passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } plugin { antispam_backend = pipe antispam_debug_target = syslog antispam_pipe_program = /bin/bash antispam_pipe_program_args = /usr/local/bin/sa-learn-pipe.sh antispam_pipe_program_notspam_arg = --ham antispam_pipe_program_spam_arg = --spam antispam_pipe_tmpdir = /tmp antispam_spam_pattern_ignorecase = SPAM;JUNK antispam_trash_pattern_ignorecase = trash;Deleted * antispam_verbose_debug = 1 quota = dict:user::file:/var/vmail/%d/%n/.quotausage quota_rule2 = Trash:storage=+100M quota_rule3 = Junk:ignore quota_rule4 = INBOX:storage=+100M quota_warning = storage=100%% quota-reached 100 %u %d quota_warning2 = storage=95%% quota-warning 95 %u %d quota_warning3 = storage=80%% quota-warning 80 %u %d quota_warning4 = -storage=100%% quota-below below %u %d sieve = /var/vmail/%d/%n/.sieve sieve_max_redirects = 25 } postmaster_address = postmaster@example.com protocols = imap pop3 service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } user = root } service imap-login { client_limit = 1000 process_limit = 512 } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } } service quota-below { executable = script /usr/local/bin/quota-below.sh unix_listener quota-below { group = vmail mode = 0666 user = vmail } user = vmail } service quota-reached { executable = script /usr/local/bin/quota-reached.sh unix_listener quota-reached { group = vmail mode = 0666 user = vmail } user = vmail } service quota-warning { executable = script /usr/local/bin/quota-warning.sh unix_listener quota-warning { group = vmail mode = 0666 user = vmail } user = vmail } ssl_cert = </etc/postfix/smtpd.cert ssl_key = </etc/postfix/smtpd.key ssl_protocols = !SSLv2 !SSLv3 userdb { driver = prefetch } userdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } protocol imap { mail_plugins = quota imap_quota antispam } protocol pop3 { mail_plugins = quota pop3_uidl_format = %08Xu%08Xv } protocol lda { mail_plugins = sieve quota } protocol lmtp { mail_plugins = quota sieve postmaster_address = webmaster@localhost }
If it's helpful, I've tried replacing the pipe script's contents with the following:
##################################################################### #!/bin/bash
/usr/lib/dovecot/deliver -d "sa-training@example.com" -m "Training.$mode" -p /var/vmail/gtube.txt
# Exit with "success" status code. exit 0 #####################################################################
And still, the message never arrives when I drag a message from "Inbox" to "Spam" in the IMAP client, despite the messages logged to syslog, which "all seem to be correct".
Yet, if I do this using the above content for the script, the "gtube" message is delivered successfully:
# su vmail $ /usr/local/bin/sa-learn-pipe-test.sh
The message arrives in the Inbox, presumably because -m "Training.$mode" does not "resolve" to anything, and per the documentation, the delivery destination will default to the Inbox in such cases.
What's wrong with my the custom pipe script to which I posted a link in my previous message, I wonder?
Thanks again for any assistance,
-Ben
I don't know how to solve your problem, but I have an alternate method.
Create a Junk folder and a ham folder then add the following cron jobs:
30 5 * * * /usr/local/bin/sa-learn --spam /var/vmail/Maildir/.Junk 30 6 * * 6 /usr/local/bin/sa-learn --backup > /etc/mail/spamassassin/spam.db 30 7 * * * /usr/local/bin/sa-learn --ham /var/vmail/Maildir/.Ham
Thank you for taking a look, Edgar!
I already have the very setup that you describe.
The challenge at hand, however, is with regard to *how* messages end-up in the "ham" and "spam" folders in the first place.
This is a multi-user system and each user must be able to contribute to ham/spam training. To make this as easy as possible, and transparent to the end-users, I need for the simple act of dragging a message from Inbox -> Spam (or vice versa) to submit the sample on the user's behalf, automatically. This has all worked swimmingly in the past.
In any case, I've made some progress on this go-around.
I tested my backend pipe script by doing this, as root, and it works:
# /bin/bash /usr/local/bin/sa-learn-pipe.sh --spam < /var/vmail/gtube.txt
The message is delivered to the appropriate mailbox and all is well in the world.
Of course, in practice, Dovecot does not run as root. So, let's try the same as the vmail user:
# su vmail $ /bin/bash /usr/local/bin/sa-learn-pipe.sh --spam < /var/vmail/gtube.txt /usr/local/bin/sa-learn-pipe.sh: line 8: /tmp/sa-learn-pipe.log: Permission denied /usr/local/bin/sa-learn-pipe.sh: line 10: /tmp/sa-learn-pipe.log: Permission denied /usr/local/bin/sa-learn-pipe.sh: line 12: /tmp/sa-learn-pipe.log: Permission denied /usr/local/bin/sa-learn-pipe.sh: line 16: /tmp/sendmail-parms.txt: Permission denied /usr/local/bin/sa-learn-pipe.sh: line 22: /tmp/sa-learn-pipe.log: Permission denied /usr/local/bin/sa-learn-pipe.sh: line 37: /tmp/sa-learn-pipe.log: Permission denied /usr/local/bin/sa-learn-pipe.sh: line 40: /tmp/sa-learn-pipe.log: Permission denied /usr/local/bin/sa-learn-pipe.sh: line 43: /tmp/strace.txt: Permission denied /usr/local/bin/sa-learn-pipe.sh: line 47: /tmp/sa-learn-pipe.log: Permission denied /usr/local/bin/sa-learn-pipe.sh: line 54: /tmp/sa-learn-pipe.log: Permission denied
Aha! Clearly, the vmail user cannot read from nor write to /tmp. (Why that is, I have no idea, as the /tmp directory's permissions certainly allow for both; maybe Dovecot implements this as a security measure.)
This prompted me to change all references to /tmp in the pipe script to ~/tmp, and create this directory:
$ whoami vmail $ mkdir ~/tmp && chmod 770 ~/tmp $ /bin/bash /usr/local/bin/sa-learn-pipe.sh --ham < /var/vmail/gtube.txt
No errors this time (at least not on the console).
But I do get this in /var/log/mail.err:
Aug 19 12:04:24 example.com dovecot: lda(sa-training@example.com): Fatal: Can't open delivery mail as raw: Permission denied
I'm not sure how to interpret this message. Where is permission being denied? More importantly, what's the fix?
Thanks for any hints!
-Ben