antispam plugin, pipe backend, how to make it work?

Wed Apr 13 09:55:33 UTC 2016

Am 13.04.2016 um 11:31 schrieb Steffen Kaiser:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On Wed, 13 Apr 2016, Johannes Rohr wrote:
>
>> tweaked the wrapper script to make it work. It seems like the 
>> sa-learn call
>
> See the page again.

Can you kindly be more precise? The wrapper script at the page is 
broken, it cannot work, as it fails to write the message passed from the 
plugin to  /tmp/sendmail-msg-$$.txt. So the input passed to sa-learn is 
missing, which I fixed by inserting "cat > /tmp/sendmail-msg-$$.txt" at 
the beginning of the script. However, this is not the actual issue, as 
sa-learn even fails even when the script is fixed.
>
>> silently fails:
>>
>> "/usr/local/bin/sa-learn-pipe.sh" 10L, 350C geschrieben
>> root at vm10:/etc/dovecot/conf.d# cat /tmp/sa-learn-pipe.log
>> starting log
>> 25200-start (--username=johannes at rohr.org --spam)
>> libgcc_s.so.1 must be installed for pthread_cancel to work
>> 25200-end
>>
>> Does the above error message have anything to do with it? Running the 
>> script & sa-learn manually works fine. do virtual users change 
>> something how dovecot calls programs, such as chrooting them?
>
> Maybe, the script uses different search paths? Uncomment "# env" to 
> verify.

I'll try later, although I doubt that environment variables are to 
blame. (For now we have a cronjob which is scanning the Junk directories 
periodically as a workaround.) I have to change back a whole bunch of 
config files to enable the plugin again. It seems to be a bug which hits 
daemons that run as a limited user, see 
https://bugs.launchpad.net/ubuntu/+source/gcc-3.3/+bug/40285/comments/30

I am a bit worried because the same error is reported from newer Ubuntu 
versions, so unless the spamassassin guys have fixed it their way, it 
might not go away by upgrading

Cheers,

Johannes
>
>>
>> Cheers,
>>
>> Johannes
>>
>>
>> Am 13.04.2016 um 09:45 schrieb Johannes Rohr:
>>> Am 13.04.2016 um 09:17 schrieb Tom Hendrikx:
>>>> On 13-04-16 08:27, Johannes Rohr wrote:
>>>>> Hi List,
>>>>>
>>>>> does the antispam plugin any way of actually seeing how it is
>>>>> calling the pipe program and how it is failing? I can't reproduce the
>>>>> failure when I call sa-learn manually. Can someone tell me, with
>>>>> which UID the call happens, is it the UID the current dovecot process
>>>>> is running under (which would be vmail, as it is a virtual user
>>>>> setup) Cheers,
>>>>>
>>>> Hi Johannes,
>>>> http://wiki2.dovecot.org/Pigeonhole/Sieve/Troubleshooting
>>>> This is exactly what the wrapper script is supposed to log for you, or
>>>> which is trivially to add to its output. The wrapper script is not
>>>> actually useful for production, but it is for debugging. That is why I
>>>> asked for its logging, and any differences between manual run and a
>>>> dovecot run.
>>>
>>> When I used the wrapper script recommended at 
>>> http://wiki2.dovecot.org/Plugins/Antispam, it merely reported the 
>>> PID and parameters (--username=some at na.me --ham). The wrapper script 
>>> noted in the dovecot wiki seems broken. Note the commented out line:
>>>
>>> #echo $* > /tmp/sendmail-parms.txt cat<&0 >> /tmp/sendmail-msg-$$.txt
>>>
>>> It seems like the second half of the line "cat<&0 >> 
>>> /tmp/sendmail-msg-$$.txt" was originally a separate line, catting 
>>> the message, which has been passed by the plugin via STDOUT to a 
>>> file. Without that command, the call of sa-learn in the next line 
>>> has no file to read from:
>>>
>>> /usr/bin/sa-learn $* /tmp/sendmail-msg-$$.txt && rm -f 
>>> /tmp/sendmail-msg-$$.txt &
>>>
>>> So I moved the cat command to a separate uncommented line and to be 
>>> sure I also commented out the
>>>
>>> rm -f /tmp/sendmail-msg-$$.txt &
>>>
>>> to verify that the file actually contains something, (why is there 
>>> an "&" terminating the line, which would have the rm command 
>>> executed in the background?)
>>>
>>> But it seems the
>>>
>>> /tmp/sendmail-msg-$$.txt &
>>>
>>> was empty and the whole operation continued to fail. Could it be 
>>> that the plugin fails to pass the actual message?
>>>
>>> Cheers,
>>>
>>> Johannes
>>>
>>>>
>>>> Regards,
>>>> Tom
>>>>
>>>>
>>>>> Johannes
>>>>>
>>>>> Am 12.04.2016 um 23:09 schrieb Johannes Rohr:
>>>>>> Hi Tom,
>>>>>>
>>>>>> Am 12.04.2016 um 22:26 schrieb Tom Hendrikx:
>>>>>>> On 12-04-16 20:14, Johannes Rohr wrote:
>>>>>>>> Hi, my setup is a dovecot 2.0.19 IMAP server on Ubuntu Precise
>>>>>>>> with the antispam plugin in version  2.0+20120225-2 and
>>>>>>>> spamassassin at version 3.2.2
>>>>>>>>
>>>>>>>> I have been trying and failed to get the pipe backend of the
>>>>>>>> antispam plugin to work. Spamassin by itself works, a manual
>>>>>>>> call of sa-learn works fine. Bayes data is stored in a mysql
>>>>>>>> DB.
>>>>>>>>
>>>>>>>> I have the following configuration in
>>>>>>>> /etc/dovecot/conf.d/90-plugin.conf
>>>>>>>>
>>>>>>>> plugin { #setting_name = value sieve=~/.dovecot.sieve
>>>>>>>> sieve_dir=~/sieve antispam_pipe_program_spam_arg = --spam
>>>>>>>> antispam_pipe_program_notspam_arg  = --ham
>>>>>>>> antispam_pipe_program = /usr/local/bin/sa-learn-pipe.sh
>>>>>>>> antispam_pipe_program_args = --username=%u # % expansion done
>>>>>>>> by dovecot
>>>>>>> You need to specify the argument list as a list seperated by
>>>>>>> semicolons, per example on the wiki:
>>>>>>>
>>>>>>> antispam_pipe_program_args = --username;%u
>>>>>> Not sure about that, because sa-learn expects the parameter
>>>>>> --username=user at na.me, this is one parameter, not two, and it
>>>>>> works for me locally, where I also set up dovecot to test. The
>>>>>> main differences are that the versions I have here are newer, I'm
>>>>>> running Debian sid, the server is on Ubuntu Precise, and second,
>>>>>> the server is handling virtual users, locally I have only real Unix
>>>>>> users.
>>>>>>
>>>>>> Also, dovecot doesn't complain about configuration errors regard
>>>>>> --username=%u. What other way would be there to get the parameter
>>>>>> passed to sa-learn the way I want?
>>>>>>
>>>>>> btw, I tried this, just to be sure, but the result was the same.
>>>>>>
>>>>>>> The verbose debug should give you some logging to work with.
>>>>>> Yes, and I pasted the output. I can of course repeat:
>>>>>>
>>>>>> Apr 12 22:53:02 vm10 imap: antispam: mailbox_is_unsure(Junk): 0 Apr
>>>>>> 12 22:53:02 vm10 imap: antispam: 
>>>>>> mailbox_is_trash(INBhttp://wiki2.dovecot.org/Pigeonhole/Sieve/TroubleshootingOX): 
>>>>>> 0 Apr 12
>>>>>> 22:53:02 vm10 imap: antispam: mailbox_is_trash(Junk): 0 Apr 12
>>>>>> 22:53:02 vm10 imap: antispam: mail copy: from trash: 0, to trash:
>>>>>> 0 Apr 12 22:53:02 vm10 imap: antispam: mailbox_is_spam(INBOX): 0
>>>>>> Apr 12 22:53:02 vm10 imap: antispam: mailbox_is_spam(Junk): 1 Apr
>>>>>> 12 22:53:02 vm10 imap: antispam: mailbox_is_unsure(INBOX): 0 Apr 12
>>>>>> 22:53:02 vm10 imap: antispam: mail copy: src spam: 0, dst spam: 1,
>>>>>> src unsure: 0 Apr 12 22:53:02 vm10 imap: antispam: running
>>>>>> mailtrain backend program /usr/bin/sa-learn Apr 12 22:53:02 vm10
>>>>>> imap: antispam: running mailtrain backend program
>>>>>> /usr/bin/sa-learn Apr 12 22:53:02 vm10 imap: antispam: running
>>>>>> mailtrain backend program parameter 1 --username=johannes at rohr.org
>>>>>> Apr 12 22:53:02 vm10 imap: antispam: running mailtrain backend
>>>>>> program parameter 2 --spam Apr 12 22:53:03 vm10 imap: antispam: run
>>>>>> program failed with exit code -1
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Also, what does the wrapper script log when dovecot runs it?
>>>>>> I have actually ditched  the wrapper script because it works
>>>>>> without locally, I have configured the plugin to call sa-learn
>>>>>> directly.
>>>>>>
>>>>>>> How does that differ from a manual run?
>>>>>> sa-learn works just fine with a manual run. It seems as if the
>>>>>> plugin is passing the parameters wrongly, but the debugging output
>>>>>> doesn't reveal the actual command, so it's just guesswork.
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Johannes
>>>>>>
>>>>>>>
>>>>>>>> antispam_trash = trash;Trash;Deleted Items;Deleted Messages
>>>>>>>> antispam_spam = SPAM;Junk antispam_backend = pipe
>>>>>>>> antispam_verbose_debug = 1 antispam_debug_target = syslog
>>>>>>>> antispam_pipe_tmpdir = /tmp }
>>>>>>>>
>>>>>>>> I have tried the following wrapper script: #!/bin/bash # set
>>>>>>>> -x LOGFILE=/tmp/sa-learn-pipe.log echo "$$-start ($*)" >>
>>>>>>>> $LOGFILE #echo $* > /tmp/sendmail-parms.txt cat<&0 >>
>>>>>>>> /tmp/sendmail-msg-$$.txt /usr/bin/sa-learn $* -D
>>>>>>>> /tmp/sendmail-msg-$$.txt >> $LOGFILE 2>&1 && \ rm -f
>>>>>>>> /tmp/sendmail-msg-$$.txt echo "$$-end" >> $LOGFILE exit 0
>>>>>>>>
>>>>>>>> It somehow isn't working. I can call the wrapper script
>>>>>>>> manually e.g. sudo -u vmail /usr/local/bin/sa-learn-pipe.sh
>>>>>>>> --username=some at mail.user --ham \ < someham
>>>>>>>>
>>>>>>>> But it doesn't work from dovecot. It seems the script is
>>>>>>>> called. but no new patterns show up in the bayes DB.
>>>>>>>> Unfortunately, the plugin seems to be documented poorly. How
>>>>>>>> does it call the programme it pipes to? Is it something like
>>>>>>>> $programme --par1 --par2 < message ? Why can't / shouldn't I
>>>>>>>> call sa-learn directly? All the instructions I found on the in
>>>>>>>> internet contained a wrapper script, but none bothered to
>>>>>>>> explain why you would actually need it.
>>>>>>>>
>>>>>>>> I would be very grateful, if someone could enlighten me on
>>>>>>>> that...
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>> Johannes
>>
>
> - -- Steffen Kaiser
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
>
> iQEVAwUBVw4R8Hz1H7kL/d9rAQKs7gf9F0QTDOugkaoIFCbzPlFohhhXGOKt6awv
> AWVCqJdZgwmnYqH2WTn+xyueIG+HDZUrHLQB7CirjOIB2J8YWyprcsaBkcnZwxBW
> kzklgjp9zK9ttHD7HHhkw7o4LwVjkOGtXJR49vz9bWL0P0N05dtdpI+HZuLS/Inx
> qz+F7FLd7th2SFrFSRSKTlaVRyv/Ta6fh50tra+DQe5f4NblUwHe7dOUmXKYxs+U
> Ievm4eKTxQA0GuQwyQSgKVqIKWZdAmA1bbmhbREK+7AEROCDt8XBYK1EJsRsG7LH
> fGh6+kLi5QHihbumJtv14ba6P137UjnCwQ+6y8X38tDY50hOnVXASw==
> =bmyk
> -----END PGP SIGNATURE-----