[Dovecot] Problems with dovecot 2.1.7, spamassassin 3.3.2 and antispam plugin
This might be a fairly long message, but I wanted to be sure to include as much information as possible. I'm having an issue with the dovecot-antispam plugin in that it seems to be unable to successfully run anything from the pipe backend. To qualify that, they run, but they fail ...
Running /usr/bin/sa-learn directly always returns with an error code of 1, and the bayes DB isn't actually updated. Running the /usr/local/bin/sa-learn-pipe.sh script from the example will run sa-learn successfully, but sa-learn fails partway through, right on a sql DB access. I have both bayes and FuzzyOcr data stored in mysql, and the error occurs on accessing either one. That is, I've tested with FuzzyOcr enabled, and it fails on the db access to the FuzzyOcr DB, and I've tried it with FuzzyOcr disabled. In that case, it fails on the access to the bayes DB.
The error line right there is :
libgcc_s.so.1 must be installed for pthread_cancel to workSince sa-learn terminated prematurely, the bayes DB isn't updated at all.
Running /usr/bin/sa-learn or the script /usr/local/bin/sa-learn-pipe.sh directly from cmdline works fine though - sa-learn completes correctly and updates the bayes DB properly, I can see the new tokens.
$ sa-learn-pipe.sh --spam </usr/share/doc/spamassassin/examples/sample-spam.txt $ sa-learn --spam /usr/share/doc/spamassassin/examples/sample-spam.txt
I think it would be great to have a dedicated logfile for antispam, one that would capture both stdout as well as stderr.
These tests are being done on a clean Ubuntu 13.04 system. I'm rebuilding the VM repeatedly as I configure the various pieces, and can test any changes or whatever you might suggest very easily. I've fully scripted the install and configuration of Exim4, Dovecot, Spamassassin, Clamav, Percona/Mysql, Roundcube, and can build a full VM under Virtualbox or any VPS in less than 10 minutes.
Ubuntu 13.04 Dovecot 2.1.7 dovecot-antispam 2.0+20120225-3
Anyone else using the antispam plugin with spamassassin 3.3.2 ? Anything I can check to see what's causing this problem ?
$ cat /usr/local/bin/sa-learn-pipe.sh #!/bin/bash
echo /usr/bin/sa-learn $* /tmp/sendmail-msg-$$.txt echo "$$-start ($*)" >> /var/log/sa-learn-pipe.log cat<&0 >> /tmp/sendmail-msg-$$.txt /usr/bin/sa-learn -D $* /tmp/sendmail-msg-$$.txt >> /tmp/sa-learn-pipe.$$.log 2>&1 echo $$ sa-learn rc=$? id=$(id) HOME=$HOME >> /var/log/sa-learn-pipe.log
while read line; do echo $$-sa-learn "$line" >> /var/log/sa-learn-pipe.log done < /tmp/sa-learn-pipe.$$.log
rm -f /tmp/sendmail-msg-$$.txt echo "$$-end" >> /var/log/sa-learn-pipe.log exit 0
With FuzzyOcr DISabled, fails right after Bayes DB access
$ cat /var/log/sa-learn-pipe.log 4505-start (--spam) 4505 sa-learn rc=134 id=uid=108(Debian-exim) gid=113(Debian-exim) groups=113(Debian-exim) HOME= 4505-sa-learn Nov 8 09:05:26.134 [4507] dbg: logger: adding facilities: all 4505-sa-learn Nov 8 09:05:26.134 [4507] dbg: logger: logging level is DBG 4505-sa-learn Nov 8 09:05:26.134 [4507] dbg: generic: SpamAssassin version 3.3.2 4505-sa-learn Nov 8 09:05:26.134 [4507] dbg: generic: Perl 5.014002, PREFIX=/usr, DEF_RULES_DIR=/usr/share/spamassassin, LOCAL_RULES_DIR=/etc/spamassassin, LOCAL_STATE_DIR=/var/lib/spamassassin : : 4505-sa-learn Nov 8 09:05:27.131 [4507] dbg: replacetags: done replacing tags 4505-sa-learn Nov 8 09:05:27.131 [4507] dbg: FreeMail: loaded freemail_domains entries: 2470 normal, 29 wildcard 4505-sa-learn Nov 8 09:05:27.132 [4507] dbg: plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x3879e88) implements 'learner_new', priority 0 4505-sa-learn Nov 8 09:05:27.132 [4507] dbg: bayes: learner_new self=Mail::SpamAssassin::Plugin::Bayes=HASH(0x3879e88), bayes_store_module=Mail::SpamAssassin::BayesStore::MySQL 4505-sa-learn Nov 8 09:05:27.151 [4507] dbg: bayes: using username: debian-spamd 4505-sa-learn Nov 8 09:05:27.151 [4507] dbg: bayes: learner_new: got store=Mail::SpamAssassin::BayesStore::MySQL=HASH(0x3ec0388) 4505-sa-learn Nov 8 09:05:27.152 [4507] dbg: plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x3879e88) implements 'learner_is_scan_available', priority 0 4505-sa-learn libgcc_s.so.1 must be installed for pthread_cancel to work 4505-end 7235-start ()
With FuzzyOcr ENabled, fails right after FuzzyOcr DB access
$ cat /var/log/sa-learn-pipe.log : : 7804-sa-learn Nov 8 09:37:09.612 [7806] info: FuzzyOcr: Using scan ocrad-invert: /usr/bin/ocrad -s5 -i $input 7804-sa-learn Nov 8 09:37:09.612 [7806] info: FuzzyOcr: Using scan ocrad-decolorize-invert: /usr/bin/ocrad -s5 -i $input 7804-sa-learn Nov 8 09:37:09.612 [7806] info: FuzzyOcr: Using scan ocrad-decolorize: /usr/bin/ocrad -s5 $input 7804-sa-learn Nov 8 09:37:09.612 [7806] info: FuzzyOcr: Using scan gocr: /usr/bin/gocr -i $input 7804-sa-learn Nov 8 09:37:09.612 [7806] info: FuzzyOcr: Using scan gocr-180: /usr/bin/gocr -l 180 -d 2 -i $input 7804-sa-learn Nov 8 09:37:09.612 [7806] info: FuzzyOcr: Using scan tesseract: /usr/bin/tesseract $input $output 7804-sa-learn Nov 8 09:37:09.612 [7806] info: FuzzyOcr: Importing for MLDBM databases not available (dependencies missing) 7804-sa-learn Nov 8 09:37:09.612 [7806] dbg: FuzzyOcr: Connecting to: dbi:mysql:database=FuzzyOcr;mysql_socket=/var/run/mysqld/mysqld.sock 7804-sa-learn libgcc_s.so.1 must be installed for pthread_cancel to work 7804-end
syslog entries from running with sa-learn-pipe.sh script antispam_pipe_program = /usr/local/bin/sa-learn-pipe.sh
Nov 8 09:36:41 newgalah1 imap: antispam: plugin initialising (2.0-notgit)
Nov 8 09:36:41 newgalah1 imap: antispam: verbose debug enabled
Nov 8 09:36:41 newgalah1 imap: antispam: "Junk" is exact match spam folder
Nov 8 09:36:41 newgalah1 imap: antispam: "Spam" is exact match spam folder
Nov 8 09:36:41 newgalah1 imap: antispam: no unsure folders
Nov 8 09:36:41 newgalah1 imap: antispam: "Trash" is wildcard match
trash folder
Nov 8 09:36:41 newgalah1 imap: antispam: "Deleted *" is wildcard match
trash folder
Nov 8 09:36:41 newgalah1 imap: antispam: pipe backend spam argument =
--spam
Nov 8 09:36:41 newgalah1 imap: antispam: pipe backend not-spam argument
= --ham
Nov 8 09:36:41 newgalah1 imap: antispam: pipe backend program =
/usr/local/bin/sa-learn-pipe.sh
Nov 8 09:36:41 newgalah1 imap: antispam: pipe backend tmpdir /tmpNov 8
09:37:08 newgalah1 imap: antispam: mailbox_is_unsure(Junk): 0
Nov 8 09:37:08 newgalah1 imap: antispam: mailbox_is_trash(INBOX): 0
Nov 8 09:37:08 newgalah1 imap: antispam: mailbox_is_trash(Junk): 0
Nov 8 09:37:08 newgalah1 imap: antispam: mail copy: from trash: 0, to
trash: 0
Nov 8 09:37:08 newgalah1 imap: antispam: mailbox_is_spam(INBOX): 0
Nov 8 09:37:08 newgalah1 imap: antispam: mailbox_is_spam(Junk): 1
Nov 8 09:37:08 newgalah1 imap: antispam: mailbox_is_unsure(INBOX): 0
Nov 8 09:37:08 newgalah1 imap: antispam: mail copy: src spam: 0, dst
spam: 1, src unsure: 0
Nov 8 09:37:08 newgalah1 imap: antispam: running mailtrain backend
program /usr/local/bin/sa-learn-pipe.sh
Nov 8 09:37:08 newgalah1 imap: antispam: running mailtrain backend
program /usr/local/bin/sa-learn-pipe.sh
Nov 8 09:37:08 newgalah1 imap: antispam: running mailtrain backend
program parameter 1 --spam
syslog entries from running with sa-learn directly antispam_pipe_program = /usr/bin/sa-learn
Nov 8 09:44:57 newgalah1 imap: antispam: plugin initialising (2.0-notgit)
Nov 8 09:44:57 newgalah1 imap: antispam: verbose debug
enabled
Nov 8 09:44:57 newgalah1 imap: antispam: "Junk" is exact match spam
folder
Nov 8 09:44:57 newgalah1 imap: antispam: "Spam" is exact match spam
folder
Nov 8 09:44:57 newgalah1 imap: antispam: no unsure
folders
Nov 8 09:44:57 newgalah1 imap: antispam: "Trash" is wildcard match
trash folder
Nov 8 09:44:57 newgalah1 imap: antispam: "Deleted *" is wildcard match
trash folder
Nov 8 09:44:57 newgalah1 imap: antispam: pipe backend spam argument =
--spam
Nov 8 09:44:57 newgalah1 imap: antispam: pipe backend not-spam argument
= --ham
Nov 8 09:44:57 newgalah1 imap: antispam: pipe backend program =
/usr/bin/sa-learn
Nov 8 09:44:57 newgalah1 imap: antispam: pipe backend tmpdir
/tmp
Nov 8 09:44:57 newgalah1 imap: antispam: mailbox_is_unsure(Junk):
0
Nov 8 09:44:57 newgalah1 imap: antispam: mailbox_is_trash(INBOX):
0
Nov 8 09:44:57 newgalah1 imap: antispam: mailbox_is_trash(Junk):
0
Nov 8 09:44:57 newgalah1 imap: antispam: mail copy: from trash: 0, to
trash: 0
Nov 8 09:44:57 newgalah1 imap: antispam: mailbox_is_spam(INBOX):
0
Nov 8 09:44:57 newgalah1 imap: antispam: mailbox_is_spam(Junk):
1
Nov 8 09:44:57 newgalah1 imap: antispam: mailbox_is_unsure(INBOX):
0
Nov 8 09:44:57 newgalah1 imap: antispam: mail copy: src spam: 0, dst
spam: 1, src unsure: 0
Nov 8 09:44:57 newgalah1 imap: antispam: running mailtrain backend
program /usr/bin/sa-learn
Nov 8 09:44:57 newgalah1 imap: antispam: running mailtrain backend
program /usr/bin/sa-learn
Nov 8 09:44:57 newgalah1 imap: antispam: running mailtrain backend
program parameter 1 --spam
Nov 8 09:44:58 newgalah1 imap: antispam: run program failed with exit
code -1
$ cat 90-plugin.conf plugin { antispam_debug_target = syslog antispam_verbose_debug = 1 antispam_backend = pipe antispam_trash_pattern = Trash;Deleted * antispam_spam = Junk;Spam antispam_pipe_program = /usr/local/bin/sa-learn-pipe.sh # antispam_pipe_program_args = --progress antispam_pipe_program_spam_arg = --spam antispam_pipe_program_notspam_arg = --ham antispam_pipe_tmpdir = /tmp }
$ dovecot -n # 2.1.7: /etc/dovecot/dovecot.conf # OS: Linux 3.8.0-32-generic x86_64 Ubuntu 13.04 auth_mechanisms = cram-md5 login plain first_valid_uid = 108 mail_location = maildir:~/Maildir:LAYOUT=fs:INBOX=~/Maildir/INBOX managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;as cii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave namespace inbox { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Spam { auto = subscribe special_use = \Junk } mailbox Trash { auto = subscribe special_use = \Trash } prefix = } passdb { args = scheme=plain username_format=%u /var/mail/%d/PASSWD driver = passwd-file } plugin { antispam_backend = pipe antispam_debug_target = syslog antispam_pipe_program = /usr/local/bin/sa-learn-pipe.sh antispam_pipe_program_notspam_arg = --ham antispam_pipe_program_spam_arg = --spam antispam_pipe_tmpdir = /tmp antispam_spam = Junk;Spam antispam_trash_pattern = Trash;Deleted * antispam_verbose_debug = 1 sieve = ~/.dovecot.sieve sieve_dir = ~/sieve } protocols = " imap lmtp sieve" service auth { unix_listener auth-client { mode = 0666 } } ssl = required ssl_cert =
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Fri, 8 Nov 2013, Dean Carpenter wrote:
Running /usr/bin/sa-learn directly always returns with an error code of 1, and the bayes DB isn't actually updated. Running the /usr/local/bin/sa-learn-pipe.sh script from the example will run sa-learn successfully, but sa-learn fails partway through, right on a sql DB access. I have both bayes and FuzzyOcr data stored in mysql, and the error occurs on accessing either one. That is, I've tested with FuzzyOcr enabled, and it fails on the db access to the FuzzyOcr DB, and I've tried it with FuzzyOcr disabled. In that case, it fails on the access to the bayes DB.
The error line right there is :
libgcc_s.so.1 must be installed for pthread_cancel to workSince sa-learn terminated prematurely, the bayes DB isn't updated at all.
But this is not a Dovecot problem, isn't it? Antispam calls sa-learn, but that program fails.
Do you have the system library libgcc_s installed as the error message says?
Search the net for that error message, see: https://lists.ubuntu.com/archives/foundations-bugs/2012-March/070050.html
If you are on a 64bit system, maybe sa-learn is compiled 32bit only, then the library must be installed as 32bit version as well.
4505 sa-learn rc=134 id=uid=108(Debian-exim) gid=113(Debian-exim) groups=113(Debian-exim) HOME= 4505-sa-learn libgcc_s.so.1 must be installed for pthread_cancel to work 4505-end
Nov 8 09:44:57 newgalah1 imap: antispam: running mailtrain backend program parameter 1 --spam Nov 8 09:44:58 newgalah1 imap: antispam: run program failed with exit code -1
Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux)
iQEVAwUBUoCH0F3r2wJMiz2NAQKypwf+M63gGswfoTkNqB0VB/TgK47hb8GQfwNj S6WpypdiSUbVU9NkHezdKmQNmSEyGP/34UChacNCtUbezRV77j/vc1kb/Uap+Z76 cETR5m0zFt5Svl84lm14027ZU9Ph5O0ddr2NTi1NJqmos4V8zBw2C6OfD85vFN1T HDPz6e1h8mIFKhoisudbHt/2KcQkkPA9NinGnSgtpyyGwX7pVghQ6Hb4XD5xD/7F HTXqOit52NvzDn2tflIWxdyOIbAxN5wU4q5tWvA0ZckCbAqpc9o+zbAYZm8nOqmo lm+oaaBpKNbPyScQTFTn7ccmm2GDsxxUEQ6ubX7GxPXOnQvuHjMefg== =Xew9 -----END PGP SIGNATURE-----
On Fri, 8 Nov 2013, Dean Carpenter wrote:
Running /usr/bin/sa-learn directly always returns with an error code of 1, and the bayes DB isn't actually updated. Running the /usr/local/bin/sa-learn-pipe.sh script from the example will run sa-learn successfully, but sa-learn fails partway through, right on a sql DB access. I have both bayes and FuzzyOcr data stored in mysql, and the error occurs on accessing either one. That is, I've tested with FuzzyOcr enabled, and it fails on the db access to the FuzzyOcr DB, and I've tried it with FuzzyOcr disabled. In that case, it fails on the access to the bayes DB.
The error line right there is :
libgcc_s.so.1 must be installed for pthread_cancel to workSince sa-learn terminated prematurely, the bayes DB isn't updated at all.
But this is not a Dovecot problem, isn't it? Antispam calls sa-learn, but that program fails. Exactly true. But the circumstance is particular to Dovecot, and I'm sure that *someone* else out there is using this config (dovecot and spamassassin). So hoping someone has seen this and gotten around it.
Do you have the system library libgcc_s installed as the error message says? Yup. $ locate
On 11/11/2013 02:31 AM, Steffen Kaiser wrote: libgcc_s
/lib/x86_64-linux-gnu/libgcc_s.so.1 /usr/lib/gcc/x86_64-linux-gnu/4.7/libgcc_s.so
Search the net for that error message, see: https://lists.ubuntu.com/archives/foundations-bugs/2012-March/070050.html
If you are on a 64bit system, maybe sa-learn is compiled 32bit only, then the library must be installed as 32bit version as well. /usr/bin/sa-learn is a perl script, calling the various Mail::SpamAssassin modules. No 32/64 bit there afaik.
-- Dean Carpenter deano is at areyes dot com 94TT :)
So, is anyone out there actually *using* the antispam plugin with spamassassin ? What is your recipe for success ?
The issue I'm seeing here appears to be the running of a perl script that does mysql access from dovecot. I'm running the exact same script from roundcube using the markasjunk2 plugin, and it works fine. The only difference is that roundcube provides a file argument for the email to learn, while dovecot pipes the email into the script.
dovecot sa-learn-pipe.sh --spam < /tmp/sometmpfile roundcube sa-learn-pipe.sh --spam /tmp/someotherfile
The script handles either case just fine. Testing on the cmdline works cleanly. From antispam/dovecot, it always fails on the mysql access :
27877-sa-learn Nov 16 10:05:44.887 [27879] dbg: FuzzyOcr: Connecting to: dbi:mysql:database=FuzzyOcr;mysql_socket=/var/run/mysqld/mysqld.sock
27877-sa-learn libgcc_s.so.1 must be installed for pthread_cancel to work 27877-end
Something about the perl DBI interface seems to want something from the libgcc_s.so lib and can't find it, even though it's installed. But why would it work fine from the cmdline, with any userid ? And from roundcube/markasjunk2 ?
On 11/11/2013 04:00 PM, Dean wrote:
On 11/11/2013 02:31 AM, Steffen Kaiser wrote:
On Fri, 8 Nov 2013, Dean Carpenter wrote:
Running /usr/bin/sa-learn directly always returns with an error code of 1, and the bayes DB isn't actually updated. Running the /usr/local/bin/sa-learn-pipe.sh script from the example will run sa-learn successfully, but sa-learn fails partway through, right on a sql DB access. I have both bayes and FuzzyOcr data stored in mysql, and the error occurs on accessing either one. That is, I've tested with FuzzyOcr enabled, and it fails on the db access to the FuzzyOcr DB, and I've tried it with FuzzyOcr disabled. In that case, it fails on the access to the bayes DB. The error line right there is : libgcc_s.so.1 must be installed for pthread_cancel to work Since sa-learn terminated prematurely, the bayes DB isn't updated at
all.
But this is not a Dovecot problem, isn't it? Antispam calls sa-learn, but that program fails.
Exactly true. But the circumstance is particular to Dovecot, and I'm sure that *someone* else out there is using this config (dovecot and spamassassin). So hoping someone has seen this and gotten around it.
Do you have the system library libgcc_s installed as the error message says?
Yup. $ locate libgcc_s
/lib/x86_64-linux-gnu/libgcc_s.so.1 /usr/lib/gcc/x86_64-linux-gnu/4.7/libgcc_s.so
Search the net for that error message, see: https://lists.ubuntu.com/archives/foundations-bugs/2012-March/070050.html
If you are on a 64bit system, maybe sa-learn is compiled 32bit only, then the library must be installed as 32bit version as well.
/usr/bin/sa-learn is a perl script, calling the various Mail::SpamAssassin modules. No 32/64 bit there afaik.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Mon, 11 Nov 2013, Dean wrote:
If you are on a 64bit system, maybe sa-learn is compiled 32bit only, then the library must be installed as 32bit version as well. /usr/bin/sa-learn is a perl script, calling the various Mail::SpamAssassin modules. No 32/64 bit there afaik.
I have a 64bit system and use the spamassassin demon to train ham/spam without such problem. I have the 32bit lib installed:
locate libgcc_s /lib/libgcc_s.so.1 /usr/lib/gcc/x86_64-linux-gnu/4.3/libgcc_s.so /usr/lib/gcc/x86_64-linux-gnu/4.3/libgcc_s_32.so /usr/lib/gcc/x86_64-linux-gnu/4.3/32/libgcc_s.so /usr/lib/gcc/x86_64-linux-gnu/4.4/libgcc_s.so /usr/lib/gcc/x86_64-linux-gnu/4.4/libgcc_s_32.so /usr/lib/gcc/x86_64-linux-gnu/4.4/32/libgcc_s.so /usr/lib32/libgcc_s.so.1
The 32bit library got onto the system via the "Suggests" of Debian's gcc package.
Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux)
iQEVAwUBUoothl3r2wJMiz2NAQLeYQgAmyR8m7sMJD0pFnwHNXJtZ2+ENokgYz1h yC4B9QxIe6gqA4ITdp8hx3EGQII3JYQaci+qq0IsAnqq9YimCkVE9zTBrVc6aU19 yEJ6l+lWSgoyPdMVSF41ysfv37U8BlUfqsmE6I3Yj7Pp2SkBv3P4aAxt9n0VFm6J HZUbmjdRqhBhZvl2pDAdyUmTGzQU+v8SMknllHfsuC1OYHHGJlFbCpQBobZoEkkY KxOUfGVgIxekk1o0hA+7jPzCeaxfEjXgYkLKFzQeHin8T9fEZ4ibdRG3nLezJS45 1OwX+6qlAfjTyJShwYZ/BednXIWB9m7FvBXFgbS6L1D2onIyf/Xfvg== =spQn -----END PGP SIGNATURE-----
On 11/18/2013 10:08 AM, Steffen Kaiser wrote:
On Mon, 11 Nov 2013, Dean wrote:
If you are on a 64bit system, maybe sa-learn is compiled 32bit only, then the library must be installed as 32bit version as well. /usr/bin/sa-learn is a perl script, calling the various Mail::SpamAssassin modules. No 32/64 bit there afaik.
I have a 64bit system and use the spamassassin demon to train ham/spam without such problem. I have the 32bit lib installed:
locate libgcc_s /lib/libgcc_s.so.1 /usr/lib/gcc/x86_64-linux-gnu/4.3/libgcc_s.so /usr/lib/gcc/x86_64-linux-gnu/4.3/libgcc_s_32.so /usr/lib/gcc/x86_64-linux-gnu/4.3/32/libgcc_s.so /usr/lib/gcc/x86_64-linux-gnu/4.4/libgcc_s.so /usr/lib/gcc/x86_64-linux-gnu/4.4/libgcc_s_32.so /usr/lib/gcc/x86_64-linux-gnu/4.4/32/libgcc_s.so /usr/lib32/libgcc_s.so.1
The 32bit library got onto the system via the "Suggests" of Debian's gcc package.
I installed the 32bit version too.
$ locate libgcc_s.so /lib/x86_64-linux-gnu/libgcc_s.so.1 /usr/lib/gcc/x86_64-linux-gnu/4.7/libgcc_s.so /usr/lib32/libgcc_s.so.1
Still no luck. This is with FuzzyOcr enabled, fails on the mysql DB connect
12570-sa-learn Nov 20 13:39:27.817 [12572] info: FuzzyOcr: Using scan gocr-180: /usr/bin/gocr -l 180 -d 2 -i $input 12570-sa-learn Nov 20 13:39:27.817 [12572] info: FuzzyOcr: Using scan tesseract: /usr/bin/tesseract $input $output 12570-sa-learn Nov 20 13:39:27.817 [12572] dbg: FuzzyOcr: Connecting to: dbi:mysql:database=FuzzyOcr;mysql_socket=/var/run/mysqld/mysqld.sock 12570-sa-learn libgcc_s.so.1 must be installed for pthread_cancel to work 12570-end
And this is with FuzzyOcr disabled, so it goes right to the Bayes DB and fails
14826-sa-learn Nov 20 13:57:38.246 [14828] dbg: bayes: learner_new self=Mail::SpamAssassin::Plugin::Bayes=HASH(0x476eae0), bayes_store_module=Mail::SpamAssassin::BayesStore::MySQL 14826-sa-learn Nov 20 13:57:38.265 [14828] dbg: bayes: using username: debian-spamd 14826-sa-learn Nov 20 13:57:38.265 [14828] dbg: bayes: learner_new: got store=Mail::SpamAssassin::BayesStore::MySQL=HASH(0x5039840) 14826-sa-learn Nov 20 13:57:38.266 [14828] dbg: plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x476eae0) implements 'learner_is_scan_available', priority 0 14826-sa-learn libgcc_s.so.1 must be installed for pthread_cancel to work 14826-end
This is what is *supposed* to happen (using cmdline /usr/local/bin/sa-learn-pipe.sh --spam < /tmp/email-to-learn)
15412-sa-learn Nov 20 14:02:55.095 [15414] dbg: bayes: using username: debian-spamd 15412-sa-learn Nov 20 14:02:55.095 [15414] dbg: bayes: learner_new: got store=Mail::SpamAssassin::BayesStore::MySQL=HASH(0x3ef55f0) 15412-sa-learn Nov 20 14:02:55.095 [15414] dbg: plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x3631ba0) implements 'learner_is_scan_available', priority 0 15412-sa-learn Nov 20 14:02:55.118 [15414] dbg: bayes: database connection established 15412-sa-learn Nov 20 14:02:55.118 [15414] dbg: bayes: found bayes db version 3
The markasjunk2 plugin for roundcube calls sa-learn-pipe.sh and it works fine. The difference is that it is called with a file parameter that contains the email to be learned, while Dovecot/antispam pipes the email into the script. Both methods work fine, as checked on the cmdline.
/usr/local/bin/sa-learn-pipe.sh --spam /tmp/email-to-learn /usr/local/bin/sa-learn-pipe.sh --spam < /tmp/email-to-learn
It's only when it's called from the context of Dovecot/antispam that we see the libgcc_s.so.1 error pop up, and the mysql connection apparently fail. I think that context is the key, but I don't what it is. Something about the environment is causing those mysql connections to fail. I believe it's permissions - the socket is 777, and the cmdline runs are done with a regular ID and work fine.
The antispam functionality is a great feature for remote email clients like thunderbird. Just need to get it working :) Any ideas ?
If anyone wants to test I can provide a small 32meg bootable ISO (Ubuntu mini.iso) that installs 13.04 and the various apps, all preconfigured - works fine for a bare machine or a VM/VPS. I can also provide the installer script that does all the installs/configuration. Run it on a bare 13.04 server/mini install and it does the rest.
-- Dean Carpenter deano is at areyes dot com 94TT :)
Darn. Thought I had found a good point about pthread_cancel. See the last post here :
https://bugs.launchpad.net/ubuntu/+source/gcc-3.3/+bug/40285
Juergen Perlinger (juergen-perlinger)
<https://launchpad.net/%7Ejuergen-perlinger> wrote on 2013-10-25:
#30
<https://bugs.launchpad.net/ubuntu/+source/gcc-3.3/+bug/40285/comments/30>
I finally tracked it down, I think. The pthread code tries to load
libgcc_s on pthread_exit(), and this dos not work when the effective
user was changed -- many daemons switch from root to a restricted
user, and that's one way to end up in this problem. If the library
is loaded globally for the process before the user is changed,
everything works.
This is a problem of the 'pthread_exit()' implementation
(delayed/on-demand load of libgcc_s) and/or the 'mmap()'
implementation in the kernel.I've tried putting /lib/x86_64-linux-gnu/libgcc_s.so.1 into /etc/ld.so.preload, I've changed the exec line in /etc/init/dovecot to
exec LD_PRELOAD=/lib/x86_64-linux-gnu/libgcc_s.so.1/usr/sbin/dovecot -F -c /etc/dovecot/dovecot.conf
and the sa-learn line in /usr/local/bin/sa-learn-pipe.sh to
LD_PRELOAD=/lib/x86_64-linux-gnu/libgcc_s.so.1 /usr/bin/sa-learn -D--progress $* /tmp/sendmail-msg-$$.txt >> /tmp/sa-learn-pipe.$$.log 2>&1
So far to no avail. I'm going to try the spool2dir backend with incron. Cumbersome, but it should work ...
On 11/20/2013 02:17 PM, Dean wrote:
On 11/18/2013 10:08 AM, Steffen Kaiser wrote:
On Mon, 11 Nov 2013, Dean wrote:
If you are on a 64bit system, maybe sa-learn is compiled 32bit only, then the library must be installed as 32bit version as well.
/usr/bin/sa-learn is a perl script, calling the various Mail::SpamAssassin modules. No 32/64 bit there afaik.
I have a 64bit system and use the spamassassin demon to train ham/spam without such problem. I have the 32bit lib installed:
locate libgcc_s /lib/libgcc_s.so.1 /usr/lib/gcc/x86_64-linux-gnu/4.3/libgcc_s.so /usr/lib/gcc/x86_64-linux-gnu/4.3/libgcc_s_32.so /usr/lib/gcc/x86_64-linux-gnu/4.3/32/libgcc_s.so /usr/lib/gcc/x86_64-linux-gnu/4.4/libgcc_s.so /usr/lib/gcc/x86_64-linux-gnu/4.4/libgcc_s_32.so /usr/lib/gcc/x86_64-linux-gnu/4.4/32/libgcc_s.so /usr/lib32/libgcc_s.so.1
The 32bit library got onto the system via the "Suggests" of Debian's gcc package.
I installed the 32bit version too.
$ locate libgcc_s.so /lib/x86_64-linux-gnu/libgcc_s.so.1 /usr/lib/gcc/x86_64-linux-gnu/4.7/libgcc_s.so /usr/lib32/libgcc_s.so.1
Still no luck. This is with FuzzyOcr enabled, fails on the mysql DB connect
12570-sa-learn Nov 20 13:39:27.817 [12572] info: FuzzyOcr: Using scan gocr-180: /usr/bin/gocr -l 180 -d 2 -i $input 12570-sa-learn Nov 20 13:39:27.817 [12572] info: FuzzyOcr: Using scan tesseract: /usr/bin/tesseract $input $output 12570-sa-learn Nov 20 13:39:27.817 [12572] dbg: FuzzyOcr: Connecting to: dbi:mysql:database=FuzzyOcr;mysql_socket=/var/run/mysqld/mysqld.sock 12570-sa-learn libgcc_s.so.1 must be installed for pthread_cancel to work 12570-end
And this is with FuzzyOcr disabled, so it goes right to the Bayes DB and fails
14826-sa-learn Nov 20 13:57:38.246 [14828] dbg: bayes: learner_new self=Mail::SpamAssassin::Plugin::Bayes=HASH(0x476eae0), bayes_store_module=Mail::SpamAssassin::BayesStore::MySQL 14826-sa-learn Nov 20 13:57:38.265 [14828] dbg: bayes: using username: debian-spamd 14826-sa-learn Nov 20 13:57:38.265 [14828] dbg: bayes: learner_new: got store=Mail::SpamAssassin::BayesStore::MySQL=HASH(0x5039840) 14826-sa-learn Nov 20 13:57:38.266 [14828] dbg: plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x476eae0) implements 'learner_is_scan_available', priority 0 14826-sa-learn libgcc_s.so.1 must be installed for pthread_cancel to work 14826-end
This is what is *supposed* to happen (using cmdline /usr/local/bin/sa-learn-pipe.sh --spam < /tmp/email-to-learn)
15412-sa-learn Nov 20 14:02:55.095 [15414] dbg: bayes: using username: debian-spamd 15412-sa-learn Nov 20 14:02:55.095 [15414] dbg: bayes: learner_new: got store=Mail::SpamAssassin::BayesStore::MySQL=HASH(0x3ef55f0) 15412-sa-learn Nov 20 14:02:55.095 [15414] dbg: plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x3631ba0) implements 'learner_is_scan_available', priority 0 15412-sa-learn Nov 20 14:02:55.118 [15414] dbg: bayes: database connection established 15412-sa-learn Nov 20 14:02:55.118 [15414] dbg: bayes: found bayes db version 3
The markasjunk2 plugin for roundcube calls sa-learn-pipe.sh and it works fine. The difference is that it is called with a file parameter that contains the email to be learned, while Dovecot/antispam pipes the email into the script. Both methods work fine, as checked on the cmdline.
/usr/local/bin/sa-learn-pipe.sh --spam /tmp/email-to-learn /usr/local/bin/sa-learn-pipe.sh --spam < /tmp/email-to-learn
It's only when it's called from the context of Dovecot/antispam that we see the libgcc_s.so.1 error pop up, and the mysql connection apparently fail. I think that context is the key, but I don't what it is. Something about the environment is causing those mysql connections to fail. I believe it's permissions - the socket is 777, and the cmdline runs are done with a regular ID and work fine.
The antispam functionality is a great feature for remote email clients like thunderbird. Just need to get it working :) Any ideas ?
If anyone wants to test I can provide a small 32meg bootable ISO (Ubuntu mini.iso) that installs 13.04 and the various apps, all preconfigured - works fine for a bare machine or a VM/VPS. I can also provide the installer script that does all the installs/configuration. Run it on a bare 13.04 server/mini install and it does the rest.
-- Dean Carpenter deano is at areyes dot com 94TT :)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Wed, 20 Nov 2013, Dean wrote:
Date: Wed, 20 Nov 2013 18:05:25 -0500 From: Dean deano-dovecot@areyes.com Cc: dovecot@dovecot.org Subject: Re: [Dovecot] Problems with dovecot 2.1.7, spamassassin 3.3.2 and antispam plugin
Darn. Thought I had found a good point about pthread_cancel. See the last post here :
https://bugs.launchpad.net/ubuntu/+source/gcc-3.3/+bug/40285
Juergen Perlinger (juergen-perlinger) https://launchpad.net/%7Ejuergen-perlinger wrote on 2013-10-25: #30 https://bugs.launchpad.net/ubuntu/+source/gcc-3.3/+bug/40285/comments/30
I finally tracked it down, I think. The pthread code tries to load libgcc_s on pthread_exit(), and this dos not work when the effective user was changed -- many daemons switch from root to a restricted user, and that's one way to end up in this problem. If the library is loaded globally for the process before the user is changed, everything works.
This is a problem of the 'pthread_exit()' implementation (delayed/on-demand load of libgcc_s) and/or the 'mmap()' implementation in the kernel.
I've tried putting /lib/x86_64-linux-gnu/libgcc_s.so.1 into /etc/ld.so.preload, I've changed the exec line in /etc/init/dovecot to
exec LD_PRELOAD=/lib/x86_64-linux-gnu/libgcc_s.so.1/usr/sbin/dovecot -F -c /etc/dovecot/dovecot.conf
and the sa-learn line in /usr/local/bin/sa-learn-pipe.sh to
LD_PRELOAD=/lib/x86_64-linux-gnu/libgcc_s.so.1 /usr/bin/sa-learn -D --progress $* /tmp/sendmail-msg-$$.txt >> /tmp/sa-learn-pipe.$$.log 2>&1
Does sa-learn change UID? Maybe you can make sa-learn set-uid or pass a specific -u /-x .
Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux)
iQEVAwUBUo26A13r2wJMiz2NAQJ8FwgAl/i1dxQL0XqKwP+Xzwea33LfK39p4Rqw BpNw2iedsK4EhY3dkpOGb1tpcdl9PVAZK2z8WxdTqgXJWugOfmVJRMymp+1nKVYQ lMCQp9LIMAasAl9BDa9OCN6L99qNclYSEH8SNPj1PDvxpyAeshroz2GQY1R+JuEl +I7EsZsVEkrm+Zq4ET6Jjbbv6n/nWCVRGvvUi83vb4oGUcdvpyQoJ6QmhEZDV3Gc kIkB1a4PhnhYqweuyZ2s6Xo+kYjjSIOvPcNkbDG3xwX1g2re6MlNTAHNSJbk7u21 8z10IwSlDRD6H13vWvtwfQL4v1tPafJp+RF0kRb9qO3C8DwE+7d2EQ== =+ICP -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Am 21.11.2013 08:45, schrieb Steffen Kaiser:
On Wed, 20 Nov 2013, Dean wrote:
Date: Wed, 20 Nov 2013 18:05:25 -0500 From: Dean deano-dovecot@areyes.com Cc: dovecot@dovecot.org Subject: Re: [Dovecot] Problems with dovecot 2.1.7, spamassassin 3.3.2 and antispam plugin
Darn. Thought I had found a good point about pthread_cancel. See the last post here :
https://bugs.launchpad.net/ubuntu/+source/gcc-3.3/+bug/40285
Juergen Perlinger (juergen-perlinger) https://launchpad.net/%7Ejuergen-perlinger wrote on 2013-10-25: #30
https://bugs.launchpad.net/ubuntu/+source/gcc-3.3/+bug/40285/comments/30
I finally tracked it down, I think. The pthread code tries to load libgcc_s on pthread_exit(), and this dos not work when the effective user was changed -- many daemons switch from root to a restricted user, and that's one way to end up in this problem. If the library is loaded globally for the process before the user is changed, everything works.
This is a problem of the 'pthread_exit()' implementation (delayed/on-demand load of libgcc_s) and/or the 'mmap()' implementation in the kernel.
I've tried putting /lib/x86_64-linux-gnu/libgcc_s.so.1 into /etc/ld.so.preload, I've changed the exec line in /etc/init/dovecot to
exec LD_PRELOAD=/lib/x86_64-linux-gnu/libgcc_s.so.1 /usr/sbin/dovecot -F -c /etc/dovecot/dovecot.conf
and the sa-learn line in /usr/local/bin/sa-learn-pipe.sh to
LD_PRELOAD=/lib/x86_64-linux-gnu/libgcc_s.so.1 /usr/bin/sa-learn -D --progress $* /tmp/sendmail-msg-$$.txt >> /tmp/sa-learn-pipe.$$.log 2>&1
Does sa-learn change UID? Maybe you can make sa-learn set-uid or pass a specific -u /-x .
-- Steffen Kaiser
may off topic , but you may manage spam learn ham/spam for users another way too, they only need simply forward mail, i use it like that way ,since years, it also works with sieve rules. For sure you have to manage only authed users from your system are able to send to learn aliases.
http://gtmp.org/doku.php/publications:sa-postfix-en
for other stuff i use spamass-milter or amavis milter and clamav-milter with sanesecurity antispam signatures, i never had urgent need for the dovecot antispam plugin
Best Regards MfG Robert Schetterer
[*] sys4 AG
http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQEcBAEBAgAGBQJSjbywAAoJEP8jBObu0LlEMwwH/1E1Mra8510edneAohvkc6Yj tIlGUaKIIpmddml2H5/1yql7nIMoGH389mDhJKYaMaG8qLb9tvCgdaF7s+auaJd+ fyK44BHr+8Je3CBLeeWfIfRk7QbnmbW5HkdTBoghoa6GBShfvxxYEEdDWMhN3xyq ThDbpfJSwsl0TV1x0Egz1FJ43mNzq6EPmqJdbNa+BKXb5LJUBLouFI+YQnDt48oQ q8wTOo/rd0yOYSUu+VulEAPx5rtlMKJXQxgMxRbiGqWJq+ybbvL13ah4y13bMsYi nflgllMxoFox1aFenzXMskMTeUx83FXKVJUZQo/IaO6tr+4/JGRK785w0Ky9iyM= =rYae -----END PGP SIGNATURE-----
On Thu Nov 21 08:56:39 2013, Robert Schetterer wrote:
may off topic , but you may manage spam learn ham/spam for users another way too, they only need simply forward mail, i use it like that way ,since years, it also works with sieve rules.
The problem: Users don't (at least ours). If they need a mail, they pull it out of the spam folder, and they use the "Junk" button in their client, but they can't be bothered to do more. And the antispam plugin is the only halfway effective way to automate this for all clients.
Hint: We make the plugin send a mail and process it later.
Yours Jost Krieger
| Jost.Krieger+sig@ruhr-uni-bochum.de Please help stamp out spam! | | Postmaster, JAPH, resident answer machine at RUB Comp. Center | | Sincere words are not sweet, sweet words are not sincere. | | Lao Tse, Tao Te King 81 |
i had the same problem.
solution: increase default_vsz_limit in /etc/dovecot/conf.d/10-master.conf.
for me 384M did the job.
On 11/21/2013 10:01 AM, Jost Krieger wrote:
On Thu Nov 21 08:56:39 2013, Robert Schetterer wrote:
may off topic , but you may manage spam learn ham/spam for users another way too, they only need simply forward mail, i use it like that way ,since years, it also works with sieve rules.
The problem: Users don't (at least ours). If they need a mail, they pull it out of the spam folder, and they use the "Junk" button in their client, but they can't be bothered to do more. And the antispam plugin is the only halfway effective way to automate this for all clients.
Hint: We make the plugin send a mail and process it later.
Yours Jost Krieger
participants (6)
-
Bane Ivosev
-
Dean
-
Dean Carpenter
-
Jost Krieger
-
Robert Schetterer
-
Steffen Kaiser