[Dovecot] Running spamc during LMTP delivery
Hello list,
we are using dovecot-lmtp for delivery to virtual users, and vmm[1] to manage them.
One nice feature of vmm is that every virtual user has their own UID/GID on the UNIX filesystem. There is no passwd entry, so no shell login, but each user effectively has a home directory and can run commands in isolation and with lowered privileges.
This requires dovecot-lmtp because only the LMTP LDA runs with root rights and can drop privileges to become the specific user for which it is delivering mail.
We would now like to run SpamAssassin with the possibility of using the antispam plugin to enable virtual users to train their databases.
Do you have an idea how I could let dovecot-lmtp invoke spamc? Or an external command as a filter, to be run by the final user?
Thanks,
-- martin | http://madduck.net/ | http://two.sentenc.es/
"truth is stranger than fiction, but it is because fiction is obliged to stick to possibilities; truth isnt." -- mark twain
spamtraps: madduck.bogus@madduck.net
On 08/08/12 10:08, martin f krafft wrote:
Hello list,
we are using dovecot-lmtp for delivery to virtual users, and vmm[1] to manage them.
One nice feature of vmm is that every virtual user has their own UID/GID on the UNIX filesystem. There is no passwd entry, so no shell login, but each user effectively has a home directory and can run commands in isolation and with lowered privileges.
This requires dovecot-lmtp because only the LMTP LDA runs with root rights and can drop privileges to become the specific user for which it is delivering mail.
We would now like to run SpamAssassin with the possibility of using the antispam plugin to enable virtual users to train their databases.
Do you have an idea how I could let dovecot-lmtp invoke spamc? Or an external command as a filter, to be run by the final user?
Thanks,
Hi Martin,
Have you taken a look at Pigeonhole and Sieve? There's experimental support for running spam filters and external commands. You'll need at least 0.3.0 which requires Dovecot 2.1.
http://wiki2.dovecot.org/Pigeonhole/Sieve
HTH, Dan
Daniel Piddock, Systems Administrator, CoreFiling Limited http://www.corefiling.com
also sprach Daniel Piddock <dgp-dove@corefiling.co.uk> [2012.08.08.1201 +0200]:
Have you taken a look at Pigeonhole and Sieve? There's experimental support for running spam filters and external commands. You'll need at least 0.3.0 which requires Dovecot 2.1.
I have found this since and am working on it. Thanks! This looks like the ticket, and I will post back when I know more…
-- martin | http://madduck.net/ | http://two.sentenc.es/
"we all know linux is great... it does infinite loops in 5 seconds." -- linus torvalds
spamtraps: madduck.bogus@madduck.net
also sprach Daniel Piddock <dgp-dove@corefiling.co.uk> [2012.08.08.1201 +0200]:
Have you taken a look at Pigeonhole and Sieve? There's experimental support for running spam filters and external commands. You'll need at least 0.3.0 which requires Dovecot 2.1.
Dear list,
here is an update. Indeed, the pigeonhole filters suggested by Daniel were the ticket. It took me a while to figure it all out though. Therefore, for posterity, at least for those running Debian systems:
First, I compiled the extprograms plugin, running into a segfault, which Stephan helped me solve. Essentially, the extprograms are not in Debian, but everything else is (install dovecot-dev!), so I checked out revision 058de395713a [1], ran
./configure --with-dovecot=/usr/lib/dovecot --with-pigeonhole=/usr/include/dovecot/sieve --with-prefix=/usr/local/stow/pigeonhole-extprograms make make install cd /usr/local/stow/ stow pigeonhole-extprograms
Since dovecot only allows a single plugin dir to be specified, I had to hack around this by creating a symlink:
ln -s /usr/local/lib/dovecot/sieve /usr/lib/dovecot/modules
Now, after enabling the following in /etc/dovecot/conf.d/90-sieve:
sieve_extensions = +vnd.dovecot.filter sieve_plugins = sieve_extprograms
you should see 'vnd.dovecot.filter' in the dovecot -n output:
dovecot -n | grep vnd.dovecot managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave vnd.dovecot.filter ^^^^^^^^^^^^^^^^^^
I used sieve_global_extensions instead, because I wanted to enable spamc globally for all users using the dovecot LDA, which includes all virtual users on this machine. I also set (in 90-sieve.conf)
sieve_before = /etc/dovecot/sieve.before.d
and then put the following sieve script into /etc/dovecot/sieve.before.d/spamassassin.sieve:
require [ "vnd.dovecot.filter" ]; filter "spamc" [ "--no-safe-fallback" ];
and then ran
sievec spamassassin.sieve && chmod 444 spamassassin.svbin
To tell the LDA where to find the spamc filter, I added (to 90-sieve.conf):
sieve_filter_bin_dir = /etc/dovecot/sieve-filter
and symlinked spamc there
ln -s /usr/bin/spamc /etc/dovecot/sieve-filter/spamc
So far so good, this now works for real system users, but it would not work for virtual users. The reason for that is that while vmm uses UIDs/GIDs above 70000 to tighten permissions per-virtual-user, spamc and spamd were unable to deal with the lack of libnss integration.
Stephan suggested simply to let libnss know about the virtual mail accounts, and I did. After installing libnss-pgsql2 and extending /etc/nsswitch.conf to read
passwd: compat pgsql group: compat pgsql
I wrote the attached configuration file (/etc/nss-pgsql.conf), which I will submit to the vmm project. The advantage is that now, while the users cannot log in, their UIDs/GIDs on the filesystems are properly mapped, and root *can* /bin/su to them (using -s to override the shell).
After a restart of spamd and dovecot, spamc is now run for every user as part of the LDA process before the user's sieve script runs.
Thanks to Stephan for his help. Comments welcome. I hope I did not forget anything.
PS: I know there are good reasons against running a spamfilter post-queue. There are also several reasons for that. The most important for me is that spam is subjective, especially in combination with training, and I never want to reject spam for fear of false positives, and since I do not want to overload the queues of mail servers (like debian.org) that forward to my account.
-- martin | http://madduck.net/ | http://two.sentenc.es/
"the unexamined life is not worth living" -- platon
spamtraps: madduck.bogus@madduck.net
also sprach martin f krafft <madduck@madduck.net> [2012.08.09.1423 +0200]:
and then put the following sieve script into /etc/dovecot/sieve.before.d/spamassassin.sieve:
require [ "vnd.dovecot.filter" ]; filter "spamc" [ "--no-safe-fallback" ];
and then ran
sievec spamassassin.sieve && chmod 444 spamassassin.svbin
To tell the LDA where to find the spamc filter, I added (to 90-sieve.conf):
sieve_filter_bin_dir = /etc/dovecot/sieve-filter
and symlinked spamc there
ln -s /usr/bin/spamc /etc/dovecot/sieve-filter/spamc
Btw, this won't work reliably. The reason is that vnd.dovecot.filter currently requires the filter executable to soak up all of the input before it even bothers reading its output. Under certain conditions, however — for instance when the input is larger than the SpamAssassin max_size setting, spamc just pipes input to output, without buffering. This would yield a deadlock because vnd.dovecot.filter would not be reading output yet but still be writing input, whereas spamc would no longer be willing to handle input while its output was not being read.
Stephan Bosch has indicated a solution to this problem (asynchronous IO), but until that's in place, I simply put the following shell script in place:
-----8<----8<----8<----- #!/bin/sh set -eu
TMPFILE=$(tempfile -p spamc) cleanup() { rm -f $TMPFILE; trap - EXIT; } trap cleanup EXIT
cat > $TMPFILE spamc -x < $TMPFILE
cleanup -----8<----8<----8<-----
This is ugly because Dovecot itself may already have spooled the mail message to disk (if it was too large for in-memory buffering), and now we're spooling it a second time. Asynchronous IO will alleviate this need for the second roundtrip via the filesystem.
Hope this helps,
--
.''. martin f. krafft <madduck@d.o> Related projects: : :' : proud Debian developer http://debiansystem.info . ' http://people.debian.org/~madduck http://vcs-pkg.org
`- Debian - when you have better things to do than fixing systems
also sprach Timo Sirainen <tss@iki.fi> [2012.08.08.1509 +0200]:
Do you have an idea how I could let dovecot-lmtp invoke spamc? Or an external command as a filter, to be run by the final user?
Use spampd LMTP proxy?
That's a system-wide filtering daemon, it does not run per-user. Since spam is a subjective classification, especially as soon as you add training to the mix, we require per-user configuration.
-- martin | http://madduck.net/ | http://two.sentenc.es/
warning: dates in calendar are closer than they appear.
spamtraps: madduck.bogus@madduck.net
participants (4)
-
Daniel Piddock
-
martin f krafft
-
martin f krafft
-
Timo Sirainen