[Dovecot] Trouble implementing Antispam plug-in for Dovecot
Hello,
I am attempting to use the Antispam plug-in for Dovecot and am having trouble with the implementation.
My configuration is Dovecot (1.2.9) + Sieve + SpamAssassin on Ubuntu 10.04. Mail that SpamAssassin tags as SPAM is moved into the affected user's "Junk" folder (/var/vmail/example.com/user/Maildir/.Junk).
I followed the Wiki article at http://wiki2.dovecot.org/Plugins/Antispam . After restarting Dovecot, my IMAP client (Thunderbird v15) reports the following whenever I attempt to move mail from Inbox to "Junk" directory:
"[CANNOT] antispam plugin not configured"
That exact phrase appears only once in a Google search ( http://dovecot-antispam.sourcearchive.com/documentation/1.1plus-p20090218.gi... ):
if (!hamaddr || !spamaddr) { mail_storage_set_error(t->box->storage, ME(NOTPOSSIBLE) "antispam plugin not configured"); return -1; }
Based on the surrounding source code, which I realize may be old, it seems that Ham and Spam addresses are required, or similar.
I am using the following configuration options:
/etc/dovecot/dovecot.conf
## IMAP specific settings protocol imap { mail_executable = /usr/lib/dovecot/rawlog /usr/lib/dovecot/imap mail_plugins = quota imap_quota antispam }
# [...]
plugin { antispam_debug_target = syslog antispam_verbose_debug = 1 antispam_backend = pipe antispam_pipe_program = /usr/bin/sa-learn-pipe.sh antispam_pipe_program_spam_arg = --spam antispam_pipe_program_notspam_arg = --ham antispam_spam_pattern_ignorecase = spam;junk }
For what it's worth, the logging directives seem not to have any effect (and perhaps the rest of the directives do not, either, hence the problem I'm having).
I tried changing the above directives to their older directive names (I also changed the order slightly, which may be irrelevant):
plugin { antispam_debug_target = syslog antispam_verbose_debug = 1 antispam_backend = pipe antispam_mail_spam = --spam antispam_mail_notspam = --ham antispam_mail_sendmail = /usr/bin/sa-learn-pipe.sh antispam_spam_pattern_ignorecase = spam;junk }
This changed the message in my IMAP client, when attempting to move a message from my Inbox to the Junk folder, to:
"[SERVERBUG] failed to send mail"
Does this indicate a problem with the "/usr/bin/sa-learn-pipe.sh" script that is provided on the Wiki (cited above)?
Also, it was not clear from the documentation ( http://johannes.sipsolutions.net/files/antispam.html ) whether or not the Pipe back-end requires the X-DSPAM-Signature header, and if so, to what value it should be set when using SpamAssassin.
Thanks for any pointers,
-Ben
On 2012-09-05 6:20 PM, Ben Johnson ben@indietorrent.org wrote:
My configuration is Dovecot (1.2.9) + Sieve + SpamAssassin on Ubuntu 10.04.
1.2.9 is really old... you really need to upgrade to a recent/stable version.
I followed the Wiki article at http://wiki2.dovecot.org/Plugins/Antispam
That is for version 2.x (note the wiki2)...
I recommend going ahead and upgrading to the latest 2.1.9 and starting over.
--
Best regards,
Charles
On 9/6/2012 6:10 AM, Charles Marcus wrote:
On 2012-09-05 6:20 PM, Ben Johnson ben@indietorrent.org wrote:
My configuration is Dovecot (1.2.9) + Sieve + SpamAssassin on Ubuntu 10.04.
1.2.9 is really old... you really need to upgrade to a recent/stable version.
Thanks, Charles. I do see your point. One of the challenges we face in this regard is that we're using a Long-Term-Support version of Ubuntu (10.04) and 1.2.9 is the latest package in the OS's repository.
That said, we could upgrade manually, but this is a production server on which downtime must be minimized, and we all know how unexpected issues arise during installation (even when the procedure is tested in a closely equivalent development environment).
I followed the Wiki article at http://wiki2.dovecot.org/Plugins/Antispam
That is for version 2.x (note the wiki2)...
Noted; thank you.
I recommend going ahead and upgrading to the latest 2.1.9 and starting over.
Maybe I'll wait until we upgrade from Ubuntu 10.04 to 12.04 to fiddle with this further.
Thanks again,
-Ben
On 06/09/2012 18:56, Ben Johnson wrote:
On 2012-09-05 6:20 PM, Ben Johnson ben@indietorrent.org wrote:
My configuration is Dovecot (1.2.9) + Sieve + SpamAssassin on Ubuntu 10.04.
1.2.9 is really old... you really need to upgrade to a recent/stable version. Thanks, Charles. I do see your point. One of the challenges we face in
On 9/6/2012 6:10 AM, Charles Marcus wrote: this regard is that we're using a Long-Term-Support version of Ubuntu (10.04) and 1.2.9 is the latest package in the OS's repository.
That said, we could upgrade manually, but this is a production server on which downtime must be minimized, and we all know how unexpected issues arise during installation (even when the procedure is tested in a closely equivalent development environment).
I personally use (lightweight) virtualisation on any new machine, I really don't see any reason why NOT to. I would typically also setup my mounts such that the operating system is separate from "the data". This makes it easy to upgrade the OS/services, but without touching the data (test before/after on the same data for example)
So in my situation I would boot a fairly small (gentoo in my case)
virtual environment that runs only dovecot + postfix, it mounts the mail
spools separately - I say "boot", but because I'm using linux-vservers,
it's really a fancy chroot, and so the instance will start in 2-3
seconds (restarts are similarly near instant). I would upgrade by
cloning this installation, upgrading it, testing it to bits, and then to
make it live basically you swap this "machine" for the live machine.
There are various ways it could be made near seamless, but in my
situation I can bear a couple of seconds whilst I literally restart the
"machine"
Similarly I segregate all my services into a dozen or so "virtual machines", so DNS has it's own "machine" and so does logging, databases, almost every webservice gets its own virtual environment, etc. You could use a full blown vmware/kvm/etc if that floats your boat better, but the point remains it's so trivial to install, makes upgrades to trivial and massively decreases your downtime risk that it's very hard to find a reason NOT to do it...
I haven't tried too hard to keep my instances tiny, so each is probably
around 400-600MB in my case. However, if it were important this could
easily be reduced to 10-100s MB each using various hardlink features.
As you can see it's easy to snapshot a whole machine to manage
upgrades/backups, etc
This is more about infrastructure, but I honestly can't get over how many people are sitting on their hands shackled by "I'm on Debian xxx and I can't install any software newer than 5 years old"... It's so easy to escape from that trap...!!
Good luck
Ed W
On 9/6/2012 2:19 PM, Ed W wrote:
On 06/09/2012 18:56, Ben Johnson wrote:
On 2012-09-05 6:20 PM, Ben Johnson ben@indietorrent.org wrote:
My configuration is Dovecot (1.2.9) + Sieve + SpamAssassin on Ubuntu 10.04.
1.2.9 is really old... you really need to upgrade to a recent/stable version. Thanks, Charles. I do see your point. One of the challenges we face in
On 9/6/2012 6:10 AM, Charles Marcus wrote: this regard is that we're using a Long-Term-Support version of Ubuntu (10.04) and 1.2.9 is the latest package in the OS's repository.
That said, we could upgrade manually, but this is a production server on which downtime must be minimized, and we all know how unexpected issues arise during installation (even when the procedure is tested in a closely equivalent development environment).
I personally use (lightweight) virtualisation on any new machine, I really don't see any reason why NOT to. I would typically also setup my mounts such that the operating system is separate from "the data". This makes it easy to upgrade the OS/services, but without touching the data (test before/after on the same data for example)
Thanks for your valuable insights, Ed. That seems like a worthwhile approach.
So in my situation I would boot a fairly small (gentoo in my case) virtual environment that runs only dovecot + postfix, it mounts the mail spools separately - I say "boot", but because I'm using linux-vservers, it's really a fancy chroot, and so the instance will start in 2-3 seconds (restarts are similarly near instant). I would upgrade by cloning this installation, upgrading it, testing it to bits, and then to make it live basically you swap this "machine" for the live machine. There are various ways it could be made near seamless, but in my situation I can bear a couple of seconds whilst I literally restart the "machine"
Similarly I segregate all my services into a dozen or so "virtual machines", so DNS has it's own "machine" and so does logging, databases, almost every webservice gets its own virtual environment, etc. You could use a full blown vmware/kvm/etc if that floats your boat better, but the point remains it's so trivial to install, makes upgrades to trivial and massively decreases your downtime risk that it's very hard to find a reason NOT to do it...
While I'm with you here, and I understand the theory (and practice, to some extent), doesn't all of this require a true, physical machine?
We can't justify the expense associated with a physical machine in a hosted environment, so we're left with so-called VPSs. My understanding is that OpenVZ cannot be installed on a VPS (for seemingly obvious reasons -- namely, that the VPS is itself an OpenVZ container).
I haven't tried too hard to keep my instances tiny, so each is probably around 400-600MB in my case. However, if it were important this could easily be reduced to 10-100s MB each using various hardlink features. As you can see it's easy to snapshot a whole machine to manage upgrades/backups, etc
This is more about infrastructure, but I honestly can't get over how many people are sitting on their hands shackled by "I'm on Debian xxx and I can't install any software newer than 5 years old"... It's so easy to escape from that trap...!!
Perhaps easy, but not necessarily inexpensive. ;-)
Thanks again for sharing the details of your strategy; I'll bear all of this in mind moving forward.
Good luck
Ed W
-Ben
On 9/6/2012 1:56 PM, Ben Johnson wrote:
On 9/6/2012 6:10 AM, Charles Marcus wrote:
On 2012-09-05 6:20 PM, Ben Johnson ben@indietorrent.org wrote:
My configuration is Dovecot (1.2.9) + Sieve + SpamAssassin on Ubuntu 10.04.
1.2.9 is really old... you really need to upgrade to a recent/stable version.
Thanks, Charles. I do see your point. One of the challenges we face in this regard is that we're using a Long-Term-Support version of Ubuntu (10.04) and 1.2.9 is the latest package in the OS's repository.
That said, we could upgrade manually, but this is a production server on which downtime must be minimized, and we all know how unexpected issues arise during installation (even when the procedure is tested in a closely equivalent development environment).
I followed the Wiki article at http://wiki2.dovecot.org/Plugins/Antispam
That is for version 2.x (note the wiki2)...
Noted; thank you.
I recommend going ahead and upgrading to the latest 2.1.9 and starting over.
Maybe I'll wait until we upgrade from Ubuntu 10.04 to 12.04 to fiddle with this further.
Thanks again,
-Ben
So, I setup a development environment that mimics the production environment in question.
Must I install Dovecot from source to install Antispam from source?
Because I installed Dovecot using a back-ported package ( https://launchpad.net/~christian-roessner-net/+archive/dovecot ) and when I go to install the Antispam plug-in from source, executing the "configure" script yields the following error:
# ~/dovecot-antispam-plugin# ./configure [...] checking for dovecot-config in "/usr/local/lib/dovecot"... not found configure: configure: Use --with-dovecot=DIR to provide the path to the dovecot-config file. configure: error: dovecot-config not found
I'm assuming that this directory does not exist on my system because I didn't install Dovecot from source.
Is this accurate? If so, is my only option to install Dovecot from source and then attempt the same with the Antispam plug-in?
Thanks for any pointers,
-Ben
At 3PM -0400 on 10/09/12 Ben Johnson wrote:
So, I setup a development environment that mimics the production environment in question.
Must I install Dovecot from source to install Antispam from source?
Because I installed Dovecot using a back-ported package ( https://launchpad.net/~christian-roessner-net/+archive/dovecot ) and when I go to install the Antispam plug-in from source, executing the "configure" script yields the following error:
# ~/dovecot-antispam-plugin# ./configure [...] checking for dovecot-config in "/usr/local/lib/dovecot"... not found configure: configure: Use --with-dovecot=DIR to provide the path to the dovecot-config file. configure: error: dovecot-config not found
I'm assuming that this directory does not exist on my system because I didn't install Dovecot from source.
It looks to me like you need to install the dovecot-dev package from that site (if you haven't already), and pass --with-dovecot=/usr/lib/dovecot to configure.
Ben
On 9/10/2012 9:14 PM, Ben Morrow wrote:
At 3PM -0400 on 10/09/12 Ben Johnson wrote:
So, I setup a development environment that mimics the production environment in question.
Must I install Dovecot from source to install Antispam from source?
Because I installed Dovecot using a back-ported package ( https://launchpad.net/~christian-roessner-net/+archive/dovecot ) and when I go to install the Antispam plug-in from source, executing the "configure" script yields the following error:
# ~/dovecot-antispam-plugin# ./configure [...] checking for dovecot-config in "/usr/local/lib/dovecot"... not found configure: configure: Use --with-dovecot=DIR to provide the path to the dovecot-config file. configure: error: dovecot-config not found
I'm assuming that this directory does not exist on my system because I didn't install Dovecot from source.
It looks to me like you need to install the dovecot-dev package from that site (if you haven't already), and pass --with-dovecot=/usr/lib/dovecot to configure.
Ben
You nailed it, Ben. Thanks!
I downloaded/installed the dovecot-dev package from the same maintainer and was able to build Antispam without issue.
The hang-up now seems to be with my configuration.
I'm seeing the following error in my email client whenever I attempt to move an email message from, say, Inbox to Junk folder:
[SERVERBUG] BUG Unknown internal error.
Nothing at all appears in syslog when this happens.
I followed the instructions at http://wiki2.dovecot.org/Plugins/Antispam , to the letter, I believe.
The relevant portions of my configuration are as follows:
protocol imap { mail_plugins = quota imap_quota antispam }
plugin { quota = dict:user::file:/var/vmail/%d/%n/.quotausage sieve = /var/vmail/%d/%n/.sieve
antispam_backend = MAILTRAIN antispam_mail_sendmail = /usr/bin/sa-learn-pipe.sh antispam_mail_spam = --spam antispam_mail_notspam = --ham antispam_spam_pattern_ignorecase = spam;junk }
The permissions and execute bit are set in accordance with other executables in the same directory:
# ls -lah /usr/bin/ | grep -i "sa-learn-pipe.sh" -rwxr-xr-x 1 root root 297 Sep 5 11:17 sa-learn-pipe.sh
I would provide more log content, but the debug options seem to be gone in the forked, >= 2.0 version of this plug-in.
Any ideas are very much appreciated.
Thanks,
-Ben
On 9/11/2012 10:40 AM, Ben Johnson wrote:
You nailed it, Ben. Thanks!
I downloaded/installed the dovecot-dev package from the same maintainer and was able to build Antispam without issue.
The hang-up now seems to be with my configuration.
I'm seeing the following error in my email client whenever I attempt to move an email message from, say, Inbox to Junk folder:
[SERVERBUG] BUG Unknown internal error.
Nothing at all appears in syslog when this happens.
I followed the instructions at http://wiki2.dovecot.org/Plugins/Antispam , to the letter, I believe.
The relevant portions of my configuration are as follows:
protocol imap { mail_plugins = quota imap_quota antispam }
plugin { quota = dict:user::file:/var/vmail/%d/%n/.quotausage sieve = /var/vmail/%d/%n/.sieve
antispam_backend = MAILTRAIN antispam_mail_sendmail = /usr/bin/sa-learn-pipe.sh antispam_mail_spam = --spam antispam_mail_notspam = --ham antispam_spam_pattern_ignorecase = spam;junk }
The permissions and execute bit are set in accordance with other executables in the same directory:
# ls -lah /usr/bin/ | grep -i "sa-learn-pipe.sh" -rwxr-xr-x 1 root root 297 Sep 5 11:17 sa-learn-pipe.sh
I would provide more log content, but the debug options seem to be gone in the forked, >= 2.0 version of this plug-in.
Any ideas are very much appreciated.
Thanks,
-Ben
Given that nobody shared any insight here, I decided to stand-up a completely new VM with Ubuntu 12.04 LTS and see if the same error occurs.
While the error is different this time around, I am still unable to get any variant of the dovecot-antispam plug-in working.
When I attempt to use the current packages from Ubuntu 12.04.1 LTS, and I use the above configuration, my syslog fills-up with the following:
dovecot: imap-login: Login: user=user@example.com, method=PLAIN, rip=10.0.1.57, lip=10.0.1.8, mpid=2879, TLS dovecot: master: Error: service(imap): child 2879 returned error 3
Surprisingly, I can find only two other mentions of "error 3" anywhere on the Internet. Those mentions were from another user to this mailing list:
http://www.dovecot.org/list/dovecot/2012-August/067787.html http://www.dovecot.org/list/dovecot/2012-August/067814.html
The lack of "error 3" ubiquity leads me to believe that I must be doing something stupid.
There's no question that this behavior is related to the antispam-plugin, because when I commented-out the relevant lines in Dovecot's configuration file, the error does not occur.
I have followed the instructions at http://wiki2.dovecot.org/Plugins/Antispam to the letter, and this is the result.
The only instance in which I diverged from the Wiki article is that I used the Ubuntu-provided dovecot-antispam package (instead of compiling the plug-in from source).
I am trying to use the "mailtrain" back-end, per the above-cited Wiki article, to call SpamAssassin's "sa-learn" utility for real-time learning.
I am not using the SpamAssassin daemon (spamd).
I'm about to throw in the towel on this plug-in, so any help would be greatly appreciated.
Thank you,
-Ben
On 19/09/12 23:41, Ben Johnson wrote:
On 9/11/2012 10:40 AM, Ben Johnson wrote:
You nailed it, Ben. Thanks!
I downloaded/installed the dovecot-dev package from the same maintainer and was able to build Antispam without issue.
The hang-up now seems to be with my configuration.
I'm seeing the following error in my email client whenever I attempt to move an email message from, say, Inbox to Junk folder:
[SERVERBUG] BUG Unknown internal error.
Nothing at all appears in syslog when this happens.
I followed the instructions at http://wiki2.dovecot.org/Plugins/Antispam , to the letter, I believe.
The relevant portions of my configuration are as follows:
protocol imap { mail_plugins = quota imap_quota antispam }
plugin { quota = dict:user::file:/var/vmail/%d/%n/.quotausage sieve = /var/vmail/%d/%n/.sieve
antispam_backend = MAILTRAIN antispam_mail_sendmail = /usr/bin/sa-learn-pipe.sh antispam_mail_spam = --spam antispam_mail_notspam = --ham antispam_spam_pattern_ignorecase = spam;junk }
The permissions and execute bit are set in accordance with other executables in the same directory:
# ls -lah /usr/bin/ | grep -i "sa-learn-pipe.sh" -rwxr-xr-x 1 root root 297 Sep 5 11:17 sa-learn-pipe.sh
I would provide more log content, but the debug options seem to be gone in the forked, >= 2.0 version of this plug-in.
Any ideas are very much appreciated.
Thanks,
-Ben
Given that nobody shared any insight here, I decided to stand-up a completely new VM with Ubuntu 12.04 LTS and see if the same error occurs.
While the error is different this time around, I am still unable to get any variant of the dovecot-antispam plug-in working.
When I attempt to use the current packages from Ubuntu 12.04.1 LTS, and I use the above configuration, my syslog fills-up with the following:
dovecot: imap-login: Login: user=user@example.com, method=PLAIN, rip=10.0.1.57, lip=10.0.1.8, mpid=2879, TLS dovecot: master: Error: service(imap): child 2879 returned error 3
Surprisingly, I can find only two other mentions of "error 3" anywhere on the Internet. Those mentions were from another user to this mailing list:
http://www.dovecot.org/list/dovecot/2012-August/067787.html http://www.dovecot.org/list/dovecot/2012-August/067814.html
The lack of "error 3" ubiquity leads me to believe that I must be doing something stupid.
There's no question that this behavior is related to the antispam-plugin, because when I commented-out the relevant lines in Dovecot's configuration file, the error does not occur.
I have followed the instructions at http://wiki2.dovecot.org/Plugins/Antispam to the letter, and this is the result.
The only instance in which I diverged from the Wiki article is that I used the Ubuntu-provided dovecot-antispam package (instead of compiling the plug-in from source).
I am trying to use the "mailtrain" back-end, per the above-cited Wiki article, to call SpamAssassin's "sa-learn" utility for real-time learning.
I am not using the SpamAssassin daemon (spamd).
I'm about to throw in the towel on this plug-in, so any help would be greatly appreciated.
Thank you,
-Ben
There are 2 versions of the dovecot-antispam package. The original version was created by Johannes Berg, then died almost a silent death, and was forked by Eugene Paskevitch. Later Johannes revived his original version somewhat.
The ubuntu package is based on the 'revived Johannes fork' [1], while the documentation on the dovecot wiki describes the 'Eugene fork'. [2]
[1] https://launchpad.net/ubuntu/precise/+source/dovecot-antispam/+copyright [2] http://wiki2.dovecot.org/Plugins/Antispam
Thus you'll need to choose which plugin you want, then follow the complete procedure for that plugin only. They both can do what you want, but building and configuration are slightly different.
It would be nice if Johannes and Eugene would either agree on a single version of the plugin, or rename one of the forks so they don't get mixed up that easily. I'm not sure if that has been discussed (on- or off-list) in the past.
FWIW: I used the 'original Johannes fork' with dovecot 1.x, and the 'Eugene fork' with 2.0.x, both with delight. Didn't get myself to upgrade to 2.1 yet, so no experience there.
-- Tom
On 9/19/2012 6:19 PM, Tom Hendrikx wrote:
On 19/09/12 23:41, Ben Johnson wrote:
On 9/11/2012 10:40 AM, Ben Johnson wrote:
You nailed it, Ben. Thanks!
I downloaded/installed the dovecot-dev package from the same maintainer and was able to build Antispam without issue.
The hang-up now seems to be with my configuration.
I'm seeing the following error in my email client whenever I attempt to move an email message from, say, Inbox to Junk folder:
[SERVERBUG] BUG Unknown internal error.
Nothing at all appears in syslog when this happens.
I followed the instructions at http://wiki2.dovecot.org/Plugins/Antispam , to the letter, I believe.
The relevant portions of my configuration are as follows:
protocol imap { mail_plugins = quota imap_quota antispam }
plugin { quota = dict:user::file:/var/vmail/%d/%n/.quotausage sieve = /var/vmail/%d/%n/.sieve
antispam_backend = MAILTRAIN antispam_mail_sendmail = /usr/bin/sa-learn-pipe.sh antispam_mail_spam = --spam antispam_mail_notspam = --ham antispam_spam_pattern_ignorecase = spam;junk }
The permissions and execute bit are set in accordance with other executables in the same directory:
# ls -lah /usr/bin/ | grep -i "sa-learn-pipe.sh" -rwxr-xr-x 1 root root 297 Sep 5 11:17 sa-learn-pipe.sh
I would provide more log content, but the debug options seem to be gone in the forked, >= 2.0 version of this plug-in.
Any ideas are very much appreciated.
Thanks,
-Ben
Given that nobody shared any insight here, I decided to stand-up a completely new VM with Ubuntu 12.04 LTS and see if the same error occurs.
While the error is different this time around, I am still unable to get any variant of the dovecot-antispam plug-in working.
When I attempt to use the current packages from Ubuntu 12.04.1 LTS, and I use the above configuration, my syslog fills-up with the following:
dovecot: imap-login: Login: user=user@example.com, method=PLAIN, rip=10.0.1.57, lip=10.0.1.8, mpid=2879, TLS dovecot: master: Error: service(imap): child 2879 returned error 3
Surprisingly, I can find only two other mentions of "error 3" anywhere on the Internet. Those mentions were from another user to this mailing list:
http://www.dovecot.org/list/dovecot/2012-August/067787.html http://www.dovecot.org/list/dovecot/2012-August/067814.html
The lack of "error 3" ubiquity leads me to believe that I must be doing something stupid.
There's no question that this behavior is related to the antispam-plugin, because when I commented-out the relevant lines in Dovecot's configuration file, the error does not occur.
I have followed the instructions at http://wiki2.dovecot.org/Plugins/Antispam to the letter, and this is the result.
The only instance in which I diverged from the Wiki article is that I used the Ubuntu-provided dovecot-antispam package (instead of compiling the plug-in from source).
I am trying to use the "mailtrain" back-end, per the above-cited Wiki article, to call SpamAssassin's "sa-learn" utility for real-time learning.
I am not using the SpamAssassin daemon (spamd).
I'm about to throw in the towel on this plug-in, so any help would be greatly appreciated.
Thank you,
-Ben
There are 2 versions of the dovecot-antispam package. The original version was created by Johannes Berg, then died almost a silent death, and was forked by Eugene Paskevitch. Later Johannes revived his original version somewhat.
The ubuntu package is based on the 'revived Johannes fork' [1], while the documentation on the dovecot wiki describes the 'Eugene fork'. [2]
[1] https://launchpad.net/ubuntu/precise/+source/dovecot-antispam/+copyright [2] http://wiki2.dovecot.org/Plugins/Antispam
Thus you'll need to choose which plugin you want, then follow the complete procedure for that plugin only. They both can do what you want, but building and configuration are slightly different.
It would be nice if Johannes and Eugene would either agree on a single version of the plugin, or rename one of the forks so they don't get mixed up that easily. I'm not sure if that has been discussed (on- or off-list) in the past.
FWIW: I used the 'original Johannes fork' with dovecot 1.x, and the 'Eugene fork' with 2.0.x, both with delight. Didn't get myself to upgrade to 2.1 yet, so no experience there.
-- Tom
Thank you for your timely response, Tom. Much appreciated.
Your remarks clear-up a lot of the confusion for me. I didn't realize that the "revived Johannes fork" made a comeback for Dovecot 2. Nor did I realize that this is the version that is packaged with Ubuntu 12.04.
Given that I'm more-or-less forced to work with Ubuntu LTS, I would prefer to use the OS's package management system wherever possible. I suppose, by extension, that I would prefer to use the "revived Johannes fork".
I've tried the Johannes versions (for both Dovecot 1 and Dovecot 2), but am unable to get either to work the way I want (mailtrain back-end with real-time calls to sa-learn, as described in the Dovecot 2/Eugene Wiki article). Shouldn't the basic tenets of each version of the plug-in apply?
Where should I start, in the way of seeking support? Basically, I intend to use the Johannes version, but I want to configure it in the manner described in the Eugene version Wiki article.
I've already described (rather exhaustively) what happens when I attempt to use the Johannes version and adapt the Eugene version instructions.
I'm curious to know what method you've used over the years with such glowing success. Which back-end did you use with Dovecot 1/Johannes? How about with Dovecot 2/Eugene?
Thanks again,
-Ben
On 9/19/2012 7:17 PM, Ben Johnson wrote:
On 9/19/2012 6:19 PM, Tom Hendrikx wrote:
On 19/09/12 23:41, Ben Johnson wrote:
On 9/11/2012 10:40 AM, Ben Johnson wrote:
You nailed it, Ben. Thanks!
I downloaded/installed the dovecot-dev package from the same maintainer and was able to build Antispam without issue.
The hang-up now seems to be with my configuration.
I'm seeing the following error in my email client whenever I attempt to move an email message from, say, Inbox to Junk folder:
[SERVERBUG] BUG Unknown internal error.
Nothing at all appears in syslog when this happens.
I followed the instructions at http://wiki2.dovecot.org/Plugins/Antispam , to the letter, I believe.
The relevant portions of my configuration are as follows:
protocol imap { mail_plugins = quota imap_quota antispam }
plugin { quota = dict:user::file:/var/vmail/%d/%n/.quotausage sieve = /var/vmail/%d/%n/.sieve
antispam_backend = MAILTRAIN antispam_mail_sendmail = /usr/bin/sa-learn-pipe.sh antispam_mail_spam = --spam antispam_mail_notspam = --ham antispam_spam_pattern_ignorecase = spam;junk }
The permissions and execute bit are set in accordance with other executables in the same directory:
# ls -lah /usr/bin/ | grep -i "sa-learn-pipe.sh" -rwxr-xr-x 1 root root 297 Sep 5 11:17 sa-learn-pipe.sh
I would provide more log content, but the debug options seem to be gone in the forked, >= 2.0 version of this plug-in.
Any ideas are very much appreciated.
Thanks,
-Ben
Given that nobody shared any insight here, I decided to stand-up a completely new VM with Ubuntu 12.04 LTS and see if the same error occurs.
While the error is different this time around, I am still unable to get any variant of the dovecot-antispam plug-in working.
When I attempt to use the current packages from Ubuntu 12.04.1 LTS, and I use the above configuration, my syslog fills-up with the following:
dovecot: imap-login: Login: user=user@example.com, method=PLAIN, rip=10.0.1.57, lip=10.0.1.8, mpid=2879, TLS dovecot: master: Error: service(imap): child 2879 returned error 3
Surprisingly, I can find only two other mentions of "error 3" anywhere on the Internet. Those mentions were from another user to this mailing list:
http://www.dovecot.org/list/dovecot/2012-August/067787.html http://www.dovecot.org/list/dovecot/2012-August/067814.html
The lack of "error 3" ubiquity leads me to believe that I must be doing something stupid.
There's no question that this behavior is related to the antispam-plugin, because when I commented-out the relevant lines in Dovecot's configuration file, the error does not occur.
I have followed the instructions at http://wiki2.dovecot.org/Plugins/Antispam to the letter, and this is the result.
The only instance in which I diverged from the Wiki article is that I used the Ubuntu-provided dovecot-antispam package (instead of compiling the plug-in from source).
I am trying to use the "mailtrain" back-end, per the above-cited Wiki article, to call SpamAssassin's "sa-learn" utility for real-time learning.
I am not using the SpamAssassin daemon (spamd).
I'm about to throw in the towel on this plug-in, so any help would be greatly appreciated.
Thank you,
-Ben
There are 2 versions of the dovecot-antispam package. The original version was created by Johannes Berg, then died almost a silent death, and was forked by Eugene Paskevitch. Later Johannes revived his original version somewhat.
The ubuntu package is based on the 'revived Johannes fork' [1], while the documentation on the dovecot wiki describes the 'Eugene fork'. [2]
[1] https://launchpad.net/ubuntu/precise/+source/dovecot-antispam/+copyright [2] http://wiki2.dovecot.org/Plugins/Antispam
Thus you'll need to choose which plugin you want, then follow the complete procedure for that plugin only. They both can do what you want, but building and configuration are slightly different.
It would be nice if Johannes and Eugene would either agree on a single version of the plugin, or rename one of the forks so they don't get mixed up that easily. I'm not sure if that has been discussed (on- or off-list) in the past.
FWIW: I used the 'original Johannes fork' with dovecot 1.x, and the 'Eugene fork' with 2.0.x, both with delight. Didn't get myself to upgrade to 2.1 yet, so no experience there.
-- Tom
Thank you for your timely response, Tom. Much appreciated.
Your remarks clear-up a lot of the confusion for me. I didn't realize that the "revived Johannes fork" made a comeback for Dovecot 2. Nor did I realize that this is the version that is packaged with Ubuntu 12.04.
Given that I'm more-or-less forced to work with Ubuntu LTS, I would prefer to use the OS's package management system wherever possible. I suppose, by extension, that I would prefer to use the "revived Johannes fork".
I've tried the Johannes versions (for both Dovecot 1 and Dovecot 2), but am unable to get either to work the way I want (mailtrain back-end with real-time calls to sa-learn, as described in the Dovecot 2/Eugene Wiki article). Shouldn't the basic tenets of each version of the plug-in apply?
Where should I start, in the way of seeking support? Basically, I intend to use the Johannes version, but I want to configure it in the manner described in the Eugene version Wiki article.
I've already described (rather exhaustively) what happens when I attempt to use the Johannes version and adapt the Eugene version instructions.
I'm curious to know what method you've used over the years with such glowing success. Which back-end did you use with Dovecot 1/Johannes? How about with Dovecot 2/Eugene?
Thanks again,
-Ben
Thanks again, Tom. After you set me straight on the various versions of the plug-in, I was able to synthesize the relevant documentation and make it all work on Ubuntu 12.04 LTS with Dovecot 2.
The key was to understand the following:
1.) The Ubuntu repositories use the Johannes versions of the plug-in (for both Dovecot 1 and 2).
Ubuntu 10.04 LTS (Dovecot 1) http://manpages.ubuntu.com/manpages/lucid/man7/dovecot-antispam.7.html
Ubuntu 12.04 LTS (Dovecot 2) http://manpages.ubuntu.com/manpages/precise/man7/dovecot-antispam.7.html
2.) The Dovecot 2 Wiki instructions for the Antispam plug-in refer to the Eugene version, and that version's configuration directives differ from the Johannes version 2 directives. So, anyone attempting to follow the Wiki article on Ubuntu will need to adapt the directives accordingly (see documentation links above for equivalent directive names).
3.) The sample "/usr/bin/sa-learn-pipe.sh" script in the Dovecot 2 Wiki instructions lacks a shebang; until a proper shebang is added to the shell script, script execution will fail with "status 1". I added the standard shebang:
#!/bin/sh
Here are the configuration directives that work for me:
plugin { antispam_debug_target = syslog antispam_backend = pipe antispam_signature = X-DSPAM-Signature antispam_signature_missing = error antispam_spam_pattern_ignorecase = SPAM;JUNK antispam_pipe_tmpdir = /tmp antispam_pipe_program = /usr/bin/sa-learn-pipe.sh antispam_pipe_program_spam_arg = --spam antispam_pipe_program_notspam_arg = --ham }
My final goal is to get the version 1 plug-in working with Dovecot 1.2.9.
I'm using the following directives, but when I move mail from Inbox to Junk, for example, I see no evidence that antispam is taking action.
plugin { antispam_signature = X-DSPAM-Signature antispam_signature_missing = error antispam_spam_pattern_ignorecase = SPAM;JUNK antispam_mail_tmpdir = /tmp antispam_mail_sendmail = /usr/bin/sa-learn-pipe.sh antispam_mail_spam = --spam antispam_mail_notspam = --ham }
Dovecot writes nothing at all to syslog. No errors, no reference to antispam. And the message is moved successfully (no failure/error notice in IMAP client). The back-end script, "/usr/bin/sa-learn-pipe.sh", attempts to create a log file in /tmp, which works with Antispam 2. But no such luck here; nothing is ever written to /tmp.
Version 2 of the plug-in includes a verbose debugging mode, but it seems that no such feature is present in version 1.
I noticed also that version 1 seems not to require a back-end to be specified explicitly (as in version 2). Is the back-end implied via the configuration directives that are present?
Respectfully,
-Ben
On Sep 20, 2012, at 8:53 AM, Ben Johnson wrote:
Thanks again, Tom. After you set me straight on the various versions of the plug-in, I was able to synthesize the relevant documentation and make it all work on Ubuntu 12.04 LTS with Dovecot 2.
The key was to understand the following:
1.) The Ubuntu repositories use the Johannes versions of the plug-in (for both Dovecot 1 and 2).
Ubuntu 10.04 LTS (Dovecot 1) http://manpages.ubuntu.com/manpages/lucid/man7/dovecot-antispam.7.html
Ubuntu 12.04 LTS (Dovecot 2) http://manpages.ubuntu.com/manpages/precise/man7/dovecot-antispam.7.html
2.) The Dovecot 2 Wiki instructions for the Antispam plug-in refer to the Eugene version, and that version's configuration directives differ from the Johannes version 2 directives.
Is there a Johannes version 2?
I understood that Johannes dropped support at dovecot 2.x and that is why Eugene's version exists.
So, anyone attempting to follow the Wiki article on Ubuntu will need to adapt the directives accordingly (see documentation links above for equivalent directive names).
3.) The sample "/usr/bin/sa-learn-pipe.sh" script in the Dovecot 2 Wiki instructions lacks a shebang; until a proper shebang is added to the shell script, script execution will fail with "status 1". I added the standard shebang:
#!/bin/sh
Here are the configuration directives that work for me:
plugin { antispam_debug_target = syslog antispam_backend = pipe antispam_signature = X-DSPAM-Signature antispam_signature_missing = error antispam_spam_pattern_ignorecase = SPAM;JUNK antispam_pipe_tmpdir = /tmp antispam_pipe_program = /usr/bin/sa-learn-pipe.sh antispam_pipe_program_spam_arg = --spam antispam_pipe_program_notspam_arg = --ham }
My final goal is to get the version 1 plug-in working with Dovecot 1.2.9.
If I were implementing Johannes antispam plugin with Dovecot 1.2.9 I would follow Johannes documentation. http://johannes.sipsolutions.net/files/antispam.html
Regards, Bradley Giesbrecht (pixilla)
On 9/20/2012 12:12 PM, Bradley Giesbrecht wrote:
On Sep 20, 2012, at 8:53 AM, Ben Johnson wrote:
Thanks again, Tom. After you set me straight on the various versions of the plug-in, I was able to synthesize the relevant documentation and make it all work on Ubuntu 12.04 LTS with Dovecot 2.
The key was to understand the following:
1.) The Ubuntu repositories use the Johannes versions of the plug-in (for both Dovecot 1 and 2).
Ubuntu 10.04 LTS (Dovecot 1) http://manpages.ubuntu.com/manpages/lucid/man7/dovecot-antispam.7.html
Ubuntu 12.04 LTS (Dovecot 2) http://manpages.ubuntu.com/manpages/precise/man7/dovecot-antispam.7.html
2.) The Dovecot 2 Wiki instructions for the Antispam plug-in refer to the Eugene version, and that version's configuration directives differ from the Johannes version 2 directives.
Is there a Johannes version 2?
Thanks for your reply, Bradley.
That seems to be the implication here: http://manpages.ubuntu.com/manpages/precise/man7/dovecot-antispam.7.html
Johannes is listed as an author, the package name is dovecot-antispam_2.0+20120225, and the manpage is nearly identical to that of version 1 at http://manpages.ubuntu.com/manpages/lucid/man7/dovecot-antispam.7.html .
I understood that Johannes dropped support at dovecot 2.x and that is why Eugene's version exists.
So, anyone attempting to follow the Wiki article on Ubuntu will need to adapt the directives accordingly (see documentation links above for equivalent directive names).
3.) The sample "/usr/bin/sa-learn-pipe.sh" script in the Dovecot 2 Wiki instructions lacks a shebang; until a proper shebang is added to the shell script, script execution will fail with "status 1". I added the standard shebang:
#!/bin/sh
Here are the configuration directives that work for me:
plugin { antispam_debug_target = syslog antispam_backend = pipe antispam_signature = X-DSPAM-Signature antispam_signature_missing = error antispam_spam_pattern_ignorecase = SPAM;JUNK antispam_pipe_tmpdir = /tmp antispam_pipe_program = /usr/bin/sa-learn-pipe.sh antispam_pipe_program_spam_arg = --spam antispam_pipe_program_notspam_arg = --ham }
My final goal is to get the version 1 plug-in working with Dovecot 1.2.9.
If I were implementing Johannes antispam plugin with Dovecot 1.2.9 I would follow Johannes documentation. http://johannes.sipsolutions.net/files/antispam.html
This documentation seems to apply to the Dovecot 2 version; the antispam_backend directive doesn't seem to exist in version 1, at least according to the official Ubuntu package documentation (cited above). In fact, this appears to be the exact document that is used for the Ubuntu package for Dovecot 2 (also cited above).
Regards, Bradley Giesbrecht (pixilla)
On 9/20/2012 11:53 AM, Ben Johnson wrote:
Thanks again, Tom. After you set me straight on the various versions of the plug-in, I was able to synthesize the relevant documentation and make it all work on Ubuntu 12.04 LTS with Dovecot 2.
The key was to understand the following:
1.) The Ubuntu repositories use the Johannes versions of the plug-in (for both Dovecot 1 and 2).
Ubuntu 10.04 LTS (Dovecot 1) http://manpages.ubuntu.com/manpages/lucid/man7/dovecot-antispam.7.html
Ubuntu 12.04 LTS (Dovecot 2) http://manpages.ubuntu.com/manpages/precise/man7/dovecot-antispam.7.html
2.) The Dovecot 2 Wiki instructions for the Antispam plug-in refer to the Eugene version, and that version's configuration directives differ from the Johannes version 2 directives. So, anyone attempting to follow the Wiki article on Ubuntu will need to adapt the directives accordingly (see documentation links above for equivalent directive names).
3.) The sample "/usr/bin/sa-learn-pipe.sh" script in the Dovecot 2 Wiki instructions lacks a shebang; until a proper shebang is added to the shell script, script execution will fail with "status 1". I added the standard shebang:
#!/bin/sh
Here are the configuration directives that work for me:
plugin { antispam_debug_target = syslog antispam_backend = pipe antispam_signature = X-DSPAM-Signature antispam_signature_missing = error antispam_spam_pattern_ignorecase = SPAM;JUNK antispam_pipe_tmpdir = /tmp antispam_pipe_program = /usr/bin/sa-learn-pipe.sh antispam_pipe_program_spam_arg = --spam antispam_pipe_program_notspam_arg = --ham }
My final goal is to get the version 1 plug-in working with Dovecot 1.2.9.
I'm using the following directives, but when I move mail from Inbox to Junk, for example, I see no evidence that antispam is taking action.
plugin { antispam_signature = X-DSPAM-Signature antispam_signature_missing = error antispam_spam_pattern_ignorecase = SPAM;JUNK antispam_mail_tmpdir = /tmp antispam_mail_sendmail = /usr/bin/sa-learn-pipe.sh antispam_mail_spam = --spam antispam_mail_notspam = --ham }
Dovecot writes nothing at all to syslog. No errors, no reference to antispam. And the message is moved successfully (no failure/error notice in IMAP client). The back-end script, "/usr/bin/sa-learn-pipe.sh", attempts to create a log file in /tmp, which works with Antispam 2. But no such luck here; nothing is ever written to /tmp.
Version 2 of the plug-in includes a verbose debugging mode, but it seems that no such feature is present in version 1.
I noticed also that version 1 seems not to require a back-end to be specified explicitly (as in version 2). Is the back-end implied via the configuration directives that are present?
Respectfully,
-Ben
Just a follow-up to report that I finally have the Antispam plug-in (the Johannes version) -- with the mailtrain/pipe back-end -- working as expected in both of the following environments:
- Ubuntu 10.04 with Dovecot 1.2.9
- Ubuntu 12.04 with Dovecot 2.0.19
In addition to the points that I mentioned in my previous note, a crucial step in the debugging process was to modify the "sa-learn-pipe.sh" script to produce more verbose output. Unlike Antispam 2, Antispam 1 lacks a verbose debugging mode, so when the plug-in does not work as expected, tracking-down the root cause can be much more difficult.
Here's the shell script from http://wiki2.dovecot.org/Plugins/Antispam , with my modifications:
After making these modifications to the shell script, I was able to determine the cause of the failure in Dovecot 1.2.9. In fact, I was incorrect earlier when I stated that version 2 was working; the same problem applied there, too.
The "vmail" user, as whom the script is executed, lacks the permissions required to change the Bayes token database (this is a "good thing").
An excerpt from the custom log file:
29722-start (--spam) Calling (as user vmail) '/usr/bin/sa-learn --spam /tmp/sendmail-msg-29722.txt' bayes: expire_old_tokens: locker: safe_lock: cannot create tmp lockfile /var/lib/amavis/.spamassassin/bayes.lock.example.com.29725 for /var/lib/amavis/.spamassassin/bayes.lock: Permission denied plugin: eval failed: bayes: (in learn) locker: safe_lock: cannot create tmp lockfile /var/lib/amavis/.spamassassin/bayes.lock.example.com.29725 for /var/lib/amavis/.spamassassin/bayes.lock: Permission denied ERROR: the Bayes learn function returned an error, please re-run with -D for more information at /usr/bin/sa-learn line 493. Learned tokens from 0 message(s) (1 message(s) examined) 29722-end
The only means by which I was able to work around this issue -- without changing filesystem permissions, users, groups, or otherwise compromising system security -- was to migrate from the standard flat-file Bayes database format to MySQL. (The guide at http://www.starbridge.org/spip/spip.php?article15 had everything I needed, and it only took me a few minutes to implement.)
After making that change, everything began working as expected; here's a sample custom log entry:
30565-start (--ham) Calling (as user vmail) '/usr/bin/sa-learn --ham /tmp/sendmail-msg-30565.txt' Learned tokens from 1 message(s) (1 message(s) examined) 30565-end
In any event, this concludes the saga, and I must say, I'm thrilled to have it all working.
Thanks again to everyone who contributed valuable input and pointed me in the right direction.
And thanks to Johannes and Eugene (and everyone else who contributed to Antispam) for solving a major hurdle in Bayes DB training.
-Ben
On 9/19/2012 5:41 PM, Ben Johnson wrote:
On 9/11/2012 10:40 AM, Ben Johnson wrote:
You nailed it, Ben. Thanks!
I downloaded/installed the dovecot-dev package from the same maintainer and was able to build Antispam without issue.
The hang-up now seems to be with my configuration.
I'm seeing the following error in my email client whenever I attempt to move an email message from, say, Inbox to Junk folder:
[SERVERBUG] BUG Unknown internal error.
Nothing at all appears in syslog when this happens.
I followed the instructions at http://wiki2.dovecot.org/Plugins/Antispam , to the letter, I believe.
The relevant portions of my configuration are as follows:
protocol imap { mail_plugins = quota imap_quota antispam }
plugin { quota = dict:user::file:/var/vmail/%d/%n/.quotausage sieve = /var/vmail/%d/%n/.sieve
antispam_backend = MAILTRAIN antispam_mail_sendmail = /usr/bin/sa-learn-pipe.sh antispam_mail_spam = --spam antispam_mail_notspam = --ham antispam_spam_pattern_ignorecase = spam;junk }
The permissions and execute bit are set in accordance with other executables in the same directory:
# ls -lah /usr/bin/ | grep -i "sa-learn-pipe.sh" -rwxr-xr-x 1 root root 297 Sep 5 11:17 sa-learn-pipe.sh
I would provide more log content, but the debug options seem to be gone in the forked, >= 2.0 version of this plug-in.
Any ideas are very much appreciated.
Thanks,
-Ben Given that nobody shared any insight here, I decided to stand-up a completely new VM with Ubuntu 12.04 LTS and see if the same error occurs.
While the error is different this time around, I am still unable to get any variant of the dovecot-antispam plug-in working.
When I attempt to use the current packages from Ubuntu 12.04.1 LTS, and I use the above configuration, my syslog fills-up with the following:
dovecot: imap-login: Login: user=user@example.com, method=PLAIN, rip=10.0.1.57, lip=10.0.1.8, mpid=2879, TLS dovecot: master: Error: service(imap): child 2879 returned error 3
Surprisingly, I can find only two other mentions of "error 3" anywhere on the Internet. Those mentions were from another user to this mailing list:
http://www.dovecot.org/list/dovecot/2012-August/067787.html http://www.dovecot.org/list/dovecot/2012-August/067814.html
The lack of "error 3" ubiquity leads me to believe that I must be doing something stupid.
There's no question that this behavior is related to the antispam-plugin, because when I commented-out the relevant lines in Dovecot's configuration file, the error does not occur.
I have followed the instructions at http://wiki2.dovecot.org/Plugins/Antispam to the letter, and this is the result.
The only instance in which I diverged from the Wiki article is that I used the Ubuntu-provided dovecot-antispam package (instead of compiling the plug-in from source).
I am trying to use the "mailtrain" back-end, per the above-cited Wiki article, to call SpamAssassin's "sa-learn" utility for real-time learning.
I am not using the SpamAssassin daemon (spamd).
I'm about to throw in the towel on this plug-in, so any help would be greatly appreciated.
Thank you,
-Ben
I've never used this plugin but let offer up my home brewed script that I call from crontab: # Process and delete .Spam and .Ham Maildirs 11 */4 * * * /usr/local/sbin/maildir.sa-learn
and /usr/local/sbin/maildir.sa-learn: #!/bin/bash
MAILDIR=Maildir
LS=/bin/ls RM=/bin/rm SU=/bin/su FIND=/bin/find SALEARN=/usr/bin/sa-learn
for i in $LS /home; do
prefs="/home/$i/.spamassassin/user_prefs"
# [ ! ."$i" == ".bill" ] && continue echo $i spamdir="/home/$i/$MAILDIR/.SystemFolders.Spam" [ ! -d "$spamdir" ] && continue
$SU -c "$SALEARN --prefspath=$prefs --spam $spamdir/new" $i
$FIND "$spamdir/new" -type f -delete
$SU -c "$SALEARN --prefspath=$prefs --spam $spamdir/cur" $i
$FIND "$spamdir/cur" -type f -delete
hamdir="/home/$i/$MAILDIR/.SystemFolders.Ham"
[ ! -d "$hamdir" ] && continue
$SU -c "$SALEARN --prefspath=$prefs --ham $hamdir/new" $i
$FIND "$hamdir/new" -type f -delete
$SU -c "$SALEARN --prefspath=$prefs --ham $hamdir/cur" $i
$FIND "$hamdir/cur" -type f -deletedone
With my setup each user has their own spamassassin user_prefs and bayes tokens.
Bill
participants (7)
-
Ben Johnson
-
Ben Morrow
-
Bill Shirley
-
Bradley Giesbrecht
-
Charles Marcus
-
Ed W
-
Tom Hendrikx