[Dovecot] Antispam plugin custom behavior?
Hello,
I currently have a setup on my system with what I call "magic folders" to enable spam filter training. Here's how it works:
1. If you have a false-negative, put the spam into the Spam.Report
folder
2. If you have a false-positive (which has all kinds of ugly
spamassassin protective markup in it), put the message into the
Spam.NotSpam folderCurrently what happens is that a cron job comes along every five
minutes and processes the messages in those folders. In the case of
the NotSpam folder, it strips the message of the spamassassin markup,
retrains the bayesian net, and redelivers the message (e.g. via
deliver). In the case of the Report folder, the message is used to
train the bayesian net (among other things) and then deleted.
I'd love to be able to trigger these actions when the mail is moved, rather than have a cron job inspecting the mailboxes.
I looked into the antispam plugin (http://johannes.sipsolutions.net/Projects/dovecot-antispam), which seems nice but doesn't appear sufficiently generic for my needs. What would really work is if I could get it set up such that putting a message into either of those directories is turned into piping the message to a script of my choosing (a different one for each folder).
Does anyone know a good way of getting my own custom behavior in here, or is my cronjob setup probably the best way?
~Kyle
The optimist thinks this is the best of all possible worlds. The pessimist fears it is true. -- J. Robert Oppenheimer
Hi
The anispam plugin does exactly what you need, and you could forget the cron script. If you use SpamAssassin, you could add a rule to Sieve to move the Spam messages when they arrives to the Spam folder. If a user moves a message from Spam folder to any other folder, then the message is considered a false possitive (when this move is detected you could run sa-learn inmediatly, without the cron script); the other way, when a user moves a message INTO the spam folder you could run again the sa-learn script, but with different parameters.
I am testing antispam plugin with dovecot 1.1rcX, but it looks there is a conflict with quota plugin. Anyway you can view the documentation i have done, but it is in spanish.
http://wiki.nutum.es/linux/samba/samba_ldap_mds/instalacion_y_configuracion_...
Regards.
2008/6/11 Kyle Wheeler <kyle-dovecot@memoryhole.net>:
Hello,
I currently have a setup on my system with what I call "magic folders" to enable spam filter training. Here's how it works:
- If you have a false-negative, put the spam into the Spam.Report folder
- If you have a false-positive (which has all kinds of ugly spamassassin protective markup in it), put the message into the Spam.NotSpam folder
Currently what happens is that a cron job comes along every five minutes and processes the messages in those folders. In the case of the NotSpam folder, it strips the message of the spamassassin markup, retrains the bayesian net, and redelivers the message (e.g. via deliver). In the case of the Report folder, the message is used to train the bayesian net (among other things) and then deleted.
I'd love to be able to trigger these actions when the mail is moved, rather than have a cron job inspecting the mailboxes.
I looked into the antispam plugin (http://johannes.sipsolutions.net/Projects/dovecot-antispam), which seems nice but doesn't appear sufficiently generic for my needs. What would really work is if I could get it set up such that putting a message into either of those directories is turned into piping the message to a script of my choosing (a different one for each folder).
Does anyone know a good way of getting my own custom behavior in here, or is my cronjob setup probably the best way?
~Kyle
The optimist thinks this is the best of all possible worlds. The pessimist fears it is true. -- J. Robert Oppenheimer
On Wednesday, June 11 at 05:51 PM, quoth Juan Asensio Sánchez:
The anispam plugin does exactly what you need, and you could forget the cron script. If you use SpamAssassin, you could add a rule to Sieve to move the Spam messages when they arrives to the Spam folder. If a user moves a message from Spam folder to any other folder, then the message is considered a false possitive (when this move is detected you could run sa-learn inmediatly, without the cron script); the other way, when a user moves a message INTO the spam folder you could run again the sa-learn script, but with different parameters.
Well, for one thing, this is different behavior than what my users are used to, and I'd rather not have to re-explain how things work and deal with confusion about the difference in behavior.
Plus, unless I misunderstand the antispam plugin (quite possible), it doesn't *alter* the message when you remove it from the Spam folder --- because if it did, that could confuse IMAP clients that expect messages not to change when moved.
~Kyle
The borrower is the slave of the lender. -- Proverbs 22:7
Kyle Wheeler wrote:
On Wednesday, June 11 at 05:51 PM, quoth Juan Asensio Sánchez:
The anispam plugin does exactly what you need, and you could forget the cron script. If you use SpamAssassin, you could add a rule to Sieve to move the Spam messages when they arrives to the Spam folder. If a user moves a message from Spam folder to any other folder, then the message is considered a false possitive (when this move is detected you could run sa-learn inmediatly, without the cron script); the other way, when a user moves a message INTO the spam folder you could run again the sa-learn script, but with different parameters.
Well, for one thing, this is different behavior than what my users are used to, and I'd rather not have to re-explain how things work and deal with confusion about the difference in behavior.
Plus, unless I misunderstand the antispam plugin (quite possible), it doesn't *alter* the message when you remove it from the Spam folder --- because if it did, that could confuse IMAP clients that expect messages not to change when moved.
~Kyle
No different behaviour for the end user. Your user could continue to drag&drop messages in/out of the designated Spam folder. The only different, which is not visible to the end user, is that the retrain of false positives is activated by pulling the messages out of the Spam folder, rather than having to specifically put it in a Ham folder. That said, they can continue to use the Ham folder as a placebo. ;)
Out of curiosity, why would you need to alter the message when moving it around?
Hugo Monteiro.
-- ci.fct.unl.pt:~# cat .signature
Hugo Monteiro Email : hugo.monteiro@fct.unl.pt Telefone : +351 212948300 Ext.15307
Centro de Informática Faculdade de Ciências e Tecnologia da Universidade Nova de Lisboa Quinta da Torre 2829-516 Caparica Portugal Telefone: +351 212948596 Fax: +351 212948548 www.ci.fct.unl.pt apoio@fct.unl.pt
ci.fct.unl.pt:~# _
On Wednesday, June 11 at 05:33 PM, quoth Hugo Monteiro:
Well, for one thing, this is different behavior than what my users are used to, and I'd rather not have to re-explain how things work and deal with confusion about the difference in behavior.
Plus, unless I misunderstand the antispam plugin (quite possible), it doesn't *alter* the message when you remove it from the Spam folder --- because if it did, that could confuse IMAP clients that expect messages not to change when moved.
~Kyle
No different behaviour for the end user. Your user could continue to drag&drop messages in/out of the designated Spam folder.
You misunderstood my original email. Users have two folders: one for reporting spam that wasn't identified as spam (the "Spam.Report" folder), one for rescuing ham that was misidentified as spam (the "Spam.NotSpam" folder).
Thus, there is no "implicit" behavior - putting a message in one of those folders explicitly tells the system "learn this message" or "unlearn this message".
The only different, which is not visible to the end user, is that the retrain of false positives is activated by pulling the messages out of the Spam folder, rather than having to specifically put it in a Ham folder.
How is that not visible to the end-user?
That said, they can continue to use the Ham folder as a placebo. ;)
Heh. I see. But the user expects messages in the Ham folder to disappear and be automatically redelivered to their inbox.
Out of curiosity, why would you need to alter the message when moving it around?
Because my system uses SpamAssassin's "report_safe" feature (see http://spamassassin.apache.org/full/3.1.x/doc/Mail_SpamAssassin_Conf.html#it...). When a message is identified as spam, instead of modifying the original message, the user receives a report message with the original message attached as a text/plain attachment. This makes reading the original rather difficult (on purpose). Thus, one of the benefits of moving the message to the NotSpam folder is that the message is restored to its original form and redelivered.
~Kyle
Unthinking respect for authority is the greatest enemy of truth. -- Albert Einstein
Kyle Wheeler wrote:
Hello,
I currently have a setup on my system with what I call "magic folders" to enable spam filter training. Here's how it works:
1. If you have a false-negative, put the spam into the Spam.Report folder 2. If you have a false-positive (which has all kinds of ugly spamassassin protective markup in it), put the message into the Spam.NotSpam folderCurrently what happens is that a cron job comes along every five minutes and processes the messages in those folders. In the case of the NotSpam folder, it strips the message of the spamassassin markup,
retrains the bayesian net, and redelivers the message (e.g. via deliver). In the case of the Report folder, the message is used to train the bayesian net (among other things) and then deleted.I'd love to be able to trigger these actions when the mail is moved, rather than have a cron job inspecting the mailboxes.
I looked into the antispam plugin (http://johannes.sipsolutions.net/Projects/dovecot-antispam), which seems nice but doesn't appear sufficiently generic for my needs. What would really work is if I could get it set up such that putting a message into either of those directories is turned into piping the message to a script of my choosing (a different one for each folder).
Does anyone know a good way of getting my own custom behavior in here, or is my cronjob setup probably the best way?
~Kyle
Hello Kyle,
Have you tried the plugin using the mailtrain backend? Basically it will forward the message, as attachment, to spam/notspam addresses that you define. That includes the use of a %u variable expansion, if you choose to use retrain addresses like like spam-username@domain.com or something. I've been pretty happy with with it and it scales a lot better than piping the message into a retrain command, since the mail system itself will handle the load in a more intelligent way.
Regards,
Hugo Monteiro.
-- ci.fct.unl.pt:~# cat .signature
Hugo Monteiro Email : hugo.monteiro@fct.unl.pt Telefone : +351 212948300 Ext.15307
Centro de Informática Faculdade de Ciências e Tecnologia da Universidade Nova de Lisboa Quinta da Torre 2829-516 Caparica Portugal Telefone: +351 212948596 Fax: +351 212948548 www.ci.fct.unl.pt apoio@fct.unl.pt
ci.fct.unl.pt:~# _
On Wednesday, June 11 at 05:01 PM, quoth Hugo Monteiro:
Have you tried the plugin using the mailtrain backend?
The antispam plugin? No, I haven't... mostly because it looks like no matter which backend I use, I'd have to alter the user-visible interface to my training system (which I don't really want to do), and it still doesn't handle the "altered message" problem.
Basically it will forward the message, as attachment, to spam/notspam addresses that you define. That includes the use of a %u variable expansion, if you choose to use retrain addresses like like spam-username@domain.com or something. I've been pretty happy with with it and it scales a lot better than piping the message into a retrain command, since the mail system itself will handle the load in a more intelligent way.
Hmmm, load is something I hadn't thought about... (the system I'm working with at the moment has plenty of capacity to spare). That's a good point.
However, one of the goals here is to make it so that if a user identifies a message that has been mistakenly tagged as spam (and "sanitized" by SpamAssassin, e.g. via the "report_safe" setting), they can get the message corrected (and back to its original form) immediately. As it is, they have to put it into the NotSpam folder and wait a couple minutes for the message to reappear in the INBOX (because the cron job only runs every so often).
~Kyle
No, I don't know that Atheists should be considered as citizens, nor should they be considered patriots. This is one nation under God. -- George H. W. Bush, August 27, 1987
Kyle Wheeler wrote:
On Wednesday, June 11 at 05:01 PM, quoth Hugo Monteiro:
Have you tried the plugin using the mailtrain backend?
The antispam plugin? No, I haven't... mostly because it looks like no matter which backend I use, I'd have to alter the user-visible interface to my training system (which I don't really want to do), and it still doesn't handle the "altered message" problem.
Basically it will forward the message, as attachment, to spam/notspam addresses that you define. That includes the use of a %u variable expansion, if you choose to use retrain addresses like like spam-username@domain.com or something. I've been pretty happy with with it and it scales a lot better than piping the message into a retrain command, since the mail system itself will handle the load in a more intelligent way.
Hmmm, load is something I hadn't thought about... (the system I'm working with at the moment has plenty of capacity to spare). That's a good point.
However, one of the goals here is to make it so that if a user identifies a message that has been mistakenly tagged as spam (and "sanitized" by SpamAssassin, e.g. via the "report_safe" setting), they can get the message corrected (and back to its original form) immediately. As it is, they have to put it into the NotSpam folder and wait a couple minutes for the message to reappear in the INBOX (because the cron job only runs every so often).
~Kyle
No need to reply to the other message regarding the "why". ;)
Has for what you're saying .. maybe it could be possible to extend the plugin to remove/add headers.. Though i have no idea on the impact that would have on IMAP clients...
Regards,
Hugo Monteiro.
-- ci.fct.unl.pt:~# cat .signature
Hugo Monteiro Email : hugo.monteiro@fct.unl.pt Telefone : +351 212948300 Ext.15307
Centro de Informática Faculdade de Ciências e Tecnologia da Universidade Nova de Lisboa Quinta da Torre 2829-516 Caparica Portugal Telefone: +351 212948596 Fax: +351 212948548 www.ci.fct.unl.pt apoio@fct.unl.pt
ci.fct.unl.pt:~# _
Kyle Wheeler wrote:
On Wednesday, June 11 at 05:01 PM, quoth Hugo Monteiro:
Have you tried the plugin using the mailtrain backend?
The antispam plugin? No, I haven't... mostly because it looks like no matter which backend I use, I'd have to alter the user-visible interface to my training system (which I don't really want to do), and it still doesn't handle the "altered message" problem.
Basically it will forward the message, as attachment, to spam/notspam addresses that you define. That includes the use of a %u variable expansion, if you choose to use retrain addresses like like spam-username@domain.com or something. I've been pretty happy with with it and it scales a lot better than piping the message into a retrain command, since the mail system itself will handle the load in a more intelligent way.
Hmmm, load is something I hadn't thought about... (the system I'm working with at the moment has plenty of capacity to spare). That's a good point. Just thinking out loud here...
Couple possibilities:
- Have an SMTP server dedicated to spam re-processing. This dedicated function SMTP server could use your existing scripts to strip the unwanted headers and then pass it back to your standard server. This would be trivial with Postfix, either using domain mapping or dedicated IP/port listeners.
- If you're not using an SMTP server with Postfix's flexibility, use something like smtpprox to add it.
Bottom line is use a tool that works within the IMAP server (the anti-spam plugin) to detect mail changes and then pass the message on to a separate process for treatment.
Daniel
On Wed, 2008-06-11 at 10:25 -0500, Kyle Wheeler wrote:
Hello,
I currently have a setup on my system with what I call "magic folders" to enable spam filter training. Here's how it works:
1. If you have a false-negative, put the spam into the Spam.Report folder 2. If you have a false-positive (which has all kinds of ugly spamassassin protective markup in it), put the message into the Spam.NotSpam folderCurrently what happens is that a cron job comes along every five minutes and processes the messages in those folders. In the case of the NotSpam folder, it strips the message of the spamassassin markup,
retrains the bayesian net, and redelivers the message (e.g. via deliver). In the case of the Report folder, the message is used to train the bayesian net (among other things) and then deleted.I'd love to be able to trigger these actions when the mail is moved, rather than have a cron job inspecting the mailboxes.
I looked into the antispam plugin (http://johannes.sipsolutions.net/Projects/dovecot-antispam), which seems nice but doesn't appear sufficiently generic for my needs. What would really work is if I could get it set up such that putting a message into either of those directories is turned into piping the message to a script of my choosing (a different one for each folder).
Does anyone know a good way of getting my own custom behavior in here, or is my cronjob setup probably the best way?
~Kyle
Hello Kyle,
As a temporary solution, and if your linux box as iNotify support, I suggest you use incron. incron is an "inotify cron" system. It works like the regular cron but is driven by filesystem events instead of time events.
Andre Rodier r e d 2 - Service Driven Development 34-35 Eastcastle Street, London W1W 8DW www.red2.co.uk | andre.rodier@red2.co.uk
(+44) 0870 471 8492 direct (+44) 0751 124 4961 mobile
On Wednesday, June 11 at 06:34 PM, quoth Andre Rodier:
As a temporary solution, and if your linux box as iNotify support, I suggest you use incron. incron is an "inotify cron" system. It works like the regular cron but is driven by filesystem events instead of time events.
Interesting idea... thanks!
~Kyle
It's amazing how much "mature wisdom" resembles being too tired. -- Robert A. Heinlein
Kyle,
1. If you have a false-negative, put the spam into the Spam.Report folder 2. If you have a false-positive (which has all kinds of ugly spamassassin protective markup in it), put the message into the Spam.NotSpam folder
I looked into the antispam plugin (http://johannes.sipsolutions.net/Projects/dovecot-antispam), which seems nice but doesn't appear sufficiently generic for my needs.
No, the plugin doesn't support that. You _could_ do that by loading the plugin twice (the plugin supports compiling as a different plugin name) and configuring the first as such:
- declare "Spam.NotSpam" as "spam" folder and make the spam->notspam move a no-op, the notspam->spam the reporting and the second
- declare "spam.report" as "spam" folder and make the spam->notspam move a no-op, the notspam->spam do the reporting
But that's quite complicated.
johannes
On Wed, 2008-06-11 at 23:43 +0200, Johannes Berg wrote:
Kyle,
"Obviously", I qualify as a spammer because I wrote the antispam plugin. Or something like that.
A message that you sent could not be delivered to one or more of its recipients. This is a permanent error. The following address(es) failed:
kyle-dovecot@memoryhole.net SMTP error from remote mail server after RCPT TO:<kyle-dovecot@memoryhole.net>: host smtp.memoryhole.net [64.253.106.173]: 553 You are a spammer. Suck eggs. (#5.7.1)
johannes
On Wednesday, June 11 at 11:51 PM, quoth Johannes Berg:
On Wed, 2008-06-11 at 23:43 +0200, Johannes Berg wrote:
Kyle,
"Obviously", I qualify as a spammer because I wrote the antispam plugin. Or something like that.
Heh, sorry about that - I have my server set to reject non-list-related messages to my kyle-dovecot address... which was a good idea until I started posting to the list again.
~Kyle
When you pray, do not be like the hypocrites, who love to stand and pray in the synagogues and on street corners so that others may see them. Amen, I say to you, they have received their reward. -- Matthew 6:5
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Wed, 11 Jun 2008, Kyle Wheeler wrote:
those directories is turned into piping the message to a script of my choosing (a different one for each folder).
There is a pipe plugin, which README exactly fits this sentence, search the archives of this list.
Bye,
Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux)
iD8DBQFIUOeBVJMDrex4hCIRAuGRAKCoeDuCI2c595H0ubW8BKTT4NS0EwCePkK3 fDuFvdTMtQEo7S4/BERSEn8= =YfAK -----END PGP SIGNATURE-----
participants (7)
-
Andre Rodier
-
Daniel L. Miller
-
Hugo Monteiro
-
Johannes Berg
-
Juan Asensio Sánchez
-
Kyle Wheeler
-
Steffen Kaiser