[Dovecot] Struggling with antispam
I have a dovecot installation I am building on a raspberry pi for a mail server for the home.
I am almost there, everything bar one thing is working.
In simple terms dovecot is the authentication agent and the lda (using lmtp) for postfix. My maildir mailboxed at at /var/mail - location and accounts controlled by a small sqlite database. the uid for these mailboxes are 10001 and up and are not mapped to any account on the system. The sieve plugin is working and I have a default script which works to archive outgoing mail, returned via postfix sender_bcc_maps to user+saveout into a Sent subfolder organised by date.
postfix also uses milters, to do spf, and dkim checking - and most importantly from this posts perspective passes all mail through dspam, adding an X-DSPAM-Signature header as it passes through. I plan to use another default sieve script to move mail based on the X-DPAM-Result header to move messages to the users Junk folder (when I can get antispam working).
I want to use the antispam plugin to pick up users moving mail in and out of their Junk folder to train dspam.
I initially tried using antispam's dspam backend. However I could not get it to work. I "think" the issue is that antispam is running with uid 10001 when it does this, and dspam will not process the message because it can't find this uid as a system user. But I am not sure.
The approach I am trying now, is use the pipe backend to send mail to spam@chandlerfamily.org.uk or ham@chandlerfamily.org.uk - and via postfix manipulations get that to call dspam as user dspam. If I run as root and use sendmail -bv to check that it works I get a delivery report that shows that it would indeed arrive at dspam
I have a file in /var/spool/dspam called group which has
dpsam:shared:*
In it so that all users share the same dictionary.
However, when I try to make this work with anitspam, I keep getting failure, and I can't work out what is causing it. Maybe its again because I am running with uid 10001 when calling sendmail but I don't know.
antispam config is
plugin { antispam_debug_target = syslog antispam_verbose_debug = 1 antispam_backend = pipe antispam_signature = X-DSPAM-Signature antispam_signature_missing = error antispam_trash = Trash antispam_spam = Junk antispam_pipe_program=/usr/bin/sendmail antispam_pipe_program_args=-f;dspam@chandlerfamily.org.uk antispam_pipe_program_spam_args=spam@chandlerfamily.org.uk antispam_pipe_program_notspam_args=ham@chandlerfamily.org.uk }
And what appears in syslog as a result of moving a mail from my INBOX to the Junk folder in thunderbird is ...
Mar 6 06:52:38 piserver imap: antispam: plugin initialising (2.0-notgit) Mar 6 06:52:38 piserver imap: antispam: verbose debug enabled Mar 6 06:52:38 piserver imap: antispam: "Junk" is exact match spam folder Mar 6 06:52:38 piserver imap: antispam: no unsure folders Mar 6 06:52:38 piserver imap: antispam: "Trash" is exact match trash folder Mar 6 06:52:38 piserver imap: antispam: pipe backend spam arg[0] = spam@chandlerfamily.org.uk Mar 6 06:52:38 piserver imap: antispam: pipe backend ham arg[0] = ham@chandlerfamily.org.uk Mar 6 06:52:38 piserver imap: antispam: pipe backend program = /usr/bin/sendmail Mar 6 06:52:38 piserver imap: antispam: pipe backend program arg[0] = -f Mar 6 06:52:38 piserver imap: antispam: pipe backend program arg[1] = dspam@chandlerfamily.org.uk Mar 6 06:52:38 piserver imap: antispam: pipe backend tmpdir /tmp Mar 6 06:54:11 piserver imap: antispam: mailbox_is_unsure(Junk): 0 Mar 6 06:54:11 piserver imap: antispam: mailbox_is_trash(INBOX): 0 Mar 6 06:54:11 piserver imap: antispam: mailbox_is_trash(Junk): 0 Mar 6 06:54:11 piserver imap: antispam: mail copy: from trash: 0, to trash: 0 Mar 6 06:54:11 piserver imap: antispam: mailbox_is_spam(INBOX): 0 Mar 6 06:54:11 piserver imap: antispam: mailbox_is_spam(Junk): 1 Mar 6 06:54:11 piserver imap: antispam: mailbox_is_unsure(INBOX): 0 Mar 6 06:54:11 piserver imap: antispam: mail copy: src spam: 0, dst spam: 1, src unsure: 0 Mar 6 06:54:11 piserver imap: antispam: running mailtrain backend program /usr/bin/sendmail Mar 6 06:54:11 piserver imap: antispam: running mailtrain backend program /usr/bin/sendmail Mar 6 06:54:11 piserver imap: antispam: running mailtrain backend program parameter 1 -f Mar 6 06:54:11 piserver imap: antispam: running mailtrain backend program parameter 2 dspam@chandlerfamily.org.uk Mar 6 06:54:11 piserver imap: antispam: running mailtrain backend program parameter 3 spam@chandlerfamily.org.uk Mar 6 06:54:11 piserver imap: antispam: run program failed with exit code 1
At the same time there is nothing is /var/log/mail.log indicating postfix has ever received any sendmail request.
I would prefer to use dpsam backend directly if someone can tell me how to overcome the dspam rejection issues, but this alternative would also work (if a bit more convoluted to get mail to the correct place) if I could get antispam to call sendmail correctly.
On 06/03/14 07:48, Alan Chandler wrote:
However, when I try to make this work with anitspam, I keep getting failure, and I can't work out what is causing it. Maybe its again because I am running with uid 10001 when calling sendmail but I don't know.
antispam config is
plugin { ...
antispam_pipe_program=/usr/bin/sendmail
Arrg:
I have been struggling with this all yesterday evening. Two minutes after posting this I realise it is /usr/sbin/sendmail, not /usr/bin/sendmail
Sorry for the noise (Although I would prefer to use the dspam backend if I could).
It is not working yet - postfix has rejected the mail because it came from an unknown sender (dspam@chandlerfamily.org.uk) I could use %u on the sender address - but then my sieve script to save outgoing mail would have to change to ignore mail aimed at the spam addresses.
** Alan Chandler alan@chandlerfamily.org.uk [2014-03-06 08:01]:
On 06/03/14 07:48, Alan Chandler wrote:
However, when I try to make this work with anitspam, I keep getting failure, and I can't work out what is causing it. Maybe its again because I am running with uid 10001 when calling sendmail but I don't know.
antispam config is
plugin { ...
antispam_pipe_program=/usr/bin/sendmail
Arrg:
I have been struggling with this all yesterday evening. Two minutes after posting this I realise it is /usr/sbin/sendmail, not /usr/bin/sendmail
Sorry for the noise (Although I would prefer to use the dspam backend if I could).
It is not working yet - postfix has rejected the mail because it came from an unknown sender (dspam@chandlerfamily.org.uk) I could use %u on the sender address - but then my sieve script to save outgoing mail would have to change to ignore mail aimed at the spam addresses. ** end quote [Alan Chandler]
I wouldn't worry about it. I've done this plenty of times in the past, usually when I lack somebody (often completely non-technical) to try and explain things to. Sometimes it can be worth blogging or documenting - sorry to use that word ;) - things instead of boring somebody with at technical explanation - or posting something to a mailing list :)
Anyway, as an aside to this, I've been amazed how effective greylisting has been - so much so a that on my server I've not got round to putting the full anti-spam configuration back in. I should imagine that on a raspberry pi the reduced load either of not running the anti-spam or much less mail being checked would be welcome.
Without going into technical detail, all I am doing is greylisting anything that comes in and is on a RBL. Non RBL stuff gets straight through, so if a proper mail server is spamming and not yet on a RBL it doesn't help. If a legitimate server has ended up on a RBL by mistake it will try again (although so will a compromised one) and just be slowed down. The vast majority of spam, for me at least it seems, just doesn't bother to retry so there is no need to use processing power to scan it. Of course at some point the spammers will improve their processes no doubt, but for now they just seem to want to churn it out to anyone that will accept it without question and bother the rest. My volume of mail is pretty low (only a few thousand a day I think), so the hassle of setting up scanning hasn't been worth effort - yet - for the small volume of spam that gets through (although being a long term Mutt user for my mail I find I fly through managing mail compared to using a mouse).
-- Paul Tansom | Aptanet Ltd. | http://www.aptanet.com/ | 023 9238 0001
Registered in England | Company No: 4905028 | Registered Office: Ralls House, Parklands Business Park, Forrest Road, Denmead, Waterlooville, Hants, PO7 6XP
On 07/03/14 12:14, Paul Tansom wrote:
Anyway, as an aside to this, I've been amazed how effective greylisting has been - so much so a that on my server I've not got round to putting the full anti-spam configuration back in.
That is very interesting - I was running on a 256Mb Pi (the old model B) and it would have worked with only the milter-greylist package installed. When my wife expressed an interest in the old PI as an XBMC front end I splashed out on a replacement with 512Mb, and have added dkim, clamav and dspam checks as well (just got this working and antispam working). and I am using just over the 256Mb in a pretty static state (I am routing all my outgoing e-mails through it, but I have not yet switched over from my other e-mail server)
One question I would be very interested in - and can't find much about it is how long do you greylist these people for?
Basically I only greylist people who fail the spf checks at the moment (that is specifically those who explicitly fail the spf check and those that have an spf record with a +all at the end)
I greylist a softfail for 4 hours and a hard fail or open for 12, but I plucked these figures out of the air.
Alan
On 3/7/2014 10:21 AM, Alan Chandler wrote:
One question I would be very interested in - and can't find much about it is how long do you greylist these people for?
Basically I only greylist people who fail the spf checks at the moment (that is specifically those who explicitly fail the spf check and those that have an spf record with a +all at the end)
I greylist a softfail for 4 hours and a hard fail or open for 12, but I plucked these figures out of the air.
Alan
A delay of 5..15 minutes is sufficient, a delay of hours unnecessarily delays legit mail without increasing the effectiveness. The vast majority of bots either don't retry, or retry once immediately.
It seems to me that greylisting based on spf would not be very effective since it appears many bot herders intentionally use domains without spf records.
Remember the purpose of greylisting is to reject bots, not delay "real" mail servers -- even if you don't want their mail.
-- Noel Jones
On 07/03/14 17:21, Noel wrote:
On 3/7/2014 10:21 AM, Alan Chandler wrote:
One question I would be very interested in - and can't find much about it is how long do you greylist these people for?
Basically I only greylist people who fail the spf checks at the moment (that is specifically those who explicitly fail the spf check and those that have an spf record with a +all at the end)
I greylist a softfail for 4 hours and a hard fail or open for 12, but I plucked these figures out of the air.
Alan A delay of 5..15 minutes is sufficient, a delay of hours unnecessarily delays legit mail without increasing the effectiveness. The vast majority of bots either don't retry, or retry once immediately.
It seems to me that greylisting based on spf would not be very effective since it appears many bot herders intentionally use domains without spf records.
Remember the purpose of greylisting is to reject bots, not delay "real" mail servers -- even if you don't want their mail.
-- Noel Jones Thanks
These few posts have made me rethink my strategy here.
Alan
** Noel noeldude@gmail.com [2014-03-07 17:23]:
On 3/7/2014 10:21 AM, Alan Chandler wrote:
One question I would be very interested in - and can't find much about it is how long do you greylist these people for?
Basically I only greylist people who fail the spf checks at the moment (that is specifically those who explicitly fail the spf check and those that have an spf record with a +all at the end)
I greylist a softfail for 4 hours and a hard fail or open for 12, but I plucked these figures out of the air.
Alan
A delay of 5..15 minutes is sufficient, a delay of hours unnecessarily delays legit mail without increasing the effectiveness. The vast majority of bots either don't retry, or retry once immediately.
It seems to me that greylisting based on spf would not be very effective since it appears many bot herders intentionally use domains without spf records.
Remember the purpose of greylisting is to reject bots, not delay "real" mail servers -- even if you don't want their mail. ** end quote [Noel]
That sounds about right. I'm blocking unknown hosts for 10 minutes before whitelisting them for 8 hours. If they don't retry with that time they are dropped from the whitelist, but if they do that whitelist is extended to 60 days. At least that's the way I'm reading the config. It's a pretty standard greylistd config combined with exim and dovecot. It's running on an Atom 330 based server, although that is protected to some extent via a similar setup on a lightweight VPS that routes mail in via the hubbed hosts config. I'm actually thinking of clustering the two boxes instead which may work better if my internet connection goes down (which it did for about a week thanks to BT line problems last year). Next is to improve my backup process which is untidy.
-- Paul Tansom | Aptanet Ltd. | http://www.aptanet.com/ | 023 9238 0001
Registered in England | Company No: 4905028 | Registered Office: Ralls House, Parklands Business Park, Forrest Road, Denmead, Waterlooville, Hants, PO7 6XP
On 06/03/14 07:59, Alan Chandler wrote:
On 06/03/14 07:48, Alan Chandler wrote:
However, when I try to make this work with anitspam, I keep getting failure, and I can't work out what is causing it. Maybe its again because I am running with uid 10001 when calling sendmail but I don't know.
antispam config is
plugin { ...
antispam_pipe_program=/usr/bin/sendmail
Arrg:
I have been struggling with this all yesterday evening. Two minutes after posting this I realise it is /usr/sbin/sendmail, not /usr/bin/sendmail
Sorry for the noise (Although I would prefer to use the dspam backend if I could).
It is not working yet - postfix has rejected the mail because it came from an unknown sender (dspam@chandlerfamily.org.uk) I could use %u on the sender address - but then my sieve script to save outgoing mail would have to change to ignore mail aimed at the spam addresses.
Well I am STILL struggling with this.
here is my (revised) plugin parameters (and also a temp change to call a script rather than sendmail)
antispam_debug_target = syslog
antispam_verbose_debug = 1
antispam_backend = pipe
antispam_signature = X-DSPAM-Signature
antispam_signature_missing = error
antispam_trash = Trash
antispam_spam = Junk
antispam_allow_append_to_spam = yes
antispam_pipe_program=/home/pi/callsendmail
antispam_pipe_program_args=-f;dspam@chandlerfamily.org.uk
antispam_pipe_program_spam_args=%u@spam.spam
antispam_pipe_program_notspam_args=%u@ham.hamTo try and find out what is going wrong.
the /home/pi/callsendmail script has the following in it
#!/bin/sh echo "$$-start ($*)" >> /tmp/dspam-train.log cat<&0 >> tmp/dspam-train.log echo "$$-end" >> /tmp/dspam-train.log
exit 0
So I was hoping to see my training message in the log. Instead I just get this
21536-start (-f dspam@chandlerfamily.org.uk alan@spam.spam) 21536-end
Is it my shell script that is wrong or antispam - why is nothing written in my log file?
-- Alan Chandler http://www.chandlerfamily.org.uk
On 12/03/14 21:43, Gedalya wrote:
On 03/12/2014 05:35 PM, Alan Chandler wrote:
cat<&0 >> tmp/dspam-train.log
missing leading / .. also, just cat >> /tmp/dspam-train.log would work, you don't need to redirect what is already your input.
Thanks that solved that
and indeed my training message is piped through as I hoped.
I now moved that same script to replace the call to dspam in master.cf of my postfix config to check how that was being called for retraining, and indeed it is being called
/usr/bin/dspam --client --mode=teft --class=spam --source=error --user alan@chandlerfamily.org.uk
with the correct message on the standard input.
HOWEVER the following appears in syslog when revert all the changes to the proper programs
Mar 12 22:09:54 piserver dspam[21941]: Unable to find a valid signature. Aborting. Mar 12 22:09:54 piserver dspam[21941]: process_message returned error -5. dropping message.
Any idea why dspam can't find the signature.
-- Alan Chandler http://www.chandlerfamily.org.uk
participants (4)
-
Alan Chandler
-
Gedalya
-
Noel
-
Paul Tansom