On Tue, Jun 1, 2010 at 09:54, Frank Elsner <frank@moltke28.b.shuttle.de> wrote:

On Tue, 1 Jun 2010 09:42:18 -0400 Phil Howard wrote:

...

I see no documents in the Dovecot documents wiki on how Dovecot features can be used in the fight against spam.  One hit came up in a search, and it was about how to fight spam in the MoinMoin Wiki itself (e.g. TextCHAs and such).

Fighting against SPAM is the MTA's job. Dovecot's job is to provide access to the mailboxes for the users.

Dovecot would still be involved ... getting detected spam into a folder for users that don't want false positives to be deleted, and a feedback folder for recycling false negatives into detection training.

On Tue, Jun 1, 2010 at 10:21, Frank Elsner <frank@moltke28.b.shuttle.de> wrote:

Ok, you obviously have a different environment and requirements.

• I do the delivery into special folder by the MTA (exim) and the LDA is not used at all.

• detection training is not done.

Sorry, my world may be to narrow :-)

Each of our worlds is narrow. Some may be wider than others. Programs like Dovecot support a LOT of things, and I can't see how anyone can be using more than a (perhaps significant in some cases) fraction of it. Obviously your choice of Exim for an MTA doesn't relate to my choice of Postfix for MTA. Dovecot supports both and more. So the programs can be a lot wider real world uses. But put all the variety of uses together as a set, and it can be seen that diverse tools are useful.

Since you aren't using Dovecot deliver, then you don't need to be knowledgeable about its -m option. I'm curious if that is the only way. I'm still thinking that a shim (program) between Postfix and Dovecot deliver that detects headers, markers, or tags placed on the message by spam detection, and adds a "-m Spam" option or whatever, might be what I need to do. But I guess you won't need it.

On Tue, Jun 1, 2010 at 10:04, Jerry <dovecot.user@seibercom.net> wrote:

The fight against SPAM is NOT Dovecots responsibility.

Whose responsibility is it to get detected spam delivered into the correct folder? Dovecot deliver is typically used to deliver into the Inbox(es). It would be involved.

Note that I am not saying "responsibility". There are features that can be used (or there aren't, as the case may be). Documenting them would be helpful. Dovecot has documentation for many other things. If it had documentation on ... dealing with (I'll use that term instead of fighting, as maybe that will be clearer to you) ... spam aspects (like folder selection aspects), that would be helpful.

The fight against SPAM is NOT Dovecots responsibility. There are numerous applications that work with your MTA for that purpose. I suggest that you investigate this further. You might want to start here:

       http://www.ijs.si/software/amavisd/

They also have their own mailing list that you can use to answer any questions that you might have.

Like "How do I get Dovecot deliver to put the spam in the spam folder?" ?

On Tue, Jun 1, 2010 at 12:17, Stan Hoeppner <stan@hardwarefreak.com> wrote:

Phil Howard put forth on 6/1/2010 9:15 AM:

...

On Tue, Jun 1, 2010 at 10:04, Jerry <dovecot.user@seibercom.net> wrote:

...

The fight against SPAM is NOT Dovecots responsibility.

Whose responsibility is it to get detected spam delivered into the correct folder?  Dovecot deliver is typically used to deliver into the Inbox(es).  It would be involved.

It's the MTA's responsibility to _REJECT_ spam attempts at SMTP time.  It's the mail OP's responsibility to properly configure the MTA to do so.  Spam fighting should be performed pre queue, not post queue.

I suggest you bone up on modern spam fighting techniques.  Post queue content analysis is not the proper way to fight spam, especially in 2010.

For my own personal server, I would agree. Even for a general email or freemail service I would agree.

However, for certain business environments, even a few false positives can be very troubling to management. Greylisting, and with shades of grey, is often more appropriate. Given that this puts mail where it will likely never be read or responded to, certainly doing as much as you can at SMTP time for a 5XX rejects is preferred, so that legitimate mail that would otherwise not be responded to is known to be not delivered where it would be read. But this can't be for everything, as even a FP rate of 0.01% is too much.

So in the end, there has to be an administrative policy decision as to what to do with what is detected as spam. And if I can find a way, I'd like to even set that up so end USER policy can be applied even at SMTP time (e.g. user decides if spam is blacklisted or greylisted, along with user specified whitelists). But that's pushing the boundaries for now. When I get time, I will look into that degree of control.

On Tue, Jun 1, 2010 at 12:41, Stan Hoeppner <stan@hardwarefreak.com> wrote:

Law firm or heavily FED GOV regulated industry?  Or just HUA (head up ass) management with zero understanding of email and spam?

People dealing with a large variety of outside clients and potential clients where bouncing THEIR mail gives bad impressions in too many cases (already has with the outside mail provider we were using up until the move, which was expedited for just such issues).

FYI, you forgot the marketing industry (not counting the bulk email marketing industry ... although we are not that). Also, lots of media and other industries can have this issue, as well as GOV itself.

On 2010-06-01 10:34 AM, Charles Marcus wrote:

There is also a plugin for dovecot that makes it easy for users to reclassify stuff as spam or not spam by simply moving the messages to the appropriate folder, but I think it is mainly designed for DSPAM... anyone using it with spamassassin?

Sorry, meant to include a link:

http://johannes.sipsolutions.net/Projects/dovecot-antispam

--

Best regards,

Charles

On Tue, Jun 1, 2010 at 10:45, Rainer Frey <rainer.frey@inxmail.de> wrote:

I guess you'll have the mental transfer of "fight spam" into "cause dovecot to perform this and that action" yourself. The possible actions are, in general, documented.

I cannot determine how deliver with an empty string give to -m would behave. OTOH, I don't know that an empty string would what would even happen, since I have yet to explore how the spam status from one of the spam detections would trickle through to setting up the transport variables in Postfix. I don't even know if -m is considered the appropriate means to deliver to a specific box in the case of junk spam. If it is, obviously someone must be using it.

You need to configure the MTA to accept and tag spam messages, e.g. with Amavis D_PASS policy and appropriate tag levels. Then the most flexible way to put the mails into a spam folder is to use the deliver plugin for sieve, which can sort and filter mails depending on many different conditions, like presence and/or value of any header. You can deploy a single system wide default script for all users, which can be complemented or overwritten by individual users (depending on configuration).

OK, so a delivery-time sieve.

See http://wiki.dovecot.org/LDA/Sieve If you use dovecot 1.2, using the new Dovecot Sieve plugin is highly recommended above the  old CMUsieve. See http://wiki.dovecot.org/LDA/Sieve/Dovecot

Ubuntu has me at Dovecot version 1.1.11. Ubuntu has turned out to be difficult for managing upgrades when packages it has are instead compiled from source. So being "up to date at the source level" is going to have to wait until I get some Slackware running here (originally Ubuntu was chosen because of the wider variety of ready-to-install packages).

The really recommended way nowadays is different though: *reject* as much spam as possible in the MTA in the receiving SMTP dialog, with a combination of blacklists, sender validation (e.g.DKIM), potentially greylisting (as you deem appropriate) and lastly content scanning with s.th. like amavis+spamassassin. I'd not expect a noticable amount of false positives, and in the rare cases the sender will be notified. The false positives rate of users scanning through a folder of tagged mail will be higher.

I've had periodic high FPs with content detection in the past (on other mail servers I didn't control). So greylisting for that is what I will likely be doing, at least at first. How much I shift to blacklisting will depend on the feedback from my own users ("hey, my Junk-spam folder has too much spam" or "How can I find that one false positive if there are a million spams in there").

Tagging ... yeah, clearly I would do that for anything not rejected at SMTP time. How to get Dovecot deliver to put those ... well, OK, I'll look into Sieve instead of writing my own shim.

A howto-like page to describe the usage of some dovecot features in an anti- spam context wouldn't hurt, OTOH there is not much anti-spam specific stuff at all at the dovecot side.

Unfortunately, a good document would likely have to be specific to the LDA, MTA, and spam tools. Given so many combinations, that's a lot documents, or a rather complex one. I can certainly use my notes from my own solutions and make something specific to Dovecot and Postfix and whatever spam tools I use (could be more than one).

On Wed, Jun 2, 2010 at 02:39, Rainer Frey <rainer.frey@inxmail.de> wrote:

On Tuesday 01 June 2010 17:01:16 Phil Howard wrote:

...

I cannot determine how deliver with an empty string give to -m would behave.

This is just what I mean: the behavior of deliver -m has absolutely nothing to do with fighting spam. It is the same whether you want to use it for spam- tagged mails or any other reason. Therefore you'll not find the description in thedovecot wiki bysearching for "fighting spam". You'll find it by searching for "deliver" (or LDA).

I found the man page (like) document for deliver just fine. I did not limit my search to just the "spam" keyword. I'm guessing that you are assuming I never found this document:

http://wiki.dovecot.org/LDA

But now that you know I have read it (and read it long before the previous post), re-read what I said. You should conclude that I am unable to determine fully what -m does from this. It does not describe the behavior if the value given is a null string. This is a concern because in master.cf for Postfix, there is no means to dynamically provide or omit the -m option. You can give it a value from a variable. But it may be necessary to give an empty string value in order to not deliver to a specific folder, and I want to know if that will get the mail into the INBOX.

Yes, sieve is mostly (with dovecot: only) used at delivery time (Cyrus IMAP might have support for applying sieve scripts with IMAP, but if so, it is an exception).

If sieve had been defined in terms of the mail client uploading scripts via IMAP to a designated special folder, it might have been easier to have users supply their own scripts. But at this point I don't want to have users doing things that will send mail back out (e.g. vacation scripts) unless and until I can ensure that such outbound mail only goes to known recipients (e.g. to avoid backscatter for FN spam with forged sender addresses).

This is not my experience. Esp. dovecot and postfix are easily managed from source. Installing the package 'build-essential' and then 'aptitude build-dep dovecot' should provide all you need to compile and install dovecot from source (which is documented in the wiki).

My experience with Debian and Ubuntu has several cases of packages being degraded or corrupted after building them from source code. Colleagues have reported the same with Fedora. No such issue has ever happened with Slackware, either to me, or that I have heard of. But, for the time being anyway, I am on Ubuntu without any source builds. For various reasons, I do need to eventually move mail services off the current machine and onto a pair of new machines. At that time, Slackware will be considered, as will OpenBSD.

This is why I recommended a combination of different measures. Content detection should come in last in that list, but still can be done at SMTP time (esp. with latest postfix and amavisd-new).

I'm all for the combination. But I need to do less rejection and more tagging for the before-queue tests (after-queue tests would limited to tagging, only). That's yet to be explored in Postfix.

On Wed, Jun 2, 2010 at 10:20, Tom Hendrikx <tom@whyscream.net> wrote:

Did you even try to set this up in your test environment (you have one, don't you?) to explore your own capabilities in finding out the behaviour of this edge case? It sounds trivial to do, but you cannot expect other list members to test this for you.

I wish. The mail server was the test environment before the domains were moved over to it. The move was originally planned for June 30, but got expedited to May 25 due to issues with the mail service provider. I will be moving this again probably around October. Hopefully that hardware will be in a couple months before.

This is a startup environment. It means there isn't a ton of cash up front to have the kind of environment that makes sense. But it is coming in at some pace, now. I may even get a real desktop computer in July.

When you change your attitude a bit more towards this direction, you could even add some valuable input to dovecot by suggesting a man page enhancement describing the behaviour of deliver when hitting an empty string as argument to -m (or at least fill the gap when someone else has the same question and tries google for it). Your latest posts to dovecot (and postfix) mailing lists seem to be only highly speculative and have a lot of lengthy arguing, and not much hands-on.

The attitude here is more about get as much done as possible. The mail server is a fraction of what is going on. I work on it when I can. Other things did get dropped the week or so leading up to the expedited changeover. Now that mail is off the problem provider, I do have more breathing time and can actually do some reading. But even so, I'd much rather spend less time reading only what I need, than spend more time reading stuff that includes what I don't need. The reason for that is because I have so much other stuff to read on other projects. Today was spent diagnosing why time servers were refusing to start, and why the monitoring programs on the RAID file servers were not able to access the network. If those problems weren't happening, I'd have been working on the lighttpd (new stuff to read) web servers.

A small example describing a more pro-active attitude:

<phil> Hi list! I need to put spam in a specific folder and want to use 'deliver -m' for this. However, when I set up deliver like this in postfix <config excerpt omitted>, I get the following results that I did not expect when the argument for '-m' was a null string: <log excerpt omitted>.

<list> Hi phil, could you try <suggestion>?. Also, when you do this <config omitted>, you'll most likely get what you want.

<phil> That works great, thanks guys. I'll add you suggestion to the dovecot wiki for future reference.

But that's not reflective of reality. First, I was not committed to using deliver -m at all. It was ONE option. And there was even more than one way to use it. Now I did misunderstand Sieve at that point, because I was addressing things in a different order. Sieve was on the "back burner" because of issues related to user Sieve scripts. I knew I needed to read more about Sieve to address it. But because I didn't connect the concept that Sieve would address this problem, I didn't expedite considering it.

Remember, the plate is more than just full; it is way overflowing. It is prudent to manage time the best you can. This means avoiding things that don't seem to be urgent. That is not a perfect process, since one might not know about the "perfect solution to everything" that lurks in the bowels of the documentation.

But I do try to phrase my initial questions in terms of what documentation to read, or of basic planning directions.

Planning and reading documentation are too often at odds. It's a hard juggle. The best planning is done by knowing everything. But if you spend all the time to know everything, the deadline is long past by the time the decision can be made. So it's something you have to go back and forth with. Read some. Make some decisions based on that.

From those decisions now you have narrowed the field of what is to be read. That means you now have the advantage of less to read.

And often the details are lacking in the documentation. Ways to fill that in do include testing (now, for example, I'll have to get and set up a whole new testing environment for email ... something I hadn't expected to need to do for a couple more months) ... or, asking.

For example, someone could argue that I should have chosen Exim instead of Postfix, and that if all I had done was read everything about Exim, I'd have know it was "better". Well, I'm not going to pass judgment on that. Since I have never used Exim and knew nothing about it (besides that it is another MTA), and have used Postfix some and know some about it, I save time by choosing Postfix. It might not have been the best choice. But making the choice is likely the best time saver because it omits all that time reading about Exim.

Sometimes some choices are easy, because the elements of the choice are simple. In those cases, complete reading might be done in 5 minutes or less. So from start to full knowledge of the major choices might be under an hour. But I have put in at least 12 hours reading time on just Dovecot and Postfix, and still don't know it all. And for a while I was doing a lot of configuration testing, to see what might work and what might not. Some of that even including my own scripts to construct files (ultimately ended up using passwd-file format, even though I had hoped heading into that I could have used CDB format).

To conclude: I think saw some list posting a while ago that contained an example of postfix master.cf using a shell-like ${some_arg:-default_string} expansion syntax. I'm sorry to see that I cannot find it right now. This syntax, or a 3-line wrapper script around deliver containing it, would solve your issue.

However I still think that you'd be better off checking whether deliver will already handle your issue nicely (albeit in an undocumented way).

Well, I do know this. I could write the shim code to run in front of deliver to inspect the message headers and run deliver with or without the -m option. If do that, it means I can defer the time to read about Sieve, and having something in place sooner, even if it is a temporary, inflexible, non-optimal solution.

On Tue, Jun 1, 2010 at 11:01, William Blunn <bill+dovecot@blunn.org> wrote:

Get your anti-spam software to add header records to the messages, then use Dovecot LDA Sieve to filter based on those header records.

So I can run Sieve as part of the deliver? Or does it run soon after that?

On Tue, Jun 1, 2010 at 12:55, William Blunn <bill+dovecot@blunn.org> wrote:

"The Dovecot Sieve plugin provides mail filtering facilities at time of final message delivery using the Sieve (RFC 5228 <http://www.ietf.org/rfc/rfc5228.txt>) language."

"Sieve is implemented as a plugin to deliver <http://wiki.dovecot.org/LDA>."

I hadn't gotten to reading much about Sieve, yet, as what I had read said it was something IMAP users would run. So I didn't realize (then) that I needed to read about it. It was in the list for stuff to read about, but just for later. It is now bumped up and I will be reading about it this week (the email server is a fraction of what is going on here).

On Tue, Jun 1, 2010 at 14:56, Frank Cusack <frank+lists/dovecot@linetwo.net> wrote:

Man oh man.  You don't have much experience with mail and it sounds like you are starting from scratch.  You have your work cut out for you. :)

Well, actually I do. A lot of it is way way back there with sendmail. Back then I coded sendmail.cf by hand and avoided m4. I have used Postfix previously, too, but in a very different circumstance, and even then it was several years back.

The standard way to filter spam is [all of]:

a) stop it at the border, then

For my current situation, this will be a smaller amount than I've done in the past.

b) content analysis and tagging, then

I haven't done this in the past for my own mail servers primarily because of principle (e.g. to me UCE isn't about what it said, but whether it is unsolicited ... which indirectly means what is said ... and bulk ... in summary, it's about behaviour, not about the message). But in this new situation, I have to rebalance these ideas to achieve different goals. Being principled doesn't count in this situation. A potential client isn't going to be turned down or lectured just because the ISP they use is spammer friendly. I don't really like that, but that's the way it is.

c) filtering at delivery time

per-user filters for part (b) on the server side is a dead end IMHO but there are some programs that do this and dovecot can work with them (by having folders that train the filter).  modern mail clients, however, have their own spam filters now, which are by definition personalized.

anyway, part (a) and (b) have nothing to do with dovecot.  the standard way to do part (c) is to have an X-Spam: header which a sieve script filters on.  the -m flag to deliver is not really for spam filtering.

Since sieve looks like it will be a problem right now, until I get a solution to that, I'm seriously considering this solution. A shim program I write in C will be run from Postfix master.cf just as Dovecot deliver is now. I'd basically change the executable path to the shim program. The shim program will read the message (I assume from stdin) up to 1MB or the end of headers. If the body isn't reached by 1MB it goes into the spam folder. If the X-Spam: header is present with a sufficient probability of spam, it goes into the spam folder. Else it goes into the INBOX. Set up a command argument list to run deliver, and include -m with the folder name if this goes to the spam folder. Set up pipes, fork, and child will exec deliver with that argument list. Pipe the buffer that was read in to deliver until it is empty, then pipe any remaining stdin to deliver all as one stream. Wait for deliver to exit and capture its exit status, and exit with the same status. Postfix should then know if delivery succeeded or failed.

I may set up a web mail service later, and will see if I can do Sieve within that.

On 01/06/2010 20:56, Phil Howard wrote:

Since sieve looks like it will be a problem right now, until I get a solution to that, I'm seriously considering this solution. A shim program I write in C will be run from Postfix master.cf just as Dovecot deliver is now. I'd basically change the executable path to the shim program. The shim program will read the message (I assume from stdin) up to 1MB or the end of headers. If the body isn't reached by 1MB it goes into the spam folder. If the X-Spam: header ispresent with a sufficient probability of spam, it goes into the spam folder. Else it goes into the INBOX. Set up a command argument list to run deliver, and include -m with the folder name if this goes to the spam folder. Set up pipes, fork, and child will exec deliver with that argument list. Pipe the buffer that was read in to deliver until it is empty, then pipe any remaining stdin to deliver all as one stream. Wait for deliver to exit and capture its exit status, and exit with the same status. Postfix should then know if delivery succeeded or failed.

Procmail will do all the things you say above with a few lines of simple configuration, but with the benefit of being already done, tried and tested.

Procmail is a little self-contained program which you can just plain run, have it do some matches on the message content, and then use that to invoke the LDA one way or another.

People may say that Procmail is a bit old, and it is. But it works.

I pass all of my incoming mail through Procmail.

You can make rules with conditions, such as matching header records with regular expressions.

If a condition matches (e.g. we found a spam header), then you can tell Procmail to pipe the message to a program (e.g. "deliver") with certain arguments.

:0

• ^X-Spam-Flag: yes | deliver -m spam

If none of the rules match, we can get procmail to do something different, e.g. pipe the message to a different program, e.g. "deliver" but with different arguments.

:0 | deliver

By default, Procmail will try to deliver the message exactly once. If it fails, it returns an error code so that the MTA can know that delivery failed, and can take the appropriate action.

If you want, Procmail will even pipe your message through another program first, e.g. SpamAssassin, so that the other program can change the message as required (e.g. adding header records saying whether or not it thinks it is spam).

:0 fw | spamassassin

If you want to pass data from the MTA to Procmail for use in rules, (e.g. the envelope recipient), Procmail provides a couple of ways to do this.

Documentation can be found in Procmail's four man pages:

Main procmail documentation - man procmail

Procmail configuration file - man procmailrc

Procmail configuration file examples - man procmailex

Procmail weighted scoring technique - man procmailsc

Bill