[Dovecot] writing a plugin for LDA to chain Unix pipes
Hi,
I've spent some part of the last week on trying to figure out whether dovecot LDA meets our requirements for a new mail server infrastructure (the IMAP and POP servers, as well as the SASL backend certainly do; thanks a lot!). The one thing that's holding me back is that we need per-user integration of SpamAssassin and some other, custom content filters, all of which are implemented as Unix pipe filters.
My conclusion is that since dovecot LDA cannot yet integrate such filters in the delivery chain, but it supports plugins, that such a plugin must be written. Conceptually, it's very simple: a configuration file option specifying the Unix pipe, and a plugin that does the proper popen() (or whatever is to be used these days) call, checking for errors and doing the right thing.
Unfortunately, I do not have much time at my hands -- basically it all boils down to Sunday evening and a little bit of Monday, and then it's got to be done. My C is rusty, but I do have a lot of experience with it, so I am confident...
... but what would really help is if someone who knows the internals of Dovecot would give me a starting point, or even some more information. How do I conceive to write such a plugin? I could not find the information online, and I'd rather not reverse engineer existing plugins, for that approach usually leads to ugly and hence unstable/insecure/unusable code.
Any feedback is thus warmly appreciated! Also, if someone is online on IRC on Sunday and Monday and wouldn't mind if I asked a question now and then, please let me know.
It goes without saying that I will happily submit my plugin (once its tested) to Dovecot upstream under the Artistic Licence, or if that doesn't work, another licence of your choosing.
Thanks,
-- martin; (greetings from the heart of the sun.) \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net@madduck
spamtraps: madduck.bogus@madduck.net
"common sense is the collection of prejudices acquired by age eighteen." -- albert einstein
On 2006-07-01 00:39:40 +0200, martin f krafft wrote:
I've spent some part of the last week on trying to figure out whether dovecot LDA meets our requirements for a new mail server infrastructure (the IMAP and POP servers, as well as the SASL backend certainly do; thanks a lot!). The one thing that's holding me back is that we need per-user integration of SpamAssassin and some other, custom content filters, all of which are implemented as Unix pipe filters.
run them in your MTA before pushing stuff into the LDA?
e.g. i deliver incoming via lmtp to dspam. dspam reinjects the mail into postfix. and postfix calls dovecot-lda to deliver the mail.
works flawlessly here. so i dont see why we should bloat dovecot-lda, when most mail server handle that just fine already.
just my 2 cents,
darix-- openSUSE - SUSE Linux is my linux openSUSE is good for you www.opensuse.org
On 7/1/06, Marcus Rueckert rueckert@informatik.uni-rostock.de wrote:
On 2006-07-01 00:39:40 +0200, martin f krafft wrote:
I've spent some part of the last week on trying to figure out whether dovecot LDA meets our requirements for a new mail server infrastructure (the IMAP and POP servers, as well as the SASL backend certainly do; thanks a lot!). The one thing that's holding me back is that we need per-user integration of SpamAssassin and some other, custom content filters, all of which are implemented as Unix pipe filters.
run them in your MTA before pushing stuff into the LDA?
e.g. i deliver incoming via lmtp to dspam. dspam reinjects the mail into postfix. and postfix calls dovecot-lda to deliver the mail.
works flawlessly here. so i dont see why we should bloat dovecot-lda, when most mail server handle that just fine already.
I have to agree. This isn't a LDA function. Content filters are a MTA function. You should be able to pipe all the filters together, so in your call to dovecot-lda in your mta, it just changes into one big pipe... For example
dspam --stdout| dovecot-lda -d $u
or
spamassasin | custom filter| dovecot-lda
As you said they are all pipe filters, so they should work in a pipe
just fine! (Dspam can be a pain to get working in a pipe sometimes).
The LDA is exactly that, Local Delivery Agent, and should only have to
deliver mail to the correct boxes, the fact that it also filters them
into more than just the Inbox, is a function of the sieve language,
which does not have functions for calling external unix tools.
If you need help setting up the 'pipe' with your mta, feel free to contact me off list, and if I have enough knowledge of your MTA, will help you as much as possible.
Tim
Linux Counter user #273956
Thank you, Marcus and Timothy, for your replies!
also sprach Marcus Rueckert rueckert@informatik.uni-rostock.de [2006.07.01.0517 +0200]:
run them in your MTA before pushing stuff into the LDA? e.g. i deliver incoming via lmtp to dspam. dspam reinjects the mail into postfix. and postfix calls dovecot-lda to deliver the mail.
Then it's not per-user, or at least I could not figure out how to do this properly with postfix.
also sprach Timothy White dovecot.user@weirdo.bur.st [2006.07.01.0931 +0200]:
I have to agree. This isn't a LDA function. Content filters are a MTA function.
Okay, all the better, then I don't have to write the plugin. :)
You should be able to pipe all the filters together, so in your call to dovecot-lda in your mta, it just changes into one big pipe... For example
dspam --stdout| dovecot-lda -d $uorspamassasin | custom filter| dovecot-lda
I read somewhere on the net that this is pretty flakey and will fail badly in the case of intermittent errors, but I cannot find the reference anymore. Is anyone using this with success?
Also, the above spawns an expensive shell process, doesn't it?
-- martin; (greetings from the heart of the sun.) \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net@madduck
spamtraps: madduck.bogus@madduck.net
"for art to exist, for any sort of aesthetic activity or perception to exist, a certain physiological precondition is indispensable: intoxication." -- friedrich nietzsche
martin f krafft wrote:
dspam --stdout| dovecot-lda -d $uorspamassasin | custom filter| dovecot-ldaI read somewhere on the net that this is pretty flakey and will fail badly in the case of intermittent errors, but I cannot find the reference anymore. Is anyone using this with success?
I'm using the former mode (with vpopmail) for dspam scanning at the domain level and it works fine. The only issue I know of with the above is if the first command in the pipe dumps core, then no error will be propagated back and the message will be lost without trace.
Also, the above spawns an expensive shell process, doesn't it?
At that point in the delivery process, everything is asynchronous (meaning the queue can handle each delivery at whatever rate it wants to without slowing down accepting mail), so it really doesn't matter whether spawning a shell is a performance drag.
Besides, dspam is very resource intensive (as is spamassassin) so the addition of a shell process is truly immaterial...
John
-- John Peacock Director of Information Research and Technology Rowman & Littlefield Publishing Group 4501 Forbes Blvd Suite H Lanham, MD 20706 301-459-3366 x.5010 fax 301-429-5747
also sprach John Peacock jpeacock@rowman.com [2006.07.01.1928 +0200]:
I'm using the former mode (with vpopmail) for dspam scanning at the domain level and it works fine. The only issue I know of with the above is if the first command in the pipe dumps core, then no error will be propagated back and the message will be lost without trace.
Mh, these chances are pretty slim, but I do wonder why. If the application dumps core, the shell will know and exit non-zero...
Btw, my concern was about the spamc -e flag, where the manpage says:
Note that there is a very slight chance mail will be lost here, because if the fork-and-exec fails there’s no place to put the mail message.
http://marc.theaimsgroup.com/?l=spamassassin-users&m=115185095923772&w=2
Also, the above spawns an expensive shell process, doesn't it?
At that point in the delivery process, everything is asynchronous (meaning the queue can handle each delivery at whatever rate it wants to without slowing down accepting mail), so it really doesn't matter whether spawning a shell is a performance drag.
Well, it does, if you get several thousand mails per hour.
Besides, dspam is very resource intensive (as is spamassassin) so the addition of a shell process is truly immaterial...
This is true, of course.
-- martin; (greetings from the heart of the sun.) \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net@madduck
spamtraps: madduck.bogus@madduck.net
"'this must be a thursday,' said arthur to himself, sinking low over his beer. 'i never could get the hang of thursdays.'" -- hitchhiker's guide to the galaxy
On 2006-07-01 19:00:01 +0200, martin f krafft wrote:
also sprach Marcus Rueckert rueckert@informatik.uni-rostock.de [2006.07.01.0517 +0200]:
run them in your MTA before pushing stuff into the LDA? e.g. i deliver incoming via lmtp to dspam. dspam reinjects the mail into postfix. and postfix calls dovecot-lda to deliver the mail.
Then it's not per-user, or at least I could not figure out how to do this properly with postfix.
what stuff do you need per user?
most user would run some antispam tool and than maybe procmail. but as you kinda enforce using dovecot-lda with sieve instead ... i dont see much need in per user stuff.
and having some common antispam solution isnt that bad either. and dspam handles per user tokens quite good already.
darix
-- openSUSE - SUSE Linux is my linux openSUSE is good for you www.opensuse.org
also sprach Marcus Rueckert rueckert@informatik.uni-rostock.de [2006.07.01.2024 +0200]:
what stuff do you need per user?
learning and configuration.
most user would run some antispam tool and than maybe procmail. but as you kinda enforce using dovecot-lda with sieve instead ... i dont see much need in per user stuff.
i do.
and having some common antispam solution isnt that bad either.
it's not as good as per-user. your spam may be my ham.
and dspam handles per user tokens quite good already.
Sure, but I don't want to use dspam.
-- martin; (greetings from the heart of the sun.) \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net@madduck
spamtraps: madduck.bogus@madduck.net
"funny how just when you think life can't possibly get any worse it suddenly does." -- marvin
On 2006-07-02 17:08:28 +0200, martin f krafft wrote:
also sprach Marcus Rueckert rueckert@informatik.uni-rostock.de [2006.07.01.2024 +0200]:
what stuff do you need per user?
learning and configuration.
configuration of what?
most user would run some antispam tool and than maybe procmail. but as you kinda enforce using dovecot-lda with sieve instead ... i dont see much need in per user stuff.
i do.
and having some common antispam solution isnt that bad either.
it's not as good as per-user. your spam may be my ham.
and SA cant be teached to track spam/ham per user?
and dspam handles per user tokens quite good already.
Sure, but I don't want to use dspam.
i am pretty sure you can establish the same workflow that i use with dspam with spamassassin. e.g. i know you can have the same "bounce msg here to train it as ham" with SA.
darix
-- openSUSE - SUSE Linux is my linux openSUSE is good for you www.opensuse.org
participants (4)
-
John Peacock
-
Marcus Rueckert
-
martin f krafft
-
Timothy White