Hi,

Ben Schumacher wrote:

Could you maybe explain how the configuration works in a little more detail? For some reason I'm having a bit of trouble wrapping my head around how this works:

What is the part you don't understand?

namespace private { separator = . location = maildir:/var/mail/%u inbox = yes hidden = no }

namespace public { separator = . prefix = learn. location = maildir:/var/learn/%u inbox = no hidden = no }

This is a simple namespace definition, nothing specific to my plugin. The learn.* part of the namespace is declared public and stored an a specific location. I did this to avoid counting the learnt messages to the quota.

plugin { (...) pipe = /var/learn/%u/.spam:spamc -d some.host -L spam pipe2 = /var/learn/%u/.ham:spamc -d some.host -L ham (...)

And here I define that any message stored to /var/learn/%u/.spam where %u is the username (that is learn.spam in the user's IMAP namespace) has to be piped to the "spamc -d some.host -L spam" command. And the same for ham.

Hope this helps,

Nicolas

Quoting Johannes Berg johannes@sipsolutions.net:

...

...

plugin { (...) pipe = /var/learn/%u/.spam:spamc -d some.host -L spam pipe2 = /var/learn/%u/.ham:spamc -d some.host -L ham (...)

And here I define that any message stored to /var/learn/%u/.spam where %u is the username (that is learn.spam in the user's IMAP namespace) has to be piped to the "spamc -d some.host -L spam" command. And the same for ham.

The above is why I like my antispam plugin much better: you can push into *any* folder *from* spam to train as ham.

It certainly is a matter of taste, but that also was the reason why I
did not like your plugin... ;-)

Firstly, if a message is correctly identified as a spam, but I want to
keep it a copy for some reason, copying outside the spam plugin means
learning as ham, so I have to copy it again into the spam plugin to
re-learn it as ham.

Sencondly, I don't know how dspam works, but spamassassin can identify
a message as HAM or SPAM without learning it. If I want to learn a
message correctly identified as spam but not auto-learnt, I have again
to move it out of the spam folder and then back in. The same if a
message is correctly identified by ham but not auto-learnt, I have to
move it to the spam folder and then out of it.

Lastly, I'd prefer my user to make some explicit action to learn a
message rather than have a system that somewhat "guesses" for them...

I do think, however, that the two plugins could possibly converge.

I don't think they can converge, they work in very different ways,
since yours work at the IMAP level while mine works at the storage
level.

Hence, mine can also be triggered by a delivery with dovecot's
deliver, but when a message is copied or appended, it does not know
where it comes from (but as far as I am concerned, since I don't like
triggering when moving a message out of a folder, it does not matter).
On the other hand, yours only works with IMAP, overriding the COPY
command.

That would entail being able to specify pipes with two endpoints, right now you just have the destination folder. In order to do that, the plugin would have to be configurable like this, with the pipes tried in the order they are numbered:

# learn into spam as spam pipe.1 = *>SPAM |/learn/as --spam pipe.2 = UNSURE>SPAM |/learn/as --spam # forbid into unsure pipe.3 = *>UNSURE - # learn from unsure or spam folder as ham pipe.4 = UNSURE>* |/learn/as --ham pipe.5 = SPAM>* |/learn/as --ham

Those only work if you consider messages that are copied from a folder
to a different one. But how do you deal with messages that are
appended from another source (such as messages being delivered or
appended by an IMAP client)?

This would allow writing a simple plugin, having dovecot load it (before the pipe plugin) and having a function in that plugin called, splitting the work the antispam plugin would do in that case into two plugins.

I guess that such a plugin is actually more generic than "pipe" then.

It looks to me that you only consider spam/ham learning as a use for
such a plugin. I think other uses are possible. For example, I
remember someone who was willing to implement an outbox.

Do you have the pipe plugin code in git anywhere I can pull from and see if I can do such modifications?

No, I have never used git. But you have the tarball.

Cheers,

Nicolas

Quoting Johannes Berg johannes@sipsolutions.net:

...

...

I do think, however, that the two plugins could possibly converge.

I don't think they can converge, they work in very different ways, since yours work at the IMAP level while mine works at the storage level.

Actually, I work at storage level too now, I modified it.

Oh. I missed that. I had a look at your plugin, quite some time ago, and did not think
you would have modified it in such a way. I should definitely give it a look; how to you figure out where the
mail comes from?

No, but I'm using spam as an example of how your plugin could be generalised to work the same as mine with a different configuration. It seems to me that you only consider use cases that depend on the target folder while my generalisation was to make it depend on the pair (source, target).

I did not think it would be possible to use the (source,target) pair
at the storage level. Now that I understand your proposal a little
better, I think it looks like a good idea.

Cheers,

Nicolas

Timo Sirainen wrote:

I downloaded it when you first mentioned it, but looks like I never got around to actually looking at it. Yes, something like this would be nice..

One thing at least that should be changed is to configure it using virtual mailbox names instead of full mailbox paths. This isn't really possible with v1.0, but v1.1 should make it pretty easy.

You already told me about this last time, but I haven't yet come to read the source code of dovecot 1.1...

A few things about the code:

save_dest_mail = mail_alloc(ctx->transaction, MAIL_FETCH_PHYSICAL_SIZE, NULL);

Quota plugin wanted to know the message's physical size, but your plugin doesn't need it. So you could just use 0 instead of MAIL_FETCH_PHYSICAL_SIZE.

Thanks for pointing this.

I think write() can return partially written data if the other side isn't reading it fast enough. Using write_full() instead would anyway be better/safer.

I just reread my code, and I think my use of write looks safe, since only the amount that was correctly written is skipped with i_stream_skip. Do you think I'm missing something?

Do you really need to wait for the executed process to finish? Since this is the only plugin currently creating child processes, I'd setup a SIGCHLD handler and waitpid() there to get rid of the zombie: lib_signals_set_handler(SIGCHLD, TRUE, chld_handler, NULL);

Waiting is required if we want the append to fail if the command fails. I guess it should better be a configurable option, don't you think so?

Multiple commands are now processed sequentially. I don't know if there's real need for multiple commands, but it would be faster to read the message input just once and send it to all pipes in parallel.

I don't think there is real need for multiple commands, but I think it was easier for me to program things like this... ;-)

"return 0 * WEXITSTATUS(status);" returns always 0 :)

Hummm... I guess I should avoid drinking alcohol before I program... ;-) It probably should be "return WEXITSTATUS(status);"; I have no idea why I changed this in such a strange way...

I'd also leave stderr as-is for the child process so it could log errors, and for handling syscall failures use: i_fatal("dup2() failed: %m");

OK, point taken.

One question still: would you consider merging my plugin in dovecot if I ported it to 1.1?

Cheers,

Nicolas