[Dovecot] dovecot, procmail and deliver
(Using dovecot 1.0 RC7 on Fedora Core 5)
<scene set>
Hitherto we have used UW-IMAP on a "farm" of Linux machines mounting NFS from a NetApp. (The UW-IMAP author doesn't like use of NFS, but with careful use of NFS mount arguments ('noac,actimeo=0' etc.) and trying to ensure that all activity for a given user takes place within one machine in the farm, we seem to have been OK.) We have also used UW-IMAP's 'tmail' (in sendmail.cf) and 'dmail' (from any user procmail recipes), so that access to the INBOX has been consistently UW-IMAP software.
We are now considering doing a transparent migration to dovecot to improve performance.
</scene>
First: a general question: To complement the UW->dovecot migration of the IMAP daemon (reading the INBOX and folders), ought I to do a similar delivery-side migration from [UW] tmail/dmail to [dovecot] 'deliver'? It would feel safer this way, but maybe I worry too much...
Second: "deliver" seems to add an unconditional "From " line at the start of each delivery. From sendmail, using the 'n' flag, "Mlocal F=...n..." that is OK. (Although I'm not convinced that "dovecot.deliver", nor its two-space separator (emphasis on "two") from the following date are ideal. Nor, come to that line's lack of "@somewhere".)
But from a procmail recipe, I end up with two "From " lines. Surely this is incorrect. How can this be reduced to one? Shouldn't "deliver" ensure that there is only "From " line?
Third: Would it be possible for "deliver" to do a "syslog" entry to confirm final delivery into the destination mailbox, please? (I could try and produce a patch, but someone else with more dovecot familiarity could probably do the job must more quickly and probably also more cleanly!)
(I think that's all for the moment.)
--
: David Lee I.T. Service : : Senior Systems Programmer Computer Centre : : Durham University : : http://www.dur.ac.uk/t.d.lee/ South Road : : Durham DH1 3LE : : Phone: +44 191 334 2752 U.K. :
Hello!
Try my patch from: http://dovecot.org/pipermail/dovecot/2006-July/014656.html
You can use dovecots deliver program with procmail, it also trips the from lines. It works well for me since July 2006.
Logging is done through the procmail logging.
Ciao, Gerhard
On Tue, 3 Oct 2006, David Lee wrote:
(Using dovecot 1.0 RC7 on Fedora Core 5)
<scene set>
Hitherto we have used UW-IMAP on a "farm" of Linux machines mounting NFS from a NetApp. (The UW-IMAP author doesn't like use of NFS, but with careful use of NFS mount arguments ('noac,actimeo=0' etc.) and trying to ensure that all activity for a given user takes place within one machine in the farm, we seem to have been OK.) We have also used UW-IMAP's 'tmail' (in sendmail.cf) and 'dmail' (from any user procmail recipes), so that access to the INBOX has been consistently UW-IMAP software.
We are now considering doing a transparent migration to dovecot to improve performance.
</scene>
First: a general question: To complement the UW->dovecot migration of the IMAP daemon (reading the INBOX and folders), ought I to do a similar delivery-side migration from [UW] tmail/dmail to [dovecot] 'deliver'? It would feel safer this way, but maybe I worry too much...
Second: "deliver" seems to add an unconditional "From " line at the start of each delivery. From sendmail, using the 'n' flag, "Mlocal F=...n..." that is OK. (Although I'm not convinced that "dovecot.deliver", nor its two-space separator (emphasis on "two") from the following date are ideal. Nor, come to that line's lack of "@somewhere".)
But from a procmail recipe, I end up with two "From " lines. Surely this is incorrect. How can this be reduced to one? Shouldn't "deliver" ensure that there is only "From " line?
Third: Would it be possible for "deliver" to do a "syslog" entry to confirm final delivery into the destination mailbox, please? (I could try and produce a patch, but someone else with more dovecot familiarity could probably do the job must more quickly and probably also more cleanly!)
(I think that's all for the moment.)
--
: David Lee I.T. Service : : Senior Systems Programmer Computer Centre : : Durham University : : http://www.dur.ac.uk/t.d.lee/ South Road : : Durham DH1 3LE : : Phone: +44 191 334 2752 U.K. :
On Wed, 4 Oct 2006, Gerhard Wiesinger wrote:
Try my patch from: http://dovecot.org/pipermail/dovecot/2006-July/014656.html
You can use dovecots deliver program with procmail, it also trips the from lines. It works well for me since July 2006.
Logging is done through the procmail logging.
Thanks for the reply. I've taken a look at the patch.
There are two different things going on in this discussion. These needn't be either/or but could be both/and.
Your patch is to procmail to improve its functionality. I recommend you to try to get it incorporated into procmail by its developers. It could be of benefit both to users of dovecot and to users of some non-dovecot delivery mechanisms.
My proposal is to enhance dovecot LDA to fix what seems to be a deficiency, especially when compared to UW-IMAP, both when driven from procmail and also probably when driven from some other, non-procmail, mechanisms.
(Our thoughts may address similar issues. They overlap, but by addressing both of them (possibly independently) with the respective project maintainers, both projects benefit.)
Already procmail, as is, (no patch necessary) works well with UW-IMAP's "dmail" delivery program. Email is delivered via UW-IMAP/dmail into the INBOX or folder. There are no problems with duplicated "From ". There is syslog (system-level, rather than user-level) logging to complete the system-level logging that was done as the email arrived.
But dovecot's LDA seems not to give this final, clean, completion. My suggestion is to provide it, so that dovecot's LDA can come up to the functional level of UW-IMAP's "dmail".
To complete this, I think (but I'm new to dovecot!) all we need to adjust in "deliver" is:
ensure that exactly one "From " line is present (add if necessary; don't add if already present);
have the ability (option?) for it to write a "syslog" entry.
I'd be happy to try to concoct a dovecot patch for the above if folk here think the idea is sound.
--
: David Lee I.T. Service : : Senior Systems Programmer Computer Centre : : Durham University : : http://www.dur.ac.uk/t.d.lee/ South Road : : Durham DH1 3LE : : Phone: +44 191 334 2752 U.K. :
On Wednesday 04 October 2006 11:02, David Lee wrote:
Your patch is to procmail to improve its functionality. I recommend you to try to get it incorporated into procmail by its developers.
I'm not sure, but I believe that procmail is unmaintained. My copy is dated 2001.
Offlist mail to this address is discarded unless
"/dev/rob0" or "not-spam" is in Subject: headerOn Wed, 2006-10-04 at 17:02 +0100, David Lee wrote:
- ensure that exactly one "From " line is present (add if necessary; don't add if already present);
Did you figure out this one yet?
Dovecot deliver is the one that needs to write the From_ line. You should just make sure that whatever feeds it the mail didn't already add the From_-line.
I added -f parameter to deliver today, which allows you to specify what address is used in the From-line instead of "dovecot.deliver".
On Sun, 8 Oct 2006, Timo Sirainen wrote:
On Wed, 2006-10-04 at 17:02 +0100, David Lee wrote:
- ensure that exactly one "From " line is present (add if necessary; don't add if already present);
Did you figure out this one yet?
Dovecot deliver is the one that needs to write the From_ line. You should just make sure that whatever feeds it the mail didn't already add the From_-line.
I added -f parameter to deliver today, which allows you to specify what address is used in the From-line instead of "dovecot.deliver".
Many thanks. That "-f ..." sound good. When "rc8" appears I'll try my best to test it as promptly as possible.
To other readers who helped on this thread (and to whom I'm grateful):
Timo says "Dovecot deliver is the one that needs to write the From_ line." Indeed. But that's subtly different from "[...] needs to generate [...]".
Dovecot needs to ensure that a "From_" line is written to the output mbox file. But the contents of that "From_" line should, ideally, represent the envelope sender of the incoming email, and that information simply isn't present, except in a "From_"-like field. (The visible "From: " isn't necessarily the same as the envelope-sender.)
--
: David Lee I.T. Service : : Senior Systems Programmer Computer Centre : : Durham University : : http://www.dur.ac.uk/t.d.lee/ South Road : : Durham DH1 3LE : : Phone: +44 191 334 2752 U.K. :
On Tue, 3 Oct 2006, David Lee wrote:
[...] Second: "deliver" seems to add an unconditional "From " line at the start of each delivery. From sendmail, using the 'n' flag, "Mlocal F=...n..." that is OK. (Although I'm not convinced that "dovecot.deliver", nor its two-space separator (emphasis on "two") from the following date are ideal. Nor, come to that, that line's lack of "@somewhere".)
But from a procmail recipe, I end up with two "From " lines. Surely this is incorrect. How can this be reduced to one? Shouldn't "deliver" ensure that there is only "From " line? [...]
The deeper I look into this, the more convinced I am that there is a real problem in dovecot LDA, part of which is a misunderstanding by dovecot of the "mbox" "From " line and a consequent problem in code structure (not merely in code details).
In what follows, I'll use the term "From_" (with a visible underscore) to refer to the "From " (word followed by a single space) which should actually be in the file.
References to the code are based on rc7.
The mbox format (check, for example, "mail.local" man pages) says that "mbox" delimits its messages by a prepended "From_" line at the start and an appended blank line at the end. (Dovecot does that.)
But it is also clear that the next part of the "From_" line should be both a full email name "some.one@some.here", and also that this represents the sender. But dovecot's fixed text "dovecot.deliver" breaks both those.
Naturally, I then turned to the source code, with a view to patching it. But this, I fear, seems to be much more convoluted than I had hoped.
"deliver.c", near line 525, calls "create_mbox_stream()". This sets up the stream with the fixed "From dovecot.deliver ..." It then opens the mbox, returning result "box". Only then does it attempt to read the incoming email, which may itself already contain a "From_" specification. And that reading itself uses the "box" result which references the erroneous, prematurely-set, fixed string.
What we need is something either:
1a Create the mbox stream, but without yet committing a "From_" spec.
1b Start reading the incoming email. If it already contains a "From_" spec., use that, else create some sort of default (equivalent to the exiting "dovecot.deliver" but as a full email name).
1c Put that "From_" into the mbox stream.
1d Copy remainder of incoming email (without any residual "From_") into the stream (etc.)
or:
2a Read first line of incoming email. If it is a "From_", store it and also remove it from incoming stream, else create some sort of default "From_". (We now have a "From_" variable, and an incoming stream positioned after any leading "From_".)
2b Create mbox stream, very similarly to as at present, but using the "From_" variable.
2c Copy remainder of incoming email (without any residual "From_") into the stream (etc.)
This is probably fixable by someone who knows the code. (But although I've got many years of C "under the belt", I'm new to the dovecot codebase, and this particular problem seems rather deeply embedded in the code's structure.)
Any thoughts? Timo?
--
: David Lee I.T. Service : : Senior Systems Programmer Computer Centre : : Durham University : : http://www.dur.ac.uk/t.d.lee/ South Road : : Durham DH1 3LE : : Phone: +44 191 334 2752 U.K. :
David Lee wrote:
But from a procmail recipe, I end up with two "From " lines. Surely this is incorrect. How can this be reduced to one? Shouldn't "deliver" ensure that there is only "From " line? [...]
The deeper I look into this, the more convinced I am that there is a real problem in dovecot LDA, part of which is a misunderstanding by dovecot of the "mbox" "From " line and a consequent problem in code structure (not merely in code details).
I think the real problem is that you are using two LDA's (Local Delivery Agent) which both assume they are the exclusive LDA. Dovecot's 'deliver' expects to be handed an e-mail directly from the MTA (Message Transfer Agent) directly, not via some other program.
If you are using procmail, you should use procmail exclusively to process the local delivery. If you want to use 'deliver' with server-side rules, you should use the Dovecot SIEVE 'deliver'.
John
-- John Peacock Director of Information Research and Technology Rowman & Littlefield Publishing Group 4501 Forbes Boulevard Suite H Lanham, MD 20706 301-459-3366 x.5010 fax 301-429-5748
On Thu, 5 Oct 2006, John Peacock wrote:
[...] I think the real problem is that you are using two LDA's (Local Delivery Agent) which both assume they are the exclusive LDA. Dovecot's 'deliver' expects to be handed an e-mail directly from the MTA (Message Transfer Agent) directly, not via some other program.
Thanks for the reply.
In what follows "From_" (with an underscore) refers to the "From " (with a space) traditionally seen in email transcations, especially in MBOX.
There are a few different things going on, which inter-relate:
Suppose the MTA itself generates a "From_" which then feeds into 'deliver'. Shouldn't 'deliver' detect this? Furthermore, shouldn't it actually use this preferentially? My own particular use is merely exposing what already seems to be poor, indeed (I would suggest) a bug.
The "From_" generated by 'deliver' is poor. It should be followed by a real email address, representing the originator of the email (possibly the other side of the world). Instead it inserts the fixed string, not even an email address: "dovecot.deliver". This seems to be a double bug: not an email address, nor representative of _sender_. Again, my own particular use is, I think, merely exposing an underlying flaw.
If you are using procmail, you should use procmail exclusively to process the local delivery. If you want to use 'deliver' with server-side rules, you should use the Dovecot SIEVE 'deliver'.
Perhaps. That takes us back to the very opening scene-setting of this thread. Reminder:
Very busy mail service (20,000 users, small farm on Linux email servers accessing NetApp storage via NFS).
Well-established UW-IMAP service. Because of the vital importance of locking (to ensure integrity) this has always carefully ensured that the only software which ever touches the mailbox is the UW-IMAP suite: imapd/pop3d for MUAs; tmail (from sendmail) for simple delivery; dmail (via user .procmailrc recipies) for user-guided delivery. So because simultaneous writes from the IMAP daemon (e.g. message deletion) and from delivery processes use the same UW-IMAP software suite, the locking should be (and seems to be over many years) consistent.
In other words: very cautious and conservative about what touches the mailboxes.
My fundamental question about migrating to dovecot. Following the above model, I feel I ought to use only "deliver" (not mixed with "mail.local", nor "procmail" nor anything else) to touch the mailboxes. With dovecot, is it known to be safe to relax this caution and conservatism (without worry!) and allow other software (along with dovecot imap) to dabble with the mailboxes?
(Even if mixed-supplier software is proven safe, so that I could use "mail.local" and direct "procmail" delivery, I still think dovecot 'deliver' could be improved as suggested earlier.)
Any thoughts?
--
: David Lee I.T. Service : : Senior Systems Programmer Computer Centre : : Durham University : : http://www.dur.ac.uk/t.d.lee/ South Road : : Durham DH1 3LE : : Phone: +44 191 334 2752 U.K. :
My fundamental question about migrating to dovecot. Following the above model, I feel I ought to use only "deliver" (not mixed with "mail.local", nor "procmail" nor anything else) to touch the mailboxes. With dovecot,
Most of my mailboxes are delivered via procmail. The *huge* mailboxes, I use dovecot LDA; as well as a small number of Sieve users are using it. the LDA does caching of headers at delivery time, which is why I use it on huge mailboxes (100,000 messages in the mailbox). Small mailboxes, I don't see so much benefit to using deliver (ie, <10,000 messages).
That's my experience, at least, on a high volume but small number of users mail system.
participants (6)
-
/dev/rob0
-
David Lee
-
Gerhard Wiesinger
-
Jason Fesler
-
John Peacock
-
Timo Sirainen