I realize (having read the RFC), that IMAP does not deal with the envelope "From " at all. I also realize that IMAP calls something else the "envelope", so I'll refer to it as the Unix (mbox) "From " line.

However, the mbox format does store those lines, and they even still have some meaning. If users access their mailboxes both through IMAP and through mbox files in ~/mail/ (e.g. elm/mutt/pine/mail -f), (which I assume is a rather common deployment given the option of storing mail in mbox files, and anyway is the idea in my intended deployment) compatibility should be preserved wherever possible.

When storing (copying/moving) a message from one mbox to another (using the IMAP COPY command), wouldn't it make more sense for dovecot to copy the entire message block, including the "From " line at the beginning, instead of inventing/writing its own (phony) "From <username>@<server>..." line and copying only the rest?

I tried to see if I can patch dovecot to the expected logical behaviour, however from the little I looked, I got a bit overwhelmed and confused by all the structs, with the mail_save_context and istream stuff. The only simple/trivial thing I could is have it, when copying a message, given the pointer to the data in the stream, search backward for a "From " line and copy from there. That seems not to be the most elegant solution...

So, any help with getting my expected behaviour from dovecot is still appreciated...

Thanks, -- Tom

On Mon, Apr 26, 2004 at 12:40:58AM +0300, Timo Sirainen wrote:

On Fri, 2004-04-23 at 13:48, Tom Alsberg wrote:

...

Hi there.

I am evaluating Dovecot as an IMAP server. I noticed on thing which irritates me. When a user saves a message to an IMAP folder with the Dovecot server, the envelope From line is changed to the address of that user, instead of keeping the From line of the original message.

How can I have the original envelope From line preserved when saving messages?

I'm not quite sure what you mean. The mbox From-line? From where should it keep it? IMAP protocol doesn't provide IMAP server that information, so Dovecot just adds the user name.

On Mon, Apr 26, 2004 at 11:43:59AM +0200, Miquel van Smoorenburg wrote:

On 2004.04.26 09:03, Tom Alsberg wrote:

...

I realize (having read the RFC), that IMAP does not deal with the envelope "From " at all. I also realize that IMAP calls something else the "envelope", so I'll refer to it as the Unix (mbox) "From " line.

Often called the "From_" line as well

...

However, the mbox format does store those lines, and they even still have some meaning.

Why not let your MDA or local delivery agent add a proper Return-Path: header. It contains the same information as the From_ line, but it works with all mailbox types, it's easier to parse and you can get at it using IMAP.

That is a good positive suggestion about how to maintain "envelope from." I don't think it counters the desire *not* to muck with the existing "From_" line though. I would not want that disturbed without some very good reason (and I can't really think of one).

mm

Comment below:

On Mon, Apr 26, 2004 at 11:43:59AM +0200, Miquel van Smoorenburg wrote:

On 2004.04.26 09:03, Tom Alsberg wrote:

...

<snip /> However, the mbox format does store those lines, and they even still have some meaning.

Why not let your MDA or local delivery agent add a proper Return-Path: header. It contains the same information as the From_ line, but it works with all mailbox types, it's easier to parse and you can get at it using IMAP.

Well, my MTA, as most others, does add a Return-Path: header. Usually it contains the same information as the From_ line, but not always, IIRC (the Return-Path: header, IIRC, may be altered/added to by an SMTP relay in the way, while the envelope From_ line is guaranteed to remain as it was). Never mind that, however - naturally I can have the same information stored also in the headers (X-Envelope-From: or something like that).

However, I would still like the From_ line to remain intact - I see no reason to rewrite it while copying the message, instead of keeping the original one. Actually, some user agents and other software even do read it and regard it (for filtering, marking messages, sorting and merging mailboxes, etc.), so it disturbs a bit when that's changed.

To note it - IMAP is not the only way -- at least in my intended installation -- that users access their mailboxes in. Wherever possible, the IMAP and the local mailbox access should be synchronized, and using one should not disturb the usage of the other. It is a bad assumption in my opinion, therefore, that the From_ line is completely irrelevant (and needs not be preserved) only because it is not visible in IMAP.

Maybe it isn't important to many other users, but if it is irrelevant to IMAP, any reason not to preserve it instead of rewriting it? I haven't tried any further yet, but would appreciate some tip on how to cleanly patch dovecot to do so (searching backwards for the From_ line is rather ugly...), or just have it "fixed" in CVS/the next version...

Mike.

Thanks again, -- Tom

On Tue, Apr 27, 2004 at 12:39:24PM +0200, Miquel van Smoorenburg wrote:

On 2004.04.27 09:53, Tom Alsberg wrote:

...

<snip /> Well, my MTA, as most others, does add a Return-Path: header. Usually it contains the same information as the From_ line, but not always,

I think that would be a bug.

Now that I checked it, you're right here...

...

IIRC (the Return-Path: header, IIRC, may be altered/added to by an SMTP relay in the way

No, the Return-Path header is not present at SMTP transport time. It is added the moment the message leaves the SMTP domain, i.e. the moment it drops into a local mbox (or is sent through another transport like UUCP).

True. Sorry for the confusion - had to refresh my understanding.

That is the same moment a From_ header is added, and the content should be the same.

Well, the From_ line is not a header. It is stored in Unix (traditional mbox style) mailboxes, and originally (as in - early Unix mail withing the host, where the "mail" command would just append the message into /var/mail/<username>, before Intenet/ARPAnet mail embraced with SMTP, etc.) was the only line stored except the message body, containing the username of the sender and the date. Nowadays, of course, it contains the full e-mail address (<username>@<domain>) in case the message travelled through the net.

...

while the envelope From_ line is guaranteed to remain as it was).

The From_ line is only retained over UUCP, no other transport has a From_ line in the message. SMTP certainly doesn't.

SMTP doesn't have a From_ line in the message, as it is not really part of the message, like the headers and body are. It is part of the SMTP protocol however, in a slightly different form:

In the SMTP protocol, sending a message begins with "MAIL FROM

<address>", and if the message goes through another server in the way, the relay over it will also begin with the same "MAIL FROM <address>" command. That, AFAIK, is called the SMTP envelope. At the end-point MTA, the FROM part of the SMTP envelope is written in the form of a "From " line. However, that's the same thing - so SMTP does actually retain that line in the protocol.

"MAIL FROM <address>" is the direct protocol to the "From " line leading a message in a traditional mbox file.

Well, when COPYing mailboxes over IMAP dovecot should be able to retain the From_ line I suppose

That's what I mean - when the IMAP COPY command is issued, the From_ line should be copied together with the rest of the actual message.

(if the internal API is rich enough) - but if the MUA does something like get message from folder A and write it to folder B (which many clients still do) the From_ line gets lost.

Well, if the MUA does this, then naturally dovecot can't do much else, so it makes sense that it invents its own line (although, as you mentioned, it could use the Return-Path: header). But it appears that most MUAs issue the COPY command properly.

I suppose the mbox folder code could get the basic info for the From_ line from the Return-Path: header if it is present. That's what my own mbox library for internal projects does...

That's possible. But I'd prefer it if when COPYing a message, it would just copy the "From " line verbatim as well, and not get into this trouble, even though I see now that the content should normally be the same (leave those semantics to the MTA...).

Mike.

Thanks, -- Tom

Hi there. Please read my comments below:

On Tue, Apr 27, 2004 at 10:49:51AM -0400, Amelia A Lewis wrote:

On Tue, Apr 27, 2004 at 03:53:39PM +0300, Tom Alsberg wrote:

...

That's possible. But I'd prefer it if when COPYing a message, it would just copy the "From " line verbatim as well, and not get into this trouble, even though I see now that the content should normally be the same (leave those semantics to the MTA...).

Okay, so this is the IMAP COPY command, right?

Yes...

Why are you putting mbox "From " lines into my maildir?

Uhmm, I am not. It should of course only copy it together with the "From " line if copied to an mbox...

If dovecot is the source, how does it know what format the target mailbox is in?

Well, it obviously knows what format the target mailbox is in already, as it needs it to be able to write to it properly anyway...

Should dovecot instances communicate with one another?

I'm not sure I understand you.

Well, from what I've seen in the code, in no occassion does the COPY command work between processes. When COPYing a message, it just reads the message (from an istream or something like that, don't remember exactly right now), and writes the message to the target mailbox (in the same process - called from within the same routine), and in the case of mbox, the writing routine first calls mbox_from_create to write the "From " line.

It could, when COPYing from an mbox to and mbox, copy the message together with the "From " line to the target mbox, instead of calling mbox_from_create to write a new "From " line and then copying the rest.

Given that every user can only access his mailboxes, and there is one process per connected user, I fail to see how any interaction between dovecot instances is necessary.

Is this only for mailboxes that a single dovecot instance is in charge of?

What do you mean by a "dovecot instance"? Do you mean one server, or one forked process, or what?

So the move, although issued as IMAP, should behave differently for a move between two mailboxes on one server than it does between mailboxes on two servers?

Can IMAP move messages between servers? (Can IMAP servers communicate amongst them?) I don't think that I understand...

As far as I understood IMAP (from the RFC, docs, etc.), within one session, it only operates on a defined mailbox/folder space (the folders accessible to the user after he's authenticated). There's no sending of messages through IMAP (modifying/appending to own mailboxes doesn't count)...

If dovecot is moving messages from one folder (that it "owns") to another folder (that it "owns"), how much of the format-specific information should be copied, if it has been instructed to do an IMAP copy? When does the difference in semantics between a server-internal IMAP COPY versus an inter-server IMAP COPY become anti-intuitive?

You'll have to explain me what I'm missing... I don't understand what you mean by an "inter-server IMAP COPY". However, if something like that exists, I still believe that when possible (e.g. the COPY is between two mbox folders in the same server, which is usually the case in a server based solely on mbox), it should preserve the format (the "From " line).

Modifying the "From " line with no reason is counter-intuitive in the first place... If users in the installation access their mailboxes only by IMAP, though, then it does not matter, and then some inconsistency (preserving the "From " line only when possible) does not hurt since it is not visible at all.

Amy!

I'm sorry for my ignorance, enlighten me... -- Tom

On Tue, Apr 27, 2004 at 04:25:06PM -0400, Amelia A Lewis wrote:

...

So, fine, implement for case 1, and leave the others alone, right? Well. A user could then reasonably ask "Why does my data change when I copy it from my ISP's machine to Uni's Cyrus server? It doesn't change when I copy it from folder to folder on my ISP!" I can't say that I much like the idea of a network protocol command that has enhancements to behave like a local file copy.

...

I think you're making way too much of this. The question was just "why is my existing "From_" line being altered when copying from one mbox to another." I say, why indeed. In all the cases where it needs to be fabricated, by all means, fabricate it. Where it already exists in the source and should also exist in the destination, try to preserve it. That's all.

mm

On Tue, 27 Apr 2004 23:44:07 +0300 Timo Sirainen tss@iki.fi wrote:

On Tue, 2004-04-27 at 23:25, Amelia A Lewis wrote:

...

"From " line, since it isn't a header. So, if dovecot implements it, then there are four possibilities:

1. IMAP COPY from serverA (dovecot) to serverA (dovecot)
2. IMAP COPY from serverA (dovecot) to serverB (dovecot)
3. IMAP COPY from serverA (dovecot) to serverC (not-dovecot)
4. IMAP COPY from serverC (not-dovecot) to ServerA (or B) (dovecot)

2-4 cases aren't COPY commands, it's client which does FETCH to serverA and APPEND to serverB. Unless you mean some kind of clustered server setup such as Cyrus Murder, but that doesn't have much to do with standard IMAP anymore.

My bad then. I'd thought that the copies actually worked server to server, rather than server to client to server.

Amy!

Amelia A. Lewis amyzing {at} talsever.com Did you exchange a walk-on part in the war for the lead role in a cage? -- Pink Floyd

On Tue, Apr 27, 2004 at 09:48:31PM +0300, Timo Sirainen wrote:

On Tue, 2004-04-27 at 15:53, Tom Alsberg wrote:

...

That's what I mean - when the IMAP COPY command is issued, the From_ line should be copied together with the rest of the actual message.

I don't think I'll want to do the copying exactly,

That's a shame... Any tips, though, on how I could change it in the least ugly-hackish way? (I'd like to see and know in order to understand the internal architecture of dovecot anyway, even if you are soon to implement it with the read and save API into the mainstream code track)

as the way COPY currently works is "read message" and "save message", although maildir has special code for hardlink-copying. I don't think mbox really deserves that..

Well, OK... Not sure why, though - I'd think this would be a small piece of code specific to mbox (if done right), and would warrant a special case.

However, I could change read and save APIs so that the "envelope from" can be read and written. With maildir reading would give NULL, and writing would just ignore it. With mbox it would work right though, and if it's NULL it'd do what it does now.

Sounds like a good idea. I'd also think of trying to base on Return-Path: if it's NULL when writing to an mbox, before dropping to inventing a completely fabricated one, but that's just an added bonus...

Any idea as to when I could expect to have this implemented in dovecot? (I'd love to help with it, given some guidance)

Main question in my mind now about the code: What data is kept in the context of a message, except for the istream pointing to the contents? (As in - where would you insert that "envelope from" field? Would it just be a parameter to the functions, or would it be some field in the data structure representing a message?)

I understood that in the current development dovecot, mbox support doesn't even work anymore... I do hope that'll be fixed when it becomes the current stable branch.

I must say that until now I really like dovecot, being the best IMAP server I've found until now (most suitable to my needs, but also clean and flexible, functioning well...). Hopefully mbox support won't just be dropped at some point, as it is important for me, and I believe for other users as well.

I understood the direction of development right now is into making dovecot as modular as feasible, so maybe even if mbox support won't be in the main dovecot code, it could be an external module (with a clean enough API, I could even write and maintain an mbox module)

Thank you, Timo, and the rest of the dovecot developers, for all the efforts into this neat piece of software :-)

Thanks again, -- Tom

On Tue, 2004-05-04 at 12:57, Tom Alsberg wrote:

So, Timo and others -

Any ideas, status, expectations, or hints, regarding this?

I haven't looked much since, but I'd like to fix it, and wouldn't want to do it ugly (back-searching "\nFrom ")... Where shall I try, or is a fix (1.0 mbox support?) in its way?

What I was going to do to 1.0 is make it read the address from From-line and make it available with mail->get_special(mail, MAIL_FETCH_FROM_ENVELOPE). Then add char *from_envelope to save() function, and make it save that for mbox if it's given. Then just make COPY command fetch it and feed it to save().

mbox code is still completely broken in 1.0-test releases. I was going to begin working on that soon.