On Jan 29, 2008, at 07:50, Dean Brooks wrote:

On Tue, Jan 29, 2008 at 04:09:26AM +0100, Lennart Lvstrand wrote:

...

I feel like I'm going totally crazy. Is it just me, or have embedded From_ lines really been breaking mbox messages since (at least) dovecot 1.0?

It's trivial to reproduce too -- just mail yourself a message with a valid From_ line in it (assuming that your delivery system isn't
doing

...

From-escaping), or put it in a draft plain text message and save it. Then go and look in your Drafts folder...

The "mbox" format, by definition, uses From_ lines as *the* separator. If it uses anything else, it's not conventional mbox format. There are variants of mbox, sometimes described as mboxcl2 that use Content-Length: as the defining separator, but that is *not* conventional mbox format.

Dovecot's support for the mbox format is described in http://wiki.dovecot.org/MailboxFormat/mbox . It is capable of parsing From_ lines as message delimiters, but
prefers the Content-Length header and will add one if not present (or
rewrite an old one if found incorrect). I guess you might want to
call that mboxcl2 with fallback to traditional mbox to be more specific.

You're correct that solely relying on the presence of (correct) CL
headers can be risky in an environment where old mbox tools may
rewrite the mailboxes, but let us disregard that for a moment. It is
not a concern of mine as I only run one type of IMAP server at the
time and I tend to let it manage all my mailboxes exclusively. (And
yes, I have good control over the incoming mail delivery method as
well.)

I didn't mean to start a discussion on mailbox formats. The point of
my message was that Dovecot effectively seems to be doing the fallback
backwards, i.e. is causing From_ lines to break up messages with
explicit and correct Content-Length headers. I first thought there
might be something wrong with my mbox files or my delivery agent, but
this happens even for new messages that are created directly from
within IMAP.

This might be a better way to illustrate it:

$ telnet localhost imap

• OK [CAPABILITY IMAP4rev1 SASL-IR SORT THREAD=REFERENCES
  MULTIAPPEND UNSELECT LITERAL+ IDLE CHILDREN NAMESPACE LOGIN- REFERRALS STARTTLS AUTH=PLAIN] Dovecot ready. a login lennart xxx a OK Logged in. b create foo b OK Create completed. c append foo {205}

• OK From: Me To: You Subject; Foo Date: Mon, 28 Jan 2008 23:50:00 +0100

Before

From someone Mon Jan 28 23:51:00 2008 From: You To: Me Subject: Bar Date: Mon, 28 Jan 2008 23:52:00 +0100

After

c OK Append completed. d select foo

• FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
• OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)]
  Flags permitted.
• 2 EXISTS
• 0 RECENT
• OK [UNSEEN 1] First unseen.
• OK [UIDVALIDITY 1201610707] UIDs valid
• OK [UIDNEXT 3] Predicted next UID d OK [READ-WRITE] Select completed. e fetch 1 body[text]
• 1 FETCH (FLAGS (\Seen) BODY[TEXT] {8} Before ) e OK Fetch completed. f fetch 2 body[text]
• 2 FETCH (FLAGS (\Seen) BODY[TEXT] {7} After ) f OK Fetch completed.

Note: Two messages were created in the foo mailbox instead of just one.

After the APPEND command was executed, the foo mailbox has the
following expected contents:

From lennart@Mini.SVA.Lovstrand.COM Tue Jan 29 14:18:01 2008 From: Me To: You Subject; Foo Date: Mon, 28 Jan 2008 23:50:00 +0100 X-IMAPbase: 1201612681 0000000001 X-UID: 1 Status: X-Keywords: Content-Length: 122

Before

From someone Mon Jan 28 23:51:00 2008 From: You To: Me Subject: Bar Date: Mon, 28 Jan 2008 23:52:00 +0100

After

But after the SELECT command was issued, it is clear that Dovecot now
thinks there are two messages given the two X-UID & Status headers --
despite there only being a single Content-Length header (now incorrect
because the added headers in the embedded message):

From lennart@Mini.SVA.Lovstrand.COM Tue Jan 29 14:18:01 2008 From: Me To: You Subject; Foo Date: Mon, 28 Jan 2008 23:50:00 +0100 X-IMAPbase: 1201612681 0000000002 X-UID: 1 Status: O X-Keywords: Content-Length: 122

Before

From someone Mon Jan 28 23:51:00 2008 From: You To: Me Subject: Bar Date: Mon, 28 Jan 2008 23:52:00 +0100 X-UID: 2 Status: O

After

This is reproducable under every version from Dovecot 1.0 to
1.1.beta14 that I've tried, but with the added caveat that 1.1.beta14
sometimes seems to get confused when it finds a truncated message and
may abort mid stream.

Cheers, --Lennart

On Sun, 2008-05-04 at 16:59 +0300, Timo Sirainen wrote:

On Tue, 2008-01-29 at 04:09 +0100, Lennart Lövstrand wrote:

...

I feel like I'm going totally crazy. Is it just me, or have embedded
From_ lines really been breaking mbox messages since (at least)
dovecot 1.0?

Finally fixed in v1.1: http://hg.dovecot.org/dovecot-1.1/rev/7871b6219480

And another fix to make that change not break: http://hg.dovecot.org/dovecot-1.1/rev/80d827b411c8