[Dovecot] Embedded From_ lines breaking Content-Length (and Dovecot)

[Lennart Lövstrand]

On Jan 29, 2008, at 07:50, Dean Brooks wrote:

> On Tue, Jan 29, 2008 at 04:09:26AM +0100, Lennart Lvstrand wrote:
>
>> I feel like I'm going totally crazy.  Is it just me, or have embedded
>> From_ lines really been breaking mbox messages since (at least)
>> dovecot 1.0?
>>
>> It's trivial to reproduce too -- just mail yourself a message with a
>> valid From_ line in it (assuming that your delivery system isn't  
>> doing
>>> From-escaping), or put it in a draft plain text message and save
>> it.  Then go and look in your Drafts folder...
>
> The "mbox" format, by definition, uses From_ lines as *the* separator.
> If it uses anything else, it's not conventional mbox format.  There
> are variants of mbox, sometimes described as mboxcl2 that use
> Content-Length: as the defining separator, but that is *not*
> conventional mbox format.

Dovecot's support for the mbox format is described in http://wiki.dovecot.org/MailboxFormat/mbox 
.  It is capable of parsing From_ lines as message delimiters, but  
prefers the Content-Length header and will add one if not present (or  
rewrite an old one if found incorrect).  I guess you might want to  
call that mboxcl2 with fallback to traditional mbox to be more specific.

You're correct that solely relying on the presence of (correct) CL  
headers can be risky in an environment where old mbox tools may  
rewrite the mailboxes, but let us disregard that for a moment.  It is  
not a concern of mine as I only run one type of IMAP server at the  
time and I tend to let it manage all my mailboxes exclusively.  (And  
yes, I have good control over the incoming mail delivery method as  
well.)

I didn't mean to start a discussion on mailbox formats.  The point of  
my message was that Dovecot effectively seems to be doing the fallback  
backwards, i.e. is causing From_ lines to break up messages with  
explicit and correct Content-Length headers.  I first thought there  
might be something wrong with my mbox files or my delivery agent, but  
this happens even for new messages that are created directly from  
within IMAP.

This might be a better way to illustrate it:

> $ telnet localhost imap
> * OK [CAPABILITY IMAP4rev1 SASL-IR SORT THREAD=REFERENCES  
> MULTIAPPEND UNSELECT LITERAL+ IDLE CHILDREN NAMESPACE LOGIN- 
> REFERRALS STARTTLS AUTH=PLAIN] Dovecot ready.
> a login lennart xxx
> a OK Logged in.
> b create foo
> b OK Create completed.
> c append foo {205}
> + OK
> From: Me
> To: You
> Subject; Foo
> Date: Mon, 28 Jan 2008 23:50:00 +0100
>
> Before
>
> From someone  Mon Jan 28 23:51:00 2008
> From: You
> To: Me
> Subject: Bar
> Date: Mon, 28 Jan 2008 23:52:00 +0100
>
> After
>
> c OK Append completed.
> d select foo
> * FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
> * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)]  
> Flags permitted.
> * 2 EXISTS
> * 0 RECENT
> * OK [UNSEEN 1] First unseen.
> * OK [UIDVALIDITY 1201610707] UIDs valid
> * OK [UIDNEXT 3] Predicted next UID
> d OK [READ-WRITE] Select completed.
> e fetch 1 body[text]
> * 1 FETCH (FLAGS (\Seen) BODY[TEXT] {8}
> Before
> )
> e OK Fetch completed.
> f fetch 2 body[text]
> * 2 FETCH (FLAGS (\Seen) BODY[TEXT] {7}
> After
> )
> f OK Fetch completed.

Note: Two messages were created in the foo mailbox instead of just one.

After the APPEND command was executed, the foo mailbox has the  
following expected contents:

> From lennart at Mini.SVA.Lovstrand.COM  Tue Jan 29 14:18:01 2008
> From: Me
> To: You
> Subject; Foo
> Date: Mon, 28 Jan 2008 23:50:00 +0100
> X-IMAPbase: 1201612681 0000000001
> X-UID: 1
> Status:
> X-Keywords:
> Content-Length: 122
>
> Before
>
> From someone  Mon Jan 28 23:51:00 2008
> From: You
> To: Me
> Subject: Bar
> Date: Mon, 28 Jan 2008 23:52:00 +0100
>
> After

But after the SELECT command was issued, it is clear that Dovecot now  
thinks there are two messages given the two X-UID & Status headers --  
despite there only being a single Content-Length header (now incorrect  
because the added headers in the embedded message):

> From lennart at Mini.SVA.Lovstrand.COM  Tue Jan 29 14:18:01 2008
> From: Me
> To: You
> Subject; Foo
> Date: Mon, 28 Jan 2008 23:50:00 +0100
> X-IMAPbase: 1201612681 0000000002
> X-UID: 1
> Status: O
> X-Keywords:
> Content-Length: 122
>
> Before
>
> From someone  Mon Jan 28 23:51:00 2008
> From: You
> To: Me
> Subject: Bar
> Date: Mon, 28 Jan 2008 23:52:00 +0100
> X-UID: 2
> Status: O
>
> After

This is reproducable under every version from Dovecot 1.0 to  
1.1.beta14 that I've tried, but with the added caveat that 1.1.beta14  
sometimes seems to get confused when it finds a truncated message and  
may abort mid stream.

Cheers,
--Lennart