[Dovecot] CATENATE/literal8 issue

[Michael M Slusarz]

Quoting Timo Sirainen <tss at iki.fi>:

> On 21.5.2013, at 9.40, Michael M Slusarz <slusarz at curecanti.org> wrote:
>
>> Using 2.2.2, I see this:
>>
>> C: 6 APPEND "INBOX" (\seen) "16-May-2013 22:05:14 -0600" CATENATE  
>> (URL "/INBOX;UIDVALIDITY=1255685337/;UID=48812/;SECTION=HEADER"  
>> TEXT ~{40}
>> S: 6 NO [UNKNOWN-CTE] Binary input allowed only when the first part  
>> is binary.
>>
>> Why is there this limitation?  It seems to me that CATENATE is  
>> confusing the content-type encoding of the data/part itself with  
>> the encoding of the IMAP literal.
>>
>> A literal 8 is nothing more than a series of OCTET's that *may*  
>> contain nulls, but not necessarily.  i.e., in the above example the  
>> 40 octets of data are US-ASCII text, which is perfectly acceptable  
>> to send as a literal8.  (Client rationale: If BINARY exists on the  
>> server, we don't bother to scan IMAP literal's for null data -- we  
>> just send them as literal8's.  It's an optimization that I would  
>> hate to get rid of.)
>
> Well, the problem is that if it does contain NULs, the MIME part  
> needs to be converted to something that doesn't. And to do that it  
> needs to modify the previous header, which with current code was  
> already read..

Is altering the header something that BINARY/CATENATE is allowed to  
do?  Especially regarding the header.  I know there is language about  
the server changing the CTE, but this is potentially troubling since  
cryptographic signatures may rely on the header text.  Changing things  
will break the message.

I can see the server altering the body text to match the header.  But  
I think the reverse is bothersome.

> Or are you saying that the error is fine if the text contains NULs,  
> but simply should be allowed as long as it doesn't?

This.  As mentioned before, it seems the code is simply assuming that  
the text part contains NULs without ever checking it.  My reading of  
the literal8 is that there is no requirement that NULs MUST exist in  
the string.

In our code, the append data is often from code that the IMAP library  
doesn't have access to.  So at APPEND time, it is unaware whether the  
data contains NUL or not - it just has a blob of data and a length.   
If BINARY exists, it is much easier for us to simply send as literal8  
and stream the data - no extra overhead is needed on our side.  Since  
each individual byte need to be handled by the server as it comes in,  
it seems much more efficient to do NUL checking there.

michael