[Dovecot] CATENATE/literal8 issue

[Michael M Slusarz]

Quoting Timo Sirainen <tss at iki.fi>:

> On Wed, 2013-05-22 at 09:38 -0600, Michael M Slusarz wrote:
>> Quoting Michael M Slusarz <slusarz at curecanti.org>:
>>
>> > Quoting Timo Sirainen <tss at iki.fi>:
>> >
>> >> Anyway .. the BINARY APPEND converts only the MIME parts that you
>> >> send with "Content-Transfer-Encoding: binary". Are you sending such
>> >> header to Dovecot?
>>
>> I can verify this isn't working as you described above:
>>
>> 1 APPEND "INBOX" CATENATE (TEXT {49+}
>> Content-Type: multipart/alternative; boundary="A" TEXT ~{1}
>> 1 NO [UNKNOWN-CTE] Binary input allowed only when the first part is binary.
>
> What do you do then if server advertises CATENATE but not BINARY?

Send as a regular literal.  If there truly are nulls in the output,  
there's not much we can do so we send as-is and hope for the best.

> Anyway for the other possibilities Dovecot could:
>
> a) Put all CATENATEd messages through the istream-binary-converter, but
> just not do any actual C-T-E:binary conversion until the first ~{binary}
> part is found.
>
> b) Just treat ~{n} exactly the same as ~{n}, unless it's the first part
> of CATENATE.
>
> Maybe this should be aked about in IMAP mailing list .. (Didn't I
> already ask something about CATENATE+BINARY combination?)

Yeah:  
http://mailman2.u.washington.edu/pipermail/imap-protocol/2012-June/001787.html  
   No responses :)

It is concerning because RFC 4466 indicates that literal8's are  
allowed for both APPEND and MULTIAPPEND, which is essentially an  
extended APPEND.  But RFC 4469 defines CATENATE TEXT as literal only:

RFC 4466:
    append-data     = literal / literal8 / append-data-ext

RFC 4469:
    append-data =/ "CATENATE" SP "(" cat-part *(SP cat-part) ")"
    cat-part = text-literal / url
    text-literal = "TEXT" SP literal

To me CATENATE =~ MULTIAPPEND - it is just another form of an extended  
APPEND.  Not sure why it shouldn't be allowed there.  But from a  
strict ABNF standpoint, you are correct that I shouldn't be sending  
literal8's.  I'll ask myself on the IMAP list why this design choice  
was made.

For the record... given the varying levels of BINARY support in  
different IMAP servers (UW IMAP is flat-out broken), I've gone ahead  
and bit the bullet and we now pre-scan outgoing append literals for  
null characters and only use literal8's when absolutely necessary.  I  
was probably being too clever for my own good in assuming that I can  
just send and assume the server will handle all issues.

With that being said... I was able to reliably reproduce a parsing  
issue in Dovecot 2.2.x when doing a MULTIAPPEND w/literal8's.  I need  
to track down if this is a single message causing the issue or some  
sort of cumulative bug that only appears once you've done something  
like 200-300 sequential appends.  I can verify that a switch from  
literal8 -> literal fixes the issue.  I'll try to create a  
reproducible test case.

michael