[Dovecot] message-decoder bug for attachments with charset=binary attribute in content-type?

12 May 2014


      Hello,
I have configure dovecot with solr and I wanted to let solr index content of
attachments. For testing I have used biabam command line tool to generate
emails with attachments.
I have found that dovecot with fts_decoder incorrectly decodes these
attachments from biabam and therefore pdftotext has reported corrupted PDF.
The problem is that biabam generates header with charset=binary and dovecot
message decoder tries to process it as UTF8 or non-UTF8 data.
============================================================
--biabam.ZxWVLybiabam.ZxWVLy
Content-Type: application/pdf; charset=binary
Content-Disposition: attachment; filename="bacula-jobs.pdf"
Content-Transfer-Encoding: base64
JVBERi0xLjQKJcfsj6IKNSAwIG9iago8PC9MZW5ndGggNiAwIFIvRmlsdGVy
IC9GbGF0ZURlY29kZT4+CnN0cmVhbQp4nF2PT0+EMBDF7/0U7yYYWdqFXdbe
1vgnMXpQezMeClSoQNltweh+egvLyczhN3kz703mCLpioFMtLDoSv2aoHKGo
.....
This PDF begins orginal with
============================================================
0000000: 2550 4446 2d31 2e34 0a25 c7ec 8fa2 0a35  %PDF-1.4.%.....5
0000010: 2030 206f 626a 0a3c 3c2f 4c65 6e67 7468   0 obj.<>.str
But the dovecot pass following data to fts_decoder script:
============================================================
0000000: 2550 4446 2d31 2e34 0a25 c3a4 c3bc c3b6  %PDF-1.4.%......
0000010: c39f 0a32 2030 206f 626a 0a3c 3c2f 4c65  ...2 0 obj.<>.
As you can see binary data are mangled.
Alpine and Thunderbird do not write charset=binary to content-type header and
searching works perfect.
I have searched in source code and I have found one place. If I replace the
following code in file dovecot-2.1.7/src/lib-mail/message-decoder.c on line 241
with new one, the dovecots message decoder decodes message correctly and
pdftotext can convert attached PDF.
Original code:
241:   ctx->binary_input = ctx->content_charset == NULL &&
242:     (ctx->flags & MESSAGE_DECODER_FLAG_RETURN_BINARY) != 0 &&
243:     (part->flags & (MESSAGE_PART_FLAG_TEXT |
244:         MESSAGE_PART_FLAG_MESSAGE_RFC822)) == 0;
My update:
241   ctx->binary_input = ((ctx->content_charset != NULL) && (strcasecmp(ctx->content_charset, "binary") == 0)) || (ctx->content_charset == NULL &&
242     (ctx->flags & MESSAGE_DECODER_FLAG_RETURN_BINARY) != 0 &&
243     (part->flags & (MESSAGE_PART_FLAG_TEXT |
244         MESSAGE_PART_FLAG_MESSAGE_RFC822)) == 0);
This will set ctx->binary_input for the attachment with charset set to
"binary".
I don't know if this is correct update, but the searching works with this
update for biabam binary attachments too.
Could you please verify this problem and maybe update the code?
Thank you very much.
# dovecot --version
2.1.7
Config:
plugin {
fts = solr
fts_solr = url=http://localhost:8080/solr/
fts_decoder = decode2text
}
service decode2text {
executable = script /etc/dovecot/scripts/decode2text.sh
user = dovecot
unix_listener decode2text {
mode = 0666
}
}
Regards,
Robert Wolf.

[Dovecot] message-decoder bug for attachments with charset=binary attribute in content-type?

Robert Wolf

JVBERi0xLjQKJcfsj6IKNSAwIG9iago8PC9MZW5ndGggNiAwIFIvRmlsdGVy IC9GbGF0ZURlY29kZT4+CnN0cmVhbQp4nF2PT0+EMBDF7/0U7yYYWdqFXdbe 1vgnMXpQezMeClSoQNltweh+egvLyczhN3kz703mCLpioFMtLDoSv2aoHKGo .....

============================================================ 0000000: 2550 4446 2d31 2e34 0a25 c7ec 8fa2 0a35 %PDF-1.4.%.....5 0000010: 2030 206f 626a 0a3c 3c2f 4c65 6e67 7468 0 obj.<>.str

============================================================ 0000000: 2550 4446 2d31 2e34 0a25 c3a4 c3bc c3b6 %PDF-1.4.%...... 0000010: c39f 0a32 2030 206f 626a 0a3c 3c2f 4c65 ...2 0 obj.<>.

Original code:

241: ctx->binary_input = ctx->content_charset == NULL && 242: (ctx->flags & MESSAGE_DECODER_FLAG_RETURN_BINARY) != 0 && 243: (part->flags & (MESSAGE_PART_FLAG_TEXT | 244: MESSAGE_PART_FLAG_MESSAGE_RFC822)) == 0;

My update:

241 ctx->binary_input = ((ctx->content_charset != NULL) && (strcasecmp(ctx->content_charset, "binary") == 0)) || (ctx->content_charset == NULL && 242 (ctx->flags & MESSAGE_DECODER_FLAG_RETURN_BINARY) != 0 && 243 (part->flags & (MESSAGE_PART_FLAG_TEXT | 244 MESSAGE_PART_FLAG_MESSAGE_RFC822)) == 0);