2 panics, 1 segmentation fault in dovecot 2.3.0 (fts)

[Infoomatic]

Hi,

While testing our migration to dovecot 2.3.0, we encountered 2 panics, 1 segmentation fault while we wanted to rebuild the solr indizes. Problem [1] was encountered while rebuilding the indizes, and now I don't know how many mails/documents are indexed or where it got stuck. [2] and [3] encountered while just playing around, since I did not know of "tokenize" and did not find a lot of information about it.

This is a ubuntu 16.04 machine, dovecot 2.3.0 compiled from source ( + the patch for the recent zlib problem). "doveadm index" runs on a ~ 4GB mdbox-zipped-mailbox for about 3-4 minutes before aborting with the message above, beforehand I did a "/opt/dovecot/bin/doveadm fts rescan -A". When I repeat "doveadm index" it aborts after about 4 seconds, after a "/opt/dovecot/bin/doveadm fts rescan -A" it runs again for 3-4 minutes.

Apache tika logs, but those are probably not related to the problem, however, the devs might want investigate the last line:
WARN  Parse exception Expected ';', got "," determining content disposition
javax.mail.internet.ParseException: Expected ';', got ","
and
org.apache.tika.exception.ZeroByteFileException: InputStream must have > 0 bytes


I would be happy if you could have a look at this, please, and of course just tell me what further information you need.

Thanks,
infoomatic


[1] /opt/dovecot/bin/doveadm index -u user1 '*'
doveadm(user1): Panic: file unichar.c: line 160 (uni_ucs4_to_utf8_c): assertion failed: (uni_is_valid_ucs4(chr))
doveadm(user1): Error: Raw backtrace: /opt/dovecot/lib/dovecot/libdovecot.so.0(+0xc5eea) [0x7ff27755beea] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(+0xc5f59) [0x7ff27755bf59] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7ff2774ce5e1] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(uni_ucs4_to_utf8_c+0xa0) [0x7ff277593300] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(+0xa74f9) [0x7ff27753d4f9] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(mail_html2text_more+0xc5) [0x7ff27753d705] -> /opt/dovecot/lib/dovecot/lib20_fts_plugin.so(+0xcf5c) [0x7ff276460f5c] -> /opt/dovecot/lib/dovecot/lib20_fts_plugin.so(fts_parser_more+0x27) [0x7ff276460c37] -> /opt/dovecot/lib/dovecot/lib20_fts_plugin.so(fts_build_mail+0x511) [0x7ff27645eb51] -> /opt/dovecot/lib/dovecot/lib20_fts_plugin.so(+0x110a6) [0x7ff2764650a6] -> /opt/dovecot/lib/dovecot/libdovecot-storage.so.0(mail_precache+0x2e) [0x7ff27784618e] -> /opt/dovecot/bin/doveadm(+0x2f697) [0x55dd2f912697] -> /opt/dovecot/bin/doveadm(+0x29827) [0x55dd2f90c827] -> /opt/dovecot/bin/doveadm(+0x2a4bb) [0x55dd2f90d4bb] -> /opt/dovecot/bin/doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x21d) [0x55dd2f90e2cd] -> /opt/dovecot/bin/doveadm(doveadm_cmd_run_ver2+0x575) [0x55dd2f91deb5] -> /opt/dovecot/bin/doveadm(doveadm_cmd_try_run_ver2+0x37) [0x55dd2f91df07] -> /opt/dovecot/bin/doveadm(main+0x1d2) [0x55dd2f8fd032] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7ff2770ec830] -> /opt/dovecot/bin/doveadm(_start+0x29) [0x55dd2f8fd439]
Aborted (core dumped)


[2] This was just about my curiosity since I haven't used "tokenize" and did not find too much info about it: 
/opt/dovecot/bin/doveadm fts tokenize -A '*'
Panic: file doveadm-print.c: line 142: unreached
Error: Raw backtrace: /opt/dovecot/lib/dovecot/libdovecot.so.0(+0xc5eea) [0x7f38ac5c2eea] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(+0xc5f59) [0x7f38ac5c2f59] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f38ac5355e1] -> /opt/dovecot/bin/doveadm(+0x3b3d3) [0x55c728e8b3d3] -> /opt/dovecot/bin/doveadm(+0x2a63a) [0x55c728e7a63a] -> /opt/dovecot/bin/doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x21d) [0x55c728e7b2cd] -> /opt/dovecot/bin/doveadm(doveadm_cmd_run_ver2+0x575) [0x55c728e8aeb5] -> /opt/dovecot/bin/doveadm(doveadm_cmd_try_run_ver2+0x37) [0x55c728e8af07] -> /opt/dovecot/bin/doveadm(main+0x1d2) [0x55c728e6a032] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f38ac153830] -> /opt/dovecot/bin/doveadm(_start+0x29) [0x55c728e6a439]
Aborted (core dumped)


[3] /opt/dovecot/bin/doveadm fts tokenize -u user1 "@"
Segmentation fault (core dumped)