Aw: Re: 2 panics, 1 segmentation fault in dovecot 2.3.0 (fts)

[Infoomatic]

Thanks Aki! The fix works! Will give you the debug stuff of the other error next week.

Regards,
Infoomatic

> Gesendet: Samstag, 27. Januar 2018 um 20:38 Uhr
> Von: "Aki Tuomi" <aki.tuomi at dovecot.fi>
> An: "Dovecot ML" <dovecot at dovecot.org>, Infoomatic <infoomatic at gmx.at>
> Betreff: Re: 2 panics, 1 segmentation fault in dovecot 2.3.0 (fts)
>
> 
> > On January 27, 2018 at 4:19 PM Infoomatic <infoomatic at gmx.at> wrote:
> > 
> > 
> > Hi,
> > 
> > While testing our migration to dovecot 2.3.0, we encountered 2 panics, 1 segmentation fault while we wanted to rebuild the solr indizes. Problem [1] was encountered while rebuilding the indizes, and now I don't know how many mails/documents are indexed or where it got stuck. [2] and [3] encountered while just playing around, since I did not know of "tokenize" and did not find a lot of information about it.
> > 
> > This is a ubuntu 16.04 machine, dovecot 2.3.0 compiled from source ( + the patch for the recent zlib problem). "doveadm index" runs on a ~ 4GB mdbox-zipped-mailbox for about 3-4 minutes before aborting with the message above, beforehand I did a "/opt/dovecot/bin/doveadm fts rescan -A". When I repeat "doveadm index" it aborts after about 4 seconds, after a "/opt/dovecot/bin/doveadm fts rescan -A" it runs again for 3-4 minutes.
> > 
> > Apache tika logs, but those are probably not related to the problem, however, the devs might want investigate the last line:
> > WARN  Parse exception Expected ';', got "," determining content disposition
> > javax.mail.internet.ParseException: Expected ';', got ","
> > and
> > org.apache.tika.exception.ZeroByteFileException: InputStream must have > 0 bytes
> > 
> > 
> > I would be happy if you could have a look at this, please, and of course just tell me what further information you need.
> > 
> > Thanks,
> > infoomatic
> > 
> > 
> > [1] /opt/dovecot/bin/doveadm index -u user1 '*'
> > doveadm(user1): Panic: file unichar.c: line 160 (uni_ucs4_to_utf8_c): assertion failed: (uni_is_valid_ucs4(chr))
> > doveadm(user1): Error: Raw backtrace: /opt/dovecot/lib/dovecot/libdovecot.so.0(+0xc5eea) [0x7ff27755beea] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(+0xc5f59) [0x7ff27755bf59] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7ff2774ce5e1] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(uni_ucs4_to_utf8_c+0xa0) [0x7ff277593300] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(+0xa74f9) [0x7ff27753d4f9] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(mail_html2text_more+0xc5) [0x7ff27753d705] -> /opt/dovecot/lib/dovecot/lib20_fts_plugin.so(+0xcf5c) [0x7ff276460f5c] -> /opt/dovecot/lib/dovecot/lib20_fts_plugin.so(fts_parser_more+0x27) [0x7ff276460c37] -> /opt/dovecot/lib/dovecot/lib20_fts_plugin.so(fts_build_mail+0x511) [0x7ff27645eb51] -> /opt/dovecot/lib/dovecot/lib20_fts_plugin.so(+0x110a6) [0x7ff2764650a6] -> /opt/dovecot/lib/dovecot/libdovecot-storage.so.0(mail_precache+0x2e) [0x7ff27784618e] -> /opt/dovecot/bin/doveadm(+0x2f697) [0x55dd2f912697] -> /opt/dovecot/bin/doveadm(+0x29827) [0x55dd2f90c827] -> /opt/dovecot/bin/doveadm(+0x2a4bb) [0x55dd2f90d4bb] -> /opt/dovecot/bin/doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x21d) [0x55dd2f90e2cd] -> /opt/dovecot/bin/doveadm(doveadm_cmd_run_ver2+0x575) [0x55dd2f91deb5] -> /opt/dovecot/bin/doveadm(doveadm_cmd_try_run_ver2+0x37) [0x55dd2f91df07] -> /opt/dovecot/bin/doveadm(main+0x1d2) [0x55dd2f8fd032] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7ff2770ec830] -> /opt/dovecot/bin/doveadm(_start+0x29) [0x55dd2f8fd439]
> > Aborted (core dumped)
> > 
> > 
> 
> You can fix this with https://github.com/dovecot/core/commit/5c1837529e6957da3e389683c43bd006859395e5.patch
> 
> > [2] This was just about my curiosity since I haven't used "tokenize" and did not find too much info about it: 
> > /opt/dovecot/bin/doveadm fts tokenize -A '*'
> > Panic: file doveadm-print.c: line 142: unreached
> > Error: Raw backtrace: /opt/dovecot/lib/dovecot/libdovecot.so.0(+0xc5eea) [0x7f38ac5c2eea] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(+0xc5f59) [0x7f38ac5c2f59] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f38ac5355e1] -> /opt/dovecot/bin/doveadm(+0x3b3d3) [0x55c728e8b3d3] -> /opt/dovecot/bin/doveadm(+0x2a63a) [0x55c728e7a63a] -> /opt/dovecot/bin/doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x21d) [0x55c728e7b2cd] -> /opt/dovecot/bin/doveadm(doveadm_cmd_run_ver2+0x575) [0x55c728e8aeb5] -> /opt/dovecot/bin/doveadm(doveadm_cmd_try_run_ver2+0x37) [0x55c728e8af07] -> /opt/dovecot/bin/doveadm(main+0x1d2) [0x55c728e6a032] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f38ac153830] -> /opt/dovecot/bin/doveadm(_start+0x29) [0x55c728e6a439]
> > Aborted (core dumped)
> >
> 
> This one sounds like a bug, we'll look into this.
>  
> > 
> > [3] /opt/dovecot/bin/doveadm fts tokenize -u user1 "@"
> > Segmentation fault (core dumped)
> >
> 
> Can you run this in gdb with debug symbols and provide bt full?
> 
> Aki
>