v2.3.11.3 solr plugin search via MUA fails to match accented ascii characters; cmd line exec of `doveadm fts lookup` PANICs (assertion failed) [proposed patch]

[John Fawcett]

sending to the list as well

On 02/11/2020 17:40, PGNet Dev wrote:

> On 11/1/20 10:35 AM, John Fawcett wrote:
>> Yes, getting more data about any potential problem would be useful.
>>
>> Just to clarify: I have a fully working search setup for some time now
>> over various dovecot releases, so no patches needed to get it working.
>>
>> My setup does use fts plugin and fts-solr plugin, but it does not use
>> lib-fts functionality (that has many features for example it was
>> stopping you indexing excluded words like tambien). On my setup without
>> lib-fts everything goes to solr which does the work of indexing without
>> all the features of lib-fts.
>
> withOUT libfts
>
> -    fts_solr = url=https://solr.presence-group.net:8984/solr/dovecot/
> use_libfts soft_commit=yes batch_size=250
> +    fts_solr = url=https://solr.presence-group.net:8984/solr/dovecot/
> soft_commit=yes batch_size=250
>
> and unmodified dovecot-provided schema/config,
>
>     /bin/cp -af /usr/share/doc/dovecot/solr-config-7.7.0.xml
> /path/to/solr/data/dovecot/conf/solrconfig.xml
>     /bin/cp -af /usr/share/doc/dovecot/solr-schema-7.7.0.xml
> /path/to/solr/data/dovecot/conf/schema.xml
>
> i suspect my config's now more similar to yours.
>
> checking,
>
>     doveadm fts rescan -u testuser at example.com
>     doveadm index      -u testuser at example.com -q '*'
>
> as before
>
>     doveadm fts lookup -u testuser at example.com body "tésting"
>
> panics,
>
>     doveadm(testuser at example.com): Panic: file mail-storage.c: line
> 2112 (mailbox_get_open_status): assertion failed: (box->opened)
>     doveadm(testuser at example.com): Error: Raw backtrace:
> /usr/lib64/dovecot/libdovecot.so.0(backtrace_append+0x46)
> [0x7f7829b81cc6] ->
> /usr/lib64/dovecot/libdovecot.so.0(backtrace_get+0x22)
> [0x7f7829b81de2] -> /usr/lib64/dovecot/libdovecot.so.0(+0x10025b)
> [0x7f7829b8b25b] -> /usr/lib64/dovecot/libdovecot.so.0(+0x100297)
> [0x7f7829b8b297] -> /usr/lib64/dovecot/libdovecot.so.0(+0x59bc6)
> [0x7f7829ae4bc6] ->
> /usr/lib64/dovecot/libdovecot-storage.so.0(+0x4779e) [0x7f7829c9879e]
> -> /usr/lib64/dovecot/lib21_fts_solr_plugin.so(+0x5849)
> [0x7f78296ea849] ->
> /usr/lib64/dovecot/lib20_fts_plugin.so(fts_backend_lookup+0x51)
> [0x7f782930b7c1] ->
> /usr/lib64/dovecot/doveadm/lib20_doveadm_fts_plugin.so(+0x3280)
> [0x7f78270d0280] -> doveadm(+0x343cd) [0x55aa57edc3cd] ->
> doveadm(+0x34fe0) [0x55aa57edcfe0] ->
> doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x22d) [0x55aa57edde2d]
> -> doveadm(doveadm_cmd_run_ver2+0x4e8) [0x55aa57eee8d8] ->
> doveadm(doveadm_cmd_try_run_ver2+0x3e) [0x55aa57eee92e] ->
> doveadm(main+0x1d4) [0x55aa57ecccf4] ->
> /lib64/libc.so.6(__libc_start_main+0xf2) [0x7f7829746042] ->
> doveadm(_start+0x2e) [0x55aa57ecd1ce]
>     Aborted
>
> but search, even for accented characters,
>
>     doveadm search -u testuser at example.com subject "tésting"
>         42d73837f133a05fad4d0000f8839f03 1
>         813ef60e984f1b5f5fc200005439fba4 293
>
>     doveadm search -u testuser at example.com body "tésting"
>         ba899d0cfe33a05fbe4d0000f8839f03 1
>         813ef60e984f1b5f5fc200005439fba4 293
>
> appears to work.
>
> next, to get tokenization -- at least email/url
> (UAX29URLEmailTokenizer) -- and lowercase & icu normalization working
> and verified.
>
the panic on doveadm fts lookup is to be expected and solved by my
previous patch.

I think you've now got a config very similar to mine. One last check I
did was to search for the same string with the grave accent. e.g.
tèsting, so that should NOT be found. That was just proof that it was
actually searching for accents and not folding them to e.

I saw Aki's advice about removing use_tls because of issues, and I know
I'm repeating myself here :-) but so far I have seen no concrete
evidence of issues on your original setup. Doesn't mean there aren't,
but until there is evidence, can't do much towards solving them.

John

John