v2.3.11.3 solr plugin search via MUA fails to match accented ascii characters; cmd line exec of `doveadm fts lookup` PANICs (assertion failed) [proposed patch]

[PGNet Dev]

On 11/1/20 10:35 AM, John Fawcett wrote:
> Yes, getting more data about any potential problem would be useful.
> 
> Just to clarify: I have a fully working search setup for some time now
> over various dovecot releases, so no patches needed to get it working.
> 
> My setup does use fts plugin and fts-solr plugin, but it does not use
> lib-fts functionality (that has many features for example it was
> stopping you indexing excluded words like tambien). On my setup without
> lib-fts everything goes to solr which does the work of indexing without
> all the features of lib-fts.

withOUT libfts

-	fts_solr = url=https://solr.presence-group.net:8984/solr/dovecot/ use_libfts soft_commit=yes batch_size=250
+	fts_solr = url=https://solr.presence-group.net:8984/solr/dovecot/ soft_commit=yes batch_size=250

and unmodified dovecot-provided schema/config,

	/bin/cp -af /usr/share/doc/dovecot/solr-config-7.7.0.xml /path/to/solr/data/dovecot/conf/solrconfig.xml
	/bin/cp -af /usr/share/doc/dovecot/solr-schema-7.7.0.xml /path/to/solr/data/dovecot/conf/schema.xml

i suspect my config's now more similar to yours.

checking,

	doveadm fts rescan -u testuser at example.com
	doveadm index      -u testuser at example.com -q '*'

as before

	doveadm fts lookup -u testuser at example.com body "tésting"

panics,

	doveadm(testuser at example.com): Panic: file mail-storage.c: line 2112 (mailbox_get_open_status): assertion failed: (box->opened)
	doveadm(testuser at example.com): Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(backtrace_append+0x46) [0x7f7829b81cc6] -> /usr/lib64/dovecot/libdovecot.so.0(backtrace_get+0x22) [0x7f7829b81de2] -> /usr/lib64/dovecot/libdovecot.so.0(+0x10025b) [0x7f7829b8b25b] -> /usr/lib64/dovecot/libdovecot.so.0(+0x100297) [0x7f7829b8b297] -> /usr/lib64/dovecot/libdovecot.so.0(+0x59bc6) [0x7f7829ae4bc6] -> /usr/lib64/dovecot/libdovecot-storage.so.0(+0x4779e) [0x7f7829c9879e] -> /usr/lib64/dovecot/lib21_fts_solr_plugin.so(+0x5849) [0x7f78296ea849] -> /usr/lib64/dovecot/lib20_fts_plugin.so(fts_backend_lookup+0x51) [0x7f782930b7c1] -> /usr/lib64/dovecot/doveadm/lib20_doveadm_fts_plugin.so(+0x3280) [0x7f78270d0280] -> doveadm(+0x343cd) [0x55aa57edc3cd] -> doveadm(+0x34fe0) [0x55aa57edcfe0] -> doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x22d) [0x55aa57edde2d] -> doveadm(doveadm_cmd_run_ver2+0x4e8) [0x55aa57eee8d8] -> doveadm(doveadm_cmd_try_run_ver2+0x3e) [0x55aa57eee92e] -> doveadm(main+0x1d4) [0x55aa57ecccf4] -> /lib64/libc.so.6(__libc_start_main+0xf2) [0x7f7829746042] -> doveadm(_start+0x2e) [0x55aa57ecd1ce]
	Aborted

but search, even for accented characters,

	doveadm search -u testuser at example.com subject "tésting"
		42d73837f133a05fad4d0000f8839f03 1
		813ef60e984f1b5f5fc200005439fba4 293

	doveadm search -u testuser at example.com body "tésting"
		ba899d0cfe33a05fbe4d0000f8839f03 1
		813ef60e984f1b5f5fc200005439fba4 293

appears to work.

next, to get tokenization -- at least email/url (UAX29URLEmailTokenizer) -- and lowercase & icu normalization working and verified.