v2.3.11.3 solr plugin search via MUA fails to match accented ascii characters; cmd line exec of `doveadm fts lookup` PANICs (assertion failed)

[John Fawcett]

On 19/10/2020 17:00, PGNet Dev wrote:
> On 10/19/20 1:18 AM, John Fawcett wrote:
>> I would recommend you to redo the tests after correcting the
>> configuration. To be doubly sure you can include accented and unique non
>> accented text in the same email and search for both. If the non accented
>> text is found you know you've searching against the updated index and
>> the fact that accented text is not found is not simply because the index
>> updates are not visible.
>
> temp changing,
>
>     fts_solr = url=https://solr.example.com:8984/solr/dovecot/
> use_libfts soft_commit=yes batch_size=250
> -    fts_enforced = body
> +    fts_enforced = yes
> -    fts_filters = normalizer-icu snowball stopwords
> +    fts_filters = lowercase snowball stopwords
>
> now,
>
>     doveadm fts rescan -u testuser at example.com
>     doveadm index      -u testuser at example.com -q '*'
>
> , no errors.
>
> on receipt of a test email with
>
>     subject: aausdfrhyétdwgyatrdf aausdfrhyetdwgyatrdf
>     body:    aausdfrhyétdwgyatrdf aausdfrhyetdwgyatrdf
>
> i see a solr auto-scan,
>
>     2020-10-19 14:41:50.628 INFO 
> (searcherExecutor-15-thread-1-processing-x:dovecot) [   x:dovecot]
> o.a.s.c.SolrCore [dovecot]  Registered new searcher autowarm time: 1 ms
>     2020-10-19 14:41:50.634 INFO  (qtp1533985074-18) [   x:dovecot]
> o.a.s.u.p.LogUpdateProcessorFactory [dovecot]  webapp=/solr
> path=/update params={}{commit=} 0 155
>     2020-10-19 14:41:51.571 INFO  (qtp1533985074-24) [   x:dovecot]
> o.a.s.u.p.LogUpdateProcessorFactory [dovecot]  webapp=/solr
> path=/update
> params={}{add=[135785/c92f64f79f0d1ed01e6d5b314f04886c/testuser at example.com
> (1680991596381732864)]} 0 9
>
>     ==> /var/log/solr/solr_gc.log <==
>     [2020-10-19T07:41:51.612-0700][146823.946s] GC(343) Pause Young
> (Normal) (G1 Evacuation Pause)
>     [2020-10-19T07:41:51.613-0700][146823.947s] GC(343) Using 2
> workers of 2 for evacuation
>     [2020-10-19T07:41:51.635-0700][146823.969s] GC(343)   Pre Evacuate
> Collection Set: 0.3ms
>     [2020-10-19T07:41:51.636-0700][146823.970s] GC(343)   Merge Heap
> Roots: 0.2ms
>     [2020-10-19T07:41:51.636-0700][146823.970s] GC(343)   Evacuate
> Collection Set: 18.9ms
>     [2020-10-19T07:41:51.637-0700][146823.971s] GC(343)   Post
> Evacuate Collection Set: 2.0ms
>     [2020-10-19T07:41:51.637-0700][146823.971s] GC(343)   Other: 1.5ms
>     [2020-10-19T07:41:51.637-0700][146823.971s] GC(343) Eden regions:
> 238->0(244)
>     [2020-10-19T07:41:51.638-0700][146823.972s] GC(343) Survivor
> regions: 4->2(31)
>     [2020-10-19T07:41:51.638-0700][146823.972s] GC(343) Old regions:
> 189->189
>     [2020-10-19T07:41:51.638-0700][146823.973s] GC(343) Archive
> regions: 2->2
>     [2020-10-19T07:41:51.639-0700][146823.973s] GC(343) Humongous
> regions: 10->9
>     [2020-10-19T07:41:51.639-0700][146823.973s] GC(343) Metaspace:
> 61564K(78028K)->61564K(78028K) NonClass:
> 55348K(65024K)->55348K(65024K) Class: 6216K(13004K)->6216K(13004K)
>     [2020-10-19T07:41:51.640-0700][146823.974s] GC(343) Pause Young
> (Normal) (G1 Evacuation Pause) 441M->200M(512M) 27.372ms
>     [2020-10-19T07:41:51.640-0700][146823.974s] GC(343) User=0.01s
> Sys=0.01s Real=0.03s
>
>     ==> /var/log/solr/solr.log <==
>     2020-10-19 14:41:51.702 INFO 
> (searcherExecutor-15-thread-1-processing-x:dovecot) [   x:dovecot]
> o.a.s.c.SolrCore [dovecot]  Registered new searcher autowarm time: 0 ms
>     2020-10-19 14:41:51.705 INFO  (qtp1533985074-18) [   x:dovecot]
> o.a.s.u.p.LogUpdateProcessorFactory [dovecot]  webapp=/solr
> path=/update params={}{commit=} 0 127
>
>
> search in TBird
>
>     subject: aausdfrhyetdwgyatrdf  => FOUND
>     body:    aausdfrhyétdwgyatrdf  => FOUND
>
>     subject: aausdfrhyetdwgyatrdf  => FOUND
>     body:    aausdfrhyétdwgyatrdf  => (emtpy)
>
> on header search, I'm _not_ seeing any additional activity in solr.log
>
> so, either i'm looking in the wrong place, haven't turned on
> appropriate logging, or i'm still not searching via solr ...
>
> separately,
>
>     doveadm fts lookup ...
>
> still panics; Aki's bug will hopefully deal with that

Depending how solr has been setup you could see the logging in the web
server access log. My access log is where I configured it in
/var/log/httpd/servername.access_log, yours may be different.

For searches I see things like this (one for each folder searched)

2a01:488:67:1000:523:f8eb:0:1 - john [19/Oct/2020:17:16:39 +0200] "GET
/solr/dovecot/select?wt=xml&fl=uid,score&rows=3176&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7dbody:aausdfrhy%c3%a9tdwgyatrdf&fq=%2Bbox:b1626f0fe8d9145e54100000c54a863a+%2Buser:john at voipsupport.it
HTTP/1.1" 200 910 "-" "-"

For index updates I see things like this:

2a01:488:67:1000:523:f8eb:0:1 - john [19/Oct/2020:17:10:01 +0200] "POST
/solr/dovecot/update HTTP/1.1" 200 156 "-" "-"

BTW I've noticed that Thunderbird does not always send the search to the
server even with the "search on server" flag set, which is why I look at
the access_log in solr if I want to be really sure it's going to solr.

John