v2.3.11.3 solr plugin search via MUA fails to match accented ascii characters; cmd line exec of `doveadm fts lookup` PANICs (assertion failed) [proposed patch]

[Aki Tuomi]

> On 01/11/2020 20:35 John Fawcett <john at voipsupport.it> wrote:
> 
>  
> On 01/11/2020 15:20, PGNet Dev wrote:
> > On 11/1/20 1:56 AM, John Fawcett wrote:
> >> At the moment I don't see other corrections needed in dovecot apart from
> >> command line doveadm fts which is not a show stopper. Via doveadm search
> >> I confirm - on my simple config - that search for accented or non
> >> accented characters works correctly as it does via imap connection.
> >
> > thx.  hopefully it'll get considered for a next release soon.
> >
> >> Only thing I cannot vouch for is bringing dovecot fts library and config
> >> into the equation because my setup delegates almost everything to solr.
> >
> > do i understand correctly that you're solr-indexing your dovecot mail
> > store withOUT using dovecot fts plugin, and that -- with your
> > aforementioned patch -- doveadm successfully uses the resulting indexes?
> >
> > i hadn't yet seriously considered _circumventing_ fts plugin; if this^
> > does get resolved soonish, then it's not a big deal.  if not, an
> > fts-plugin-less setup would be interesting to know more abt!
> >
> >> Can you get evidence of things not working? For example tests run with
> >> soft_commit configured - that's important since without it the updates
> >> don't show up immediately in searches, that do show that the update is
> >> happening in solr via solr log, but then search is not working on
> >> accented characters, despite it working on other text in the same
> >> message? The solr logs also show whether the text was found or not via
> >> the "hits=" value in the logged searches, for example:
> >>
> >> 2020-11-01 08:32:42.231 INFO  (qtp24119573-21) [   x:dovecot]
> >> o.a.s.c.S.Request [dovecot]  webapp=/solr path=/select
> >> params={q={!lucene+q.op%3DAND}body:también&fl=uid,score&sort=uid+asc&fq=%2Bbox:b1626f0fe8d9145e54100000c54a863a+%2Buser:john at voipsupport.it&rows=3202&wt=xml}
> >>
> >> hits=3 status=0 QTime=3
> >>
> >> But if no hits are found, then dovecot cannot be expected to display
> >> results. It still may be an indexing problem though.
> >
> > my current config has soft_commit enabled,
> >
> >   fts_solr = url=https://solr.example.com:8984/solr/dovecot/
> > use_libfts soft_commit=yes batch_size=250
> >
> > i'll see abt getting some clearer test results ...
> 
> Yes, getting more data about any potential problem would be useful.
> 
> Just to clarify: I have a fully working search setup for some time now
> over various dovecot releases, so no patches needed to get it working.
> 
> My setup does use fts plugin and fts-solr plugin, but it does not use
> lib-fts functionality (that has many features for example it was
> stopping you indexing excluded words like tambien). On my setup without
> lib-fts everything goes to solr which does the work of indexing without
> all the features of lib-fts.
> 
> My setup is like this not because of issues in lib-fts, but because I
> never had the need for it. There is no evidence at the moment however
> that even with lib-fts enabled there are issues with dovecot indexing or
> searching.
> 
> What is currently nor working is "doveadm fts" command line utility. But
> this is mitigated by being able to use a similar command line utility
> "doveadm search". The issue on "dovecot fts" command line utility has
> (so far as the available evidence suggests) no effect on indexing or
> imap searches.
> 
> fyi my working configuration includes fts and fts_solr plugins
> 
> mail_plugins =  quota notify replication fts fts_solr
> 
> (and those are also recalled in the various specific plugin settings for
> imap, lmtp ecc), The specific config I am using for fts and fts_solr is:
> 
>   fts = solr
>   fts_enforced = yes
>   fts_solr = url=https://user@server.example.com:443/solr/dovecot/
> batch_size=500 soft_commit=no
> 
> BTW I use soft_commit=no because I have periodic soft commits setup on
> solr and I accept that newly indexed text won't become searchable for up
> to that interval, but for your testing purposes much more useful as you
> have it.
> 
> John

PGNet,

you should try removing use_libfts from your config line and let solr do that part.

Aki