Dovecot - FTS Solr: disk usage & position information?

[Vincent Brillault]

Hi Alessio,

> this optimization also produce a less RAM requirements on Solr server?

Unfortunately we didn't measure this before/after the change. Since we 
are removing features (position information), I wouldn't expect the 
memory requirement to increase, but I'm no expert.

To be honest, I've not been able to measure in any sensible way the 
memory really required by Solr. The memory directly used by the Solr 
process is rather limited, but a lot of memory is used for file caches, 
which also feels (again not an expert) important for good performances.

> The attached schema and config file also works with Solr 7.7.0? Since 
> dovecot provide schema and config for 7.7.0 will be useful for many of 
> us a path based on it.

At least the solrconfig.xml shouldn't work with 7.7.0 since I increased 
the luceneMatchVersion to match 8.6 and imported a few defaults from the 
default upstream 8.6 configuration. I think these changes could be 
ignored for 7.7.0.

For schema.xml, I made quite a few changes, but all seem to be backward 
compatible:
  - Remove unused 'boolean' field type
  - Remove KeywordMarkerFilterFactory: protwords are usually empty anyway
  - User a simper 'text_basic' field type (no StopFilterFactory, 
SynonymGraphFilterFactory or PorterStemFilterFactory) for processing 
non-human fields (all but body and subject)
  - Remplace autoGeneratePhraseQueries & positionIncrementGap by 
omitTermFreqAndPositions="true" & omitPositions="true" on TextField 
fieldtypes (as discussed in this thread)
  - Minor modifications on WordDelimiterGraphFilterFactory when used in 
search to have better match (things like 'covid19' are indexed as 
['covid', '19', 'covid19'] but only searched as 'covid19')

 From taking a quick look at the documentation, I _think_ most of them 
are compatible with 7.7.0, but without testing, I can't guarantee it.

Cheers,
Vincent
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://dovecot.org/pipermail/dovecot/attachments/20210906/b5d78feb/attachment.sig>