Dovecot - FTS Solr: disk usage & position information?

[Vincent Brillault]

Dear Shawn,

Thanks for your very complete answer!

> This is completely off-topic for the dovecot list.  I am involved with
> the Solr project, so I can discuss it.  My message will also be off
> topic here.

Sorry, maybe I didn't explain myself properly. I asked on the dovecot
mailing list as I'm interested in:
- The interaction between Solr & dovecot: what dovecot really needs and
uses from Solr.
- The reasons for the settings in the schema example in the dovecot
repositories.

I think these are still interesting to be discussed on the dovecot
mailing list, but I'm extremely grateful for your feedback.

> You didn't say what version of Solr you're on.  That document for Lucene
> 6.2.0 would be relevant for Solr 6.2.0.

Indeed, I should have. I'm using Solr 8.6, which is clearly not the same
as Solr 6.2.0, but when looking at more recent versions of the
documentation, no information about the use of each file appeared.
That's why I was mentioning it was slightly outdated.

>> I don't see this kind of query being made, `~` only being used for fuzzy
>> search.
> 
> Positions are required for a phrase query -- where the query text is in
> double quotes. 

Yes, I discovered that while testing yesterday :D
```"field "body" was indexed without position data; cannot run
PhraseQuery```

> Right now you noticed that dovecot's FTS Solr plugin doesn't
> explicitly use phrase queries, but there's no guarantee that this will
> always be the case.  Position data will only be accessed if it is needed
> for a query, so if it is not needed it should not affect query
> performance. 

Of course if dovecot's FTS Solr plugin requirements change, then the
schema I'm using will to change. This is why I'm asking here. Solr is a
powerful engine, but search within IMAP are more restricted. As far as I
understand, dovecot does not make use of all the features for Solr, only
of a very small subset and thus I believe it makes sense to try to
optimize the configuration to deliver what it needs without spending to
much compute or storage on features dovecot doesn't need.

> I cannot speak as to whether the FTS Solr plugin relies on
> the autoGenereatePhraseQueries functionality, but if it does, then you
> definitely want position data in the index.  That functionality can do a
> lot to improve relevancy ranking, so I would expect it to be
> instrumental in good full-text searching -- disabling positions will
> probably not help your search results.

This is the main question and what I don't really understand. If the
query generated by dovecot from the IMAP searches it creates
significantly improve with position data, then yes, it's clearly
required. If it only marginally improves it, then a cost/benefit
analysis should be taken.

Yesterday, I've modified my test cluster to use
`omitTermFreqAndPositions="true" omitPositions="true"` instead of
`autoGeneratePhraseQueries="true"`. This is a painful operation as it
requires to drop everything and re-index all the data, but at the end of
the day, after re-indexation:
- Total disk usage for the test cluster went from 16.0 GB to 9.8 GB, so
a 39% reduction is disk usage :)
- No .pos file created in the cores

Basic tests show no obvious change in the search results (after I
removed autoGeneratePhraseQueries, before that it failed in some cases).

Did any other Dovecot user try something similar? (I've only found once
post on the internet raising the question so far :/).

> If you want an in-depth discussion beyond this email, please subscribe
> to the solr-user mailing list and ask there.

Thanks, I'll take on your offer, for the Solr specific part, as I need
to understand that autoGeneratePhraseQueries better :)

> Note that general Solr recommendations are to have enough space
> available that the Solr index can triple in size temporarily -- this is
> to accommodate all possible scenarios for Lucene segment merging. 
> Running Solr on systems with limited disk space is not recommended.

Well, it depends on what you define as "limited". I'd love to have
infinite storage, but unfortunately every resource is always limited one
way of another. Ensuring that each core can temporarily triple in size
(required e.g. if ones want to split the shards to distribute over more
nodes) is one thing (that can have a limited impact if the shards are
split in small enough sizes). Requiring double the size overall with no
operational benefit is another ;). I'm just trying to understand how
much storage we'll really need once the cluster is scaled to final use.


Thanks again Shawn for your contribution, it was quite helpful!
Cheers,
Vincent

-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://dovecot.org/pipermail/dovecot/attachments/20210805/e756de50/attachment.sig>