Solr

[Aki Tuomi]

> On 02 January 2019 at 10:59 "M. Balridge" <dovecot at r.paypc.com> wrote:
> 
> 
> 
> > The main problem is : After some time of indexing from Dovecot, Dovecot
> > returns errors (invalid SID, etc...) and Solr return "out of range
> > indexes" errors
> 
> I've been watching the progress of this thread with no small concern, mainly
> because I've been tasked with providing a server-side email search facility
> with a budget and manpower level that comes down to mainly *1*, i.e., me.
> 
> I was expecting, given the strongly worded language about "just use
> lucene/SOLR" and "ignore squat", that I should invest time + effort into this
> JAVA nightmare that is SOLR.
> 
> I started with squat and another word-indexor system that used out-of-band
> (not a dovecot plugin) software to provide rapid (sub-second) searches through
> tens-of-GB-scale mailboxes.
> 
> Unlike what I was led to believe, the squat indexes worked surprisingly well,
> once you sorted out the odd resource size (ulimit-related) issues (vsz &
> friends) limitations. I did notice the "worst-case" search performance have
> worryingly high O(x) increases in time, but I'd not seen anything that was a
> dealbreaker. It goes without saying that various substring searches worked as
> expected, for the most part.
> 
> My experiences with SOLR were similar to Messr. Moreau's: lots of startup
> errors with provided schemata files. Lots of JAVA nonsense issues. Lots of
> sensitivity to WHICH Java runtime, etc, etc. I finally fixated a specific JVM,
> version of SOLR, and dovecot to find the "best" working combination, only to
> find that the searches didn't work out as expected. I expected to be able to
> do date-ranging based searches. Didn't work. I expected to search CONTENTS of
> emails, and despite many days of tweaks, I couldn't get it to index even the
> basics like filenames/types of attachments, so I could exposed
> attachment-based searching to my users.
> 
> So, without rancour or antipathy, I ask the entire list: has ANYONE gotten a
> Dovecot/solr-fts-plugin setup to work that provides as a BASELINE, all of the
> following functionality:
> 
> 1) The ability to search for a string within any of the structured fields
> (from/subject) that returns correct results?
> 
> 2) The ability to search for any string within the BODY of emails, including
> the MIME attachment boundaries?
> 
> 3) The ability to do "ranging" searches for structures within emails that
> decompose to "dates" or other simple-numeric data?
> 
> OPTIONALLY, and this is probably way outside of the scope of the above,
> despite the fact that it's listed as a "selling point" of SOLR versus other
> full text search engines:
> 
> 4) The ability to do searches against any attachments that are able to be
> post-processed and hyper-indexed by SOLR+Tika?
> 
> -------------
> 
> SOLR seems to have "brand cachet", so presumably it actually works (for somebody).
> 
> Dovecot has not a little "brand cachet", and for me, I have innate faith and
> trust in Timo and his software. I am no stranger to the "costs" of "free"
> software, in that you sacrifice your own blood, sweat, and tears just to get
> these disparate pieces to work together.
> 
> I *DO* respect that Timo has to keep the lights (and sauna) on in Finland.
> Maybe there's a super-secret (no advertised prices, "carrier-only" price list)
> with _Dovecot, Oy_ wherein the above ARE actually available for something less
> than 6.022 x 10^23 Euros per centi-second of licencing fees.
> 
> But please, level with us faithful users.  Does this morass of Java B.S.
> actually work, and if not, please just deprecate and remove this moribund
> software, and stop trying to bury the only FTS plugin many of us HAVE actually
> gotten to work.  (Pretty please?)
> 
> I respect that Messr. Moreau has made an earnest effort to get this JAVA B.S.
> to actually work, as I have. 
> 
> He persevered where I'd given up. He's vocal about it, and now I'm chiming in
> that this ornate collection of switchblades only cuts those who try to use them.
> 
> Respectfully,
> =M=
>

We do intend to polish fts-solr before we drop fts-squat. And even then, anyone is free to pick it up and continue the work, as it works as plugin just fine, so it's not a matter of us just flushing it to oblivion. 

fts-squat is not really worth pursuing for us since it would eat away effort from our current dovecot fts plugin, which unfortunately is not currently open-sourced.

Aki