<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /></head><body style='font-size: 9pt; font-family: Verdana,Geneva,sans-serif'>
<p>Also, a description of the "to be" functions of the backend:</p>
<p><br /></p>
<p>struct fts_backend fts_backend_xapian = {<br />.name = "xapian",<br />.flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT,<strong> -> what other flags ?</strong></p>
<p>{<br />fts_backend_xapian_alloc,<br />fts_backend_xapian_init,<br />fts_backend_xapian_deinit,<br />fts_backend_xapian_get_last_uid,<br />fts_backend_xapian_update_init,<br />fts_backend_xapian_update_deinit,<br />fts_backend_xapian_update_set_mailbox,<br />fts_backend_xapian_update_expunge,<br />fts_backend_xapian_update_set_build_key,<br />fts_backend_xapian_update_unset_build_key,<br />fts_backend_xapian_update_build_more,<br />fts_backend_xapian_refresh,<br />fts_backend_xapian_rescan,<br />fts_backend_xapian_optimize,<br />fts_backend_default_can_lookup,<br />fts_backend_xapian_lookup,<br />fts_backend_xapian_lookup_multi,<br />fts_backend_xapian_lookup_done<br />}<br />};</p>
<div id="signature"> </div>
<p><br /></p>
<p id="reply-intro">On 2019-01-04 20:33, Joan Moreau via dovecot wrote:</p>
<blockquote type="cite" style="padding: 0 0.4em; border-left: #1010ff 2px solid; margin: 0"><!-- html ignored --><!-- head ignored --><!-- meta ignored -->
<p>Yes but:</p>
<p>1 - is there a documentation of the main object ? (fts_backend, mail_user, mailbox, etc..)</p>
<p>2 - What are the mandatory functions ?</p>
<p>3 - Search : Supposedly, the FTS shall have several parameters : the keyword(s), the user & mailbox, and the fields (to, from, body, etc..) to be includude in the search. What is the function called in the plugin ?</p>
<p>4 - Indexing : Somehow, what is the logic ? fts core just ask to "index me this email of this mailbox" ? or this is delegated to the plugin to sort out which emails it has indexed yet or not ?</p>
<p><br /></p>
<p>Thank you</p>
<p><br /></p>
<div id="signature"> </div>
<p><br /></p>
<p id="reply-intro">On 2019-01-04 18:49, admin wrote:</p>
<blockquote style="padding: 0 0.4em; border-left: #1010ff 2px solid; margin: 0;">
<div>A starting point would be to have a look at the current FTS plugins:</div>
<div> </div>
<div><a href="https://github.com/dovecot/core/tree/master/src/plugins/fts-solr" target="_blank" rel="noopener noreferrer">https://github.com/dovecot/core/tree/master/src/plugins/fts-solr</a></div>
<div>and</div>
<div><a href="https://github.com/dovecot/core/tree/master/src/plugins/fts-squat" target="_blank" rel="noopener noreferrer">https://github.com/dovecot/core/tree/master/src/plugins/fts-squat</a></div>
<div> </div>
<div>-M</div>
<div> </div>
<div> </div>
<div>Am Freitag, den 04.01.2019, 18:17 +0800 schrieb Joan Moreau via dovecot:</div>
<blockquote style="margin: 0 0 0 .8ex; border-left: 2px #729fcf solid; padding-left: 1ex;">
<p>Why not, but please guide me about the core structure (mandatory funcitons, etc..) of a typical Dovecot FTS plugin</p>
<p><br /></p>
<p><br /></p>
<div id="signature"> </div>
<p><br /></p>
<p id="reply-intro">On 2019-01-04 17:20, Aki Tuomi wrote:</p>
<blockquote style="margin: 0 0 0 .8ex; border-left: 2px #729fcf solid; padding-left: 1ex;">
<div class="pre" style="margin: 0; padding: 0; font-family: monospace;">I hope you are aware that "linking with Xapian" requires somewhat more work than just -lxapian in linker? If you or someone feels like writing fts_xapian, go for it. <br /><br />Aki<br /><br />
<blockquote style="margin: 0 0 0 .8ex; border-left: 2px #729fcf solid; padding-left: 1ex;">On 04 January 2019 at 08:20 Joan Moreau via dovecot <<a href="mailto:dovecot@dovecot.org" rel="noreferrer">dovecot@dovecot.org</a>> wrote:<br /><br /><br />What about consedering linking Dovecot with Xapian librairies instead of<br />going to nightmare Solr ? <br /><br /><a href="https://xapian.org/features" target="_blank" rel="noopener noreferrer">https://xapian.org/features</a><br /><br />On 2019-01-02 17:10, John Tulp wrote:<br /><br />
<blockquote style="margin: 0 0 0 .8ex; border-left: 2px #729fcf solid; padding-left: 1ex;">On Wed, 2019-01-02 at 00:59 -0800, M. Balridge wrote: The main problem is : After some time of indexing from Dovecot, Dovecot<br />returns errors (invalid SID, etc...) and Solr return "out of range<br />indexes" errors <br />I've been watching the progress of this thread with no small concern, mainly<br />because I've been tasked with providing a server-side email search facility<br />with a budget and manpower level that comes down to mainly *1*, i.e., me.<br /><br />I was expecting, given the strongly worded language about "just use<br />lucene/SOLR" and "ignore squat", that I should invest time + effort into this<br />JAVA nightmare that is SOLR.<br /><br />I started with squat and another word-indexor system that used out-of-band<br />(not a dovecot plugin) software to provide rapid (sub-second) searches through<br />tens-of-GB-scale mailboxes.<br /><br />Unlike what I was led to believe, the squat indexes worked surprisingly well,<br />once you sorted out the odd resource size (ulimit-related) issues (vsz &<br />friends) limitations. I did notice the "worst-case" search performance have<br />worryingly high O(x) increases in time, but I'd not seen anything that was a<br />dealbreaker. It goes without saying that various substring searches worked as<br />expected, for the most part.<br /><br />My experiences with SOLR were similar to Messr. Moreau's: lots of startup<br />errors with provided schemata files. Lots of JAVA nonsense issues. Lots of<br />sensitivity to WHICH Java runtime, etc, etc. I finally fixated a specific JVM,<br />version of SOLR, and dovecot to find the "best" working combination, only to<br />find that the searches didn't work out as expected. I expected to be able to<br />do date-ranging based searches. Didn't work. I expected to search CONTENTS of<br />emails, and despite many days of tweaks, I couldn't get it to index even the<br />basics like filenames/types of attachments, so I could exposed<br />attachment-based searching to my users.<br /><br />So, without rancour or antipathy, I ask the entire list: has ANYONE gotten a<br />Dovecot/solr-fts-plugin setup to work that provides as a BASELINE, all of the<br />following functionality:<br /><br />1) The ability to search for a string within any of the structured fields<br />(from/subject) that returns correct results?<br /><br />2) The ability to search for any string within the BODY of emails, including<br />the MIME attachment boundaries?<br /><br />3) The ability to do "ranging" searches for structures within emails that<br />decompose to "dates" or other simple-numeric data?<br /><br />OPTIONALLY, and this is probably way outside of the scope of the above,<br />despite the fact that it's listed as a "selling point" of SOLR versus other<br />full text search engines:<br /><br />4) The ability to do searches against any attachments that are able to be<br />post-processed and hyper-indexed by SOLR+Tika?<br /><br />-------------<br /><br />SOLR seems to have "brand cachet", so presumably it actually works (for somebody).<br /><br />Dovecot has not a little "brand cachet", and for me, I have innate faith and<br />trust in Timo and his software. I am no stranger to the "costs" of "free"<br />software, in that you sacrifice your own blood, sweat, and tears just to get<br />these disparate pieces to work together.<br /><br />I *DO* respect that Timo has to keep the lights (and sauna) on in Finland.<br />Maybe there's a super-secret (no advertised prices, "carrier-only" price list)<br />with _Dovecot, Oy_ wherein the above ARE actually available for something less<br />than 6.022 x 10^23 Euros per centi-second of licencing fees.<br /><br />But please, level with us faithful users. Does this morass of Java B.S.<br />actually work, and if not, please just deprecate and remove this moribund<br />software, and stop trying to bury the only FTS plugin many of us HAVE actually<br />gotten to work. (Pretty please?)<br /><br />I respect that Messr. Moreau has made an earnest effort to get this JAVA B.S.<br />to actually work, as I have. <br /><br />He persevered where I'd given up. He's vocal about it, and now I'm chiming in<br />that this ornate collection of switchblades only cuts those who try to use them.<br /><br />Respectfully,<br />=M=</blockquote>
Fascinating...<br /><br />SOLR says the following are powered by SOLR...<br /><br /><a href="https://wiki.apache.org/solr/PublicServers" target="_blank" rel="noopener noreferrer">https://wiki.apache.org/solr/PublicServers</a><br /><br />Perhaps if you could find out from that list which of them are using<br />SOLR in conjunction with Dovecot...<br /><br />food for thought...</blockquote>
</div>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</body></html>