<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /></head><body style='font-size: 9pt; font-family: Verdana,Geneva,sans-serif'>
<p>For "rescan " and "optimize", wouldn't it be the dovecot core who indicate which are to be dismissed (expunged), or re-ask for indexing a particular (or all) uid ? WHy would the backend be aware of the transactions on the mailbox ???</p>
<p>There is alredy "fts_backend_xxx_update_expunge", so I beleive the management of the expunged messages is *NOT* in the backend, right ?</p>
<div id="signature"> </div>
<p><br /></p>
<p id="reply-intro">On 2019-01-06 15:41, Joan Moreau wrote:</p>
<blockquote type="cite" style="padding: 0 0.4em; border-left: #1010ff 2px solid; margin: 0"><!-- html ignored --><!-- head ignored --><!-- meta ignored -->
<p>also, for fts_backend_solr_update_set_build_key -> where is the data (of the hdr_name or the body) ?</p>
<p><br /></p>
<div id="signature"> </div>
<p><br /></p>
<p id="reply-intro">On 2019-01-06 14:10, Joan Moreau wrote:</p>
<blockquote style="padding: 0 0.4em; border-left: #1010ff 2px solid; margin: 0;">
<p>for the "last uid"-> this is not the last added, but the maximum of the UID in the indexed emails, right ?</p>
<p><br /></p>
<div id="signature"> </div>
<p><br /></p>
<p id="reply-intro">On 2019-01-06 11:53, Joan Moreau via dovecot wrote:</p>
<blockquote style="padding: 0 0.4em; border-left: #1010ff 2px solid; margin: 0;">
<p>Thank you</p>
<p>I still don't get the "build_key" function. The email (body, hearders, .. and the uid) is the one (and only) to index . What "key" is that function referring to ? Or is the "key" here the actual email ?</p>
<p><br /></p>
<p><br /></p>
<div id="signature"> </div>
<p><br /></p>
<p id="reply-intro">On 2019-01-06 08:43, Stephan Bosch wrote:</p>
<blockquote style="padding: 0 0.4em; border-left: #1010ff 2px solid; margin: 0;">
<div class="pre" style="margin: 0; padding: 0; font-family: monospace;"><br /><span style="white-space: nowrap;">Op 06/01/2019 om 01:00 schreef Joan Moreau:</span>
<blockquote style="padding: 0 0.4em; border-left: #1010ff 2px solid; margin: 0;"><span style="white-space: nowrap;">Anyone willing to explain those functions ?</span><br /><br /><span style="white-space: nowrap;">Most notably " get_last_uid"</span></blockquote>
<br /><span style="white-space: nowrap;">From src/plugins/fts/fts-api.h:</span><br /><br /><span style="white-space: nowrap;">/* Get the last_uid for the mailbox. */</span><br />int fts_backend_get_last_uid(struct fts_backend *backend, struct mailbox *box,<br /><span style="white-space: nowrap;"> uint32_t *last_uid_r);</span><br /><br />The solr sources ( src/plugins/fts-solr/fts-backend-solr.c:213) tell me this returns the last UID added to the index for the given mailbox and FTS index.<br /><br />
<blockquote style="padding: 0 0.4em; border-left: #1010ff 2px solid; margin: 0;"><span style="white-space: nowrap;">"set_build_key"</span></blockquote>
<br /><span style="white-space: nowrap;">From src/plugins/fts/fts-api.h:</span><br /><br /><span style="white-space: nowrap;">/* Switch to building index for specified key. If backend doesn't want to</span><br /><span style="white-space: nowrap;"> index this key, it can return FALSE and caller will skip to next key. */</span><br />bool fts_backend_update_set_build_key(struct fts_backend_update_context *ctx,<br /><span style="white-space: nowrap;"> const struct fts_backend_build_key *key);</span><br /><br /><span style="white-space: nowrap;">Same file provides outline of what a build_key is.</span><br /><br />
<blockquote style="padding: 0 0.4em; border-left: #1010ff 2px solid; margin: 0;"><span style="white-space: nowrap;">"build_more" ,</span></blockquote>
<br /><span style="white-space: nowrap;">/* Add more content to the index for the currently specified build key.</span><br /><span style="white-space: nowrap;"> Non-BODY_PART_BINARY data must contain only full valid UTF-8 characters,</span><br /><span style="white-space: nowrap;"> but it doesn't need to be NUL-terminated. size contains the data size in</span><br /> bytes, not characters. This function may be called many times and the data<br /><span style="white-space: nowrap;"> block sizes may be small. Backend returns 0 if ok, -1 if build should be</span><br /><span style="white-space: nowrap;"> aborted. */</span><br /><span style="white-space: nowrap;">int fts_backend_update_build_more(struct fts_backend_update_context *ctx,</span><br /><span style="white-space: nowrap;"> const unsigned char *data, size_t size);</span><br /><br />You should look at the sources of a few backends like squat and solr to get a feel of what exactly this is doing.<br /><br />
<blockquote style="padding: 0 0.4em; border-left: #1010ff 2px solid; margin: 0;"><span style="white-space: nowrap;">what is refresh versus rescan ?</span></blockquote>
<br /><span style="white-space: nowrap;">From fts-api.h:</span><br /><br /><span style="white-space: nowrap;">/* Refresh index to make sure we see latest changes from lookups.</span><br /><span style="white-space: nowrap;"> Returns 0 if ok, -1 if error. */</span><br /><span style="white-space: nowrap;">int fts_backend_refresh(struct fts_backend *backend);</span><br /><span style="white-space: nowrap;">/* Go through the entire index and make sure all mails are indexed,</span><br /><span style="white-space: nowrap;"> and delete any extra mails in the index. */</span><br /><span style="white-space: nowrap;">int fts_backend_rescan(struct fts_backend *backend);</span><br /><br /><span style="white-space: nowrap;">Regards,</span><br /><br /><br />Stepham<br /><br /><br />
<blockquote style="padding: 0 0.4em; border-left: #1010ff 2px solid; margin: 0;"><br /><br />On January 5, 2019 14:23:10 Joan Moreau via dovecot <<a href="mailto:dovecot@dovecot.org" rel="noreferrer">dovecot@dovecot.org</a>> wrote:<br /><br />
<blockquote style="padding: 0 0.4em; border-left: #1010ff 2px solid; margin: 0;"><span style="white-space: nowrap;">Thank Stephan</span><br /><br />I basically need to know the role/description of each of the functions of the fts_backend:<br /><br /><br /><span style="white-space: nowrap;">struct fts_backend fts_backend_xapian = {</span><br /><span style="white-space: nowrap;">.name = "xapian",</span><br /><span style="white-space: nowrap;">.flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT,*-> what other flags ?*</span><br /><br /><span style="white-space: nowrap;">{</span><br /><span style="white-space: nowrap;">fts_backend_xapian_alloc,</span><br /><span style="white-space: nowrap;">fts_backend_xapian_init,</span><br /><span style="white-space: nowrap;">fts_backend_xapian_deinit,</span><br /><span style="white-space: nowrap;">fts_backend_xapian_get_last_uid,</span><br /><span style="white-space: nowrap;">fts_backend_xapian_update_init,</span><br /><span style="white-space: nowrap;">fts_backend_xapian_update_deinit,</span><br /><span style="white-space: nowrap;">fts_backend_xapian_update_set_mailbox,</span><br /><span style="white-space: nowrap;">fts_backend_xapian_update_expunge,</span><br /><span style="white-space: nowrap;">fts_backend_xapian_update_set_build_key,</span><br /><span style="white-space: nowrap;">fts_backend_xapian_update_unset_build_key,</span><br /><span style="white-space: nowrap;">fts_backend_xapian_update_build_more,</span><br /><span style="white-space: nowrap;">fts_backend_xapian_refresh,</span><br /><span style="white-space: nowrap;">fts_backend_xapian_rescan,</span><br /><span style="white-space: nowrap;">fts_backend_xapian_optimize,</span><br /><span style="white-space: nowrap;">fts_backend_default_can_lookup,</span><br /><span style="white-space: nowrap;">fts_backend_xapian_lookup,</span><br /><span style="white-space: nowrap;">fts_backend_xapian_lookup_multi,</span><br />fts_backend_xapian_lookup_done<br /><span style="white-space: nowrap;">}</span><br /><span style="white-space: nowrap;">};</span><br /><br /><br /><span style="white-space: nowrap;">THank you</span><br /><br /><span style="white-space: nowrap;">On 2019-01-05 08:49, Stephan Bosch wrote:</span><br /><br />
<blockquote style="padding: 0 0.4em; border-left: #1010ff 2px solid; margin: 0;"><br /><span style="white-space: nowrap;">Op 04/01/2019 om 11:17 schreef Joan Moreau via dovecot:</span>
<blockquote style="padding: 0 0.4em; border-left: #1010ff 2px solid; margin: 0;"><br />Why not, but please guide me about the core structure (mandatory funcitons, etc..) of a typical Dovecot FTS plugin<br /><br /></blockquote>
The Dovecot API documentation is not exhaustive everywhere, but the basics are documented. The remaining questions can be answered by looking at examples found in similar plugins or the relevant API sources.<br /><br /><span style="white-space: nowrap;">I know of one FTS plugin not written by Dovecot developers:</span><br /><br /><span style="white-space: nowrap;"><a href="https://github.com/atkinsj/fts-elasticsearch" target="_blank" rel="noopener noreferrer">https://github.com/atkinsj/fts-elasticsearch</a></span><br /><br />If you really wish to do something like this, just go ahead. It will not be a small effort though. As soon as you have concrete questions, we can help you (don't expect rapid responses though).<br /><br /><span style="white-space: nowrap;">Regards,</span><br /><br /><span style="white-space: nowrap;">Stephan.</span><br /><br /><br /><br /></blockquote>
</blockquote>
<br /><br /></blockquote>
</div>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</body></html>