<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /></head><body style='font-size: 9pt; font-family: Verdana,Geneva,sans-serif'>
<p>I have tried to spend some time of understanding the logic (if any !) of the fts part</p>
<p><br /></p>
<p>Honestly, the one who created this mess shall be the one to fix it, or one shall refactor it totally.</p>
<p><br /></p>
<p>Basically, the fts "core" should be able to do</p>
<p>- select the backend according to conf file</p>
<p>- send new emails/maiblox to backend</p>
<p>- send teh ID of the emails to be removed</p>
<p>- resend an entire mailbox ('rescan')</p>
<p>- send the search parameters (from client) to backend and return the email to front end based on backend results (and NOTHING more)</p>
<p><br /></p>
<p>Today, the fts part is plain wong and must be totally reviewed.</p>
<p>I do not have the time but I can participate in testing if someone is ready to roll up its sleeves on teh mater</p>
<p><br /></p>
<p>THe "loop" part seems the most urgent : It breaks everything (search timeout 100% of the time)</p>
<p><br /></p>
<p><br /></p>
<p><br /></p>
<p>On 2019-04-06 09:56, Joan Moreau via dovecot wrote:</p>
<blockquote type="cite" style="padding: 0 0.4em; border-left: #1010ff 2px solid; margin: 0"><!-- html ignored --><!-- head ignored --><!-- meta ignored -->
<p>For the point 1, this is not "suboptimal", it is plain wrong (results are damn wrong ! and this is not related to the backend, but the FTS logic in Dovecot core)</p>
<div id="signature"> </div>
<p>For the point 2 , this has been discussed already numerous times but without action. The dovecot core shall be the one re-submitting the emails to scan, not the backend to try to figure out where and which are the emails to be re-scaned</p>
<p>For the point 3, I will do a bit of research in the existing code and will get back to you</p>
<p>For the point 4, this is random. FTS backend (xapian, lucene, solr, whatever..) returns X, then dovecot core choose to select only Y emails. THis is a clear bug.</p>
<p><br /></p>
<p><br /></p>
<p id="reply-intro">On 2019-04-05 20:08, Josef 'Jeff' Sipek via dovecot wrote:</p>
<blockquote style="padding: 0 0.4em; border-left: #1010ff 2px solid; margin: 0;">
<div class="pre" style="margin: 0; padding: 0; font-family: monospace;">On Fri, Apr 05, 2019 at 19:33:57 +0800, Joan Moreau via dovecot wrote:
<blockquote style="padding: 0 0.4em; border-left: #1010ff 2px solid; margin: 0;">Hi <br /><br />If you plan to fix the FTS part of Dovecot, I will be very gratefull.</blockquote>
<br />I'm trying to figure out what is causing the 3rd issue you listed, so we can<br />decide how severe it is and therefore how quickly it needs to be fixed. At<br />the moment we are unable to reproduce it, and therefore we cannot fix it.<br /><br />
<blockquote style="padding: 0 0.4em; border-left: #1010ff 2px solid; margin: 0;">Not sure this is related to any specific commit but rahter the overall<br />design</blockquote>
<br />Ok.<br /><br />
<blockquote style="padding: 0 0.4em; border-left: #1010ff 2px solid; margin: 0;">The list of bugs so far <br /><br />1 - Double call to fts plugins with inconsistent parameter (first call<br />diferent from second call for the same request)</blockquote>
<br />Understood. It is my understanding that this is simply suboptimal rather<br />than causing crashes/etc.<br /><br />
<blockquote style="padding: 0 0.4em; border-left: #1010ff 2px solid; margin: 0;">2 - "Rescan" features for now consists of deleting indexes. SHall be<br />resending emails to rescan to the fts plugin instead</blockquote>
<br />I'm not sure I follow. The rescan operation is invoked on the fts backend<br />and it is up to the implementation to somehow ensure that after it is done<br />the fts index is up to date. The easiest way to implement it is to simply<br />delete the fts index and re-index all the mails. That is what currently<br />happens in the solr backend.<br /><br />The lucene fts backend does a more complicated matching of the fts index<br />with the emails. Finally, the deprecated squat backend seem to ignore the<br />rescan requests (its rescan vfunc is NULL).<br /><br />
<blockquote style="padding: 0 0.4em; border-left: #1010ff 2px solid; margin: 0;">3 - the loop when body search (just do a "doveadm search -u user@domain<br />mailbox inbox text whatevertexte") <br /><br />Refer to my email to Timo on 2019-04-03 18:30 on the same thread for bug<br />details <br /><br />(especially the loop)</blockquote>
<br />This seems to be the most important of the 4 issues you listed, so I'd like<br />to focus on this one for now.<br /><br />As I mentioned, we cannot reproduce this ourselves. So, we need your help<br />to narrow things down. Therefore, can you give us the commit hashes of<br />revisions that you know are good and which are bad? You can use git-bisect<br />to narrow the range down.<br /><br />
<blockquote style="padding: 0 0.4em; border-left: #1010ff 2px solid; margin: 0;">4 - Most notably, I notice that header search usually does not care<br />about fts plugin (even with fts_enforced) and rely on some internal<br />search , which si total non-sense</blockquote>
<br />You're right, that doesn't seem to make sense. Can you provide a test case?<br /><br />Jeff.<br /><br />
<blockquote style="padding: 0 0.4em; border-left: #1010ff 2px solid; margin: 0;">Let me know how can I help on thos 4 points <br /><br />On 2019-04-05 18:37, Josef 'Jeff' Sipek wrote:<br /><br />
<blockquote style="padding: 0 0.4em; border-left: #1010ff 2px solid; margin: 0;">On Fri, Apr 05, 2019 at 17:45:36 +0800, Joan Moreau wrote: <br /><br />
<blockquote style="padding: 0 0.4em; border-left: #1010ff 2px solid; margin: 0;">I am on master (very latest) <br /><br />No clue exactly when this problem appears, but <br /><br />1 - the "request twice the fts plugin instead of once" issue has always<br />been there (since my first RC release of fts-xapian)</blockquote>
<br />Ok, good to know.<br /><br />
<blockquote style="padding: 0 0.4em; border-left: #1010ff 2px solid; margin: 0;">2 - the body/text loop has appeared recently (maybe during the month of<br />March)</blockquote>
<br />Our testing doesn't seem to be able to reproduce this. Can you try to<br />git-bisect this to find which commit broke it?<br /><br />Thanks,<br /><br />Jeff.<br /><br />On 2019-04-05 16:36, Josef 'Jeff' Sipek via dovecot wrote:<br /><br />On Wed, Apr 03, 2019 at 19:02:52 +0800, Joan Moreau via dovecot wrote: <br /><br />issue seems in the Git version : <br />Which git revision?<br /><br />Before you updated to the broken revision, which revision/version were you<br />running?<br /><br />Can you try it with 5f6e39c50ec79ba8847b2fdb571a9152c71cd1b6 (the commit<br />just before the fts_enforced=body introduction)? That's the only recent fts<br />change.<br /><br />Thanks,<br /><br />Jeff.<br /><br />On 2019-04-03 18:58, @lbutlr via dovecot wrote:<br /><br />On 3 Apr 2019, at 04:30, Joan Moreau via dovecot <<a href="mailto:dovecot@dovecot.org" rel="noreferrer">dovecot@dovecot.org</a>> wrote: <br /><br />doveadm search -u <a href="mailto:jom@grosjo.net" rel="noreferrer">jom@grosjo.net</a> mailbox inbox text milan <br />Did that search over my list mail and got 83 results, not able to duplicate your issue.<br /><br />What version of dovecot and have you tried to reindex?<br /><br />dovecot-2.3.5.1 here.</blockquote>
</blockquote>
</div>
</blockquote>
</blockquote>
</body></html>