Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"

[Timo Sirainen]

On 28 May 2018, at 13.28, kfx <kadafax at gmail.com> wrote:
> 
> 
>> Especially what is in the "fts" header vs. "next uid" header? Does the UID in "fts" header keep changing every time you save a new mail? I suppose it will. 
> 
> Diff between 2 emails:
> next uid = 30104    |    next uid = 30105
> last_indexed_uid = 30103    |    last_indexed_uid = 30104

So Dovecot thinks it has indexed everything.

>> You could also monitor (e.g. tcpdump/wireshark) the network traffic between Dovecot <-> Solr what happens when a new mail arrives. I suspect Dovecot sends it to Solr, which for whatever reason just ignores the update.
> 
> ################### TCPDUMP ####################################
> POST /solr/dovecot/update HTTP/1.1
> Host: localhost:8983
> Date: Mon, 28 May 2018 10:18:05 GMT
> Transfer-Encoding: chunked
> Connection: Keep-Alive
> Content-Type: text/xml
> 
> <add><doc><field name="uid">37581</field><field name="box">e0c58a3093235153110800003ea484a8</field><field name="user">username</field><field name="id">37581/e0c58a3093235153110800003ea484a8/username</field><field name="body">Search Pattern: Kai8oovi
..
> </field></doc></add>

And Dovecot sends the mail.

> ######### SOLR'S RESPONSE ###############
> <?xml version="1.0" encoding="UTF-8"?>
> <response>
> 
> <lst name="responseHeader">
>  <int name="status">0</int>
>  <int name="QTime">0</int>
> </lst>
> </response>

And Solr receives it. Your tcpdump doesn't show <commit softCommit="true" waitSearcher="true"/> being sent though. Do you see it being sent anywhere? Does it make the mails visible if you run it yourself? Or if you run hard commit? :

curl http://<host>:8983/solr/update?commit=true

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dovecot.org/pipermail/dovecot/attachments/20180528/ae857324/attachment-0001.html>