Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"

Mon May 28 14:30:32 EEST 2018

On 28/05/2018 13:23, kfx wrote:
> On 28/05/2018 13:04, Timo Sirainen wrote:
>> On 28 May 2018, at 13.28, kfx <kadafax at gmail.com 
>> <mailto:kadafax at gmail.com>> wrote:
>>>
>>>
>>>> Especially what is in the "fts" header vs. "next uid" header? Does 
>>>> the UID in "fts" header keep changing every time you save a new 
>>>> mail? I suppose it will.
>>>
>>> Diff between 2 emails:
>>> next uid = 30104    |    next uid = 30105
>>> last_indexed_uid = 30103    |    last_indexed_uid = 30104
>>
>> So Dovecot thinks it has indexed everything.
>>
>>>> You could also monitor (e.g. tcpdump/wireshark) the network traffic 
>>>> between Dovecot <-> Solr what happens when a new mail arrives. I 
>>>> suspect Dovecot sends it to Solr, which for whatever reason just 
>>>> ignores the update.
>>>
>>> ################### TCPDUMP ####################################
>>> POST /solr/dovecot/update HTTP/1.1
>>> Host: localhost:8983
>>> Date: Mon, 28 May 2018 10:18:05 GMT
>>> Transfer-Encoding: chunked
>>> Connection: Keep-Alive
>>> Content-Type: text/xml
>>>
>>> <add><doc><field name="uid">37581</field><field 
>>> name="box">e0c58a3093235153110800003ea484a8</field><field 
>>> name="user">username</field><field 
>>> name="id">37581/e0c58a3093235153110800003ea484a8/username</field><field 
>>> name="body">Search Pattern: Kai8oovi
>> ..
>>> </field></doc></add>
>>
>> And Dovecot sends the mail.
>>
>>> ######### SOLR'S RESPONSE ###############
>>> <?xml version="1.0" encoding="UTF-8"?>
>>> <response>
>>>
>>> <lst name="responseHeader">
>>>  <int name="status">0</int>
>>>  <int name="QTime">0</int>
>>> </lst>
>>> </response>
>>
>> And Solr receives it. Your tcpdump doesn't show <commit 
>> softCommit="true" waitSearcher="true"/> being sent though. Do you see 
>> it being sent anywhere? 
> 
> Yes:
> 
> ################### TCPDUMP ###################################
> POST /solr/dovecot/update HTTP/1.1
> Host: localhost:8983
> Date: Mon, 28 May 2018 10:18:05 GMT
> Expect: 100-continue
> Content-Length: 47
> Connection: Keep-Alive
> Content-Type: text/xml
> 
> <commit softCommit="true" waitSearcher="true"/>
> ################### /TCPDUMP ###################################
> 
> 
> 
>> Does it make the mails visible if you run it yourself? Or if you run 
>> hard commit? :
>>
>> curl http://<host>:8983/solr/update?commit=true
>>
> 
> # curl http://127.0.0.1:8983/solr/update?commit=true
> <html>
> 
> 
>                          [0/0]
> <head>
> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
> <title>Error 404 Not Found</title>
> </head>
> <body><h2>HTTP ERROR 404</h2>
> <p>Problem accessing /solr/update. Reason:
> <pre>    Not Found</pre></p>
> </body>
> </html>
> 
> # curl http://127.0.0.1:8983/solr/dovecot/update?commit=true
> <?xml version="1.0" encoding="UTF-8"?>
> <response>
> 
> <lst name="responseHeader">
>    <int name="status">0</int>
>    <int name="QTime">0</int>
> </lst>
> </response>
> 
> # doveadm search -u username mailbox INBOX body Kai8oovi
> ==> No result ('Kai8oovi' is the search pattern, it should returns 4 
> results)

In the web interface of solr at http://127.0.0.1:8983/solr/#/~cores/dovecot

I can see:
> lastModified: less than a minute ago
> version:1428772
> numDocs:6353615
> maxDoc:6356213
> deletedDocs:2598

So it IS indexing :(
Just below I see the "optimized:" parameter followed by an icon which 
seems saying that is NOT "optimized". Don't know if it's relevant.
This is driving me crazy :(