Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"

Mon May 28 14:23:06 EEST 2018

On 28/05/2018 13:04, Timo Sirainen wrote:
> On 28 May 2018, at 13.28, kfx <kadafax at gmail.com 
> <mailto:kadafax at gmail.com>> wrote:
>>
>>
>>> Especially what is in the "fts" header vs. "next uid" header? Does 
>>> the UID in "fts" header keep changing every time you save a new mail? 
>>> I suppose it will.
>>
>> Diff between 2 emails:
>> next uid = 30104    |    next uid = 30105
>> last_indexed_uid = 30103    |    last_indexed_uid = 30104
> 
> So Dovecot thinks it has indexed everything.
> 
>>> You could also monitor (e.g. tcpdump/wireshark) the network traffic 
>>> between Dovecot <-> Solr what happens when a new mail arrives. I 
>>> suspect Dovecot sends it to Solr, which for whatever reason just 
>>> ignores the update.
>>
>> ################### TCPDUMP ####################################
>> POST /solr/dovecot/update HTTP/1.1
>> Host: localhost:8983
>> Date: Mon, 28 May 2018 10:18:05 GMT
>> Transfer-Encoding: chunked
>> Connection: Keep-Alive
>> Content-Type: text/xml
>>
>> <add><doc><field name="uid">37581</field><field 
>> name="box">e0c58a3093235153110800003ea484a8</field><field 
>> name="user">username</field><field 
>> name="id">37581/e0c58a3093235153110800003ea484a8/username</field><field name="body">Search 
>> Pattern: Kai8oovi
> ..
>> </field></doc></add>
> 
> And Dovecot sends the mail.
> 
>> ######### SOLR'S RESPONSE ###############
>> <?xml version="1.0" encoding="UTF-8"?>
>> <response>
>>
>> <lst name="responseHeader">
>>  <int name="status">0</int>
>>  <int name="QTime">0</int>
>> </lst>
>> </response>
> 
> And Solr receives it. Your tcpdump doesn't show <commit 
> softCommit="true" waitSearcher="true"/> being sent though. Do you see it 
> being sent anywhere? 

Yes:

################### TCPDUMP ###################################
POST /solr/dovecot/update HTTP/1.1
Host: localhost:8983
Date: Mon, 28 May 2018 10:18:05 GMT
Expect: 100-continue
Content-Length: 47
Connection: Keep-Alive
Content-Type: text/xml

<commit softCommit="true" waitSearcher="true"/>
################### /TCPDUMP ###################################


> Does it make the mails visible if you run it 
> yourself? Or if you run hard commit? :
> 
> curl http://<host>:8983/solr/update?commit=true
> 

# curl http://127.0.0.1:8983/solr/update?commit=true
<html> 
 
 
                         [0/0]
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Not Found</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/update. Reason:
<pre>    Not Found</pre></p>
</body>
</html>

# curl http://127.0.0.1:8983/solr/dovecot/update?commit=true
<?xml version="1.0" encoding="UTF-8"?>
<response>

<lst name="responseHeader">
   <int name="status">0</int>
   <int name="QTime">0</int>
</lst>
</response>

# doveadm search -u username mailbox INBOX body Kai8oovi
==> No result ('Kai8oovi' is the search pattern, it should returns 4 
results)