Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"

[kfx]

Okay I'm progressing here. If I copy an email within my MUA to another 
folder and then make another copy back to my Inbox, the new copied email 
is correctly and immediately indexed ! If I delete an email, the search 
result is also correct and immediate.

Is it possible that my problem come from Dovecot LDA ?

In 10-mail.conf I have:
# cat dovecot/conf.d/10-mail.conf| grep -v -e "^$" -e "^#"  -e "^  #"
mail_location = maildir:/var/vmail/%Ln
namespace {
   type = private
   location = maildir:/var/vmail/%Ln
   inbox = yes
mailbox Trash {
   auto = subscribe
   special_use = \Trash
}
mailbox Drafts {
   auto = subscribe
   special_use = \Drafts
}
mailbox Sent {
   auto = subscribe # autocreate, autosubscribe
   special_use = \Sent
}
mailbox "Sent Messages" {
   special_use = \Sent
}
}
mail_uid=vmail
mail_gid=vmail
auth_socket_path = /var/run/dovecot/auth-userdb
mail_plugins = $mail_plugins quota stats fts fts_solr

And in 15-lda.conf:
# cat dovecot/conf.d/15-lda.conf| grep -v -e "^$" -e "^#"  -e "^  #"
postmaster_address = postmaster at mydomain
hostname = mydomain
recipient_delimiter = +
lda_mailbox_autocreate = yes
lda_mailbox_autosubscribe = yes
protocol lda {
   mail_plugins = $mail_plugins sieve virtual
}

And the lda portion of "doveconf -n" become:
protocol lda {
   mail_plugins = " quota stats fts fts_solr sieve virtual"
}



  Here is my postfix relevant conf:

################## MAIN.CF ##############################
# postconf -n | grep dovecot
dovecot_destination_recipient_limit = 1
smtpd_sasl_type = dovecot
virtual_transport = dovecot
#########################################################

################## MASTER.CF ##############################
dovecot unix    -       n       n       -       -      pipe
   flags=DRhu user=vmail:vmail 
argv=/usr/local/libexec/dovecot/dovecot-lda -f ${sender} -a ${recipient} 
-d ${user}@${nexthop}
#########################################################



On 28/05/2018 13:30, kfx wrote:
> On 28/05/2018 13:23, kfx wrote:
>> On 28/05/2018 13:04, Timo Sirainen wrote:
>>> On 28 May 2018, at 13.28, kfx <kadafax at gmail.com 
>>> <mailto:kadafax at gmail.com>> wrote:
>>>>
>>>>
>>>>> Especially what is in the "fts" header vs. "next uid" header? Does 
>>>>> the UID in "fts" header keep changing every time you save a new 
>>>>> mail? I suppose it will.
>>>>
>>>> Diff between 2 emails:
>>>> next uid = 30104    |    next uid = 30105
>>>> last_indexed_uid = 30103    |    last_indexed_uid = 30104
>>>
>>> So Dovecot thinks it has indexed everything.
>>>
>>>>> You could also monitor (e.g. tcpdump/wireshark) the network traffic 
>>>>> between Dovecot <-> Solr what happens when a new mail arrives. I 
>>>>> suspect Dovecot sends it to Solr, which for whatever reason just 
>>>>> ignores the update.
>>>>
>>>> ################### TCPDUMP ####################################
>>>> POST /solr/dovecot/update HTTP/1.1
>>>> Host: localhost:8983
>>>> Date: Mon, 28 May 2018 10:18:05 GMT
>>>> Transfer-Encoding: chunked
>>>> Connection: Keep-Alive
>>>> Content-Type: text/xml
>>>>
>>>> <add><doc><field name="uid">37581</field><field 
>>>> name="box">e0c58a3093235153110800003ea484a8</field><field 
>>>> name="user">username</field><field 
>>>> name="id">37581/e0c58a3093235153110800003ea484a8/username</field><field 
>>>> name="body">Search Pattern: Kai8oovi
>>> ..
>>>> </field></doc></add>
>>>
>>> And Dovecot sends the mail.
>>>
>>>> ######### SOLR'S RESPONSE ###############
>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>> <response>
>>>>
>>>> <lst name="responseHeader">
>>>>  <int name="status">0</int>
>>>>  <int name="QTime">0</int>
>>>> </lst>
>>>> </response>
>>>
>>> And Solr receives it. Your tcpdump doesn't show <commit 
>>> softCommit="true" waitSearcher="true"/> being sent though. Do you see 
>>> it being sent anywhere? 
>>
>> Yes:
>>
>> ################### TCPDUMP ###################################
>> POST /solr/dovecot/update HTTP/1.1
>> Host: localhost:8983
>> Date: Mon, 28 May 2018 10:18:05 GMT
>> Expect: 100-continue
>> Content-Length: 47
>> Connection: Keep-Alive
>> Content-Type: text/xml
>>
>> <commit softCommit="true" waitSearcher="true"/>
>> ################### /TCPDUMP ###################################
>>
>>
>>
>>> Does it make the mails visible if you run it yourself? Or if you run 
>>> hard commit? :
>>>
>>> curl http://<host>:8983/solr/update?commit=true
>>>
>>
>> # curl http://127.0.0.1:8983/solr/update?commit=true
>> <html>
>>
>>
>>                          [0/0]
>> <head>
>> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
>> <title>Error 404 Not Found</title>
>> </head>
>> <body><h2>HTTP ERROR 404</h2>
>> <p>Problem accessing /solr/update. Reason:
>> <pre>    Not Found</pre></p>
>> </body>
>> </html>
>>
>> # curl http://127.0.0.1:8983/solr/dovecot/update?commit=true
>> <?xml version="1.0" encoding="UTF-8"?>
>> <response>
>>
>> <lst name="responseHeader">
>>    <int name="status">0</int>
>>    <int name="QTime">0</int>
>> </lst>
>> </response>
>>
>> # doveadm search -u username mailbox INBOX body Kai8oovi
>> ==> No result ('Kai8oovi' is the search pattern, it should returns 4 
>> results)
> 
> In the web interface of solr at http://127.0.0.1:8983/solr/#/~cores/dovecot
> 
> I can see:
>> lastModified: less than a minute ago
>> version:1428772
>> numDocs:6353615
>> maxDoc:6356213
>> deletedDocs:2598
> 
> So it IS indexing :(
> Just below I see the "optimized:" parameter followed by an icon which 
> seems saying that is NOT "optimized". Don't know if it's relevant.
> This is driving me crazy :(
>