Okay I'm progressing here. If I copy an email within my MUA to another folder and then make another copy back to my Inbox, the new copied email is correctly and immediately indexed ! If I delete an email, the search result is also correct and immediate.
Is it possible that my problem come from Dovecot LDA ?
In 10-mail.conf I have: # cat dovecot/conf.d/10-mail.conf| grep -v -e "^$" -e "^#" -e "^ #" mail_location = maildir:/var/vmail/%Ln namespace { type = private location = maildir:/var/vmail/%Ln inbox = yes mailbox Trash { auto = subscribe special_use = \Trash } mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Sent { auto = subscribe # autocreate, autosubscribe special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } } mail_uid=vmail mail_gid=vmail auth_socket_path = /var/run/dovecot/auth-userdb mail_plugins = $mail_plugins quota stats fts fts_solr
And in 15-lda.conf: # cat dovecot/conf.d/15-lda.conf| grep -v -e "^$" -e "^#" -e "^ #" postmaster_address = postmaster@mydomain hostname = mydomain recipient_delimiter = + lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes protocol lda { mail_plugins = $mail_plugins sieve virtual }
And the lda portion of "doveconf -n" become: protocol lda { mail_plugins = " quota stats fts fts_solr sieve virtual" }
Here is my postfix relevant conf:
################## MAIN.CF ############################## # postconf -n | grep dovecot dovecot_destination_recipient_limit = 1 smtpd_sasl_type = dovecot virtual_transport = dovecot #########################################################
################## MASTER.CF ############################## dovecot unix - n n - - pipe flags=DRhu user=vmail:vmail argv=/usr/local/libexec/dovecot/dovecot-lda -f ${sender} -a ${recipient} -d ${user}@${nexthop} #########################################################
On 28/05/2018 13:30, kfx wrote:
On 28/05/2018 13:23, kfx wrote:
On 28/05/2018 13:04, Timo Sirainen wrote:
On 28 May 2018, at 13.28, kfx <kadafax@gmail.com <mailto:kadafax@gmail.com>> wrote:
Especially what is in the "fts" header vs. "next uid" header? Does the UID in "fts" header keep changing every time you save a new mail? I suppose it will.
Diff between 2 emails: next uid = 30104 | next uid = 30105 last_indexed_uid = 30103 | last_indexed_uid = 30104
So Dovecot thinks it has indexed everything.
You could also monitor (e.g. tcpdump/wireshark) the network traffic between Dovecot <-> Solr what happens when a new mail arrives. I suspect Dovecot sends it to Solr, which for whatever reason just ignores the update.
################### TCPDUMP #################################### POST /solr/dovecot/update HTTP/1.1 Host: localhost:8983 Date: Mon, 28 May 2018 10:18:05 GMT Transfer-Encoding: chunked Connection: Keep-Alive Content-Type: text/xml
<add><doc><field name="uid">37581</field><field name="box">e0c58a3093235153110800003ea484a8</field><field name="user">username</field><field name="id">37581/e0c58a3093235153110800003ea484a8/username</field><field name="body">Search Pattern: Kai8oovi .. </field></doc></add>
And Dovecot sends the mail.
######### SOLR'S RESPONSE ############### <?xml version="1.0" encoding="UTF-8"?> <response>
<lst name="responseHeader"> <int name="status">0</int> <int name="QTime">0</int> </lst> </response>
And Solr receives it. Your tcpdump doesn't show <commit softCommit="true" waitSearcher="true"/> being sent though. Do you see it being sent anywhere?
Yes:
################### TCPDUMP ################################### POST /solr/dovecot/update HTTP/1.1 Host: localhost:8983 Date: Mon, 28 May 2018 10:18:05 GMT Expect: 100-continue Content-Length: 47 Connection: Keep-Alive Content-Type: text/xml
<commit softCommit="true" waitSearcher="true"/> ################### /TCPDUMP ###################################
Does it make the mails visible if you run it yourself? Or if you run hard commit? :
# curl http://127.0.0.1:8983/solr/update?commit=true <html>
[0/0] <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 404 Not Found</title> </head> <body><h2>HTTP ERROR 404</h2> <p>Problem accessing /solr/update. Reason: <pre> Not Found</pre></p> </body> </html>
# curl http://127.0.0.1:8983/solr/dovecot/update?commit=true <?xml version="1.0" encoding="UTF-8"?> <response>
<lst name="responseHeader"> <int name="status">0</int> <int name="QTime">0</int> </lst> </response>
# doveadm search -u username mailbox INBOX body Kai8oovi ==> No result ('Kai8oovi' is the search pattern, it should returns 4 results)
In the web interface of solr at http://127.0.0.1:8983/solr/#/~cores/dovecot
I can see:
lastModified: less than a minute ago version:1428772 numDocs:6353615 maxDoc:6356213 deletedDocs:2598
So it IS indexing :( Just below I see the "optimized:" parameter followed by an icon which seems saying that is NOT "optimized". Don't know if it's relevant. This is driving me crazy :(