Looking further at tcpdumps of the Dovecot->Solr traffic and Solr metrics it doesn't seem like there's anything suspicious apart from the TCP windows running full and Dovecot backing of ... until it times out and close the connection.
From my understanding of how Dovecot operates towards Solr it will flush ~1000 documents towards Solr in /update request until it has traversed the mailbox (let's say 20.000 mails), doing softCommits after each.
But is it really reasonable for Dovecot to expect that no request will take more than 60s to process by Solr? It doesn't seem like my Solr can handle that, although it does process documents and it does reasonably fast clear pending documents after Dovecot closes the connection.
On the surface it looks like Dovecot is too impatient.
/Peter
On 4/10/19 6:25 PM, Peter Mogensen wrote:
On 4/4/19 6:57 PM, Peter Mogensen wrote:
On 4/4/19 6:47 PM, dovecot-request@dovecot.org wrote:
For a typical Solr index, 60 seconds is an eternity. Most people aim for query times of 100 milliseconds or less, and they often achieve that goal.
I'm pretty sure I get these while indexing, not querying.
Apr 04 16:44:50 host dovecot[114690]: indexer-worker(me@example.com): Error: fts_solr: Indexing failed: Request timed out (Request queued 66.015 secs ago, 1 attempts in 66.005 secs, 63.146 in http ioloop, 0.000 in other ioloops, connected 94.903 secs ago)
Doing a TCP dump on indexing operations which consistently fail, I see that there's a lot of softCommits which never get an HTTP answer:
============================================== POST /solr/dovebody/update HTTP/1.1 Host: localhost:8983 Date: Wed, 10 Apr 2019 14:22:29 GMT Expect: 100-continue Content-Length: 47 Connection: Keep-Alive Content-Type: text/xml
HTTP/1.1 100 Continue
<commit softCommit="true" waitSearcher="true"/>
....