On 12/04/2019 12:09, Peter Mogensen via dovecot wrote:
Looking further at tcpdumps of the Dovecot->Solr traffic and Solr metrics it doesn't seem like there's anything suspicious apart from the TCP windows running full and Dovecot backing of ... until it times out and close the connection.
From my understanding of how Dovecot operates towards Solr it will flush ~1000 documents towards Solr in /update request until it has traversed the mailbox (let's say 20.000 mails), doing softCommits after each.
But is it really reasonable for Dovecot to expect that no request will take more than 60s to process by Solr? It doesn't seem like my Solr can handle that, although it does process documents and it does reasonably fast clear pending documents after Dovecot closes the connection.
On the surface it looks like Dovecot is too impatient.
/Peter
If this value was made configurable people could set it to what they want. However the underlying problem is likely on solr configuration.
Is this a problem only on initial indexing or an ongoing problem after initial indexing?
The parameters that the solr plugin are using are designed to make documents visible to searches quickly.
If there is an issue on initial indexing, where you are not really concerned about qucik visibility but just getting things into the index efficiently, a better approach would be for dovecot plugin not to send any commit or softCommit (or waitSearcher either) and that should speed things up. You'd need to configure solr with a long autoSoftCommit maxTime and a reasonable autoCommit maxTime, which you could then reconfigure when the load was done.
If you're using dovecot built from source code it should be possible to test that by some minor modification of the code in fts-backend-solr.c.
John