Solr connection timeout hardwired to 60s

[John Fawcett]

On 12/04/2019 12:09, Peter Mogensen via dovecot wrote:
> Looking further at tcpdumps of the Dovecot->Solr traffic and Solr
> metrics it doesn't seem like there's anything suspicious apart from the
> TCP windows running full and Dovecot backing of ... until it times out
> and close the connection.
>
> From my understanding of how Dovecot operates towards Solr it will flush
> ~1000 documents towards Solr in /update request until it has traversed
> the mailbox (let's say 20.000 mails), doing softCommits after each.
>
> But is it really reasonable for Dovecot to expect that no request will
> take more than 60s to process by Solr?
> It doesn't seem like my Solr can handle that, although it does process
> documents and it does reasonably fast clear pending documents after
> Dovecot closes the connection.
>
> On the surface it looks like Dovecot is too impatient.
>
> /Peter

If this value was made configurable people could set it to what they
want. However the underlying problem is likely on solr configuration.

Is this a problem only on initial indexing or an ongoing problem after
initial indexing?

The parameters that the solr plugin are using are designed to make
documents visible to searches quickly.

If there is an issue on initial indexing, where you are not really
concerned about qucik visibility but just getting things into the index
efficiently, a better approach would be for dovecot plugin not to send
any commit or softCommit (or waitSearcher either) and that should speed
things up. You'd need to configure solr with a long autoSoftCommit
maxTime and a reasonable autoCommit maxTime, which you could then
reconfigure when the load was done.

If you're using dovecot built from source code it should be possible to
test that by some minor modification of the code in  fts-backend-solr.c.

John