On 4/14/19 4:04 PM, dovecot-request@dovecot.org wrote:
Solr ships with autoCommit set to 15 seconds and openSearcher set to false on the autoCommit.? The autoSoftCommit setting is not enabled by default, but depending on how the index was created, Solr might try to set autoSoftCommit to 3 seconds ... which is WAY too short.
I just run with the default. 15s autoCommit and no autoSoftCommit
This thread says that dovecot is sending explicit commits.?
I see explicit /update req. with softCommit and waitSearcer=true in a tcpdump.
One thing that might be happening to exceed 60 seconds is an extremely long commit, which is usually caused by excessive cache autowarming, but might be related to insufficient memory.? The max heap setting on an out-of-the-box Solr install (5.0 and later) is 512MB.? That's VERY small, and it doesn't take much index data before a much larger heap is required.
I run with
SOLR_JAVA_MEM="-Xmx8g -Xms2g"
I looked into the code (version 2.3.5.1):
This is 2.2.35. I haven't checked the source difference to 2.3.x I must admit.
I immagine that one of the reasons dovecot sends softCommits is because without autoindex active and even if mailboxes are periodically indexed from cron, the last emails received with be indexed at the moment of the search.?
I expect that dovecot has to because of it's default behavior by only bringing the index up-to-date just before search. So it has towait for the index result to be available if there's been any new mails indexed.
- a configurable batch size would enable to tune the number of emails per request and help stay under the 60 seconds hard coded http request timeout. A configurable http timeout would be less useful, since this will potentially run into other timeouts on solr side.
Being able to configure it is great. But I don't think it solves much. I recompiled with 100 as batch size and it still ended in timeouts. Then I recompiled with 10min timeout and now I see all the batches completing and their processesing time is mostly between 1 and 2 minutes (so all would have failed).
To me it looks like Solr just takes too long time to index. This is no small machine. It's a 20 core Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz and for this test it's not doing anything else, so I'm a bit surprised that even with only a few users this takes so long time.
/Peter