dovecot Digest, Vol 192, Issue 52

[Peter Mogensen]

On 4/14/19 4:04 PM, dovecot-request at dovecot.org wrote:

>> Solr ships with autoCommit set to 15 seconds and openSearcher set to
>> false on the autoCommit.? The autoSoftCommit setting is not enabled by
>> default, but depending on how the index was created, Solr might try to
>> set autoSoftCommit to 3 seconds ... which is WAY too short.

I just run with the default. 15s autoCommit and no autoSoftCommit

>> This thread says that dovecot is sending explicit commits.? 

I see explicit /update req. with softCommit and waitSearcer=true in a
tcpdump.

>> One thing
>> that might be happening to exceed 60 seconds is an extremely long
>> commit, which is usually caused by excessive cache autowarming, but
>> might be related to insufficient memory.? The max heap setting on an
>> out-of-the-box Solr install (5.0 and later) is 512MB.? That's VERY
>> small, and it doesn't take much index data before a much larger heap
>> is required.

I run with

SOLR_JAVA_MEM="-Xmx8g -Xms2g"

> I looked into the code (version 2.3.5.1):

This is 2.2.35. I haven't checked the source difference to 2.3.x I must
admit.

> I immagine that one of the reasons dovecot sends softCommits is because
> without autoindex active and even if mailboxes are periodically indexed
> from cron, the last emails received with be indexed at the moment of the
> search.? 

I expect that dovecot has to because of it's default behavior by only
bringing the index up-to-date just before search. So it has towait for
the index result to be available if there's been any new mails indexed.

> 1) a configurable batch size would enable to tune the number of emails
> per request and help stay under the 60 seconds hard coded http request
> timeout. A configurable http timeout would be less useful, since this
> will potentially run into other timeouts on solr side.

Being able to configure it is great.
But I don't think it solves much. I recompiled with 100 as batch size
and it still ended in timeouts.
Then I recompiled with 10min timeout and now I see all the batches
completing and their processesing time is mostly between 1 and 2 minutes
(so all would have failed).

To me it looks like Solr just takes too long time to index. This is no
small machine. It's a 20 core Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz
and for this test it's not doing anything else, so I'm a bit surprised
that even with only a few users this takes so long time.

/Peter