[enhancement] fts-solr low performance

[Aki Tuomi]

On 05.03.2018 11:07, azurIt wrote:
>> Hi,
>>
>> we have activated fts-solr about a week ago and immediately started to  
>> experience really *low* performance with MOVE and EXPUNGE commands.  
>> After several days of googling, tcpdumping and straceing i was able to  
>> find and resolve the problem.
>>
>> We are using Dovecot 2.2.27 from Debian Jessie (jessie-backports),  
>> which is doing a soft commit in solr after every MOVE or EXPUNGE  
>> command - this behavior cannot be, currently, changed. The problem is  
>> that this was causing every MOVE/EXPUNGE to take about 6 seconds to  
>> complete. The problem appears to be in very old version of Solr -  
>> 3.6.2 (!!). This is the only version which is shipped with current  
>> (Jessie) and also next (Stretch) version of Debian, don't ask my why,  
>> i don't understand it either. Solr versions below 4.0 are NOT  
>> supporting soft commits, so all commits are hard and this was the  
>> problem. Finally, i decided to patch our Dovecot to not send a commit  
>> at all and everything started to be super fast. I'm doing hard commits  
>> every minute via cron so the only consequence of this is that you  
>> cannot search for messages delivered before less then a minute (which  
>> you, usually, don't need to do anyway).
>>
>> While googling i also find out that Solr supports autoCommit function  
>> (and from version 4.0 also autoSoftCommit), so there's no reason for  
>> Dovecot to handle this on it's own (and potentially doing hundreds or  
>> thousands of soft commits every second) - you can just set Solr to,  
>> for example, do autoSoftCommit every second and autoCommit every minute:
>> https://cwiki.apache.org/confluence/display/solr/UpdateHandlers+in+SolrConfig#UpdateHandlersinSolrConfig-autoCommit
>>
>> Also this wiki page should be updated with warning about old versoins  
>> of Solr not supporting soft commits (you could also mention the  
>> auto[Soft]Commit function):
>> http://wiki2.dovecot.org/Plugins/FTS/Solr
>>
>> I suggest to allow completely disable Solr commits in Dovecot by  
>> configuration, so people like me can handle this easily. What do you  
>> think?
>>
>> azur
>
>
>
> Hi,
>
> any news on this? Even Solr documentation suggests to NOT doing commits from applications:
> https://lucene.apache.org/solr/guide/6_6/shards-and-indexing-data-in-solrcloud.html#ShardsandIndexingDatainSolrCloud-IgnoringCommitsfromClientApplicationsinSolrCloud
>
> Thanks for not ignoring me.
>
> azur

You are not being ignored. We'll attend to this eventually.

Aki