[enhancement] fts-solr low performance

[azurit at pobox.sk]

Hi,

we have activated fts-solr about a week ago and immediately started to  
experience really *low* performance with MOVE and EXPUNGE commands.  
After several days of googling, tcpdumping and straceing i was able to  
find and resolve the problem.

We are using Dovecot 2.2.27 from Debian Jessie (jessie-backports),  
which is doing a soft commit in solr after every MOVE or EXPUNGE  
command - this behavior cannot be, currently, changed. The problem is  
that this was causing every MOVE/EXPUNGE to take about 6 seconds to  
complete. The problem appears to be in very old version of Solr -  
3.6.2 (!!). This is the only version which is shipped with current  
(Jessie) and also next (Stretch) version of Debian, don't ask my why,  
i don't understand it either. Solr versions below 4.0 are NOT  
supporting soft commits, so all commits are hard and this was the  
problem. Finally, i decided to patch our Dovecot to not send a commit  
at all and everything started to be super fast. I'm doing hard commits  
every minute via cron so the only consequence of this is that you  
cannot search for messages delivered before less then a minute (which  
you, usually, don't need to do anyway).

While googling i also find out that Solr supports autoCommit function  
(and from version 4.0 also autoSoftCommit), so there's no reason for  
Dovecot to handle this on it's own (and potentially doing hundreds or  
thousands of soft commits every second) - you can just set Solr to,  
for example, do autoSoftCommit every second and autoCommit every minute:
https://cwiki.apache.org/confluence/display/solr/UpdateHandlers+in+SolrConfig#UpdateHandlersinSolrConfig-autoCommit

Also this wiki page should be updated with warning about old versoins  
of Solr not supporting soft commits (you could also mention the  
auto[Soft]Commit function):
http://wiki2.dovecot.org/Plugins/FTS/Solr

I suggest to allow completely disable Solr commits in Dovecot by  
configuration, so people like me can handle this easily. What do you  
think?

azur