<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /></head><body style='font-size: 9pt; font-family: Verdana,Geneva,sans-serif'>
<p>Hi Stephan,</p>
<p>What's up with that ?</p>
<p>Thank you so much</p>
<p><br /></p>
<div id="signature"> </div>
<p><br /></p>
<p id="reply-intro">On 2019-01-05 02:04, Stephan Bosch wrote:</p>
<blockquote type="cite" style="padding: 0 0.4em; border-left: #1010ff 2px solid; margin: 0"><!-- html ignored --><!-- head ignored --><!-- meta ignored -->
<div class="pre" style="margin: 0; padding: 0; font-family: monospace"><span style="white-space: nowrap;">Hi,</span><br /><br /><span style="white-space: nowrap;">Op 04/01/2019 om 05:36 schreef Joan Moreau via dovecot:</span>
<blockquote type="cite" style="padding: 0 0.4em; border-left: #1010ff 2px solid; margin: 0"><br />Hi<br /><br />This is the summary of my work with SOLR-Dovecot, in my *quest to reproduce the previoulsy excellent work of fts_squat*<br /><br /><br />@Aki : Based on the time I have spent on this, I would love to see you updating the Wiki with those improvements, and adding my name somewhere<br /><br /><span style="white-space: nowrap;">@All : Hope it helps</span><br /><br /></blockquote>
I'll be going through the description below soon. I've recently independently installed fts-solr from scratch. Although this wasn't a flawless effort, I managed to get some basic indexing going. From this mail thread I understand that there are quite a few more problems than I've seen myself so far. Then again, I didn't perform extensive tests with actual searches.<br /><br />Maybe we can turn all this into a test suite that we can run internally here at Dovecot. At the very least, the described Dovecot bugs need to be addressed and the wiki needs to be updated.<br /><br /><span style="white-space: nowrap;">I'll get back to you.</span><br /><br /><br /><span style="white-space: nowrap;">Regards,</span><br /><br /><span style="white-space: nowrap;">Stephan.</span><br /><br /><br /><br />
<blockquote type="cite" style="padding: 0 0.4em; border-left: #1010ff 2px solid; margin: 0"><br /><span style="white-space: nowrap;">*- Installation:*</span><br /><br />-> Create a clean install using the default, (at least in the Archlinux package), and do a "sudo -u solr solr create -c dovecot ". The config files are then in /opt/solr/server/solr/dovecot/conf and datafiles in /opt/solr/server/solr/dovecot/data<br /><br /><span style="white-space: nowrap;">-> In /opt/solr/server/solr/dovecot/conf/solrconfig.xml:</span><br /><br /> * around line 313, change <openSearcher>false</openSearcher> to <openSearcher>true</openSearcher><br /><br /> * around line 147, set <writeLockTimeout>2000</writeLockTimeout> (or above)<br /><br /><span style="white-space: nowrap;"> * around line 696 : uncomment <str name="df">hdr</str></span><br /><br /> * around line 1127, before <updateProcessor class="solr.UUIDUpdateProcessorFactory" name="uuid"/>, add <schemaFactory class="ClassicIndexSchemaFactory"></schemaFactory><br /><br /> * around line 1161, delete the whole <updateProcessor class="solr.AddSchemaFieldsUpdateProcessorFactory" name="add-schema-fields"><br /><br /> * around line 1192, remove the whole <updateRequestProcessorChain name="add-unknown-fields-to-the-schema" ... /><br /><br /><span style="white-space: nowrap;">-> Remove /opt/solr/server/solr/dovecot/conf/managed-schema</span><br /><br />-> Change "schema.xml" by the one below to reproduce fts_squat behavior (equivalent to " fts_squat = partial=3 full=25" in dovecot.conf) (note : such a huge trouble to replace a single line setup, anyway...)<br /><br />-> Move /opt/solr/server/solr (or the subfolder data) to a partition with *space*, ideally ext4 or faster file system (it looks like Solr is not considering using a simple mysql database, which would make sense to avoid all the fuzz and let it transit to a non-java state, but that is another story)<br /><br /><span style="white-space: nowrap;">-> Config of dovecot.conf is as below</span><br /><br />-> The systemd unit shall specify high ulimit for files and proc (see below)<br /><br />-> Increase the memory available for the JavaVM (I put 12Gb as I have quite a space on my server, but you may adapt it as per your specs) : in /opt/solr/bin/solr.in.sh, set SOLR_HEAP="12288m"<br /><br />-> As Solr is complaining a lot, you may consider a filter for it in your syslog-ng or journald as it pollutes greatly your audit files<br /><br /><span style="white-space: nowrap;">-> (re)Start solr (first) and dovecot by systemctl</span><br /><br /><span style="white-space: nowrap;">-> Launch redindex ( doveadm fts rescan -u <username> )</span><br /><br /><span style="white-space: nowrap;">-> wait for a big while to let the system re-index all your mail boxes</span><br /><br /><br /><span style="white-space: nowrap;">*- Bugs so far*</span><br /><br />-> Line 620 of fts_solr dovecot plugin : the size oof header is improperly calculated ("huge header" warning for a simple email, which kilss the index of that considered email, so basically MOST emails as the calculation is wrong)<br /><br />-> The UID returned by SOlr is to be considered as a STRING (and that is maybe the source of problem of the "out of bound" errors in fts_solr dovecot, as "long" is not enough)<br /><br />-> Java errors : A lot of non sense for me, I am not expert in Java. But, with increased memory, it seems not crashing, even if complaining quite a lot in the logs<br /><br /><br /><br /><br /><span style="white-space: nowrap;">*-------SCHEMA.XML in /opt/solr/server/solr/dovecot/conf*</span><br /><br /><span style="white-space: nowrap;"><?xml version="1.0" encoding="UTF-8"?></span><br /><span style="white-space: nowrap;"><schema name="dovecot" version="2.0"></span><br /><span style="white-space: nowrap;"><uniqueKey>id</uniqueKey></span><br /><fieldType name="dovecottext" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"><br /><span style="white-space: nowrap;"><analyzer type="index"></span><br /><span style="white-space: nowrap;"><tokenizer class="solr.ClassicTokenizerFactory"/></span><br /><filter class="solr.WordDelimiterGraphFilterFactory" catenateNumbers="1" generateNumberParts="1" splitOnCaseChange="1" generateWordParts="1" splitOnNumerics="1" catenateAll="1" catenateWords="1" preserveOriginal="1"/><br /><span style="white-space: nowrap;"><filter class="solr.FlattenGraphFilterFactory"/></span><br /><span style="white-space: nowrap;"><filter class="solr.LowerCaseFilterFactory"/></span><br /><span style="white-space: nowrap;"><filter class="solr.TrimFilterFactory"/></span><br /><span style="white-space: nowrap;"><filter class="solr.RemoveDuplicatesTokenFilterFactory"/></span><br /><span style="white-space: nowrap;"></analyzer></span><br /><span style="white-space: nowrap;"><analyzer type="query"></span><br /><span style="white-space: nowrap;"><tokenizer class="solr.KeywordTokenizerFactory"/></span><br /><span style="white-space: nowrap;"><filter class="solr.LowerCaseFilterFactory"/></span><br /><span style="white-space: nowrap;"><filter class="solr.TrimFilterFactory"/></span><br /><span style="white-space: nowrap;"><filter class="solr.RemoveDuplicatesTokenFilterFactory"/></span><br /><span style="white-space: nowrap;"></analyzer></span><br /><span style="white-space: nowrap;"></fieldType></span><br /><fieldType name="dovecotfield" class="solr.TextField" autoGeneratePhraseQueries="true"><br /><span style="white-space: nowrap;"><analyzer type="index"></span><br /><span style="white-space: nowrap;"><tokenizer class="solr.ClassicTokenizerFactory"/></span><br /><span style="white-space: nowrap;"><filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="25"/></span><br /><span style="white-space: nowrap;"><filter class="solr.TrimFilterFactory"/></span><br /><span style="white-space: nowrap;"><filter class="solr.LowerCaseFilterFactory"/></span><br /><span style="white-space: nowrap;"><filter class="solr.RemoveDuplicatesTokenFilterFactory"/></span><br /><span style="white-space: nowrap;"></analyzer></span><br /><span style="white-space: nowrap;"><analyzer type="query"></span><br /><span style="white-space: nowrap;"><tokenizer class="solr.KeywordTokenizerFactory"/></span><br /><span style="white-space: nowrap;"><filter class="solr.LowerCaseFilterFactory"/></span><br /><span style="white-space: nowrap;"><filter class="solr.TrimFilterFactory"/></span><br /><span style="white-space: nowrap;"><filter class="solr.RemoveDuplicatesTokenFilterFactory"/></span><br /><span style="white-space: nowrap;"></analyzer></span><br /><span style="white-space: nowrap;"></fieldType></span><br /><br /><span style="white-space: nowrap;"><fieldType name="string" class="solr.StrField"/></span><br /><span style="white-space: nowrap;"><field name="_version_" type="string" indexed="true" stored="true"/></span><br /><span style="white-space: nowrap;"><field name="bcc" type="string" indexed="false" stored="false"/></span><br /><span style="white-space: nowrap;"><field name="body" type="dovecottext" indexed="true" stored="false"/></span><br /><field name="box" type="string" indexed="true" required="true" stored="true"/><br /><span style="white-space: nowrap;"><field name="cc" type="dovecotfield" indexed="true" stored="false"/></span><br /><span style="white-space: nowrap;"><field name="from" type="dovecotfield" indexed="true" stored="false"/></span><br /><span style="white-space: nowrap;"><field name="hdr" type="string" indexed="false" stored="false"/></span><br /><field name="id" type="string" indexed="true" required="true" stored="true"/><br /><span style="white-space: nowrap;"><field name="subject" type="dovecottext" indexed="true" stored="false"/></span><br /><span style="white-space: nowrap;"><field name="to" type="dovecotfield" indexed="true" stored="false"/></span><br /><field name="uid" type="string" indexed="true" required="true" stored="true"/><br /><field name="user" type="string" indexed="true" required="true" stored="true"/><br /><span style="white-space: nowrap;"></schema></span><br /><br /><br /><span style="white-space: nowrap;">*-- DOVECOT.CONF*</span><br /><br /><span style="white-space: nowrap;">mail_plugins = fts fts_solr</span><br /><br /><span style="white-space: nowrap;">plugin {</span><br /><span style="white-space: nowrap;">plugin = fts fts_solr managesieve sieve</span><br /><br /><span style="white-space: nowrap;">fts = solr</span><br /><span style="white-space: nowrap;">fts_autoindex = yes</span><br /><span style="white-space: nowrap;">fts_enforced = yes</span><br /><span style="white-space: nowrap;">fts_solr = url=<a href="http://127.0.0.1:8983/solr/dovecot/" target="_blank" rel="noopener noreferrer">http://127.0.0.1:8983/solr/dovecot/</a></span><br /><br />(replace 127.0.0.1 by your solr server if you want to use an external server)<br /><span style="white-space: nowrap;">(...)</span><br /><br /><span style="white-space: nowrap;">}</span><br /><br /><br /><br /><span style="white-space: nowrap;">*-- /etc/systemd/system/multi-user.target.wants/solr.service*</span><br /><br /><span style="white-space: nowrap;">[Unit]</span><br /><span style="white-space: nowrap;">Description=Solr full text search engine</span><br /><span style="white-space: nowrap;">After=network.target</span><br /><br /><span style="white-space: nowrap;">[Service]</span><br /><span style="white-space: nowrap;">Type=simple</span><br /><span style="white-space: nowrap;">User=solr</span><br /><span style="white-space: nowrap;">Group=solr</span><br /><span style="white-space: nowrap;">PrivateTmp=yes</span><br /><span style="white-space: nowrap;">WorkingDirectory=/opt/solr</span><br /><span style="white-space: nowrap;">*LimitNOFILE=65000*</span><br /><span style="white-space: nowrap;">*LimitNPROC=65000*</span><br /><span style="white-space: nowrap;">ExecStart=/opt/solr/bin/solr start -f</span><br /><br /><span style="white-space: nowrap;">[Install]</span><br /><span style="white-space: nowrap;">WantedBy=multi-user.target</span><br /><br /><br /></blockquote>
</div>
</blockquote>
</body></html>