<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Joan,</p>
<p>The reason for dropping squat, I'm assuming, is that Lucene and
Solr potentially provide superior features & performance and
as they are 3rd-party libraries & apps it reduces the
maintenance responsibilities and let's the Dovecot team focus on
mail server specific stuff - and let others focus on FTS. There
is a *huge* difference between a functional Solr setup & squat
- and if I'm able to get it working we should be able to get you
there as well.<br>
</p>
<p>I don't recall what OS you're running - I'm on Ubuntu 18.04. My
Java version is OpenJDK 10.0.2. Attached is my complete Solr
config. Try one more time - stop the server, delete the data
folder, unpack the attached into the conf folder - and restart. I
also have</p>
<p><br>
</p>
/etc/default/solr.in.sh:<br>
SOLR_OPTS="$SOLR_OPTS -Dsolr.autoSoftCommit.maxTime=3000"<br>
SOLR_OPTS="$SOLR_OPTS -Dsolr.autoCommit.maxTime=60000"<br>
SOLR_PID_DIR=/run/solr<br>
SOLR_HOME=/usr/local/lib<br>
<p>Adjust the above folders as appropriate - or don't use them at
all if you're using the defaults.</p>
<p><br>
</p>
/etc/systemd/system/solr.service:<br>
# put this file in /etc/systemd/system/ as root<br>
# below paths assume solr installed in /opt/solr, SOLR_PID_DIR is
/data<br>
# and that all configuration exists in /etc/default/solr.in.sh which
is the case if previously installed as an init.d service<br>
# change port in pid file if differs<br>
# note that it is configured to auto restart solr if it fails
(Restart=on-faliure) and that's the motivation indeed :)<br>
# to switch from systemv (init.d) to systemd, do the following after
creating this file:<br>
# sudo systemctl daemon-reload<br>
# sudo service solr stop # if already running<br>
# sudo systemctl enable solr<br>
# systemctl start solr<br>
# this was inspired by
<a class="moz-txt-link-freetext" href="https://confluence.t5.fi/display/~stefan.roos/2015/04/01/Creating+systemd+unit+(service)+for+Apache+Solr">https://confluence.t5.fi/display/~stefan.roos/2015/04/01/Creating+systemd+unit+(service)+for+Apache+Solr</a><br>
[Unit]<br>
Description=Apache SOLR 7.5.0<br>
After=syslog.target network.target remote-fs.target
nss-lookup.target systemd-journald-dev-log.socket<br>
Before=multi-user.target graphical.target nginx.service
dovecot.service<br>
Conflicts=shutdown.target<br>
[Service]<br>
LimitNOFILE=65000<br>
User=vmail<br>
Group=mail<br>
ExecStartPre=/bin/mkdir -p /run/solr<br>
ExecStartPre=/bin/chown -R vmail.mail /run/solr<br>
PermissionsStartOnly=true<br>
PIDFile=/run/solr/solr-8983.pid<br>
Environment=SOLR_INCLUDE=/etc/default/solr.in.sh<br>
ExecStart=/opt/solr/bin/solr start<br>
ExecStop=/opt/solr/bin/solr stop<br>
Restart=on-failure<br>
RestartSec=15s<br>
TimeoutStopSec=30s<br>
[Install]<br>
WantedBy=multi-user.target graphical.target dovecot.service<br>
<p>If you don't use systemd disregard - but see if any of the above
applies for your setup.</p>
Let me know what happens. I agree this can be a mortal pain to
setup - but it's worth it.<br>
<pre class="moz-signature" cols="72">Daniel
</pre>
<div class="moz-cite-prefix">On 12/21/2018 4:33 AM, Joan Moreau
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:80493d57e80f87acae498315bb2e04f8@grosjo.net">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<p>Dear Daniel.</p>
<p>Thank you for your kind reply.</p>
<p>Regarding NFS, no, there is nothing like this in my setup.</p>
<p>Deleteing SOLR and recreating it, I did it so many times
already.</p>
<p>I started with *your* setup in the first place, as FTS_squat
(which actually works very well and very straightforward, I have
no clue why going for SOlr which is just a pain and not
maintaining squat), and it leads to totally funny results (for
instance, I type "emirates" in my "Air Companies" subfolder and
get a lot of results .. but of competing companies :D )</p>
<p>I added the fts_enforce following AKi advice.</p>
<p>I removed fts_decoder for the time being.</p>
<p>I don't know where to go now. Dovcot still returning errors and
SOlr still companinig with "Out of range" and other Java errors.</p>
<p>Bottom line, I am back to squat, but as it is not maintained so
crashed also times to times.</p>
<p><br>
</p>
<p>I think we should discuss on</p>
<p>(1) Why the damn choice of Solr has been main. As you
empahised, maintainend so many independent software is a pain</p>
<p>(2) If there is a real reason why going for SOlr, how to have a
working (i.e. getting the right results to the end user) setup ?</p>
<p>(3) If there iare no tangible reason, what about maintaining
fts_squat , which did the job nicely for years and no complains
about.</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<div id="signature"> </div>
<p><br>
</p>
<p id="reply-intro">On 2018-12-16 08:51, Daniel Miller via dovecot
wrote:</p>
<blockquote type="cite" style="padding: 0 0.4em; border-left:
#1010ff 2px solid; margin: 0"><!-- html ignored -->
<!-- head ignored --><!-- meta ignored -->
<p>Joan,</p>
<p>I understand and sympathize with your frustration - trying to
get multiple applications to work together, particularly given
the lack of documentation for some of them, can be extremely
challenging. That said, I suggest you consider an alternative
viewpoint. Frequently being misunderstood myself I apologize
in advance if I'm reading you wrong - but it appears your view
towards the situation is there is a bug in Dovecot related to
this problem. That may well be - but I generally approach
these matters from the assumption that *I* made the error in
configuration and go from there. I'm not an official rep for
any product nor claim to be any form of expert in these
matters - but I do have a working setup and I'd like to help
you if I can. If you're willing to - take a deep breath and
let's try starting over.</p>
<p>Looking back through your emails there were two items that
stood out - your Dovecot config has two settings I don't use:
"fts_decoder" and "fts_enforced". I also asked you earlier
whether or not NFS is involved here and I didn't see an answer
- please clarify.</p>
<p>I suggest you try once more: delete Solr completely.
Re-install per the directions and use *my* managed-schema.
Also comment out the Dovecot directives for "fts_decoder" and
"fts_enforced" so you're closer to my setup. Try running
again and then post back - I'll do what I can. Based on the
fact that Dovecot+Solr 7.5+my schema is working for me leads
me to believe we can get it working for you as well.</p>
<pre class="moz-signature">Daniel
</pre>
<div class="moz-cite-prefix">On 12/15/2018 2:42 PM, Joan Moreau
wrote:</div>
<blockquote type="cite" style="padding: 0 0.4em; border-left:
#1010ff 2px solid; margin: 0">
<p>here my latest schema.xml (remove the "long" type hich
seems to be very deprecated in 7.x)</p>
<p><?xml version="1.0" encoding="UTF-8"?><br>
<schema name="dovecot" version="2.0"><br>
<uniqueKey>id</uniqueKey><br>
<types><br>
<fieldType name="string" class="solr.StrField" /><br>
<fieldType name="gjlong" class="solr.LongPointField"
positionIncrementGap="0" /><br>
<fieldType name="gjtext" class="solr.TextField"
autoGeneratePhraseQueries="true"
positionIncrementGap="100"><br>
<analyzer type="index"><br>
<tokenizer class="solr.StandardTokenizerFactory"/><br>
<filter class="solr.StopFilterFactory"
words="stopwords.txt" ignoreCase="true"/><br>
<filter class="solr.WordDelimiterGraphFilterFactory"
generateWordParts="1" generateNumberParts="1"
splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1"
catenateNumbers="1" catenateAll="1"/><br>
<filter class="solr.FlattenGraphFilterFactory"/>
<!-- required on index analyzers after graph filters
--><br>
<filter class="solr.LowerCaseFilterFactory"/><br>
<filter class="solr.NGramFilterFactory" minGramSize="3"
maxGramSize="15" /><br>
<filter class="solr.KeywordMarkerFilterFactory"
protected="protwords.txt"/><br>
<filter class="solr.PorterStemFilterFactory"/><br>
</analyzer><br>
<analyzer type="query"><br>
<tokenizer class="solr.StandardTokenizerFactory"/><br>
<filter class="solr.SynonymGraphFilterFactory"
expand="true" ignoreCase="true" synonyms="synonyms.txt"/><br>
<filter class="solr.FlattenGraphFilterFactory"/>
<!-- required on index analyzers after graph filters
--><br>
<filter class="solr.StopFilterFactory"
words="stopwords.txt" ignoreCase="true"/><br>
<filter class="solr.WordDelimiterGraphFilterFactory"
generateWordParts="1" generateNumberParts="1"
splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1"
catenateNumbers="1" catenateAll="1"/><br>
<filter class="solr.LowerCaseFilterFactory"/><br>
<filter class="solr.NGramFilterFactory" minGramSize="3"
maxGramSize="15" /><br>
<filter class="solr.KeywordMarkerFilterFactory"
protected="protwords.txt"/><br>
<filter class="solr.PorterStemFilterFactory"/><br>
</analyzer><br>
</fieldType><br>
</types><br>
<fields><br>
<field name="_version_" type="string" indexed="true"
stored="true"/><br>
<field name="bcc" type="string" indexed="false"
stored="false"/><br>
<field name="body" type="gjtext" indexed="true"
stored="false"/><br>
<field name="box" type="string" indexed="true"
required="true" stored="true"/><br>
<field name="hdr" type="gjtext" indexed="false"
stored="false"/><br>
<field name="cc" type="gjtext" indexed="true"
stored="false"/><br>
<field name="from" type="gjtext" indexed="true"
stored="false"/><br>
<field name="id" type="string" indexed="true"
required="true" stored="true"/><br>
<field name="subject" type="gjtext" indexed="true"
stored="false"/><br>
<field name="to" type="gjtext" indexed="true"
stored="false"/><br>
<field name="uid" type="string" indexed="true"
required="true" stored="true"/><br>
<field name="user" type="string" indexed="true"
required="true" stored="true"/><br>
</fields><br>
</schema></p>
<p><br>
</p>
<div id="signature"> </div>
<p><br>
</p>
<p id="reply-intro">On 2018-12-15 20:54, Joan Moreau wrote:</p>
<blockquote style="padding: 0 0.4em; border-left: #1010ff 2px
solid; margin: 0;">
<div dir="auto">
<div dir="auto"> </div>
<div dir="auto">Daniel,</div>
<div dir="auto"> </div>
<div dir="auto">I have done that so any times (deleteing
the data folders, recreating the instance, restarting
etc...)</div>
<div dir="auto"> </div>
<div dir="auto">But this is really not the issue</div>
<div dir="auto"> </div>
<div dir="auto">The issue is</div>
<div dir="auto">1 - fts_solr reports errors in the log
file (this is a pure dovecot issue) : how to have much
more details on what fts_solr sends to Slor server and
what does it returns ?</div>
<div dir="auto">2 - Solr returns properly for a few hours,
then starts crashing or responding non-sense after some
time </div>
<div dir="auto"> </div>
<div dir="auto"> </div>
<div dir="auto">Additionally, is there a doc of fts-squat
in order to adjust the code to new releases of dovect ?</div>
<div dir="auto"> </div>
<div dir="auto"> </div>
<div id="aqm-original" style="color: black;">
<div class="aqm-original-body">
<div style="color: black;">
<p style="color: black; font-size: 10pt;
font-family: sans-serif; margin: 8pt 0;">On
December 12, 2018 4:44:10 PM Daniel Miller via
dovecot <a class="moz-txt-link-rfc2396E"
href="mailto:dovecot@dovecot.org"
rel="noreferrer" moz-do-not-send="true"><dovecot@dovecot.org></a>
wrote:</p>
<blockquote class="gmail_quote" style="margin: 0 0 0
0.75ex; border-left: 1px solid #808080;
padding-left: 0.75ex;">On 12/11/2018 4:46 AM, Joan
Moreau via dovecot wrote:<br>
<blockquote style="padding: 0 0.4em; border-left:
#1010ff 2px solid; margin: 0;">
<p>I shared the errors already so many times
(check this mailinling for "solr" in teh
title)</p>
<p>Contrary to what you say, with SOlr 7.5 and
Dovecot git, I had to remove the
"managed-schema" to make solr respond a bit
properly. It relies on schema.xml</p>
<p>In order to create the instance, no, it
copies the default config in the dovecot
instance.</p>
</blockquote>
<p>I'm not a Solr expert by any means but I
believe you are incorrect.</p>
<p>As of Solr 5.x the managed-schema file is the
primary method for configuration. The method I
detailed previously for setting up a config
helps automate creating new Solr instances - but
as I stated you can either setup a Solr template
and then create the instance from that or create
an instance using the default template and then
adjust it.</p>
<p>The part that you *must* do after creating from
the default template is stop the server, delete
the entire "<prefix>/solr/dovecot/data"
folder, then install the correct managed-schema
file, then restart the server. The server will
not function with mismatched schema/data.</p>
<p>If you'll try that - explicitly "rm -rf
<prefix>/solr/dovecot/data", copy the
managed-schema file into the conf folder, and
restart - things will either work or there's
something else that needs correction.</p>
--<br>
Daniel</blockquote>
</div>
</div>
</div>
<div dir="auto"> </div>
</div>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</body>
</html>