dovecot-fts-solr Solr9 support
I run
dovecot-2.3.18, & am deploying Solr search with 'dovecot-fts-solr'
I run Solr standalone.
When creating a dedicated 'dovecot' core, cp'ing from Dovecot's included solr schemas
ls -al /usr/share/doc/dovecot/solr-*
-rw-r--r-- 1 root root 12K Dec 3 06:48 /usr/share/doc/dovecot/solr-config-7.7.0.xml
-rw-r--r-- 1 root root 2.7K Dec 3 06:48 /usr/share/doc/dovecot/solr-schema-7.7.0.xml
-rw-r--r-- 1 root root 2.9K Dec 3 06:48 /usr/share/doc/dovecot/solr-schema.xmlis tested only for Solr 7.7.0
https://doc.dovecot.org/configuration_manual/fts/solr/
"The steps described in this page are tested for Solr 7.7.0. For other versions, this these steps may need to be adjusted."which, per solr docs
https://solr.apache.org/downloads.html
About versions and support
Version Description
9.x Current major version for feature releases (STABLE)
8.11.x Previous major version may sometimes receive critical bugfix releases
10 Next major version, yet to be released (UNSTABLE)
<8.11 All older versions are End Of Life (EOL)is EOL'd and unsupported.
In the dovecot 7.7.0 schema, FastLRUCache/LRUCache are used as the caching class(es),
grep class=.*LRU /usr/share/doc/dovecot/solr-config-7.7.0.xml
<filterCache class="solr.FastLRUCache"
<queryResultCache class="solr.LRUCache"
<documentCache class="solr.LRUCache"
class="solr.search.LRUCache"For Solr 9,
cat CHANGES.txt
Upgrade Notes
---------------------
...
* SOLR-13817: Deprecate legacy SolrCache implementations. Users are encouraged to transition their
configurations to use org.apache.solr.search.CaffeineCache instead. (ab)running solr9 + dovecot-fts-solr search, based on the 7.7.0 schemas, results in errors, failed scan & search.
Modifying the config to use CaffeineCache
cp solr-config-7.7.0.xml solr-config-9.0.0.xml
perl -pi -e '\
s|solr.FastLRUCache|solr.CaffeineCache|g; \
s|solr.LRUCache|solr.CaffeineCache|g; \
s|solr.search.LRUCache|solr.search.CaffeineCache|g;' \
solr-config-9.0.0.xmlappears to do the trick. Atm, indexing & search -- both manually triggered, and on delivery receipt to Dovecot store -- is working with no errors (yet).
I've not tested exhaustively yet, & don't know if this Cache class change in the schema is both necessary & sufficient for dovecot-fts-solr search use with solr 9.0.0
Given that for Solr
"<8.11 All older versions are End Of Life (EOL)"it'd be useful for Dovecot to 'retest' for current, not-EOL'd Solr, and publish/package compatible schemas accordingly.
On 5/17/22 09:02, PGNet Dev wrote:
Modifying the config to use CaffeineCache
cp solr-config-7.7.0.xml solr-config-9.0.0.xml perl -pi -e '
s|solr.FastLRUCache|solr.CaffeineCache|g;
s|solr.LRUCache|solr.CaffeineCache|g;
s|solr.search.LRUCache|solr.search.CaffeineCache|g;'
solr-config-9.0.0.xmlappears to do the trick. Atm, indexing & search -- both manually triggered, and on delivery receipt to Dovecot store -- is working with no errors (yet).
Yesterday I upgraded my Solr install for dovecot from 8.11.2-SNAPSHOT to 10.0.0-SNAPSHOT. It is still working and the config did not need further changes from what I already had.
I have been customizing my solrconfig.xml and managed-schema files ... I had made the change to CaffeineCache quite a while back. I have also changed the analyzer chain in the schema to use a contrib component, so my config won't be directly usable -- it requires loading some extra jars. When I upgraded, I did need to change the extra jars that were loaded.
I can come up with a new config that should work in 8.x, 9.x, and the current dev branch by almost anyone.
I should probably see if I can write some end to end instructions for installing Solr and configuring dovecot to use it. Something that is missing from the current instructions is how to initiate a FULL reindex of all data in the dovecot install.
Thanks, Shawn
On 5/17/22 12:17 PM, Shawn Heisey wrote:
I had made the change to CaffeineCache quite a while back.
iiuc, solr8x still has support for *LRUCache classes, as well as CaffeineCache.
in my 8x installs, i'd simply never noticed related search problems ... only after my 9x install, with the full deprecation of legacy cache(s), did I notice these
... /var/log/solr/solr.log.6:2022-05-17 10:08:09.402 ERROR (qtp1165791284-17) [ x:dovecot] o.a.s.s.CacheConfig Error instantiating cache => org.apache.solr.common.SolrException: Error loading class 'solr.search.LRUCache' /var/log/solr/solr.log.6:org.apache.solr.common.SolrException: Error loading class 'solr.search.LRUCache' /var/log/solr/solr.log.6:Caused by: java.lang.ClassNotFoundException: solr.search.LRUCache /var/log/solr/solr.log.6:2022-05-17 10:09:13.534 ERROR (qtp1165791284-23) [ x:dovecot] o.a.s.s.CacheConfig Error instantiating cache => org.apache.solr.common.SolrException: Error loading class 'solr.LRUCache' /var/log/solr/solr.log.6:org.apache.solr.common.SolrException: Error loading class 'solr.LRUCache' /var/log/solr/solr.log.6:Caused by: java.lang.ClassNotFoundException: solr.LRUCache /var/log/solr/solr.log.6:2022-05-17 10:09:13.556 ERROR (qtp1165791284-23) [ x:dovecot] o.a.s.s.CacheConfig Error instantiating cache => org.apache.solr.common.SolrException: Error loading class 'solr.FastLRUCache' /var/log/solr/solr.log.6:org.apache.solr.common.SolrException: Error loading class 'solr.FastLRUCache' /var/log/solr/solr.log.6:Caused by: java.lang.ClassNotFoundException: solr.FastLRUCache ...
I can come up with a new config that should work in 8.x, 9.x, and the current dev branch by almost anyone.
CaffeineCache _should_ be a solution for both series. I'm just starting to do a bit of testing, but so far, indexing/search with Caffeine seems quicker, at least. I do not yet know in which case(s) CaffeineCache is _not_ an appropriate solution.
I should probably see if I can write some end to end instructions for installing Solr and configuring dovecot to use it.
Having something well-vetted and doc'd will be useful.
Something that is missing from the current instructions is how to initiate a FULL reindex of all data in the dovecot install.
Beyond looping over each $acct
doveadm fts rescan -u ${acct} doveadm index -u ${acct} -q '*'
?
On 5/17/22 11:32, PGNet Dev wrote:
Something that is missing from the current instructions is how to initiate a FULL reindex of all data in the dovecot install.
Beyond looping over each $acct
doveadm fts rescan -u ${acct} doveadm index -u ${acct} -q '*'
I hadn't seen the fts rescan before. That was what I was missing to make things really clean. Before I was deleting all the dovecot.*index* files, and now I don't need to do that. This is my new script, with the old way commented out:
#!/bin/sh DOVECOT_SOLR_URL_BASE="http://localhost:8983/solr/dovecot" #MAIL_ROOT=/var/vmail
# DO NOT MODIFY BELOW
# WITHOUT GOOD REASON
#
DEL_ALL_QUERY_XML="<delete><query>*:*</query></delete>"
service dovecot stop
curl
"${DOVECOT_SOLR_URL_BASE}/update?commit=true&optimize=true"
-H "Content-Type: text/xml"
--data-binary "${DEL_ALL_QUERY_XML}"
#find ${MAIL_ROOT} -name "dovecot.*index*" -print0 | xargs -0 rm -f
service dovecot start
doveadm force-resync -A '*'
doveadm fts rescan -A
doveadm index -A -q '*'
The script stops Dovecot, wipes out the Solr index, then restarts Dovecot and does the doveadm commands necessary for a full reindex of all mailboxes, without specifying them one by one. The force-resync probably isn't required, but I figure it can't hurt. I've tested this new version a few times. Good thing a full reindex only takes about ten minutes for my system.
Thanks, Shawn
On 5/17/2022 11:32 AM, PGNet Dev wrote:
CaffeineCache _should_ be a solution for both series. I'm just starting to do a bit of testing, but so far, indexing/search with Caffeine seems quicker, at least. I do not yet know in which case(s) CaffeineCache is _not_ an appropriate solution.
CaffeineCache was added in Solr 8.3.0. So it is not available in all 8.x releases.
https://issues.apache.org/jira/browse/SOLR-8241
I know more than a little bit about Solr's caches. I am the author of LFUCache, added way back in Solr 3.6.0. That is an extremely naive "intro to programming" implementation, but it works.
https://issues.apache.org/jira/browse/SOLR-2906
Caffeine is a highly optimized and battle-tested cache, with very good performance. Shortly after it was added to Solr, all the other cache implementations were deprecated, and then removed in 9.0.0.
I should probably see if I can write some end to end instructions for installing Solr and configuring dovecot to use it.
Having something well-vetted and doc'd will be useful.
If I ever find any time I will definitely work on that.
Thanks, Shawn
I was trying a new install of dovecot w/ solr9. I've manually fixed the file linking to the proper directories, however one plugin is no longer shipped. Since the solr files aren't updated yet to 9, can anyone tell me if I need the discontinued velocity plugin that was default in the dovecot solr 7.7 config file. It appears it is now a third party plugin that hasn't been updated for 3 years.
On 28/09/2022 04:32 EEST Nathanael Anderson nathan@andersonsplace.net wrote:
I was trying a new install of dovecot w/ solr9. I've manually fixed the file linking to the proper directories, however one plugin is no longer shipped. Since the solr files aren't updated yet to 9, can anyone tell me if I need the discontinued velocity plugin that was default in the dovecot solr 7.7 config file. It appears it is now a third party plugin that hasn't been updated for 3 years.
Probably not needed. Solr schema is maintained mostly by community users, we do occasional checks.
Aki
On 9/27/22 19:32, Nathanael Anderson wrote:
I was trying a new install of dovecot w/ solr9. I've manually fixed the file linking to the proper directories, however one plugin is no longer shipped. Since the solr files aren't updated yet to 9, can anyone tell me if I need the discontinued velocity plugin that was default in the dovecot solr 7.7 config file. It appears it is now a third party plugin that hasn't been updated for 3 years.
The velocity stuff that Solr ships with is a templating system that allows Solr to host a little website showcasing its capabilities. It is strongly recommended to never use this in production, as it requires that end users have direct network access to the Solr install, which is never a good idea.
Dovecot accesses the API directly and does not need velocity.
I am running a dev version of Solr 9.1.0 with the config and schema stripped down to just what is needed for Dovecot. I have added the jars necessary for the ICU analysis components and I am using two of those analysis components in my schema.
I installed Solr on Ubuntu Server using the service installer script included in the download. This extracts the tarball in /opt, and then sets up /opt/solr as a symlink to the version-specific directory in /opt. It creates a directory structure under /var/solr and creates /etc/default/solr.in.sh. If you use a service name other than solr, that will be named /etc/default/${servicename}.in.sh and I believe the data will go to /var/${servicename}.
For ICU, I created /var/solr/data/lib, then copied icu4j-70.1.jar and lucene-analysis-icu-9.3.0.jar from /opt/solr/modules/analysis-extras/lib to that new lib directory. Solr 9.0.0 would have lucene jars from Lucene 9.0.0, but the 9.x branch is currently using Lucene 9.3.0. Do not use <lib> config elements in solrconfig.xml to load the jars. My solrconfig.xml and managed-schema.xml files can be found here:
https://paste.elyograg.org/view/97597ed3 https://paste.elyograg.org/view/dca55086
My index is quite small by Solr standards, which is why I have such a low maxTime on autoSoftCommit. Larger indexes may do better with a larger interval there.
I use LATEST for luceneMatchVersion, which generates a warning when Solr starts. I am also using 2.0 for the schema version so that it will automatically pick up new defaults after the 1.6 version when those versions are created in later versions of Solr.
This is the current contents of /etc/default/solr.in.sh with commented lines removed:
SOLR_PID_DIR="/var/solr"
SOLR_HOME="/var/solr/data"
LOG4J_PROPS="/var/solr/log4j2.xml"
SOLR_LOGS_DIR="/var/solr/logs"
SOLR_PORT="8983"
SOLR_HEAP="1g"
GC_TUNE="
-XX:+UseG1GC
-XX:+ParallelRefProcEnabled
-XX:MaxGCPauseMillis=100
-XX:+UseLargePages
-XX:+AlwaysPreTouch
-XX:+ExplicitGCInvokesConcurrent
-XX:ParallelGCThreads=2
-XX:+UseStringDeduplication
-XX:+UseNUMA
"
SOLR_JAVA_STACK_SIZE="-Xss1m"
SOLR_ULIMIT_CHECKS=false
SOLR_GZIP_ENABLED=true
SOLR_JETTY_HOST=0.0.0.0
Once you have all that in place, start and stop solr using service or systemctl. Don't run the solr script directly except to create the index ... and for that you must run it as the solr user. Running it as root is prohibited by default, and forcing it will cause problems.
My Solr install is running in cloud mode, but I have removed the things that configure that to make this info easier to use.
One final note: Solr 9 cannot use indexes touched by Solr 7 or earlier. You will need to completely reindex.
Thanks, Shawn
Shawn -
You had mentioned in another email (somewhere) that were hopefully going to do a write-up of setting up Solr 9.x with Dovecot. Any chance you've had time for that ?
Thanks -
On 2022-09-30 1:52 pm, Shawn Heisey wrote:
On 9/27/22 19:32, Nathanael Anderson wrote:
I was trying a new install of dovecot w/ solr9. I've manually fixed the file linking to the proper directories, however one plugin is no longer shipped. Since the solr files aren't updated yet to 9, can anyone tell me if I need the discontinued velocity plugin that was default in the dovecot solr 7.7 config file. It appears it is now a third party plugin that hasn't been updated for 3 years.
The velocity stuff that Solr ships with is a templating system that allows Solr to host a little website showcasing its capabilities. It is strongly recommended to never use this in production, as it requires that end users have direct network access to the Solr install, which is never a good idea.
Dovecot accesses the API directly and does not need velocity.
I am running a dev version of Solr 9.1.0 with the config and schema stripped down to just what is needed for Dovecot. I have added the jars necessary for the ICU analysis components and I am using two of those analysis components in my schema.
I installed Solr on Ubuntu Server using the service installer script included in the download. This extracts the tarball in /opt, and then sets up /opt/solr as a symlink to the version-specific directory in /opt. It creates a directory structure under /var/solr and creates /etc/default/solr.in.sh. If you use a service name other than solr, that will be named /etc/default/${servicename}.in.sh and I believe the data will go to /var/${servicename}.
For ICU, I created /var/solr/data/lib, then copied icu4j-70.1.jar and lucene-analysis-icu-9.3.0.jar from /opt/solr/modules/analysis-extras/lib to that new lib directory. Solr 9.0.0 would have lucene jars from Lucene 9.0.0, but the 9.x branch is currently using Lucene 9.3.0. Do not use <lib> config elements in solrconfig.xml to load the jars. My solrconfig.xml and managed-schema.xml files can be found here:
https://paste.elyograg.org/view/97597ed3 [1] https://paste.elyograg.org/view/dca55086 [2]
My index is quite small by Solr standards, which is why I have such a low maxTime on autoSoftCommit. Larger indexes may do better with a larger interval there.
I use LATEST for luceneMatchVersion, which generates a warning when Solr starts. I am also using 2.0 for the schema version so that it will automatically pick up new defaults after the 1.6 version when those versions are created in later versions of Solr.
This is the current contents of /etc/default/solr.in.sh with commented lines removed:
SOLR_PID_DIR="/var/solr" SOLR_HOME="/var/solr/data" LOG4J_PROPS="/var/solr/log4j2.xml" SOLR_LOGS_DIR="/var/solr/logs" SOLR_PORT="8983" SOLR_HEAP="1g" GC_TUNE=" -XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:MaxGCPauseMillis=100 -XX:+UseLargePages -XX:+AlwaysPreTouch -XX:+ExplicitGCInvokesConcurrent -XX:ParallelGCThreads=2 -XX:+UseStringDeduplication -XX:+UseNUMA " SOLR_JAVA_STACK_SIZE="-Xss1m" SOLR_ULIMIT_CHECKS=false SOLR_GZIP_ENABLED=true SOLR_JETTY_HOST=0.0.0.0
Once you have all that in place, start and stop solr using service or systemctl. Don't run the solr script directly except to create the index ... and for that you must run it as the solr user. Running it as root is prohibited by default, and forcing it will cause problems.
My Solr install is running in cloud mode, but I have removed the things that configure that to make this info easier to use.
One final note: Solr 9 cannot use indexes touched by Solr 7 or earlier. You will need to completely reindex.
Thanks, Shawn
Links:
[1] https://paste.elyograg.org/view/97597ed3 [2] https://paste.elyograg.org/view/dca55086
participants (5)
-
Aki Tuomi
-
deano-dovecot＠areyes.com
-
Nathanael Anderson
-
PGNet Dev
-
Shawn Heisey