Hi,
I am finally trying to use an fts engine with dovecot. My version is 2.3.7.2 under Ubuntu 20.04.
I installed solr 7.7.3 and then 8.6.0 to see if this was a version-related error. I copied the schema from 7.7.0 as many people said this was fine.
I get the following error when trying to reindex a user's mailbox:
doveadm(francis@francisaugusto.com): Error: fts_solr: Indexing failed: 400 Bad Request doveadm(francis@francisaugusto.com): Error: Mailbox INBOX: Transaction commit failed: FTS transaction commit failed: backend deinit doveadm(francis@francisaugusto.com): Debug: auth-master: conn unix:/var/run/dovecot/auth-userdb: Disconnected: Connection closed (fd=10)
On Solr I get this error:
org.apache.solr.common.SolrException: Exception writing document id 210/9fd7941e8297d25d91600000c3fdd3da/francis@francisaugusto.com to the index; possible analysis error: cannot change field "box" from index options=DOCS_AND_FREQS_AND_POSITIONS to inconsistent index options=DOCS at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:246)
Parallel to this, I got some log messages on Solr before attempting to reindex the user (sorry for the garbadged text:
TIME (LOCAL)
LEVEL
CORE
LOGGER
MESSAGE
7/21/2020, 6:38:47 PM
WARN false
x:dovecot
SolrResourceLoader
Solr loaded a deprecated plugin/analysis class [solr.TrieLongField].Please consult documentation how to replace it accordingly.
7/21/2020, 6:38:47 PM
WARN false
x:dovecot
SolrResourceLoader
Solr loaded a deprecated plugin/analysis class[solr.SynonymFilterFactory]. Please consult documentation how to replace it accordingly.
7/21/2020, 6:38:47 PM
WARN false
x:dovecot
SolrResourceLoader
Solr loaded a deprecated plugin/analysis class[solr.WordDelimiterFilterFactory]. Please consult documentation how to replace it accordingly.
7/22/2020, 6:43:46 PM
ERROR false
x:dovecot
RequestHandlerBase
java.lang.IllegalStateException: Type mismatch: uid was indexed asSORTED_NUMERIC
7/22/2020, 6:43:46 PM
ERROR false
x:dovecot
HttpSolrCall
null:java.lang.IllegalStateException: Type mismatch: uid was indexedas SORTED_NUMERIC
7/22/2020, 6:43:49 PM
ERROR false
x:dovecot
RequestHandlerBase
java.lang.IllegalStateException: Type mismatch: uid was indexed asSORTED_NUMERIC
7/22/2020, 6:43:49 PM
ERROR false
x:dovecot
HttpSolrCall
null:java.lang.IllegalStateException: Type mismatch: uid was indexedas SORTED_NUMERIC
7/22/2020, 6:43:56 PM
ERROR false
x:dovecot
RequestHandlerBase
java.lang.IllegalStateException: Type mismatch: uid was indexed asSORTED_NUMERIC
7/22/2020, 6:43:56 PM
ERROR false
x:dovecot
HttpSolrCall
null:java.lang.IllegalStaCould anyone point me the right direction to make this work?
Best,
Just one thing I forgot to mention: using break-imap-search got me an error: dovecot would say this is an invalid option for ftr_solr.
On 22.07.2020 19:13, Francis Augusto Medeiros-Logeay wrote:
Hi,
I am finally trying to use an fts engine with dovecot. My version is 2.3.7.2 under Ubuntu 20.04.
I installed solr 7.7.3 and then 8.6.0 to see if this was a version-related error. I copied the schema from 7.7.0 as many people said this was fine.
I get the following error when trying to reindex a user's mailbox:
doveadm(francis@francisaugusto.com): Error: fts_solr: Indexing failed: 400 Bad Request doveadm(francis@francisaugusto.com): Error: Mailbox INBOX: Transaction commit failed: FTS transaction commit failed: backend deinit doveadm(francis@francisaugusto.com): Debug: auth-master: conn unix:/var/run/dovecot/auth-userdb: Disconnected: Connection closed (fd=10)
On Solr I get this error:
org.apache.solr.common.SolrException: Exception writing document id 210/9fd7941e8297d25d91600000c3fdd3da/francis@francisaugusto.com to the index; possible analysis error: cannot change field "box" from index options=DOCS_AND_FREQS_AND_POSITIONS to inconsistent index options=DOCS at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:246)
Parallel to this, I got some log messages on Solr before attempting to reindex the user (sorry for the garbadged text:
TIME (LOCAL) LEVEL CORE LOGGER MESSAGE 7/21/2020, 6:38:47 PM WARN false x:dovecot SolrResourceLoader Solr loaded a deprecated plugin/analysis class[solr.TrieLongField]. Please consult documentation how to replace it accordingly.
7/21/2020, 6:38:47 PM WARN false x:dovecot SolrResourceLoader Solr loaded a deprecated plugin/analysis class[solr.SynonymFilterFactory]. Please consult documentation how to replace it accordingly.
7/21/2020, 6:38:47 PM WARN false x:dovecot SolrResourceLoader Solr loaded a deprecated plugin/analysis class[solr.WordDelimiterFilterFactory]. Please consult documentation how to replace it accordingly.
7/22/2020, 6:43:46 PM ERROR false x:dovecot RequestHandlerBase java.lang.IllegalStateException: Type mismatch: uid was indexed asSORTED_NUMERIC
7/22/2020, 6:43:46 PM ERROR false x:dovecot HttpSolrCall null:java.lang.IllegalStateException: Type mismatch: uid wasindexed as SORTED_NUMERIC
7/22/2020, 6:43:49 PM ERROR false x:dovecot RequestHandlerBase java.lang.IllegalStateException: Type mismatch: uid was indexed asSORTED_NUMERIC
7/22/2020, 6:43:49 PM ERROR false x:dovecot HttpSolrCall null:java.lang.IllegalStateException: Type mismatch: uid wasindexed as SORTED_NUMERIC
7/22/2020, 6:43:56 PM ERROR false x:dovecot RequestHandlerBase java.lang.IllegalStateException: Type mismatch: uid was indexed asSORTED_NUMERIC
7/22/2020, 6:43:56 PM ERROR false x:dovecot HttpSolrCall null:java.lang.IllegalStaCould anyone point me the right direction to make this work?
Best,
On 7/22/2020 11:13 AM, Francis Augusto Medeiros-Logeay wrote:
On Solr I get this error:
org.apache.solr.common.SolrException: Exception writing document id 210/9fd7941e8297d25d91600000c3fdd3da/francis@francisaugusto.com mailto:210/9fd7941e8297d25d91600000c3fdd3da/francis@francisaugusto.com to the index; possible analysis error: cannot change field "box" from index options=DOCS_AND_FREQS_AND_POSITIONS to inconsistent index options=DOCS <snip> Parallel to this, I got some log messages on Solr before attempting to reindex the user (sorry for the garbadged text: Time (Local) Level Core Logger Message <snip> 7/22/2020, 6:43:46 PM ERROR false x:dovecot RequestHandlerBase java.lang.IllegalStateException: Type mismatch: uid was indexed as SORTED_NUMERIC
These errors are a LOT longer than what we see here. It appears that you are showing us what you can see in the logging tab of the admin UI ... you will see far more detail if you find the solr.log file and get information from there. Full detail can be seen on a single log entry in the admin UI by clicking on the little "i" icon, but it will close again *VERY* quickly, so that's not a very reliable option.
The full detail is usually required for deciphering log messages.
As for the source of the problem ... these look like the kind of errors that can be encountered if you complete some indexing with one schema, then change the schema without deleting the existing index, restart/reload, and try to index more data. Most changes to the schema will require full reindexing, and some changes require actually deleting the index directory entirely and allowing it to be rebuilt. In some rare cases, upgrading Solr *might* result in a change to the schema that requires deleting the index even though the schema config file itself is the same. I do not know if your setup is one that would be affected in this way.
https://cwiki.apache.org/confluence/display/solr/HowToReindex
Your question is more appropriate for the solr-user mailing list at Apache than this list ... but I did not expect you to know that in advance. I can try to help you.
Thanks, Shawn
__ Francis Augusto Medeiros-Logeay francis@francisaugusto.com Mobile: +47 45 17 14 91 Oslo, Norway Sent from a mobile device / Enviado a partir de dispositivo móvel
On 24 Jul 2020, at 19:34, Shawn Heisey elyograg@elyograg.org wrote:
On 7/22/2020 11:13 AM, Francis Augusto Medeiros-Logeay wrote:
On Solr I get this error: org.apache.solr.common.SolrException: Exception writing document id 210/9fd7941e8297d25d91600000c3fdd3da/francis@francisaugusto.com mailto:210/9fd7941e8297d25d91600000c3fdd3da/francis@francisaugusto.com to the index; possible analysis error: cannot change field "box" from index options=DOCS_AND_FREQS_AND_POSITIONS to inconsistent index options=DOCS <snip> Parallel to this, I got some log messages on Solr before attempting to reindex the user (sorry for the garbadged text: Time (Local) Level Core Logger Message <snip> 7/22/2020, 6:43:46 PM ERROR false x:dovecot RequestHandlerBase java.lang.IllegalStateException: Type mismatch: uid was indexed as SORTED_NUMERIC
These errors are a LOT longer than what we see here. It appears that you are showing us what you can see in the logging tab of the admin UI ... you will see far more detail if you find the solr.log file and get information from there. Full detail can be seen on a single log entry in the admin UI by clicking on the little "i" icon, but it will close again *VERY* quickly, so that's not a very reliable option.
The full detail is usually required for deciphering log messages.
As for the source of the problem ... these look like the kind of errors that can be encountered if you complete some indexing with one schema, then change the schema without deleting the existing index, restart/reload, and try to index more data. Most changes to the schema will require full reindexing, and some changes require actually deleting the index directory entirely and allowing it to be rebuilt. In some rare cases, upgrading Solr *might* result in a change to the schema that requires deleting the index even though the schema config file itself is the same. I do not know if your setup is one that would be affected in this way.
https://cwiki.apache.org/confluence/display/solr/HowToReindex
Your question is more appropriate for the solr-user mailing list at Apache than this list ... but I did not expect you to know that in advance. I can try to help you.
Thanks a lot Shawn. I will send the full error when I get home a week from now. But this was my first attempt to use solr - is it still subject to those conflicts?
Best,
Francis
On 7/26/2020 3:48 AM, r_f@med-lo.eu wrote:
Thanks a lot Shawn. I will send the full error when I get home a week from now. But this was my first attempt to use solr - is it still subject to those conflicts?
You'll have to be more specific ... but I believe the answer to your question is likely to be "yes.
I am the person who wrote the "HowToReindex" wiki page that I linked. Nothing in that wiki page is addressed at any particular version of Solr. The possibility of a schema change requiring a complete reindex (and sometimes deleting the index directory entirely) applies to ANY version.
Thanks, Shawn
On 26.07.2020 22:45, Shawn Heisey wrote:
On 7/26/2020 3:48 AM, r_f@med-lo.eu wrote:
Thanks a lot Shawn. I will send the full error when I get home a week from now. But this was my first attempt to use solr - is it still subject to those conflicts?
You'll have to be more specific ... but I believe the answer to your question is likely to be "yes.
I am the person who wrote the "HowToReindex" wiki page that I linked. Nothing in that wiki page is addressed at any particular version of Solr. The possibility of a schema change requiring a complete reindex (and sometimes deleting the index directory entirely) applies to ANY version.
Ok, I am trying again. I get no errors when doing a doveadm fts rescan, but get errors when trying this:
doveadm index -u myuser@mydomain.com INBOX doveadm(myuser@mydomain.com): Error: fts_solr: Indexing failed: 400 Bad Request doveadm(myuser@mydomain.com): Error: Mailbox INBOX: Transaction commit failed: FTS transaction commit failed: backend deinit
I guess this is a matter of waiting the reindex to be over?
I get so many "Type mismatch" errors in Solr, except for this one that looks different and showed up after trying the doveadm index command above:
ERROR true x:dovecot RequestHandlerBase org.apache.solr.common.SolrException: Exception writing document id 210/9fd7941e8297d25d91600000c3fdd3da/myuser@mydomain.com to the index; possible analysis error: cannot change field "box" from index options=DOCS_AND_FREQS_AND_POSITIONS to inconsistent index options=DOCS org.apache.solr.common.SolrException: Exception writing document id 210/9fd7941e8297d25d91600000c3fdd3da/myuser@mydomain.com to the index; possible analysis error: cannot change field "box" from index options=DOCS_AND_FREQS_AND_POSITIONS to inconsistent index options=DOCS at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:246) at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:76) at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:259) at org.apache.solr.update.processor.DistributedUpdateProcessor.doVersionAdd(DistributedUpdateProcessor.java:489) at org.apache.solr.update.processor.DistributedUpdateProcessor.lambda$versionAdd$0(DistributedUpdateProcessor.java:339) at org.apache.solr.update.VersionBucket.runWithLock(VersionBucket.java:50) at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:339) at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:225) at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103) at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:261) at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:188) at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97) at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:198) at org.apache.solr.core.SolrCore.execute(SolrCore.java:2576) at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:803) at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:582) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:424) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:505) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: java.lang.IllegalArgumentException: cannot change field "box" from index options=DOCS_AND_FREQS_AND_POSITIONS to inconsistent index options=DOCS at org.apache.lucene.index.FieldInfos$FieldNumbers.verifyConsistent(FieldInfos.java:375) at org.apache.lucene.index.FieldInfos$FieldNumbers.setIndexOptions(FieldInfos.java:441) at org.apache.lucene.index.DefaultIndexingChain.initIndexOptions(DefaultIndexingChain.java:701) at org.apache.lucene.index.DefaultIndexingChain.getOrAddField(DefaultIndexingChain.java:664) at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:440) at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:406) at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:250) at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:495) at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1594) at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1586) at org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocValues(DirectUpdateHandler2.java:971) at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:344) at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:291) at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:238) ... 53 more Last Check: 8/2/2020, 1:51:12 PM Show dates in UTC Auto-Refresh
Any clue on what I could do here?
Best,
Francis
PS - I can show the full log of one of the type mismatch error as well...
participants (3)
-
Francis Augusto Medeiros-Logeay
-
r_f＠med-lo.eu
-
Shawn Heisey