Trying to use solr

newer
Question about migration 2.2 - 2.3...

Francis Augusto Medeiros-Logeay

22 Jul 2020 22 Jul '20

5:13 p.m.

Hi,

I am finally trying to use an fts engine with dovecot. My version is 2.3.7.2 under Ubuntu 20.04.

I installed solr 7.7.3 and then 8.6.0 to see if this was a version-related error. I copied the schema from 7.7.0 as many people said this was fine.

I get the following error when trying to reindex a user's mailbox:

doveadm(francis@francisaugusto.com): Error: fts_solr: Indexing failed: 400 Bad Request doveadm(francis@francisaugusto.com): Error: Mailbox INBOX: Transaction commit failed: FTS transaction commit failed: backend deinit doveadm(francis@francisaugusto.com): Debug: auth-master: conn unix:/var/run/dovecot/auth-userdb: Disconnected: Connection closed (fd=10)

On Solr I get this error:

org.apache.solr.common.SolrException: Exception writing document id 210/9fd7941e8297d25d91600000c3fdd3da/francis@francisaugusto.com to the index; possible analysis error: cannot change field "box" from index options=DOCS_AND_FREQS_AND_POSITIONS to inconsistent index options=DOCS at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:246)

Parallel to this, I got some log messages on Solr before attempting to reindex the user (sorry for the garbadged text:

	TIME (LOCAL)
	LEVEL
	CORE
	LOGGER
	MESSAGE

	7/21/2020, 6:38:47 PM
	WARN false
	x:dovecot
	SolrResourceLoader
	Solr loaded a deprecated plugin/analysis class [solr.TrieLongField].

Please consult documentation how to replace it accordingly.

	7/21/2020, 6:38:47 PM
	WARN false
	x:dovecot
	SolrResourceLoader
	Solr loaded a deprecated plugin/analysis class

[solr.SynonymFilterFactory]. Please consult documentation how to replace it accordingly.

	7/21/2020, 6:38:47 PM
	WARN false
	x:dovecot
	SolrResourceLoader
	Solr loaded a deprecated plugin/analysis class

[solr.WordDelimiterFilterFactory]. Please consult documentation how to replace it accordingly.

	7/22/2020, 6:43:46 PM
	ERROR false
	x:dovecot
	RequestHandlerBase
	java.lang.IllegalStateException: Type mismatch: uid was indexed as

SORTED_NUMERIC

	7/22/2020, 6:43:46 PM
	ERROR false
	x:dovecot
	HttpSolrCall
	null:java.lang.IllegalStateException: Type mismatch: uid was indexed

as SORTED_NUMERIC

	7/22/2020, 6:43:49 PM
	ERROR false
	x:dovecot
	RequestHandlerBase
	java.lang.IllegalStateException: Type mismatch: uid was indexed as

SORTED_NUMERIC

	7/22/2020, 6:43:49 PM
	ERROR false
	x:dovecot
	HttpSolrCall
	null:java.lang.IllegalStateException: Type mismatch: uid was indexed

as SORTED_NUMERIC

	7/22/2020, 6:43:56 PM
	ERROR false
	x:dovecot
	RequestHandlerBase
	java.lang.IllegalStateException: Type mismatch: uid was indexed as

SORTED_NUMERIC

	7/22/2020, 6:43:56 PM
	ERROR false
	x:dovecot
	HttpSolrCall
	null:java.lang.IllegalSta

Could anyone point me the right direction to make this work?

Best,

Attachments:

attachment.html (text/html — 8.1 KB)

Show replies by date

Francis Augusto Medeiros-Logeay

22 Jul 22 Jul

5:17 p.m.

Just one thing I forgot to mention: using break-imap-search got me an error: dovecot would say this is an invalid option for ftr_solr.

On 22.07.2020 19:13, Francis Augusto Medeiros-Logeay wrote:

...

Hi,

I am finally trying to use an fts engine with dovecot. My version is 2.3.7.2 under Ubuntu 20.04.

I installed solr 7.7.3 and then 8.6.0 to see if this was a version-related error. I copied the schema from 7.7.0 as many people said this was fine.

I get the following error when trying to reindex a user's mailbox:

doveadm(francis@francisaugusto.com): Error: fts_solr: Indexing failed: 400 Bad Request doveadm(francis@francisaugusto.com): Error: Mailbox INBOX: Transaction commit failed: FTS transaction commit failed: backend deinit doveadm(francis@francisaugusto.com): Debug: auth-master: conn unix:/var/run/dovecot/auth-userdb: Disconnected: Connection closed (fd=10)

On Solr I get this error:

org.apache.solr.common.SolrException: Exception writing document id 210/9fd7941e8297d25d91600000c3fdd3da/francis@francisaugusto.com to the index; possible analysis error: cannot change field "box" from index options=DOCS_AND_FREQS_AND_POSITIONS to inconsistent index options=DOCS at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:246)

Parallel to this, I got some log messages on Solr before attempting to reindex the user (sorry for the garbadged text:
  TIME (LOCAL)
  LEVEL
  CORE
  LOGGER
  MESSAGE

  7/21/2020, 6:38:47 PM
  WARN false
  x:dovecot
  SolrResourceLoader
  Solr loaded a deprecated plugin/analysis class
[solr.TrieLongField]. Please consult documentation how to replace it accordingly.
  7/21/2020, 6:38:47 PM
  WARN false
  x:dovecot
  SolrResourceLoader
  Solr loaded a deprecated plugin/analysis class
[solr.SynonymFilterFactory]. Please consult documentation how to replace it accordingly.
  7/21/2020, 6:38:47 PM
  WARN false
  x:dovecot
  SolrResourceLoader
  Solr loaded a deprecated plugin/analysis class
[solr.WordDelimiterFilterFactory]. Please consult documentation how to replace it accordingly.
  7/22/2020, 6:43:46 PM
  ERROR false
  x:dovecot
  RequestHandlerBase
  java.lang.IllegalStateException: Type mismatch: uid was indexed as
SORTED_NUMERIC
  7/22/2020, 6:43:46 PM
  ERROR false
  x:dovecot
  HttpSolrCall
  null:java.lang.IllegalStateException: Type mismatch: uid was
indexed as SORTED_NUMERIC
  7/22/2020, 6:43:49 PM
  ERROR false
  x:dovecot
  RequestHandlerBase
  java.lang.IllegalStateException: Type mismatch: uid was indexed as
SORTED_NUMERIC
  7/22/2020, 6:43:49 PM
  ERROR false
  x:dovecot
  HttpSolrCall
  null:java.lang.IllegalStateException: Type mismatch: uid was
indexed as SORTED_NUMERIC
  7/22/2020, 6:43:56 PM
  ERROR false
  x:dovecot
  RequestHandlerBase
  java.lang.IllegalStateException: Type mismatch: uid was indexed as
SORTED_NUMERIC
  7/22/2020, 6:43:56 PM
  ERROR false
  x:dovecot
  HttpSolrCall
  null:java.lang.IllegalSta
Could anyone point me the right direction to make this work?

Best,

Shawn Heisey

24 Jul 24 Jul

5:33 p.m.

On 7/22/2020 11:13 AM, Francis Augusto Medeiros-Logeay wrote:

...

These errors are a LOT longer than what we see here. It appears that you are showing us what you can see in the logging tab of the admin UI ... you will see far more detail if you find the solr.log file and get information from there. Full detail can be seen on a single log entry in the admin UI by clicking on the little "i" icon, but it will close again *VERY* quickly, so that's not a very reliable option.

The full detail is usually required for deciphering log messages.

As for the source of the problem ... these look like the kind of errors that can be encountered if you complete some indexing with one schema, then change the schema without deleting the existing index, restart/reload, and try to index more data. Most changes to the schema will require full reindexing, and some changes require actually deleting the index directory entirely and allowing it to be rebuilt. In some rare cases, upgrading Solr *might* result in a change to the schema that requires deleting the index even though the schema config file itself is the same. I do not know if your setup is one that would be affected in this way.

https://cwiki.apache.org/confluence/display/solr/HowToReindex

Your question is more appropriate for the solr-user mailing list at Apache than this list ... but I did not expect you to know that in advance. I can try to help you.

Thanks, Shawn

r_f＠med-lo.eu

26 Jul 26 Jul

9:48 a.m.

__ Francis Augusto Medeiros-Logeay francis@francisaugusto.com Mobile: +47 45 17 14 91 Oslo, Norway Sent from a mobile device / Enviado a partir de dispositivo móvel

...

On 24 Jul 2020, at 19:34, Shawn Heisey <elyograg@elyograg.org> wrote:

On 7/22/2020 11:13 AM, Francis Augusto Medeiros-Logeay wrote:

...
On Solr I get this error: org.apache.solr.common.SolrException: Exception writing document id 210/9fd7941e8297d25d91600000c3fdd3da/francis@francisaugusto.com <mailto:210/9fd7941e8297d25d91600000c3fdd3da/francis@francisaugusto.com> to the index; possible analysis error: cannot change field "box" from index options=DOCS_AND_FREQS_AND_POSITIONS to inconsistent index options=DOCS <snip> Parallel to this, I got some log messages on Solr before attempting to reindex the user (sorry for the garbadged text: Time (Local) Level Core Logger Message <snip> 7/22/2020, 6:43:46 PM ERROR false x:dovecot RequestHandlerBase java.lang.IllegalStateException: Type mismatch: uid was indexed as SORTED_NUMERIC

These errors are a LOT longer than what we see here. It appears that you are showing us what you can see in the logging tab of the admin UI ... you will see far more detail if you find the solr.log file and get information from there. Full detail can be seen on a single log entry in the admin UI by clicking on the little "i" icon, but it will close again *VERY* quickly, so that's not a very reliable option.

The full detail is usually required for deciphering log messages.

As for the source of the problem ... these look like the kind of errors that can be encountered if you complete some indexing with one schema, then change the schema without deleting the existing index, restart/reload, and try to index more data. Most changes to the schema will require full reindexing, and some changes require actually deleting the index directory entirely and allowing it to be rebuilt. In some rare cases, upgrading Solr *might* result in a change to the schema that requires deleting the index even though the schema config file itself is the same. I do not know if your setup is one that would be affected in this way.

https://cwiki.apache.org/confluence/display/solr/HowToReindex

Your question is more appropriate for the solr-user mailing list at Apache than this list ... but I did not expect you to know that in advance. I can try to help you.

Thanks a lot Shawn. I will send the full error when I get home a week from now. But this was my first attempt to use solr - is it still subject to those conflicts?

Best,

Francis

Shawn Heisey

8:45 p.m.

On 7/26/2020 3:48 AM, r_f@med-lo.eu wrote:

...

You'll have to be more specific ... but I believe the answer to your question is likely to be "yes.

I am the person who wrote the "HowToReindex" wiki page that I linked. Nothing in that wiki page is addressed at any particular version of Solr. The possibility of a schema change requiring a complete reindex (and sometimes deleting the index directory entirely) applies to ANY version.

Thanks, Shawn

Francis Augusto Medeiros-Logeay

2 Aug 2 Aug

11:53 a.m.

On 26.07.2020 22:45, Shawn Heisey wrote:

...

Ok, I am trying again. I get no errors when doing a doveadm fts rescan, but get errors when trying this:

doveadm index -u myuser@mydomain.com INBOX doveadm(myuser@mydomain.com): Error: fts_solr: Indexing failed: 400 Bad Request doveadm(myuser@mydomain.com): Error: Mailbox INBOX: Transaction commit failed: FTS transaction commit failed: backend deinit

I guess this is a matter of waiting the reindex to be over?

I get so many "Type mismatch" errors in Solr, except for this one that looks different and showed up after trying the doveadm index command above:

ERROR true x:dovecot RequestHandlerBase org.apache.solr.common.SolrException: Exception writing document id 210/9fd7941e8297d25d91600000c3fdd3da/myuser@mydomain.com to the index; possible analysis error: cannot change field "box" from index options=DOCS_AND_FREQS_AND_POSITIONS to inconsistent index options=DOCS org.apache.solr.common.SolrException: Exception writing document id 210/9fd7941e8297d25d91600000c3fdd3da/myuser@mydomain.com to the index; possible analysis error: cannot change field "box" from index options=DOCS_AND_FREQS_AND_POSITIONS to inconsistent index options=DOCS at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:246) at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:76) at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:259) at org.apache.solr.update.processor.DistributedUpdateProcessor.doVersionAdd(DistributedUpdateProcessor.java:489) at org.apache.solr.update.processor.DistributedUpdateProcessor.lambda$versionAdd$0(DistributedUpdateProcessor.java:339) at org.apache.solr.update.VersionBucket.runWithLock(VersionBucket.java:50) at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:339) at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:225) at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103) at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:261) at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:188) at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97) at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:198) at org.apache.solr.core.SolrCore.execute(SolrCore.java:2576) at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:803) at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:582) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:424) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:505) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: java.lang.IllegalArgumentException: cannot change field "box" from index options=DOCS_AND_FREQS_AND_POSITIONS to inconsistent index options=DOCS at org.apache.lucene.index.FieldInfos$FieldNumbers.verifyConsistent(FieldInfos.java:375) at org.apache.lucene.index.FieldInfos$FieldNumbers.setIndexOptions(FieldInfos.java:441) at org.apache.lucene.index.DefaultIndexingChain.initIndexOptions(DefaultIndexingChain.java:701) at org.apache.lucene.index.DefaultIndexingChain.getOrAddField(DefaultIndexingChain.java:664) at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:440) at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:406) at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:250) at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:495) at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1594) at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1586) at org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocValues(DirectUpdateHandler2.java:971) at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:344) at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:291) at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:238) ... 53 more Last Check: 8/2/2020, 1:51:12 PM Show dates in UTC Auto-Refresh

Any clue on what I could do here?

Best,

Francis

PS - I can show the full log of one of the type mismatch error as well...

1727

Age (days ago)

1738

Last active (days ago)

List overview

5 comments

3 participants

participants (3)

Francis Augusto Medeiros-Logeay
r_f＠med-lo.eu
Shawn Heisey