Dovecot+Solr+Tika segmentation fault

[Manuel Josupeit-Walter]

Hi all,

I am running a mail server for family and friends based on Dovecot and Docker. Today I made a weird observation which I must have overlooked for a long time so far because they logs were written to the journal. However, I am usually only investigating the Docker logs.

It seems that some e-mail indexing leads to segfaults and it seems to be related only with Solr and Tika. I have auto-indexing turned on so sometimes the indexer-worker crashes when moving mails from one folder to the other or searching for mails otherwise. Also, if I try to manually reindex using doveadm index, the doveadm process quits with a segfault.

I am always running the most recent Dovecot version, in this case 2.3.11.3, but I am not neccessarily up-to-date with Solr or Tika. I am on Solr Version 7.4 and Tika 1.24.1. On the console I am receiving Tika errors and on the Solr Webinterface I see a couple of Ealry EOFs as well.

I am not a Linux expert from a dev perspective, but it seems to me that the client (Dovecot) closes the connection prematurely.

Does anyone have a hint? I am happy to provide more information if required. I need to know how to collect them, though.

Best,
Manuel

The following can be found in the journald logs during the attempt to reindex my Sent mailbox with doveadm index:

Nov 24 16:59:49 server.name tika[988]: INFO tika/ (application/pdf)
Nov 24 16:59:49 server.name tika[988]: INFO tika/ (application/pdf)
Nov 24 16:59:50 server.name tika[988]: INFO tika/ (application/pdf)
Nov 24 16:59:50 server.name tika[988]: INFO tika/ (application/pdf)
Nov 24 16:59:50 server.name tika[988]: INFO tika/ (application/pdf)
Nov 24 16:59:50 server.name tika[988]: INFO tika/ (application/pdf)
Nov 24 16:59:50 server.name tika[988]: INFO tika/ (application/pdf)
Nov 24 16:59:50 server.name tika[988]: INFO tika/ (application/pdf)
Nov 24 16:59:50 server.name tika[988]: INFO tika/ (application/pdf)
Nov 24 16:59:50 server.name tika[988]: INFO tika/ (application/pdf)
Nov 24 16:59:50 server.name tika[988]: WARN Using fallback font LiberationSans for Helvetica
Nov 24 16:59:50 server.name tika[988]: WARN Using fallback font LiberationSans for Helvetica
Nov 24 16:59:50 server.name tika[988]: WARN Using fallback font LiberationSans for Helvetica
Nov 24 16:59:50 server.name tika[988]: WARN Using fallback font LiberationSans for Helvetica
Nov 24 16:59:50 server.name tika[988]: WARN Using fallback font LiberationSans for Helvetica
Nov 24 16:59:50 server.name tika[988]: WARN Using fallback font LiberationSans for Helvetica
Nov 24 16:59:50 server.name tika[988]: WARN Using fallback font LiberationSans for Helvetica
Nov 24 16:59:50 server.name tika[988]: WARN Using fallback font LiberationSans for Helvetica
Nov 24 16:59:50 server.name tika[988]: WARN Using fallback font LiberationSans for Helvetica
Nov 24 16:59:50 server.name tika[988]: WARN Using fallback font LiberationSans for Helvetica
Nov 24 16:59:50 server.name tika[988]: WARN Using fallback font LiberationSans for Helvetica
Nov 24 16:59:50 server.name tika[988]: WARN Using fallback font LiberationSans for Helvetica
Nov 24 16:59:50 server.name tika[988]: WARN Using fallback font LiberationSans for Helvetica
Nov 24 16:59:50 server.name tika[988]: WARN Using fallback font LiberationSans for Helvetica
Nov 24 16:59:50 server.name tika[988]: WARN Using fallback font LiberationSans for Helvetica
Nov 24 16:59:50 server.name tika[988]: WARN Using fallback font LiberationSans for Helvetica
Nov 24 16:59:50 server.name tika[988]: WARN Using fallback font LiberationSans for Helvetica
Nov 24 16:59:50 server.name tika[988]: WARN Using fallback font LiberationSans for Helvetica
Nov 24 16:59:50 server.name tika[988]: WARN Using fallback font LiberationSans for Helvetica
Nov 24 16:59:50 server.name tika[988]: WARN Using fallback font LiberationSans for Helvetica
Nov 24 16:59:50 server.name tika[988]: WARN Using fallback font LiberationSans for Helvetica
Nov 24 16:59:50 server.name tika[988]: WARN Using fallback font LiberationSans for Helvetica
Nov 24 16:59:50 server.name tika[988]: WARN Using fallback font LiberationSans for Helvetica
Nov 24 16:59:50 server.name tika[988]: WARN Using fallback font LiberationSans for Helvetica
Nov 24 16:59:50 server.name kernel: doveadm[400]: segfault at 0 ip 00007fb9f4fa2b64 sp 00007ffc927694f0 error 4 in libdovecot.so.0.0.0[7fb9f4f58000+db000]
Nov 24 16:59:50 server.name kernel: Code: 89 f5 48 8d b8 f0 01 00 00 48 8b 52 08 4c 8b 62 48 e8 90 89 fb ff 48 8b 44 24 08 be 00 20 00 00 4c 89 ef 48 8b 80 10 01 00 00 <4c> 8b 38 e8 64 b2 fb ff 48 8b 44 24 08 4c 89 ef 48 8b b0 10 01 00
Nov 24 16:59:50 server.name tika[988]: INFO tika/ (application/pdf)
Nov 24 16:59:50 server.name tika[988]: WARN tika/: Text extraction failed (null)
Nov 24 16:59:50 server.name tika[988]: org.apache.tika.exception.TikaException: TIKA-198: Illegal IOException from org.apache.tika.parser.DefaultParser at 4dcc5b82
Nov 24 16:59:50 server.name tika[988]: at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:286)
Nov 24 16:59:50 server.name tika[988]: at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:143)
Nov 24 16:59:50 server.name tika[988]: at org.apache.tika.server.resource.TikaResource.parse(TikaResource.java:409)
Nov 24 16:59:50 server.name tika[988]: at org.apache.tika.server.resource.TikaResource$4.write(TikaResource.java:521)
Nov 24 16:59:50 server.name tika[988]: at org.apache.cxf.jaxrs.provider.BinaryDataProvider.writeTo(BinaryDataProvider.java:177)
Nov 24 16:59:50 server.name tika[988]: at org.apache.cxf.jaxrs.utils.JAXRSUtils.writeMessageBody(JAXRSUtils.java:1472)
Nov 24 16:59:50 server.name tika[988]: at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.serializeMessage(JAXRSOutInterceptor.java:249)
Nov 24 16:59:50 server.name tika[988]: at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.processResponse(JAXRSOutInterceptor.java:122)
Nov 24 16:59:50 server.name tika[988]: at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.handleMessage(JAXRSOutInterceptor.java:84)
Nov 24 16:59:50 server.name tika[988]: at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
Nov 24 16:59:50 server.name tika[988]: at org.apache.cxf.interceptor.OutgoingChainInterceptor.handleMessage(OutgoingChainInterceptor.java:90)
Nov 24 16:59:50 server.name tika[988]: at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
Nov 24 16:59:50 server.name tika[988]: at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
Nov 24 16:59:50 server.name tika[988]: at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:267)
Nov 24 16:59:50 server.name tika[988]: at org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(JettyHTTPDestination.java:247)
Nov 24 16:59:50 server.name tika[988]: at org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHandler.java:79)
Nov 24 16:59:50 server.name tika[988]: at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
Nov 24 16:59:50 server.name tika[988]: at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
Nov 24 16:59:50 server.name tika[988]: at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300)
Nov 24 16:59:50 server.name tika[988]: at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:190)
Nov 24 16:59:50 server.name tika[988]: at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1215)
Nov 24 16:59:50 server.name tika[988]: at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
Nov 24 16:59:50 server.name tika[988]: at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
Nov 24 16:59:50 server.name tika[988]: at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
Nov 24 16:59:50 server.name tika[988]: at org.eclipse.jetty.server.Server.handle(Server.java:500)
Nov 24 16:59:50 server.name tika[988]: at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
Nov 24 16:59:50 server.name tika[988]: at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
Nov 24 16:59:50 server.name tika[988]: at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
Nov 24 16:59:50 server.name tika[988]: at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
Nov 24 16:59:50 server.name tika[988]: at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
Nov 24 16:59:50 server.name tika[988]: at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
Nov 24 16:59:50 server.name tika[988]: at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
Nov 24 16:59:50 server.name tika[988]: at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
Nov 24 16:59:50 server.name tika[988]: at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
Nov 24 16:59:50 server.name tika[988]: at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
Nov 24 16:59:50 server.name tika[988]: at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
Nov 24 16:59:50 server.name tika[988]: at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
Nov 24 16:59:50 server.name tika[988]: at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
Nov 24 16:59:50 server.name tika[988]: at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
Nov 24 16:59:50 server.name tika[988]: at java.lang.Thread.run(Thread.java:748)
Nov 24 16:59:50 server.name tika[988]: Caused by: org.eclipse.jetty.io.EofException: Early EOF
Nov 24 16:59:50 server.name tika[988]: at org.eclipse.jetty.server.HttpInput$3.getError(HttpInput.java:1141)
Nov 24 16:59:50 server.name tika[988]: at org.eclipse.jetty.server.HttpInput$3.noContent(HttpInput.java:1129)
Nov 24 16:59:50 server.name tika[988]: at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:321)
Nov 24 16:59:50 server.name tika[988]: at java.io.FilterInputStream.read(FilterInputStream.java:133)
Nov 24 16:59:50 server.name tika[988]: at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
Nov 24 16:59:50 server.name tika[988]: at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
Nov 24 16:59:50 server.name tika[988]: at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
Nov 24 16:59:50 server.name tika[988]: at java.io.FilterInputStream.read(FilterInputStream.java:107)
Nov 24 16:59:50 server.name tika[988]: at org.apache.tika.io.ProxyInputStream.read(ProxyInputStream.java:78)
Nov 24 16:59:50 server.name tika[988]: at org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:78)
Nov 24 16:59:50 server.name tika[988]: at org.apache.pdfbox.io.ScratchFile.createBuffer(ScratchFile.java:443)
Nov 24 16:59:50 server.name tika[988]: at org.apache.pdfbox.pdmodel.PDDocument.load(PDDocument.java:1220)
Nov 24 16:59:50 server.name tika[988]: at org.apache.pdfbox.pdmodel.PDDocument.load(PDDocument.java:1196)
Nov 24 16:59:50 server.name tika[988]: at org.apache.tika.parser.pdf.PDFParser.getPDDocument(PDFParser.java:190)
Nov 24 16:59:50 server.name tika[988]: at org.apache.tika.parser.pdf.PDFParser.parse(PDFParser.java:149)
Nov 24 16:59:50 server.name tika[988]: at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:280)
Nov 24 16:59:50 server.name tika[988]: at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:280)
Nov 24 16:59:50 server.name tika[988]: ... 39 more
Nov 24 16:59:50 server.name tika[988]: ERROR Problem with writing the data, class org.apache.tika.server.resource.TikaResource$4, ContentType: text/plain