- dovecot - dovecot.org

dovecot passes zero byte input stream when passing email with .eml attachment to apache tika parser, causes 'SEVERE' error
by pgnd 13 Aug '23

13 Aug '23

revisiting a prior, never resolved issue tika 2.4.1 'Text extraction failed' errors when dovecot+fts 2.3.19.1 passes embedded *.eml (message/rfc822) files ; org.apache.tika.parser.mail.RFC822Parser or dovecot ? https://dovecot.org/list/dovecot/2022-August/125085.html i now run dovecot --version 2.3.20 (80a5ac675d) with dovecot23_fts_flatcurve built 20230812 from git.HEAD on lsb_release -rd Description: Fedora release 38 (Thirty Eight) Release: 38 apache/tika 2.8.0 runs on another box dovecot passes mail to tika, via config fts_tika = http://192.0.2.20:9998/tika/ indexing & search work with simple mail passed, and with attachments ... except with .eml attachments. sending this message (plain txt email, with .eml as attachment) Content-Type: multipart/mixed; boundary="------------O6z0p865JVYqnvxPyzc0aSLA" Message-ID: <0b7f5053-05b5-4b4f-8c7e-4a640b79a5f4(a)fastmail.fm> Date: Sat, 12 Aug 2023 19:37:44 -0400 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Reply-To: sender(a)example1.com From: sender <sender(a)example1.com> Content-Language: en-US To: recipient(a)example2.com Subject: tika test This is a multi-part message in MIME format. --------------O6z0p865JVYqnvxPyzc0aSLA Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Â tika test --------------O6z0p865JVYqnvxPyzc0aSLA Content-Type: message/rfc822; name="t2.eml" Content-Disposition: attachment; filename="t2.eml" Content-Transfer-Encoding: 8bit Message-ID: <89e5914a-e928-422d-824f-5b4e90170cd7(a)fastmail.fm> Date: Sat, 12 Aug 2023 14:32:48 -0400 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Reply-To: sender(a)example1.com From: sender <sender(a)example1.com> Content-Language: en-US To: recipient(a)example2.com Subject: t2 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Â t2 --------------O6z0p865JVYqnvxPyzc0aSLA-- from an off-site server to local postfix+dovecot instance tika & dovecot logs, journalctl -f -u tika Aug 12 19:52:36 svr04tika[2035]: INFO [qtp1472012329-175] 19:52:36,918 org.apache.tika.server.core.TikaLoggingFilter Request URI: http://192.0.2.20:9998/tika/ Aug 12 19:52:36 svr04tika[2035]: INFO [qtp1472012329-175] 19:52:36,918 org.apache.tika.server.core.resource.TikaResource /tika (message/rfc822) Aug 12 19:52:36 svr04tika[2035]: WARN [qtp1472012329-175] 19:52:36,918 org.apache.tika.server.core.resource.TikaResource tika/: Text extraction failed (t2.eml) Aug 12 19:52:36 svr04tika[2035]: org.apache.tika.exception.ZeroByteFileException: InputStream must have > 0 bytes Aug 12 19:52:36 svr04tika[2035]: at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:185) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at org.apache.tika.parser.ParserDecorator.parse(ParserDecorator.java:152) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at org.apache.tika.parser.DigestingParser.parse(DigestingParser.java:57) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at org.apache.tika.server.core.resource.TikaResource.parse(TikaResource.java:357) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at org.apache.tika.server.core.resource.TikaResource.lambda$produceText$1(TikaResource.java:507) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at org.apache.cxf.jaxrs.provider.BinaryDataProvider.writeTo(BinaryDataProvider.java:177) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at org.apache.cxf.jaxrs.utils.JAXRSUtils.writeMessageBody(JAXRSUtils.java:1651) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.serializeMessage(JAXRSOutInterceptor.java:249) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.processResponse(JAXRSOutInterceptor.java:122) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.handleMessage(JAXRSOutInterceptor.java:84) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at org.apache.cxf.interceptor.OutgoingChainInterceptor.handleMessage(OutgoingChainInterceptor.java:90) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:265) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(JettyHTTPDestination.java:247) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHandler.java:79) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:190) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) ~[tika-server-standard-2.8.0.jar:2.8.0] Aug 12 19:52:36 svr04tika[2035]: at java.lang.Thread.run(Thread.java:1623) [?:?] Aug 12 19:52:36 svr04tika[2035]: Aug 12, 2023 7:52:36 PM org.apache.cxf.jaxrs.utils.JAXRSUtils logMessageHandlerProblem Aug 12 19:52:36 svr04tika[2035]: SEVERE: Problem with writing the data, class org.apache.tika.server.core.resource.TikaResource$$Lambda$387/0x00000008012a9300, ContentType: text/plain tail -f /var/log/dovecot/*log | grep http-client 2023-08-12 19:52:36 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: host 192.0.2.20: Host created 2023-08-12 19:52:36 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: host 192.0.2.20: Host session created 2023-08-12 19:52:36 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: peer 192.0.2.20:9998 (shared): Peer created 2023-08-12 19:52:36 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: peer 192.0.2.20:9998: Peer pool created 2023-08-12 19:52:36 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: peer 192.0.2.20:9998: Peer created 2023-08-12 19:52:36 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: queue http://192.0.2.20:9998: Setting up connection to 192.0.2.20:9998 (1 requests pending) 2023-08-12 19:52:36 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: peer 192.0.2.20:9998: Linked queue http://192.0.2.20:9998 (1 queues linked) 2023-08-12 19:52:36 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: queue http://192.0.2.20:9998: Started new connection to 192.0.2.20:9998 2023-08-12 19:52:36 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: request [Req1: PUT http://192.0.2.20/tika/] Submitted (requests left=1) 2023-08-12 19:52:36 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: request [Req1: PUT http://192.0.2.20/tika/] Waiting for request to finish 2023-08-12 19:52:36 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: peer 192.0.2.20:9998: Creating 1 new connections to handle requests (already 0 usable, connecting to 0, closing 0) 2023-08-12 19:52:36 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: peer 192.0.2.20:9998: Making new connection 1 of 1 (0 connections exist, 0 pending) 2023-08-12 19:52:36 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: conn 192.0.2.20:9998 [1]: Connecting 2023-08-12 19:52:36 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: conn 192.0.2.20:9998 [1]: Waiting for connect (fd=28) to finish for max 0 msecs 2023-08-12 19:52:36 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: conn 192.0.2.20:9998 [1]: HTTP connection created (1 parallel connections exist) 2023-08-12 19:52:36 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: conn 192.0.2.20:9998 [1]: Client connected (fd=28) 2023-08-12 19:52:36 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: conn 192.0.2.20:9998 [1]: Connected 2023-08-12 19:52:36 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: conn 192.0.2.20:9998 [1]: Ready for requests 2023-08-12 19:52:36 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: peer 192.0.2.20:9998: Successfully connected (1 connections exist, 0 pending) 2023-08-12 19:52:36 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: peer 192.0.2.20:9998: Successfully connected (1 connections exist, 0 pending) 2023-08-12 19:52:36 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: peer 192.0.2.20:9998: Using 1 idle connections to handle 1 requests (1 total connections ready) 2023-08-12 19:52:36 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: queue http://192.0.2.20:9998: Connection to peer 192.0.2.20:9998 claimed request [Req1: PUT http://192.0.2.20:9998/tika/] 2023-08-12 19:52:36 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: conn 192.0.2.20:9998 [1]: Claimed request [Req1: PUT http://192.0.2.20:9998/tika/] 2023-08-12 19:52:36 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: request [Req1: PUT http://192.0.2.20/tika/] Sent header 2023-08-12 19:52:36 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: peer 192.0.2.20:9998: No more requests to service for this peer (1 connections exist, 0 pending) 2023-08-12 19:52:36 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: conn 192.0.2.20:9998 [1]: Got 422 response for request [Req1: PUT http://192.0.2.20:9998/tika/] Unprocessable Entity (took 3 ms + 106 ms in queue) 2023-08-12 19:52:36 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: request [Req1: PUT http://192.0.2.20/tika/] Finished 2023-08-12 19:52:36 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: queue http://192.0.2.20:9998: Dropping request [Req1: PUT http://192.0.2.20:9998/tika/] 2023-08-12 19:52:36 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: host 192.0.2.20: Host is idle (timeout = 100 msecs) 2023-08-12 19:52:36 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: request [Req1: PUT http://192.0.2.20/tika/] Free (requests left=1) 2023-08-12 19:52:36 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: peer 192.0.2.20:9998: No requests to service for this peer (1 connections exist, 0 pending) 2023-08-12 19:52:36 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: conn 192.0.2.20:9998 [1]: No more requests queued; going idle (timeout = 100 msecs) 2023-08-12 19:52:37 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: host 192.0.2.20: Idle host timed out 2023-08-12 19:52:37 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: host 192.0.2.20: Host destroy 2023-08-12 19:52:37 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: host 192.0.2.20: Host session destroy 2023-08-12 19:52:37 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: queue http://192.0.2.20:9998: Destroy 2023-08-12 19:52:37 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: peer 192.0.2.20:9998: Unlinked queue http://192.0.2.20:9998 (0 queues linked) 2023-08-12 19:52:37 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: conn 192.0.2.20:9998 [1]: Idle connection timed out 2023-08-12 19:52:37 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: conn 192.0.2.20:9998 [1]: Connection close 2023-08-12 19:52:37 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: conn 192.0.2.20:9998 [1]: Connection disconnect 2023-08-12 19:52:37 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: conn 192.0.2.20:9998 [1]: Disconnected: Connection closed (fd=28) 2023-08-12 19:52:37 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: conn 192.0.2.20:9998 [1]: Detached peer 2023-08-12 19:52:37 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: peer 192.0.2.20:9998: Lost a connection (0 queues linked, 0 connections left, 0 connections pending, 0 requests pending, 0 requests urgent) 2023-08-12 19:52:37 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: conn 192.0.2.20:9998 [1]: Connection destroy 2023-08-12 19:52:37 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: peer 192.0.2.20:9998: Dropping peer now 2023-08-12 19:52:37 indexer-worker(recipient(a)example2.com)<lrVsHUQb2GR1dgAA+IOfAw:xKZLMEQb2GR3dgAA+IOfAw>: Debug: http-client: peer 192.0.2.20:9998: Peer close unclear 2 me whether this is a bug in dovecot or in tika, or a matter of (mis)config. ??

1 0

sso / sharing authenticated sessions how does it work
by Marc 12 Aug '23

12 Aug '23

I am totally not experienced with setting up sso etc. I was wondering if/how it is possible to share an authenticated session between dovecot and the mail client with the webmail so a user does not need to login again.

1 0

Dovecot submission panics on SMTP RCPT error
by IB Development Team 10 Aug '23

10 Aug '23

Hi, If SMTP server rejects RCPT with message aaa bbbbbbbbbb ccccccc dd eeeeeeeeeeee ffffffffffff gggggggggg hh iiiiii jj kk llll mmmmmmmmmm nnn o ppppppppppppppp rrrrrr ssssss t uuuu w xxxxxxx yyyyyy zzzzzzzz aa bbbbbbbbbbb cccccccc dd eeeeee fff gggggg hhhhhhh iiiiiiii jj kkkkkk ll mm nnnn ooooo ppppppp qqqq r ssss tttttt uuuuuuuuuuuuu www xxx yyyyyyy zzzzz aaaaa bbbbbbbbb ccc ddddddd when submitting message via dovecot submisstion server, dovecot panics: 2023-08-10 17:10:04 myhost exim[...] rejected RCPT <[...]>: aa bbbbbbbbbb ccccccc dd eeeeeeeeeeee ffffffffffff gggggggggg hh iiiiii jj kk llll mmmmmmmmmm nnn o ppppppppppppppp rrrrrr ssssss t uuuu w xxxxxxx yyyyyy zzzzzzzz aa bbbbbbbbbbb cccccccc dd eeeeee fff gggggg hhhhhhh iiiiiiii jj kkkkkk ll mm nnnn ooooo ppppppp qqqq r ssss tttttt uuuuuuuuuuuuu www xxx yyyyyyy zzzzz aaaaa bbbbbbbbb ccc ddddddd 2023-08-10 17:10:04 myhost dovecot: submission(test)<7274><bAOu/ZICuuDAqHoB>: Panic: file smtp-server-reply.c: line 625 (smtp_server_reply_send_real): assertion failed: (text[0] == '-') 2023-08-10 17:10:04 myhost dovecot: submission(test)<7274><bAOu/ZICuuDAqHoB>: Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(backtrace_append+0x42) [0x7f64c8e484e2] -> /usr/lib/dovecot/libdovecot.so.0(backtrace_get+0x1e) [0x7f64c8e485fe] -> /usr/lib/dovecot/libdovecot.so.0(+0xfc49b) [0x7f64c8e5449b] -> /usr/lib/dovecot/libdovecot.so.0(+0xfc531) [0x7f64c8e54531] -> /usr/lib/dovecot/libdovecot.so.0(+0x53aee) [0x7f64c8dabaee] -> /usr/lib/dovecot/libdovecot.so.0(+0x4751c) [0x7f64c8d9f51c] -> /usr/lib/dovecot/libdovecot.so.0(+0x7b605) [0x7f64c8dd3605] -> /usr/lib/dovecot/libdovecot.so.0(+0x122171) [0x7f64c8e7a171] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x69) [0x7f64c8e69f59] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x132) [0x7f64c8e6b592] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x50) [0x7f64c8e6a000] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x40) [0x7f64c8e6a1c0] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f64c8de04e3] -> dovecot/submission(main+0x4df) [0x562f3ac7c11f] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f64c8ba4d0a] -> dovecot/submission(_start+0x2a) [0x562f3ac7c1aa] Error message is configured in exim4.conf with begin routers mytestrouter: [...] cannot_route_message = aa bbbbbbbbbb ccccccc dd eeeeeeeeeeee ffffffffffff gggggggggg hh iiiiii jj kk llll mmmmmmmmmm nnn o ppppppppppppppp rrrrrr ssssss t uuuu w xxxxxxx yyyyyy zzzzzzzz aa bbbbbbbbbbb cccccccc dd eeeeee fff gggggg hhhhhhh iiiiiiii jj kkkkkk ll mm nnnn ooooo ppppppp qqqq r ssss tttttt uuuuuuuuuuuuu www xxx yyyyyyy zzzzz aaaaa bbbbbbbbb ccc ddddddd Dovecot installed from up-to-date Debian 11 package: # dpkg -s dovecot-core | grep Version Version: 1:2.3.13+dfsg1-2+deb11u1 -- Regards, Paweł Bogusławski IB Development Team E: dev(a)ib.pl

2 2

Dovecot and Thunderbird 15
by Elise 10 Aug '23

10 Aug '23

Hi, I recently had some issues with Thunderbird 15 when removing e-mail. With my other mailclient Outlook I could without any issue. As I don't grand POP3 mail pickup, I disabled that in dovecot.conf #service pop3-login { # inet_listener pop3 { # address = 10.10.10.52 # } # inet_listener pop3s { # address = 10.10.10.52 # } #} Enabling the above and restarting Dovecot solved the issue. Is it TB related or do I overlook a config parameter? E.

2 3

Question about custom flags
by d.meister-braeuer＠heinlein-support.de 09 Aug '23

09 Aug '23

Hello all, RFC3501 describes among other things that custom flags can be set for mails. These can be allowed or forbidden in the server: "Servers MAY permit the client to define new keywords in the mailbox". When I am on our server, it also looks like dovecot basically allows this: a select INBOX FLAGS (\Answered \Flagged \Deleted \Seen \Draft $Forwarded Old $ITipAnalyzed $ITipProcessed Forwarded NonJunk $cl_1 $cl_2 $cl_3) OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $Forwarded Old $ITipAnalyzed $ITipProcessed Forwarded NonJunk $cl_1 $cl_2 $cl_3 \*)] Flags permitted. The only option that goes in this direction is this one: mail_max_keyword_length But basically describes something else as well, if I see it correctly. Is there a way to prevent dovecot from giving custom flags, or what is the name of the parameter where you can set this? Best

1 0

Outlook not showing all email in inbox at random
by David Koski 09 Aug '23

09 Aug '23

I manage IT for a small office with about 10 workstations. Dovecot has been installed for years and is now at 2.3.4.1-5 (Debian). I just discovered that the user cannot see many emails in the inbox. I was alerted when email was not appearing to be coming in for many hours. Other users observed the same thing. On investigation, I noticed many inbox emails missing in the Outlook GUI from times past that were on the server. It appears to be a broken client (Outlook 2019) to me but troubleshooting has yielded no solid leads. The profile and OST files have been re-created many times with no change in outcome. Each time, the inbox does not synchronize about 40 most recent emails but does for the last one for the day before and many before that. It may well be an Outlook issue so I'm looking for information to work around an Outlook bug if necessary. Thunderbird works perfectly. Also, interestingly, I can copy email from Thunderbird (or move emails on the server) to a temporary directory form the inbox, then move them back and they will appear if I don't move too many at a time. Regards, David Koski dkoski(a)sutinen.com

4 7

Application to automate mail-crypt plugin with user keys
by Benedikt Zumtobel 08 Aug '23

08 Aug '23

Hi everyone, I run a very small (about 10 users) mailserver setup with the mail-crypt plugin activated and wanted to transition from a global key setup to user keys. Since I couldn't find a satisfying solution to reencrypt mailboxes when users change their password I started writing a script that ended up being a small webservice giving users a simple selfservice portal to change their password. I consider it pretty much done by now and it's public on github. Maybe it's useful for somebody else. However its use case is very narrow since I initially wrote this just for myself. It does only support a Postgres user store for instance. The main reason, however, for this post is me wondering if anybody would bother to give me some feedback. Especially for the following aspects: I execute doveadm from within my application. The advantage is that I can keep the code very light (~650 lines of code) while at the same time don't have to care about the crypto stuff which I wouldn't dare trying anyways. The disadvantage of this approach is that I had to write a doveadm wrapper that needs the setuid bit to execute doveadm mailbox commands. I consider this a bit hacky but a better solution didn't come to my mind. Another aspect is the password that is being used to encrypt the private key. When creating a new user in the database I generate a unique salt. This salt is prefixed to the user's clear text password and the resulting string is run trough the sha3-512 hash function. The output is the mail_crypt_private_password. You find the project here: https://github.com/nonce9/pwch Criticism is welcome. Benedikt

4 3

Dovecot bug submit - Dovecot stats
by Brent Clark 07 Aug '23

07 Aug '23

Good day Guys Just a brief background, Im looking to enable and run doveadm stats, as per https://doc.dovecot.org/configuration_manual/stats/ When I run the command: doveadm stats add --description "IMAP SELECT commands" --exporter log-exporter --exporter-include "name timestamps" --fields "bytes_in bytes_out" --group-by "cmd_name tagged_reply_state" imap_cmd_select "event=imap_command_finished AND name=SELECT" I get: ==> /var/log/mail.info <== Aug 3 05:52:36 bullseye dovecot: stats: Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(backtrace_append+0x42) [0x7f08c7097582] -> /usr/lib/dovecot/libdovecot.so.0(backtrace_get+0x1e) [0x7f08c709769e] -> /usr/lib/dovecot/libdovecot.so.0(+0x1022fb) [0x7f08c70a42fb] -> /usr/lib/dovecot/libdovecot.so.0(+0x102391) [0x7f08c70a4391] -> /usr/lib/dovecot/libdovecot.so.0(+0x55589) [0x7f08c6ff7589] -> dovecot/stats(+0x5aa5) [0x561765217aa5] -> dovecot/stats(stats_metrics_add_dynamic+0x11f) [0x56176521a80f] -> dovecot/stats(+0x6724) [0x561765218724] -> /usr/lib/dovecot/libdovecot.so.0(connection_input_default+0x155) [0x7f08c709bd05] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x69) [0x7f08c70ba529] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x132) [0x7f08c70bbc12] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x50) [0x7f08c70ba5d0] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x40) [0x7f08c70ba790] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f08c702d353] -> dovecot/stats(main+0x128) [0x561765217e38] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f08c6ca8d0a] -> dovecot/stats(_start+0x2a) [0x561765217eba] ==> /var/log/mail.warn <== Aug 3 06:23:51 bullseye dovecot: stats: Fatal: master: service(stats): child 38121 killed with signal 6 (core dumped) I installed and run systemd-coredump, dovecot-sysreport Please will you take a look at the various attached files (coredumps etc) Many thanks. Kind Regards Brent Clark

2 1

doveadm fts lookup: Is "search query" documented anywhere?
by Graham Leggett 07 Aug '23

07 Aug '23

Hi all, According to the —help option for doveadm fts, there is a “lookup” option as follows: [root@aurora ~]# doveadm fts --help usage: doveadm [-Dv] [-f <formatter>] fts <command> [<args>] expand [-u <user>|-A] [-S <socket_path>] <search query> lookup [-u <user>|-A] [-S <socket_path>] <search query> optimize [-u <user>|-A] [-S <socket_path>] [<namespace>] rescan [-u <user>|-A] [-S <socket_path>] [<namespace>] tokenize [-u <user>|-A] [-S <socket_path>] <text> The option does not appear to be documented anywhere (eg https://doc.dovecot.org/3.0/man/doveadm-fts.1/) and “search query” doesn’t appear to be defined. Guessing that it might be just plain text, I tried searching for a known word, but this failed as follows: [root@aurora ~]# doveadm fts lookup -u test(a)example.com sweet Fatal: Unknown argument SWEET Anyone know? Regards, Graham —

2 2

Trash deleted date?
by joe a 07 Aug '23

07 Aug '23

Using Thunderbird client. Cannot sort "Trash" Folder by time/date deleted. I can display both "Received" and "Date" columns, but they show identical data. Does dovecot create a "deleted" time stamp? dovecote 2.3.20 (80a5ac675d) TB 102.14.0 64 bit

1 0