- dovecot - dovecot.org

tika 2.4.1 'Text extraction failed' errors when dovecot+fts 2.3.19.1 passes embedded *.eml (message/rfc822) files ; org.apache.tika.parser.mail.RFC822Parser or dovecot ?
by PGNet Dev 01 Aug '22

01 Aug '22

i'm running dovecot 2.3.19.1 + fts tika-server-standard 2.4.1 dovecot is feeding tika backend via fts_tika when dovecot passes data with *.eml attachments embedded, tika fails to correctly parse/extract content not clear if the issue is with tika, or what dovecot's passing in this case. other non-.eml attachments are fine. here's the current failing procedure, (1) create a simple pdf enscript -p mime.ps /etc/mime.types ps2pdf mime.ps mime.pdf (2) send an email *with* mime.pdf attachment to echo "test" | mailx -s "test" -a ./mime.pdf testuser(a)example.com tika processes OK journalctl -f -u tika ... Jul 30 19:09:24 mx-test tika[19682]: INFO [qtp2112135199-30] 19:09:24,165 org.apache.tika.server.core.resource.TikaResource /tika (application/pdf) ... save the just-received email with .pdf attachment as mime.eml (3) send an email with NO .pdf attachment save the just-received email with .pdf attachment as mime2.eml (4) send an email with mime.eml attachment, containing the embedded mime.pdf echo "test" | mailx -s "test" -a ./mime.eml testuser(a)example.com tika fails to extract message/rfc822 journalctl -f -u tika | grep -v StatusLogger ... Jul 30 19:28:00 mx-test tika[20049]: INFO [qtp2112135199-30] 19:28:00,834 org.apache.tika.server.core.resource.TikaResource /tika (message/rfc822) Jul 30 19:28:00 mx-test tika[20049]: WARN [qtp2112135199-30] 19:28:00,840 org.apache.tika.server.core.resource.TikaResource tika/: Text extraction failed (mime.eml) Jul 30 19:28:00 mx-test tika[20049]: org.apache.tika.exception.ZeroByteFileException: InputStream must have > 0 bytes Jul 30 19:28:00 mx-test tika[20049]: at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:153) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.apache.tika.parser.ParserDecorator.parse(ParserDecorator.java:152) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.apache.tika.parser.DigestingParser.parse(DigestingParser.java:55) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.apache.tika.server.core.resource.TikaResource.parse(TikaResource.java:352) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.apache.tika.server.core.resource.TikaResource.lambda$produceText$1(TikaResource.java:502) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.apache.cxf.jaxrs.provider.BinaryDataProvider.writeTo(BinaryDataProvider.java:177) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.apache.cxf.jaxrs.utils.JAXRSUtils.writeMessageBody(JAXRSUtils.java:1616) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.serializeMessage(JAXRSOutInterceptor.java:249) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.processResponse(JAXRSOutInterceptor.java:122) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.handleMessage(JAXRSOutInterceptor.java:84) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.apache.cxf.interceptor.OutgoingChainInterceptor.handleMessage(OutgoingChainInterceptor.java:90) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:265) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(JettyHTTPDestination.java:247) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHandler.java:79) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:190) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at java.lang.Thread.run(Thread.java:833) ~[?:?] Jul 30 19:28:00 mx-test tika[20049]: ERROR [qtp2112135199-30] 19:28:00,845 org.apache.cxf.jaxrs.utils.JAXRSUtils Problem with writing the data, class org.apache.tika.server.core.resource.TikaResource$$Lambda$338/0x0000000800eb4a38, ContentType: text/plain (5) send an email with mime2.eml attachment, WITHOUT an embedded .pdf echo "test" | mailx -s "test" -a ./mime.eml testuser(a)example.com again, tika fails to extract message/rfc822 journalctl -f -u tika | grep -v StatusLogger ... Jul 30 19:28:33 mx-test tika[20049]: INFO [qtp2112135199-30] 19:28:33,607 org.apache.tika.server.core.resource.TikaResource /tika (message/rfc822) Jul 30 19:28:33 mx-test tika[20049]: WARN [qtp2112135199-30] 19:28:33,616 org.apache.tika.server.core.resource.TikaResource tika/: Text extraction failed (mime2.eml) Jul 30 19:28:33 mx-test tika[20049]: org.apache.tika.exception.ZeroByteFileException: InputStream must have > 0 bytes Jul 30 19:28:33 mx-test tika[20049]: at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:153) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.apache.tika.parser.ParserDecorator.parse(ParserDecorator.java:152) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.apache.tika.parser.DigestingParser.parse(DigestingParser.java:55) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.apache.tika.server.core.resource.TikaResource.parse(TikaResource.java:352) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.apache.tika.server.core.resource.TikaResource.lambda$produceText$1(TikaResource.java:502) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.apache.cxf.jaxrs.provider.BinaryDataProvider.writeTo(BinaryDataProvider.java:177) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.apache.cxf.jaxrs.utils.JAXRSUtils.writeMessageBody(JAXRSUtils.java:1616) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.serializeMessage(JAXRSOutInterceptor.java:249) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.processResponse(JAXRSOutInterceptor.java:122) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.handleMessage(JAXRSOutInterceptor.java:84) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.apache.cxf.interceptor.OutgoingChainInterceptor.handleMessage(OutgoingChainInterceptor.java:90) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:265) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(JettyHTTPDestination.java:247) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHandler.java:79) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:190) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at java.lang.Thread.run(Thread.java:833) ~[?:?] Jul 30 19:28:33 mx-test tika[20049]: ERROR [qtp2112135199-30] 19:28:33,630 org.apache.cxf.jaxrs.utils.JAXRSUtils Problem with writing the data, class org.apache.tika.server.core.resource.TikaResource$$Lambda$338/0x0000000800eb4a38, ContentType: text/plain (6) submit mime.eml directly to tika curl -T ./mime.eml http://127.0.0.1:9998/tika journalctl -f -u tika | grep -v StatusLogger ... Jul 30 19:30:08 mx-test tika[20049]: INFO [qtp2112135199-34] 19:30:08,073 org.apache.tika.server.core.resource.TikaResource /tika (autodetecting type) (7) submit mime2.eml directly to tika curl -T ./mime2.eml http://127.0.0.1:9998/tika journalctl -f -u tika | grep -v StatusLogger ... Jul 30 19:30:52 mx-test tika[20049]: INFO [qtp2112135199-30] 19:30:52,349 org.apache.tika.server.core.resource.TikaResource /tika (autodetecting type) (8) where, cat mime.eml Return-Path: <msmtp(a)pgnd.example.com> Delivered-To: testuser(a)example.com ... From: msmtp(a)pgnd.example.com Date: Sat, 30 Jul 2022 18:53:38 -0400 To: testuser(a)example.com Subject: test User-Agent: Heirloom mailx 12.5 7/5/10 Content-Type: multipart/mixed; boundary="=_62e5b672.wAyBX+sGMbS7ZcNv8O/A1QeYuseaJ2NDRf8hfdbm/x8Vayp+" Message-Id: <4LwKS35QWSzWf3Q(a)mx-test.example.com> This is a multi-part message in MIME format. --=_62e5b672.wAyBX+sGMbS7ZcNv8O/A1QeYuseaJ2NDRf8hfdbm/x8Vayp+ Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline test --=_62e5b672.wAyBX+sGMbS7ZcNv8O/A1QeYuseaJ2NDRf8hfdbm/x8Vayp+ Content-Type: application/pdf Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="mime.pdf" JVBERi0xLjQKJcfsj6IKJSVJbnZvY2F0aW9uOiBwYXRoL2dzIC1QLSAtZFNBRkVSIC1kQ29t ... Rgo= --=_62e5b672.wAyBX+sGMbS7ZcNv8O/A1QeYuseaJ2NDRf8hfdbm/x8Vayp+-- and, cat mime2.eml Return-Path: <msmtp(a)pgnd.example.com> Delivered-To: testuser(a)example.com ... From: msmtp(a)pgnd.example.com Date: Sat, 30 Jul 2022 19:14:59 -0400 To: testuser(a)example.com Subject: test User-Agent: Heirloom mailx 12.5 7/5/10 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-Id: <4LwKwh5brVzWf3Q(a)mx-test.example.com> test

2 2

Re: Dovecot folder problem while deleting via IMAP
by Christian Mack 29 Jul '22

29 Jul '22

Hello Am 29.07.22 um 10:42 schrieb Pascal Bonny: > Dovecot Version: > 2.3.17 (8414c38c5c) > > Dear, > Some troubles occur while we delete a folder via IMAP ssl command or via a IMAP > client (different clients tested). > > When we want to delete subfolder we observe following issues. > > > some examples following: > > we created a folder with subfolders x/y/z > > <IMAP command create> > a02 CREATE x.y.z > a02 OK Create completed (0.003 + 0.000 + 0.002 secs). > <IMAP command create> > > <doveadm mailbox list -u "user(a)domain.tld"> > ====response from dovecot cli==== > x > x.y > x.y.z > > then we try to delete the z subfolder of x > > <IMAP command delete> > a04 DELETE x.y.z > a05 OK Delete completed (0.002 + 0.000 + 0.001 secs). > <IMAP command delete> > > The IMAP console returns success and the dovecot listing is ok > > <doveadm mailbox list -u "user(a)domain.tld"> > ====response from dovecot cli ===== > x > x.y > > But dovecot status on x.y returns the folder is not existent > [cut] That is normal IMAP behaviour. With creating x.y.z you really create only folder z and a folder hirarchy x.y, in which you can not store emails (nor have any access). You have to create x and x.y, in order to get complete folders and subfolders usable for emails and be able to delete them :-) Kind regards, Christian Mack -- Christian Mack Universität Konstanz Kommunikations-, Informations-, Medienzentrum (KIM) Abteilung IT-Dienste Forschung, Lehre, Infrastruktur 78457 Konstanz +49 7531 88-4416

1 0

large search indexer tasks, submitted to flatcurve+tika+tesseract backend for attachment scanning, timeout even with "fts_index_timeout = 0"; how to increase/remove timeouts?
by PGNet Dev 28 Jul '22

28 Jul '22

i'm running dovecot 2.3.19.1 search indexing is enabled, via fts_flatcurve (dovecot23_fts_flatcurve) attachment processing is via fts_tika, with tika-server-standard-2.4.2 backend & tesseract5 OCR i'm working getting this optimized on a smaller, 4-core server that opt includes capping tesseract @ single-thread, Environment=OMP_THREAD_LIMIT=1 still, some attachments are OCR-scanned, and can take some time. when i exec large reindex jobs, i get occassional timeout errors on dovecot's indexer-worker connection to tiks backend, e.g., 2022-07-23 09:54:43 indexer-worker(postmaster(a)example.com)<DIcjEWb922JhXAAA+IOfAw>: Error: fts_tika: PUT http://127.0.0.1:9998/tika/ failed: Request timed out (Request queued 61.031 secs ago, 1 send attempts in 60.103 secs, 60.080 in http ioloop, 0.000 in other ioloops, connected 60.103 secs ago) 2022-07-23 09:54:43 indexer-worker(postmaster(a)example.com)<DIcjEWb922JhXAAA+IOfAw>: Error: Mailbox Sent: Precache for UID=90782 failed: Internal error occurred. Refer to server log for more information. [2022-07-23 09:54:43] (attempted to index 2 messages between UIDs 90778..90782) i don't see any fts timeout info @ https://wiki.dovecot.org/Timeouts here https://doc.dovecot.org/settings/plugin/fts-plugin/#plugin_setting-fts-fts_… " fts_index_timeout Default: 0 Values: Unsigned integer When the full text search backend detects that the index isn’t up-to-date, the indexer is told to index the messages and is given this much time to do so. If this time limit is reached, an error is returned, indicating that the search timed out during waiting for the indexing to complete: NO [INUSE] Timeout while waiting for indexing to finish A value of 0 means no timeout. " double-checking my config, /usr/sbin/dovecot -c /etc/dovecot/dovecot.conf -n submission_relay_connect_timeout = 10 secs submission_timeout = 10 secs there's no tika/fts timeout override, so should be "0" by default. i.e., no timeout @ Patch: enhancements for solr/tika integration https://dovecot.org/list/dovecot/2022-January/123828.html a patch was proposed for similar issue with fts-solr indexer (not sure if it's in current release AND effective for my config), but that *limits* the size of attachments submitted. i _do_ want to index all/large attachments, so, iiuc, need to ensure no, or longer, timeout. where do I set that timeout to not fail, as above, on large index tasks?

2 2

RHEL9 Latest Repo?
by dovecot＠ptld.com 28 Jul '22

28 Jul '22

Any plans or timeline for when there will be a latest repo for RHEL9? [dovecot-2.3-latest] name=Dovecot 2.3 RHEL $releasever - $basearch baseurl=http://repo.dovecot.org/ce-2.3-latest/rhel/$releasever/RPMS/$basear… gpgkey=https://repo.dovecot.org/DOVECOT-REPO-GPG gpgcheck=1 enabled=1

3 2

test-crypto.c - Assert failed
by Tamsy 27 Jul '22

27 Jul '22

Dear List, Please pardon me if this has been already discussed before. I couldn't find the matter with a quick search. On a new standard Ubuntu 22.04 LTS installation Dovecot's "configure && make" runs through but "make check" fails. Is dovecot-2.3.19.1 not yet compatible with openSSL 3.0.2 (openssl 3.0.2-0ubuntu1.6) or is this just happening here? Distributor ID: Ubuntu Description: Ubuntu 22.04 LTS Release: 22.04 Codename: jammy Linux 5.15.0-41-generic #44-Ubuntu SMP Wed Jun 22 14:20:53 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux $ make check <SNIP> test_cipher_test_vectors ............................................. : ok test_cipher_aead_test_vectors ........................................ : ok test_hmac_test_vectors ............................................... : ok test_load_v1_keys .................................................... : ok test_load_v1_key ..................................................... : ok test_load_v1_public_key .............................................. : ok test_load_v2_key ..................................................... : ok test_load_v2_public_key .............................................. : ok test_get_info_v2_key ................................................. : ok test_gen_and_get_info_rsa_pem ........................................ : ok test_get_info_rsa_private_key ........................................ : ok test_get_info_invalid_keys ........................................... : ok test_get_info_key_encrypted .......................................... : ok test_get_info_pw_encrypted ........................................... : ok test-crypto.c:827: Assert failed: ret == TRUE Panic: file dcrypt-openssl.c: line 2639 (dcrypt_openssl_private_to_public_key): assertion failed: (priv_key != NULL && pub_key_r != NULL) Error: Raw backtrace: ./test-crypto(+0x60704) [0x168704] -> ./test-crypto(backtrace_append+0x1c) [0x168893] -> ./test-crypto(backtrace_get+0x2a) [0x1688bf] -> ./test-crypto(+0x28ef4) [0x130ef4] -> ./test-crypto(default_fatal_handler+0) [0x130fc6] -> ./test-crypto(default_error_handler+0) [0x131014] -> ./test-crypto(i_fatal+0) [0x1312ae] -> .libs/libdcrypt_openssl.so(+0xe795) [0x4ea3795] -> ./test-crypto(dcrypt_key_convert_private_to_public+0x7b) [0x11eeb1] -> ./test-crypto(+0x21655) [0x129655] -> ./test-crypto(+0x23d66) [0x12bd66] -> ./test-crypto(test_run+0x21) [0x12c13a] -> ./test-crypto(main+0x83) [0x12af16] -> /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x4893d90] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x4893e40] -> ./test-crypto(_start+0x25) [0x11c1a5] ../../run-test.sh: line 39: 164971 Aborted (core dumped) /usr/bin/valgrind -q $trace_children --error-exitcode=213 --leak-check=full --gen-suppressions=all --suppressions="$supp_path" --log-file=$test_out $noundef $* ==164971== Conditional jump or move depends on uninitialised value(s) ==164971== at 0x514E234: ??? (in /usr/lib/x86_64-linux-gnu/libcrypto.so.3) ==164971== by 0x514E511: ??? (in /usr/lib/x86_64-linux-gnu/libcrypto.so.3) ==164971== by 0x504F0F4: EVP_DecryptFinal_ex (in /usr/lib/x86_64-linux-gnu/libcrypto.so.3) ==164971== by 0x4E9CD3F: dcrypt_openssl_ctx_sym_final (in /usr/local/src/dovecot-2.3.19.1/src/lib-dcrypt/.libs/libdcrypt_openssl.so) ==164971== by 0x11E3DB: dcrypt_ctx_sym_final (in /usr/local/src/dovecot-2.3.19.1/src/lib-dcrypt/test-crypto) ==164971== by 0x1270DE: test_cipher_aead_test_vectors (in /usr/local/src/dovecot-2.3.19.1/src/lib-dcrypt/test-crypto) ==164971== by 0x12BD65: test_run_funcs (in /usr/local/src/dovecot-2.3.19.1/src/lib-dcrypt/test-crypto) ==164971== by 0x12C139: test_run (in /usr/local/src/dovecot-2.3.19.1/src/lib-dcrypt/test-crypto) ==164971== by 0x12AF15: main (in /usr/local/src/dovecot-2.3.19.1/src/lib-dcrypt/test-crypto) ==164971== { <insert_a_suppression_name_here> Memcheck:Cond obj:/usr/lib/x86_64-linux-gnu/libcrypto.so.3 obj:/usr/lib/x86_64-linux-gnu/libcrypto.so.3 fun:EVP_DecryptFinal_ex fun:dcrypt_openssl_ctx_sym_final fun:dcrypt_ctx_sym_final fun:test_cipher_aead_test_vectors fun:test_run_funcs fun:test_run fun:main } ==164971== 2,304 bytes in 1 blocks are possibly lost in loss record 911 of 947 ==164971== at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==164971== by 0x4005D97: malloc (rtld-malloc.h:56) ==164971== by 0x4005D97: _dlfo_mappings_segment_allocate (dl-find_object.c:217) ==164971== by 0x4005D97: _dl_find_object_update_1 (dl-find_object.c:671) ==164971== by 0x4005D97: _dl_find_object_update (dl-find_object.c:804) ==164971== by 0x400ECCF: dl_open_worker_begin (dl-open.c:735) ==164971== by 0x49DEC27: _dl_catch_exception (dl-error-skeleton.c:208) ==164971== by 0x400DF99: dl_open_worker (dl-open.c:782) ==164971== by 0x49DEC27: _dl_catch_exception (dl-error-skeleton.c:208) ==164971== by 0x400E34D: _dl_open (dl-open.c:883) ==164971== by 0x48FA6BB: dlopen_doit (dlopen.c:56) ==164971== by 0x49DEC27: _dl_catch_exception (dl-error-skeleton.c:208) ==164971== by 0x49DECF2: _dl_catch_error (dl-error-skeleton.c:227) ==164971== by 0x48FA1AD: _dlerror_run (dlerror.c:138) ==164971== by 0x48FA747: dlopen_implementation (dlopen.c:71) ==164971== by 0x48FA747: dlopen@@GLIBC_2.34 (dlopen.c:81) ==164971== { <insert_a_suppression_name_here> Memcheck:Leak match-leak-kinds: possible fun:malloc fun:malloc fun:_dlfo_mappings_segment_allocate fun:_dl_find_object_update_1 fun:_dl_find_object_update fun:dl_open_worker_begin fun:_dl_catch_exception fun:dl_open_worker fun:_dl_catch_exception fun:_dl_open fun:dlopen_doit fun:_dl_catch_exception fun:_dl_catch_error fun:_dlerror_run fun:dlopen_implementation fun:dlopen@@GLIBC_2.34 } ==164971== 2,304 bytes in 1 blocks are possibly lost in loss record 912 of 947 ==164971== at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==164971== by 0x4005D97: malloc (rtld-malloc.h:56) ==164971== by 0x4005D97: _dlfo_mappings_segment_allocate (dl-find_object.c:217) ==164971== by 0x4005D97: _dl_find_object_update_1 (dl-find_object.c:671) ==164971== by 0x4005D97: _dl_find_object_update (dl-find_object.c:804) ==164971== by 0x400ECCF: dl_open_worker_begin (dl-open.c:735) ==164971== by 0x49DEC27: _dl_catch_exception (dl-error-skeleton.c:208) ==164971== by 0x400DF99: dl_open_worker (dl-open.c:782) ==164971== by 0x49DEC27: _dl_catch_exception (dl-error-skeleton.c:208) ==164971== by 0x400E34D: _dl_open (dl-open.c:883) ==164971== by 0x49DEF90: do_dlopen (dl-libc.c:95) ==164971== by 0x49DEC27: _dl_catch_exception (dl-error-skeleton.c:208) ==164971== by 0x49DECF2: _dl_catch_error (dl-error-skeleton.c:227) ==164971== by 0x49DF0C6: dlerror_run (dl-libc.c:45) ==164971== by 0x49DF0C6: __libc_dlopen_mode (dl-libc.c:162) ==164971== by 0x498F7C0: __libc_unwind_link_get (unwind-link.c:50) ==164971== by 0x498F7C0: __libc_unwind_link_get (unwind-link.c:40) ==164971== { <insert_a_suppression_name_here> Memcheck:Leak match-leak-kinds: possible fun:malloc fun:malloc fun:_dlfo_mappings_segment_allocate fun:_dl_find_object_update_1 fun:_dl_find_object_update fun:dl_open_worker_begin fun:_dl_catch_exception fun:dl_open_worker fun:_dl_catch_exception fun:_dl_open fun:do_dlopen fun:_dl_catch_exception fun:_dl_catch_error fun:dlerror_run fun:__libc_dlopen_mode fun:__libc_unwind_link_get fun:__libc_unwind_link_get } Failed to run: ./test-crypto make[3]: *** [Makefile:1136: check-local] Error 1 make[3]: Leaving directory '/usr/local/src/dovecot-2.3.19.1/src/lib-dcrypt' make[2]: *** [Makefile:983: check-am] Error 2 make[2]: Leaving directory '/usr/local/src/dovecot-2.3.19.1/src/lib-dcrypt' make[1]: *** [Makefile:573: check-recursive] Error 1 make[1]: Leaving directory '/usr/local/src/dovecot-2.3.19.1/src' make: *** [Makefile:702: check-recursive] Error 1

2 2

Ubuntu 22.04/Jammy repo coming soon ?
by Dean Carpenter 25 Jul '22

25 Jul '22

Is there any idea of when a repo for Ubuntu 22.04/Jammy will be built ? I've tried a couple of test installs onto a 22.04 system using the 20.04/Focal repo, but no luck yet. libssl issues etc. -- Dean Carpenter

2 2

Re: How to correctly use readonly=yes with the sqlite driver?
by oddlama 24 Jul '22

24 Jul '22

On 24/07/2022 10:52, Eduardo M KALINOWSKI wrote: > The documentation mentions that "Prior to v2.3.18, Dovecot uses the > whole value as filename to connect, whitespace included.". Are you > running a new enough version? > > -- > You may get an opportunity for advancement today. Watch it! > > Eduardo M KALINOWSKI > eduardo at kalinowski.com.br I am running the newest available version from the main gentoo repository, which is 2.3.19.1 - unfortunately, nobody else on the internet seems to have written about this feature before, so I cannot easily check if I made a mistake. Do you require any further information? Best regards, oddlama

2 1

How to correctly use readonly=yes with the sqlite driver?
by oddlama 24 Jul '22

24 Jul '22

Hi! I want to use a sqlite database in readonly mode, which I found mentionend in the docs [1]. Unfortunately no examples were given, so I tried to infer the usage by inspecting core/driver-sqlite.c. If I interpret it correctly, this should have worked: /etc/dovecot/dovecot-sql.conf.ext: driver = sqlite connect = readonly=yes /var/vmail/virtual.sqlite user_query = #... Unfortunately, dovecot tries to open a file named "readonly=yes /var/vmail/virtual.sqlite" instead. I have no idea why the option is not parsed. Is this a bug or am I using the feature incorrectly? Best regards, oddlama [1]: https://doc.dovecot.org/settings/plugin/sql-sqlite/

2 1

Dovecot sync stopped working since 2.3.18-r1 when .maildir has 700 permissions
by Martin Kuchta 23 Jul '22

23 Jul '22

Hello, Since upgrading to dovecot 2.3.18-r1 my sync setup using replicator plugin stopped working. It seems there is a problem accessing a .maildir with 700 permissions, only accessible by the owner. Everything worked fine prior to this version and I made no configuration changes. # 2.3.19.1 (9b53102964): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.19 (4eae2f79) # OS: Linux 5.10.74-gentoo x86_64 Gentoo Base System release 2.8 # Hostname: www.example.com auth_mechanisms = plain login auth_username_format = %Ln doveadm_password = # hidden, use -P to show it hostname = www.example.xom listen = * login_greeting = Dovecot ready. mail_location = maildir:~/.maildir mail_plugins = notify replication managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext imapflags notify vnd.dovecot.pipe namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = * driver = pam } plugin { mail_replica = tcps:www.example.com:8000 sieve = file:~/sieve;active=~/.dovecot.sieve sieve_extensions = +notify +imapflags +vnd.dovecot.pipe sieve_pipe_bin_dir = /usr/lib/dovecot/sieve-pipe sieve_plugins = sieve_extprograms } postmaster_address = postmaster(a)example.com protocols = imap lmtp sieve service aggregator { fifo_listener replication-notify-fifo { mode = 0666 } unix_listener replication-notify { mode = 0666 } } service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } } service doveadm { inet_listener { port = 8000 ssl = yes } } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0600 } } ssl_cert = </etc/letsencrypt/live/example.com/fullchain.pem ssl_cipher_list = ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA ssl_client_ca_dir = /etc/ssl/certs ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it userdb { driver = passwd } protocol lmtp { mail_plugins = notify replication sieve postmaster_address = postmaster(a)example.com } protocol lda { mail_plugins = notify replication sieve } local_name mail.example.com { ssl_cert = </etc/letsencrypt/live/example.com/fullchain.pem ssl_key = # hidden, use -P to show it } local_name example.com { ssl_cert = </etc/letsencrypt/live/example.com/fullchain.pem ssl_key = # hidden, use -P to show it } -- roughgrain.com - Mastering Mentoring +447780565902

1 1

Tools to get a report of which folders have new mail?
by Steve Litt 20 Jul '22

20 Jul '22

Hi all, I use a Dovecot IMAP server on my Linux desktop computer, and I'm pretty good at writing shellscripts. Is there any way I could use doveadm or other tools to create a report that shows all my folders in a hierarchy? Also, is there a way to show only those with new mail? Thanks, SteveT Steve Litt Summer 2022 featured book: Thriving in Tough Times http://www.troubleshooters.com/bookstore/thrive.htm

7 10