July 2022 - dovecot - dovecot.org

Unclear how to get both user and domain quotas to work
by Hans van Zijst 10 Sep '22

10 Sep '22

Hi, First of all: my apologies for this long message. I'm trying to post all the information anybody could need to help me on my way. What I'm trying to make is an installation where both users and domains can have quotas. Ideally I would configure a quota for a domain and set a quota of 0 for all users within that domain, so that mail will be accepted up to the point where the domain quota is reached. Setting up user quota was fairly easy, there's lots of documentation around and examples to take inspiration from. Not so much for domain quota however, or I'm really bad at searching the Net. I'm running Debian Bullseye here: MariaDB 10.5.12 Dovecot 2.3.13 Postfixadmin 3.3.10 I want to use dictionaries in MySQL, because eventually this will be a platform with several Dovecot and Postfix servers. At this time I'm only running one Dovecot installation and a few Postfix servers (two for communication with the outside world and one to do the virus and spam scanning). My current setup for user quota is how Postfixadmin wants it: MariaDB [postfixadmin]> desc quota2; +----------+--------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +----------+--------------+------+-----+---------+-------+ | username | varchar(100) | NO | PRI | NULL | | | bytes | bigint(20) | NO | | 0 | | | messages | int(11) | NO | | 0 | | +----------+--------------+------+-----+---------+-------+ 3 rows in set (0.001 sec) In /etc/dovecot/conf.d/90-quota.conf I have this: plugin { quota = dict:User quota::proxy::sqlquota quota_rule = *:bytes=1024M quota_rule2 = Trash:bytes=+100M quota_vsizes = yes quota_status_success = DUNNO quota_status_nouser = DUNNO quota_status_overquota = "552 5.2.2 Mailbox is full" } dict { sqlquota = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext } The SQL-configuration /etc/dovecot/dovecot-dict-sql.conf.ext is this: connect = host=... dbname=... user=... password=... map { pattern = priv/quota/storage table = quota2 username_field = username value_field = bytes } map { pattern = priv/quota/messages table = quota2 username_field = username value_field = messages } map { pattern = shared/expire/$user/$mailbox table = expires value_field = expire_stamp fields { username = $user mailbox = $mailbox } } I use LMTP to accept mail from Postfix; the quota-plugin is activated for LMTP, so every incoming mail should be counted. This works like a charm: if I don't configure a quota for a user, he gets the default 1GB, if I override it in Postfixadmin, then that's what he gets. I'm running quota-status on a TCP-port, so the gateway SMTP-servers can connect to Dovecot for a quotacheck to decide whether to accept or reject an incoming message. Again, this works flawlessly. I can't find any serious documentation about this mapping thing. In particular, what's with that "pattern"? All I can find in the documentation is: "The dict key must match exactly priv/quota/storage. The dict keys are hardcoded in the Dovecot code, so depending on what functionality you’re configuring you need to know the available dict keys used it." -- https://doc.dovecot.org/configuration_manual/dict/#dict-sql If anyone can point me at better documentation than the actual source code: please! I browsed through the quota-plugin source code, but I'm not a programmer and couldn't find what I thought I was looking for. That said, I'm not sure where I got the third "map" from, with the shared/expire pattern. I seem to have forgotten to document that bit of my journey. Now, domain quotas... I've tried all kinds of stuff, but at the moment I'm stuck with something that seems to almost work. I added a second quota root: plugin { quota2 = dict:Domain quota:%d:proxy::sqldomainquota quota2_rule = *:bytes=1024M quota2_status_success = DUNNO quota2_status_nouser = DUNNO quota2_status_overquota = "552 5.2.2 Domain quota exceeded" quota2_vsizes = yes } dict { sqldomainquota = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext } This dict refers to the same mapping as the one for user quotas, and I think that's the way it should be. Please correct me if I'm wrong. This creates a new user in the quota2 table, by the name of just the domain. Unfortunately, it simply fills it with the numbers from the first user it can find in that domain, not all users in that domain combined. What I think I should do, is change something in the dict so that it sums up all users in that domain, something like this: SELECT SUM(bytes) FROM quota2 WHERE username LIKE '%@domain'; Problem is that I can't find anything useful online with regards to this, I'm not even sure if I'm thinking in the right direction here. Any hints, suggestions or corrections? Or pointers to articles or documentation I might have missed? Kind regards, Hans

2 1

Redhat 9 Repository for Dovecot
by Istiak Ferdous 29 Aug '22

29 Aug '22

Hello, Redhat 9 is publicly released for some time. Is there any plan to provide repository for it? http://repo.dovecot.org/ce-2.3-latest/rhel/ https://developers.redhat.com/articles/2022/05/18/whats-new-red-hat-enterpr… Thanks, Istiak Ferdous

2 4

Replication not working - GUIDs conflict - will be merged later
by Patrick Westenberg 28 Aug '22

28 Aug '22

Hi everyone, I have a weird problem with replication that I'm unable to solve. A new account is sychronized from an external provider via imapsync. The mails end up on my backend1. I see that the folder structure is immediately replicated to backend2. However, a lot of mails are missing and "doveadm replicator status" also states that something failed: priority fast sync full sync success sync failed mail(a)example.com none 00:02:36 00:39:47 - y There are no error log entries regarding this user at all. After some research I tried to start the replication manually with "doveadm -D backup -u mail(a)example.com -d tcp:x.y.z.11" and finally I got an error message: Jul 31 13:55:37 doveadm(mail(a)example.com)<11341><rm1YCblt5mJNLAAAJFg/Tg>: Debug: brain M: Mailbox INBOX: local=e74590221b6ce6620d29000024583f4e/0/1, remote=20a9ce2b1c6ce662244e0000baba0ddd/0/1: GUIDs conflict - will be merged later As a matter of fact, the mails in the inbox are the ones that are missing on backend2. I always end up with this error, no matter what I do: - I completely removed the folder structure on backend2. - I removed all database entries. - I removed this user from the replication, ran imapsync again followed by "doveadm -D backup ..." I also removed fts-flatcurve to rule this out as a cause of error. I have no idea what to do. This configuration worked for months. I got this error with 2.3.17.1 and 2.3.19.1 Any hints would be highly appreciated. Regards Patrick # 2.3.19.1 (9b53102964): /usr/local/etc/dovecot/dovecot.conf # Pigeonhole version 0.5.19 (4eae2f79) auth_master_user_separator = * auth_mechanisms = plain login doveadm_password = # hidden, use -P to show it doveadm_port = 12345 listen = x.y.z.12 lmtp_save_to_detail_mailbox = yes log_debug = category=fts-flatcurve log_path = /var/log/dovecot.log mail_debug = yes mail_home = /srv/mail/%Ld/%Ln mail_location = maildir:~/Maildir mail_plugins = quota notify replication managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext editheader metric imapc_traffic { fields = bytes_in bytes_out filter = event=imap_command_finished group_by = user } namespace inbox { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Spamverdacht { auto = subscribe special_use = \Junk } mailbox Trash { auto = subscribe special_use = \Trash } prefix = INBOX/ separator = / type = private } passdb { args = /usr/local/etc/dovecot/master-user driver = passwd-file master = yes pass = yes } passdb { args = /usr/local/etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { mail_replica = tcp:x.y.z.11 quota = count:User quota quota_grace = 25M quota_rule2 = INBOX/Trash:storage=+100M quota_status_nouser = DUNNO quota_status_success = DUNNO quota_vsizes = yes quota_warning = storage=100%% quota-full %u quota_warning2 = storage=95%% quota-warning 95 %u quota_warning3 = storage=80%% quota-warning 80 %u quota_warning4 = -storage=100%% quota-ok %u sieve = ~/.dovecot.sieve sieve_after = /usr/local/etc/dovecot/sieve/sieve_after.sieve sieve_default = /usr/local/etc/dovecot/sieve/default.sieve sieve_dir = ~/sieve sieve_extensions = +editheader } protocols = imap pop3 lmtp sieve service aggregator { fifo_listener replication-notify-fifo { user = vmail } unix_listener replication-notify { user = vmail } } service auth { unix_listener auth-userdb { group = vmail mode = 0666 user = vmail } } service config { unix_listener config { mode = 0600 user = vmail } } service doveadm { inet_listener { port = 12345 } } service imap-login { process_min_avail = 2 service_count = 0 } service lmtp { executable = lmtp -L inet_listener lmtp { address = x.y.z.12 port = 24 } process_min_avail = 20 } service managesieve-login { inet_listener sieve { port = 4190 } } service quota-full { executable = script /usr/local/etc/dovecot/quota_full.sh unix_listener quota-full { user = vmail } user = root } service quota-ok { executable = script /usr/local/etc/dovecot/quota_ok.sh unix_listener quota-ok { user = vmail } user = root } service quota-status { client_limit = 1 executable = quota-status -p postfix inet_listener { port = 12340 } } service quota-warning { executable = script /usr/local/etc/dovecot/quota_warning.sh unix_listener quota-warning { user = vmail } user = root } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0600 user = vmail } } ssl = required ssl_cert = </etc/ssl/certs/bundle.pem ssl_client_ca_dir = /etc/ssl/certs ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it userdb { driver = prefetch } userdb { args = /usr/local/etc/dovecot/dovecot-sql.conf.ext driver = sql } verbose_proctitle = yes protocol lmtp { info_log_path = /var/log/dovecot-lmtp.log mail_plugins = quota notify replication sieve } protocol imap { mail_max_userip_connections = 40 mail_plugins = quota notify replication imap_quota }

5 9

namespace directives from userdb not being applied
by Brendan Braybrook 24 Aug '22

24 Aug '22

hi all, i'm having a strange issue. i have a few dovecot instances running: 1- debian 10 with debian's dovecot-core 1:2.3.4.1-5+deb10u5 2- debian 10 with debian's dovecot-core 1:2.3.4.1-5+deb10u6 3- debian 11 with dovecot's dovecot-core 2:2.3.19.1-2+debian11 for all these, i've just been testing with a static userdb (ignore if the login user was different; they all had the same userdb data.. various different passdb configs were tried (none do any userdb prefetching): userdb { driver = static args = mail=maildir:/mail/mailstore03/492/177/b@b.com/:INDEX=/mail/index03/492/177/b@b.com/ home=/mail/mailstore03/492/177/b(a)b.com/ namespace=inbox,share namespace/inbox/separator=/ namespace/share/separator=/ namespace/share/prefix=boop/ } instance #2 and #3 don't seem to pass all the namespace directives i'm passing through to the imap process. (i originally found it happening with a lua userdb script then noticed it with a dict userdb, but the simpler static userdb still shows it happening so i don't think it's related to the different userdb mechanisms themselves) debug looks like: auth: Debug: master userdb out: USER 3466461185 b(a)b.com mail=maildir:/mail/mailstore03/492/177/b@b.com/:INDEX=/mail/index03/492/177/b@b.com/ home=/mail/mailstore03/492/177/b(a)b.com/ namespace=inbox,share namespace/inbox/separator=/ namespace/share/separator=/ namespace/share/prefix=boop/ auth_mech=PLAIN auth_token=bcdb366d56f4fceee83775a1b28ca89ccdf881fd script-login(b(a)b.com)<56895><>: Debug: Added userdb setting: mail=maildir:/mail/mailstore03/492/177/b@b.com/:INDEX=/mail/index03/492/177/b@b.com/ script-login(b(a)b.com)<56895><>: Debug: Added userdb setting: namespace=inbox,share script-login(b(a)b.com)<56895><>: Debug: Added userdb setting: namespace/inbox/separator=/ script-login(b(a)b.com)<56895><>: Debug: Added userdb setting: namespace/share/prefix=boop/ script-login(b(a)b.com)<56895><>: Debug: Added userdb setting: namespace/share/separator=/ imap(b(a)b.com)<56894><9mFyornjaMcKyAUI>: Debug: Added userdb setting: mail=maildir:/mail/mailstore03/492/177/b@b.com/:INDEX=/mail/index03/492/177/b@b.com/ imap(b(a)b.com)<56894><9mFyornjaMcKyAUI>: Debug: Added userdb setting: namespace=inbox,share imap(b(a)b.com)<56894><9mFyornjaMcKyAUI>: Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:/mail/mailstore03/492/177/b@b.com/:INDEX=/mail/index03/492/177/b@b.com/ imap(b(a)b.com)<56894><9mFyornjaMcKyAUI>: Debug: Namespace : type=private, prefix=, sep=, inbox=no, hidden=no, list=yes, subscriptions=yes location=maildir:/mail/mailstore03/492/177/b@b.com/:INDEX=/mail/index03/492/177/b@b.com/ the script-login task notes that these userdb settings are being added: namespace=inbox,share namespace/inbox/separator=/ namespace/share/prefix=boop/ namespace/share/separator=/ but the imap task only notes one: namespace=inbox,share both of these two instances end up getting an error because the imap task didn't pick up the prefix: Error: namespace configuration error: Duplicate namespace prefix: "" instance #1 behaves differently. it's script-login task debug is the same, but the imap debug notes: imap(bigip-imap(a)nagios.tucows.com)<2146><s3ctO7njd6cKBS39>: Debug: Added userdb setting: mail=maildir:/mail/mailstore03/492/177/b@b.com/:INDEX=/mail/index03/492/177/b@b.com/ imap(bigip-imap(a)nagios.tucows.com)<2146><s3ctO7njd6cKBS39>: Debug: Added userdb setting: namespace=inbox,share imap(bigip-imap(a)nagios.tucows.com)<2146><s3ctO7njd6cKBS39>: Debug: Added userdb setting: namespace/inbox/separator=/ imap(bigip-imap(a)nagios.tucows.com)<2146><s3ctO7njd6cKBS39>: Debug: tc_empty_notify_plugin init imap(bigip-imap(a)nagios.tucows.com)<2146><s3ctO7njd6cKBS39>: Debug: Namespace inbox: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:/mail/mailstore03/492/177/b@b.com/:INDEX=/mail/index03/492/177/b@b.com/ imap(bigip-imap(a)nagios.tucows.com)<2146><s3ctO7njd6cKBS39>: Debug: Namespace : type=private, prefix=, sep=, inbox=no, hidden=no, list=yes, subscriptions=yes location=maildir:/mail/mailstore03/492/177/b@b.com/:INDEX=/mail/index03/492/177/b@b.com/ on this one, the imap task notes that it added ALL the namespace directives, and consequently doesn't have any namespace prefix collisions. the difference between 1:2.3.4.1-5+deb10u5 and 1:2.3.4.1-5+deb10u6 doesn't seem that great, and the bug tracker for it doesn't indicate anything around userdb would be patched (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=970386) but the problem still seems to exist in 2.3.19 - can anyone else see the same thing happening with the current builds? thanks! -brendan

3 4

Dovecot ACLs and XOAUTH2
by Felix Auringer 23 Aug '22

23 Aug '22

Hi, we're trying to set up an IMAP server using Dovecot as part of a new email system in a middle-sized company (around 150 employees). We have a keycloak identity server that is used for all logins and we would also like to use it for the new email system so that the non-technical employees (the majority) will have no issues using it. It's very nice that XOAUTH2 is already included in Dovecot, it works very well with our Roundcube web client. However, it seems that other clients like Thunderbird or Android Apps like FairEmail are not easy to set up as they have their XOAUTH providers hardcoded somewhere (https://bugzilla.mozilla.org/show_bug.cgi?id=1602166) Are you aware of any solution to integrate a dovecot server using XOAUTH2 into local clients (like Thunderbird)? We are currently trying to do it with passwords that can be set in Keycloak only for the purpose of email that can then be used to receive a valid token via a direct access grant but it's not a really nice solution as it introduces an additional password. Additionally, we would like to have all permission-related information saved in our identity server. For email, this includes shared mailboxes that a user is allowed to access. The ACL plugin that is used for shared mailboxes currently reads the permissions from disk which is not really feasible if the user base is large and setting these permissions should happen automatically. We are thinking the best way to do this is to encode the user's permissions in the token that dovecot receives in the login process. But as the token is only available in the authentication process and it does not seem to be intended to return such information from the password database (https://doc.dovecot.org/configuration_manual/authentication/password_databa…) we are unsure on how to process these information from the token. We did not find any plugin hooks in the authentication processes and the normal plugin hooks do not have access to information from the token anymore. What do you think is the best way to do this? We would be okay modifying some dovecot source code and contributing it back if desired. Best regards, Felix Auringer --- Gesellschaft für interkulturelles Zusammenleben gGmbH (GIZ) Felix Auringer IT Reformationsplatz 2 13597 Berlin Tel: 030/513 0100 00; Fax: 030/513 0100 09 www.giz.berlin; felix.auringer(a)giz.berlin Amtsgericht Charlottenburg HRB 200872 B Geschäftsführerin: Dr. Britta Marschke

2 6

rawlog data in a lua script
by michael.zork＠feierfighter.de 08 Aug '22

08 Aug '22

Hi, I'm searching for a possibility to have the rawlog feature in lua, which would be much easier for processing. Currently Dovecot, when activating rawlog for a user, writes everything to disk (which creates I/O), and I have to somehow read it from there. That's a bit complicated, because I have to get notified via inotify or similar when there are new files created, and then I have to start a "tail" or "epoll" mechanism on the files to get the contents in more or less real time (IMAP sessions can be multiple hours or days). It would be much easier to hook to the "raw request and response events" inside Dovecot and have the rawlog-data in a lua script, where I can prepare it and send it to another maschine for monitoring/collection/analysis/statistics or similar, for example via HTTP. Having the rawlog data available in lua would make things a lot easier. Is there any possibility at the moment to create a lua script and "hook" to those "request and response events"? If not, would it be possible to add that feature in the future? Kind regards Michael

5 8

tika 2.4.1 'Text extraction failed' errors when dovecot+fts 2.3.19.1 passes embedded *.eml (message/rfc822) files ; org.apache.tika.parser.mail.RFC822Parser or dovecot ?
by PGNet Dev 01 Aug '22

01 Aug '22

i'm running dovecot 2.3.19.1 + fts tika-server-standard 2.4.1 dovecot is feeding tika backend via fts_tika when dovecot passes data with *.eml attachments embedded, tika fails to correctly parse/extract content not clear if the issue is with tika, or what dovecot's passing in this case. other non-.eml attachments are fine. here's the current failing procedure, (1) create a simple pdf enscript -p mime.ps /etc/mime.types ps2pdf mime.ps mime.pdf (2) send an email *with* mime.pdf attachment to echo "test" | mailx -s "test" -a ./mime.pdf testuser(a)example.com tika processes OK journalctl -f -u tika ... Jul 30 19:09:24 mx-test tika[19682]: INFO [qtp2112135199-30] 19:09:24,165 org.apache.tika.server.core.resource.TikaResource /tika (application/pdf) ... save the just-received email with .pdf attachment as mime.eml (3) send an email with NO .pdf attachment save the just-received email with .pdf attachment as mime2.eml (4) send an email with mime.eml attachment, containing the embedded mime.pdf echo "test" | mailx -s "test" -a ./mime.eml testuser(a)example.com tika fails to extract message/rfc822 journalctl -f -u tika | grep -v StatusLogger ... Jul 30 19:28:00 mx-test tika[20049]: INFO [qtp2112135199-30] 19:28:00,834 org.apache.tika.server.core.resource.TikaResource /tika (message/rfc822) Jul 30 19:28:00 mx-test tika[20049]: WARN [qtp2112135199-30] 19:28:00,840 org.apache.tika.server.core.resource.TikaResource tika/: Text extraction failed (mime.eml) Jul 30 19:28:00 mx-test tika[20049]: org.apache.tika.exception.ZeroByteFileException: InputStream must have > 0 bytes Jul 30 19:28:00 mx-test tika[20049]: at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:153) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.apache.tika.parser.ParserDecorator.parse(ParserDecorator.java:152) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.apache.tika.parser.DigestingParser.parse(DigestingParser.java:55) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.apache.tika.server.core.resource.TikaResource.parse(TikaResource.java:352) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.apache.tika.server.core.resource.TikaResource.lambda$produceText$1(TikaResource.java:502) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.apache.cxf.jaxrs.provider.BinaryDataProvider.writeTo(BinaryDataProvider.java:177) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.apache.cxf.jaxrs.utils.JAXRSUtils.writeMessageBody(JAXRSUtils.java:1616) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.serializeMessage(JAXRSOutInterceptor.java:249) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.processResponse(JAXRSOutInterceptor.java:122) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.handleMessage(JAXRSOutInterceptor.java:84) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.apache.cxf.interceptor.OutgoingChainInterceptor.handleMessage(OutgoingChainInterceptor.java:90) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:265) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(JettyHTTPDestination.java:247) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHandler.java:79) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:190) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:00 mx-test tika[20049]: at java.lang.Thread.run(Thread.java:833) ~[?:?] Jul 30 19:28:00 mx-test tika[20049]: ERROR [qtp2112135199-30] 19:28:00,845 org.apache.cxf.jaxrs.utils.JAXRSUtils Problem with writing the data, class org.apache.tika.server.core.resource.TikaResource$$Lambda$338/0x0000000800eb4a38, ContentType: text/plain (5) send an email with mime2.eml attachment, WITHOUT an embedded .pdf echo "test" | mailx -s "test" -a ./mime.eml testuser(a)example.com again, tika fails to extract message/rfc822 journalctl -f -u tika | grep -v StatusLogger ... Jul 30 19:28:33 mx-test tika[20049]: INFO [qtp2112135199-30] 19:28:33,607 org.apache.tika.server.core.resource.TikaResource /tika (message/rfc822) Jul 30 19:28:33 mx-test tika[20049]: WARN [qtp2112135199-30] 19:28:33,616 org.apache.tika.server.core.resource.TikaResource tika/: Text extraction failed (mime2.eml) Jul 30 19:28:33 mx-test tika[20049]: org.apache.tika.exception.ZeroByteFileException: InputStream must have > 0 bytes Jul 30 19:28:33 mx-test tika[20049]: at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:153) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.apache.tika.parser.ParserDecorator.parse(ParserDecorator.java:152) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.apache.tika.parser.DigestingParser.parse(DigestingParser.java:55) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.apache.tika.server.core.resource.TikaResource.parse(TikaResource.java:352) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.apache.tika.server.core.resource.TikaResource.lambda$produceText$1(TikaResource.java:502) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.apache.cxf.jaxrs.provider.BinaryDataProvider.writeTo(BinaryDataProvider.java:177) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.apache.cxf.jaxrs.utils.JAXRSUtils.writeMessageBody(JAXRSUtils.java:1616) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.serializeMessage(JAXRSOutInterceptor.java:249) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.processResponse(JAXRSOutInterceptor.java:122) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.handleMessage(JAXRSOutInterceptor.java:84) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.apache.cxf.interceptor.OutgoingChainInterceptor.handleMessage(OutgoingChainInterceptor.java:90) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:265) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(JettyHTTPDestination.java:247) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHandler.java:79) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:190) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) ~[tika-server-standard-2.4.1.jar:2.4.1] Jul 30 19:28:33 mx-test tika[20049]: at java.lang.Thread.run(Thread.java:833) ~[?:?] Jul 30 19:28:33 mx-test tika[20049]: ERROR [qtp2112135199-30] 19:28:33,630 org.apache.cxf.jaxrs.utils.JAXRSUtils Problem with writing the data, class org.apache.tika.server.core.resource.TikaResource$$Lambda$338/0x0000000800eb4a38, ContentType: text/plain (6) submit mime.eml directly to tika curl -T ./mime.eml http://127.0.0.1:9998/tika journalctl -f -u tika | grep -v StatusLogger ... Jul 30 19:30:08 mx-test tika[20049]: INFO [qtp2112135199-34] 19:30:08,073 org.apache.tika.server.core.resource.TikaResource /tika (autodetecting type) (7) submit mime2.eml directly to tika curl -T ./mime2.eml http://127.0.0.1:9998/tika journalctl -f -u tika | grep -v StatusLogger ... Jul 30 19:30:52 mx-test tika[20049]: INFO [qtp2112135199-30] 19:30:52,349 org.apache.tika.server.core.resource.TikaResource /tika (autodetecting type) (8) where, cat mime.eml Return-Path: <msmtp(a)pgnd.example.com> Delivered-To: testuser(a)example.com ... From: msmtp(a)pgnd.example.com Date: Sat, 30 Jul 2022 18:53:38 -0400 To: testuser(a)example.com Subject: test User-Agent: Heirloom mailx 12.5 7/5/10 Content-Type: multipart/mixed; boundary="=_62e5b672.wAyBX+sGMbS7ZcNv8O/A1QeYuseaJ2NDRf8hfdbm/x8Vayp+" Message-Id: <4LwKS35QWSzWf3Q(a)mx-test.example.com> This is a multi-part message in MIME format. --=_62e5b672.wAyBX+sGMbS7ZcNv8O/A1QeYuseaJ2NDRf8hfdbm/x8Vayp+ Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline test --=_62e5b672.wAyBX+sGMbS7ZcNv8O/A1QeYuseaJ2NDRf8hfdbm/x8Vayp+ Content-Type: application/pdf Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="mime.pdf" JVBERi0xLjQKJcfsj6IKJSVJbnZvY2F0aW9uOiBwYXRoL2dzIC1QLSAtZFNBRkVSIC1kQ29t ... Rgo= --=_62e5b672.wAyBX+sGMbS7ZcNv8O/A1QeYuseaJ2NDRf8hfdbm/x8Vayp+-- and, cat mime2.eml Return-Path: <msmtp(a)pgnd.example.com> Delivered-To: testuser(a)example.com ... From: msmtp(a)pgnd.example.com Date: Sat, 30 Jul 2022 19:14:59 -0400 To: testuser(a)example.com Subject: test User-Agent: Heirloom mailx 12.5 7/5/10 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-Id: <4LwKwh5brVzWf3Q(a)mx-test.example.com> test

2 2

Re: Dovecot folder problem while deleting via IMAP
by Christian Mack 29 Jul '22

29 Jul '22

Hello Am 29.07.22 um 10:42 schrieb Pascal Bonny: > Dovecot Version: > 2.3.17 (8414c38c5c) > > Dear, > Some troubles occur while we delete a folder via IMAP ssl command or via a IMAP > client (different clients tested). > > When we want to delete subfolder we observe following issues. > > > some examples following: > > we created a folder with subfolders x/y/z > > <IMAP command create> > a02 CREATE x.y.z > a02 OK Create completed (0.003 + 0.000 + 0.002 secs). > <IMAP command create> > > <doveadm mailbox list -u "user(a)domain.tld"> > ====response from dovecot cli==== > x > x.y > x.y.z > > then we try to delete the z subfolder of x > > <IMAP command delete> > a04 DELETE x.y.z > a05 OK Delete completed (0.002 + 0.000 + 0.001 secs). > <IMAP command delete> > > The IMAP console returns success and the dovecot listing is ok > > <doveadm mailbox list -u "user(a)domain.tld"> > ====response from dovecot cli ===== > x > x.y > > But dovecot status on x.y returns the folder is not existent > [cut] That is normal IMAP behaviour. With creating x.y.z you really create only folder z and a folder hirarchy x.y, in which you can not store emails (nor have any access). You have to create x and x.y, in order to get complete folders and subfolders usable for emails and be able to delete them :-) Kind regards, Christian Mack -- Christian Mack Universität Konstanz Kommunikations-, Informations-, Medienzentrum (KIM) Abteilung IT-Dienste Forschung, Lehre, Infrastruktur 78457 Konstanz +49 7531 88-4416

1 0

large search indexer tasks, submitted to flatcurve+tika+tesseract backend for attachment scanning, timeout even with "fts_index_timeout = 0"; how to increase/remove timeouts?
by PGNet Dev 28 Jul '22

28 Jul '22

i'm running dovecot 2.3.19.1 search indexing is enabled, via fts_flatcurve (dovecot23_fts_flatcurve) attachment processing is via fts_tika, with tika-server-standard-2.4.2 backend & tesseract5 OCR i'm working getting this optimized on a smaller, 4-core server that opt includes capping tesseract @ single-thread, Environment=OMP_THREAD_LIMIT=1 still, some attachments are OCR-scanned, and can take some time. when i exec large reindex jobs, i get occassional timeout errors on dovecot's indexer-worker connection to tiks backend, e.g., 2022-07-23 09:54:43 indexer-worker(postmaster(a)example.com)<DIcjEWb922JhXAAA+IOfAw>: Error: fts_tika: PUT http://127.0.0.1:9998/tika/ failed: Request timed out (Request queued 61.031 secs ago, 1 send attempts in 60.103 secs, 60.080 in http ioloop, 0.000 in other ioloops, connected 60.103 secs ago) 2022-07-23 09:54:43 indexer-worker(postmaster(a)example.com)<DIcjEWb922JhXAAA+IOfAw>: Error: Mailbox Sent: Precache for UID=90782 failed: Internal error occurred. Refer to server log for more information. [2022-07-23 09:54:43] (attempted to index 2 messages between UIDs 90778..90782) i don't see any fts timeout info @ https://wiki.dovecot.org/Timeouts here https://doc.dovecot.org/settings/plugin/fts-plugin/#plugin_setting-fts-fts_… " fts_index_timeout Default: 0 Values: Unsigned integer When the full text search backend detects that the index isn’t up-to-date, the indexer is told to index the messages and is given this much time to do so. If this time limit is reached, an error is returned, indicating that the search timed out during waiting for the indexing to complete: NO [INUSE] Timeout while waiting for indexing to finish A value of 0 means no timeout. " double-checking my config, /usr/sbin/dovecot -c /etc/dovecot/dovecot.conf -n submission_relay_connect_timeout = 10 secs submission_timeout = 10 secs there's no tika/fts timeout override, so should be "0" by default. i.e., no timeout @ Patch: enhancements for solr/tika integration https://dovecot.org/list/dovecot/2022-January/123828.html a patch was proposed for similar issue with fts-solr indexer (not sure if it's in current release AND effective for my config), but that *limits* the size of attachments submitted. i _do_ want to index all/large attachments, so, iiuc, need to ensure no, or longer, timeout. where do I set that timeout to not fail, as above, on large index tasks?

2 2

RHEL9 Latest Repo?
by dovecot＠ptld.com 28 Jul '22

28 Jul '22

Any plans or timeline for when there will be a latest repo for RHEL9? [dovecot-2.3-latest] name=Dovecot 2.3 RHEL $releasever - $basearch baseurl=http://repo.dovecot.org/ce-2.3-latest/rhel/$releasever/RPMS/$basear… gpgkey=https://repo.dovecot.org/DOVECOT-REPO-GPG gpgcheck=1 enabled=1

3 2