- dovecot - dovecot.org

Hanging doveadm-server processes with dsync replication
by Sebastian Marske 10 Feb '22

10 Feb '22

Dear Dovecot mailing list, after updating Dovecot from 2.3.16 to 2.3.18 on a dsync-replicated server there are hanging doveadm-server processes piling up over time, eventually resulting in some of the affected users being shown as out of sync. Our setup is based on FreeBSD 13, ZFS and Dovecot (we use custom packages built with poudriere) with master/master replication using dsync. However, we use a shared ip (Carp), so that only one server is actually active. Please see the output of "doveconf -n" at the end for our config. Starting from an in-sync state, I updated Dovecot on the inactive server. Occasionally, Dovecot logs messages like: Feb 8 15:02:15 myhost dovecot[99800]: doveadm(someuser)<2090><ZLayIal3AmIqCAAADKIhQg>: Error: write(<local>) failed: Timed out after 60 seconds These occur for an increasing number of users (maybe 30 after two days), but not for every user (there are >4800 users on that server) and also only once for every affected user. Here's some more information about the process/user from the log entry: # doveadm replicator dsync-status username type status someuser incremental Waiting for dsync to finish (the type is "incremental for most users, but "normal" and "full" show up as well) # doveadm replicator status someuser username priority fast sync full sync success sync failed someuser low 00:01:51 21:56:06 69:56:10 y (again, not all are in failed state) # top -abp 2090 PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 2090 sysdov 1 20 0 26M 15M kqread 12 0:01 0.00% doveadm-server: [<local>] (doveadm-server) (didn't change for >1d) # top -m io -abp 2090 PID USERNAME VCSW IVCSW READ WRITE FAULT TOTAL PERCENT COMMAND 2090 sysdov 363 58 0 271 2 273 0.00% doveadm-server: [<local>] (doveadm-server) (didn't change for >1d) # gdb -p 2090 ... (gdb stuff; gdb complaining about missing debug symbols) (gdb) bt full #0 0x00000000416a94ca in _kevent () from /lib/libc.so.7 No symbol table info available. #1 0x00000000419b94f3 in ?? () from /lib/libthr.so.3 No symbol table info available. #2 0x000000004152a645 in io_loop_handler_run_internal () from /usr/local/lib/dovecot/libdovecot.so.0 No symbol table info available. #3 0x00000000415282fa in io_loop_handler_run () from /usr/local/lib/dovecot/libdovecot.so.0 No symbol table info available. #4 0x0000000041528138 in io_loop_run () from /usr/local/lib/dovecot/libdovecot.so.0 No symbol table info available. #5 0x000000004148ac58 in master_service_run () from /usr/local/lib/dovecot/libdovecot.so.0 No symbol table info available. #6 0x0000000001086431 in main () No symbol table info available. (gdb) So I guess it's waiting for something that * doesn't happen on my system * or it didn't wait for in 2.3.16 >From what I've seen, mails from the active server (still on 2.3.16) are replicated to this server. For non-affected users, mails are also replicated from this server to the active one. I can't tell about "outgoing" replication for affected users, yet. After downgrading back to 2.3.16, things are fine again. Most affected users jump back to being successfully synced within a couple of minutes. If not, starting replication via doveadm get's them there. Testing 2.3.18 again, it seems that the same users are affected again. I also tested 2.3.17 when it came out and had the same issue, paired with the ioloop issue [1], which was fixed in 2.3.18 and which I don't see anymore. The hanging doveadm processes remain, though. Do you have any suggestions on how to resolve this? [1] https://dovecot.org/pipermail/dovecot/2022-January/123907.html Best regards Sebastian # doveconf -n # 2.3.18 (9dd8408c18): /usr/local/etc/dovecot/dovecot.conf # Pigeonhole version 0.5.18 (0bc28b32) # OS: FreeBSD 13.0-RELEASE-p6 amd64 # Hostname: myhost... auth_cache_ttl = 0 auth_username_chars = abcdefghijklmnopqrstuvwxyz01234567890@.- auth_username_format = %n default_client_limit = 126000 default_process_limit = 50000 default_vsz_limit = 512 M doveadm_password = # hidden, use -P to show it first_valid_gid = 20 first_valid_uid = 20 imap_client_workarounds = tb-extra-mailbox-sep imap_logout_format = in=%i out=%o delflag=%{deleted} deleted=%{expunged} trashed=%{trashed} session=<%{session}> login_trusted_networks = # imap proxy ips... mail_gid = sysdov mail_location = maildir:~/maildir:INDEX=/addons/index/%u:CONTROL=~/control:LAYOUT=fs mail_plugins = acl notify replication mail_uid = sysdov managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext editheader namespace fremdeordner { list = yes location = maildir:%%h/maildir:INDEX=/addons/index/%u/FremdeOrdner/%%u:LAYOUT=fs prefix = FremdeOrdner/%%u/ separator = / subscriptions = no type = shared } namespace inbox { inbox = yes list = yes location = mailbox Archive { auto = no special_use = \Archive } mailbox Archives { special_use = \Archive } mailbox AutoCleanSpam { auto = subscribe } mailbox "Deleted Items" { special_use = \Trash } mailbox "Deleted Messages" { special_use = \Trash } mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Entwürfe { special_use = \Drafts } mailbox "Gelöschte Elemente" { special_use = \Trash } mailbox "Gesendete Elemente" { special_use = \Sent } mailbox Junk { special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Items" { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } mailbox name { special_use = \Drafts \Junk \Sent \Trash \Archive } prefix = separator = / subscriptions = yes type = private } passdb { args = /usr/local/etc/dovecot/deny-users deny = yes driver = passwd-file } passdb { args = failure_show_msg=yes dovecot driver = pam } plugin { acl = vfile acl_shared_dict = file:/addons/acl/shared-folder mail_replica = tcp:myreplica...:12345 sieve = /addons/sieve/%u.sieve sieve_dir = sieve_extensions = +imap4flags +editheader sieve_vacation_dont_check_recipient = yes } protocols = imap lmtp replication_dsync_parameters = -d -n 'inbox' -l 30 -U replication_max_conns = 150 service aggregator { fifo_listener replication-notify-fifo { user = sysdov } unix_listener replication-notify { user = sysdov } } service anvil { client_limit = 60003 unix_listener anvil-auth-penalty { mode = 00 } unix_listener anvil { group = nagios mode = 0660 } } service auth { client_limit = 126000 unix_listener auth-userdb { mode = 0644 user = sysdov } } service config { unix_listener config { user = sysdov } } service doveadm { inet_listener { port = 12345 } user = sysdov vsz_limit = 1 G } service imap-login { client_limit = 20000 process_limit = 10000 process_min_avail = 138 service_count = 0 } service imap { process_limit = 80000 process_min_avail = 10 vsz_limit = 512 M } service lmtp { client_limit = 1 executable = lmtp -L process_min_avail = 20 unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0660 user = sysdov } } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0600 user = sysdov } } ssl_cert = </usr/local/etc/letsencrypt/live/myhost/fullchain.pem ssl_key = # hidden, use -P to show it ssl_prefer_server_ciphers = yes userdb { default_fields = mail_replica=tcp:myreplica...:12345 driver = passwd override_fields = uid=29 gid=29 blocking=yes } verbose_proctitle = yes protocol imap { login_trusted_networks = # imap proxy ips... mail_max_userip_connections = 25 mail_plugins = acl notify replication acl imap_acl } protocol lmtp { mail_plugins = acl notify replication sieve postmaster_address = postmaster@... sendmail_path = /usr/local/sbin/sendmail }

1 0

Different handling of upper and lower case while indexing/searching with Solr
by Patrik Peng 10 Feb '22

10 Feb '22

Hello there We stumbled upon an user account with Solr FTS, which returned no search results for any given search query. Further investigation revealed an issue between indexing mails and querying the index. The user name contains upper and lower case characters (eg. Some.User(a)domain.net) When new mail is indexed for this user, the user name used for Solr's `user` and `id` fields are transformed into lowercase, as shown in the Solr log: webapp=/solr path=/update params={...}{add=[8543/426f3b0348d03451a3fb00008ba2b673/some.user(a)domain.net (1724281617442144256), ... (162 adds)]} 0 44298 And can be confirmed by manually querying Solr. The Solr schema in use performs no transformation for the affected fields. When a search request is performed via IMAP, Dovecot queries Solr with the original user name: GET /solr/dovecot_fts_popimap/select?wt=json&f...&fq=%2Bbox:1a30ec359dce3451b8e600008ba2b673+%2Buser:Some.User@domain.net HTTP/1.1" Which (correctly) returns zero results. To summarize, I suspect dovecot transforms any user name to lower case while indexing mails, but not when querying for results. Is this a bug, or caused by misconfiguration? Regards Patrik

3 9

Outlook 2019 and push notifications
by Laura Smith 10 Feb '22

10 Feb '22

Is there some secret config sauce I'm unaware of to get IMAP push notifications working with Outlook ? My present situation is that non-Outlook users (e.g. Apple iOS, Apple Mail etc.) work perfectly with the config below. New emails get pushed, deletes get reflected, all is happy in the world. But with Outlook nothing. You send a user an email, nothing happens in their inbox until you manually refresh by checking for new mails or waiting for the auto-check to kick in (sometimes switching between IMAP folders also does the trick). I know Outlook has a long history of being a mess when it comes to standards compliance, but surely it can't be this bad ? doveconf -n : # 2.3.18 (9dd8408c18): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.18 (0bc28b32) # OS: Linux 5.10.0-11-amd64 x86_64 Debian 11.2 xfs # Hostname: foobar.example.com auth_verbose = yes auth_verbose_passwords = sha1:7 doveadm_password = # hidden, use -P to show it imap_capability = +SPECIAL-USE imapc_features = rfc822.size fetch-headers fetch-bodystructure imapc_port = 993 imapc_ssl = imaps mail_location = maildir:/foo_data/mail/%d/%n/Maildir mail_plugins = notify replication push_notification managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext imapsieve vnd.dovecot.imapsieve namespace inbox { inbox = yes location = mailbox "Deleted Messages" { auto = no special_use = \Trash } mailbox Drafts { special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = /etc/dovecot/local_sql_users.conf driver = sql } plugin { imapsieve_mailbox1_before = file:/etc/dovecot/imap_sieve/spam.sieve imapsieve_mailbox1_causes = COPY imapsieve_mailbox1_name = Junk imapsieve_mailbox2_before = file:/etc/dovecot/imap_sieve/ham.sieve imapsieve_mailbox2_causes = COPY imapsieve_mailbox2_from = Junk imapsieve_mailbox2_name = * sieve = file:~/sieve;active=~/.dovecot.sieve sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment sieve_pipe_bin_dir = /etc/dovecot/imap_sieve sieve_plugins = sieve_imapsieve sieve_extprograms } protocols = imap lmtp sieve service auth { unix_listener /var/spool/postfix/private/dovecot-auth { group = postfix mode = 0660 user = postfix } vsz_limit = 2 G } service doveadm { inet_listener { port = 11867 ssl = yes } } service imap-login { process_min_avail = 5 service_count = 1 } service lmtp { process_min_avail = 5 unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0660 user = postfix } user = foo_virtmail } service managesieve-login { inet_listener sieve { port = 4190 } inet_listener sieves { address = port = 5190 ssl = yes } } ssl = required ssl_ca = </etc/foobar/ssl/certificates/_.example.com.issuer.crt ssl_cert = </etc/foobar/ssl/certificates/_.example.com.crt ssl_cipher_list = ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384 ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it ssl_prefer_server_ciphers = yes userdb { args = /etc/dovecot/local_sql_users.conf default_fields = uid=foo_virtmail gid=foo_virtmail home=/foo_data/mail/%d/%n driver = sql } protocol lmtp { mail_plugins = notify replication push_notification sieve postmaster_address = foo(a)foobar.com } protocol lda { deliver_log_format = msgid=%m: %$ mail_plugins = notify replication push_notification sieve postmaster_address = postmaster quota_full_tempfail = yes rejection_reason = Your message to <%t> was automatically rejected:%n%r } protocol imap { mail_max_userip_connections = 20 mail_plugins = notify replication push_notification imap_sieve }

2 1

Certificate and showing a sign-cert not there
by Wayne Spivak 10 Feb '22

10 Feb '22

Hi - I am running Postfix 3.6.4 with Dovecot 2.3.17.1 (476cd46418). I have a multi-signed cert from Entrust. The cert works fine on port 25. However, on Port 587 I get an error: c [root@mcq wbs]# openssl s_client -connect mcq.sbanetweb.com:993 -servername mcq.sbanetweb.com CONNECTED(00000003) depth=0 C = US, ST = New York, L = Bellmore, O = SBA Consulting LTD, CN = mcq.sbanetweb.com verify error:num=20:unable to get local issuer certificate verify return:1 depth=0 C = US, ST = New York, L = Bellmore, O = SBA Consulting LTD, CN = mcq.sbanetweb.com verify error:num=21:unable to verify the first certificate verify return:1 depth=0 C = US, ST = New York, L = Bellmore, O = SBA Consulting LTD, CN = mcq.sbanetweb.com verify return:1 --- Certificate chain 0 s:C = US, ST = New York, L = Bellmore, O = SBA Consulting LTD, CN = mcq.sbanetweb.com i:C = US, O = "Entrust, Inc.", OU = See www.entrust.net/legal-terms <http://www.entrust.net/legal-terms> , OU = "(c) 2012 Entrust, Inc. - for authorized use only", CN = Entrust Certification Authority - L1K [root@mcq wbs]# dovecot -n # 2.3.17.1 (476cd46418): /etc/dovecot/dovecot.conf # OS: Linux 5.16.5-200.fc35.x86_64 x86_64 Fedora release 35 (Thirty Five) # Hostname: mcq.sbanetweb.com auth_mechanisms = plain login disable_plaintext_auth = no mbox_write_locks = fcntl namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = pam } protocols = imap service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } unix_listener auth-userdb { group = postfix mode = 0666 user = postfix } } service imap-login { inet_listener imap { port = 143 } inet_listener imaps { port = 993 ssl = yes } } service submission-login { inet_listener submission { port = 587 } } ssl = required ssl_cert = </etc/postfix/tls/ServerCertificate.pem ssl_cipher_list = ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-G CM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AE S128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA25 6:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE- ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES1 28-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE -DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES12 8-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNUL L:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-D ES-CBC3-SHA:!KRB5-DES-CBC3-SHA ssl_client_ca_dir = /etc/postfix/tls/ ssl_client_ca_file = ChainBundle.pem ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it ssl_prefer_server_ciphers = yes userdb { driver = passwd } protocol imap { mail_max_userip_connections = 15 } Any ideas? Wayne Spivak SBANETWEB.com

5 9

Config for filtering (ignoring) specific mbox mail folders?
by John Hardin 08 Feb '22

08 Feb '22

Folks: I'm migrating to Dovecot to provide IMAP services for my personal network. I have more than a couple of decades of mail in multiple (1000+) mbox mail files, but I don't care to expose all of those to IMAP. Is there any way to filter out what mail folders Dovecot will LIST and provide access to using globs or regular expressions? I'd prefer to avoid rearranging the mbox files (e.g. to subdirectories) if possible. I looked through the Dovecot documentation and didn't see anything promising, and I couldn't find any discussion of the topic online. The feature that looked like it might be closest was Namespaces, but I couldn't see how to use that to ignore specific mbox files. Is there any facility within Dovecot (or a plugin) to filter out / ignore mail folders by name, RE or file glob? Thanks in advance! -- John Hardin KA7OHZ http://www.impsec.org/~jhardin/ jhardin(a)impsec.org pgpk -a jhardin(a)impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79

1 0

Re: On mailbox full, retry for 4 days or similar instead of reject
by Jorge Bastos 08 Feb '22

08 Feb '22

Hi Lucas, Oh, it's so damn simple! Thank you!! Thank you all others aswell, i've read all info sent, thanks!! Jorge On 2022-02-07 22:57, Lucas Rolff wrote: > An option is to use > https://doc.dovecot.org/settings/core/#core_setting-quota_full_tempfail > - you can configure it e.g. > > protocol lda { > quota_full_tempfail = yes > } > >> On 7 Feb 2022, at 23:41, Jorge Bastos <mysql.jorge(a)decimal.pt> wrote: >> >> Howdy, >> >> I don't know if this is dovecot specific and i guess it may not be at >> 100% so I ask for help. >> >> I want postfix not to discard the message imediatly when a mailbox is >> full, i mean when postfix tries to deliver it to dovecot lmtp. >> Is it possible to change the behavior to something like what postfix >> does when he tries to deliver a message to an external server and the >> server is unaccessible for 4 days (the default i guess), and if in >> that period discard it. >> >> Does this exists? At least i know gmail does something similar to >> this. >> >> I've tried to google a bit but didn't found info that could lead me to >> this configuration. >> >> Thanks in advanced, >> Jorge

1 1

sieve: destination folder lookup in database?
by Hadmut Danisch 08 Feb '22

08 Feb '22

Hi, is there a way to lookup the action for an incoming mail, especially which folder to move the mail in, in a database? Given Problem: I want to replace the mda mailfilter/maildrop with dovecot/sieve. The mailfilter script is rather short, but looksup the destination folder by querying a berkeley database file with the sender's address to find the target folder, using the sender's address as an index. This way, it can store the target folder for hunders or thousands of senders, without needing to compare every single one of them by if == elsif == else... Is there a similar method for dovecot / sieve? I found that dovecot's sieve has some database connections, e.g. LDAP or https://doc.dovecot.org/configuration_manual/sieve/dict/ but rather to receive the whole sieve script for a given recipient as the owner of the mailbox. Is there a way to determine the target folder where to drop a mail into quickly by the sender's address, even for thousands, without the need to compare every single one of them? regards Hadmut

2 2

sieve script to alter subject of incoming message after lookup
by Marc 08 Feb '22

08 Feb '22

Is it possible with sieve to : 1. get sender of an email 2. lookup if this sender is present in an mailbox 3. if the sender is known, change the subject of the email. if the sender is not known, do nothing. 4. put/leave message in inbox. or maybe use dictionary (not really familiar with this) 1. get sender of an email 2. lookup if this sender is present in an dictionary 3. if the sender is known, change the subject of the email. if the sender is not known, do nothing. 4. put/leave message in inbox. But then I need something like 1. if message is dragged into a specific mailbox/folder 2. add sender to dictionary 1. if message is removed from a specific mailbox/folder 2. remove sender from dictionary

1 0

Different auth logging with cache lookup. (OPEN)
by Eikås Arvid 07 Feb '22

07 Feb '22

dovecot-2.3.5 and centos 7 User and IP are logged in both scenarios. auth-worker(26145): sql(user,1.1.1.1,<XXXXXXXX>): Password mismatch auth-worker(13380): cache(user(a)online.no,1.1.1.1): Password mismatch dovecot-2.3.17 and rhel 8 Feb 03 14:42:53 auth-worker(1676266): Info: conn unix:auth-worker (pid=1675846,uid=97): auth-worker<1>: sql(user,127.0.0.1,<XXXXXXXX>): Password mismatch (given password: xxxx) Feb 03 14:43:48 auth-worker(1676266): Info: conn unix:auth-worker (pid=1675846,uid=97): auth-worker<2>: Password mismatch (given password: xxxx) User and IP is not logged in cache lookup. Is this a design change or a unplanned change? Any ide when it was changed?

1 0

Lucene support for FTS - EOL date.
by Jacek Grabowski 07 Feb '22

07 Feb '22

Hello Actually on the https://doc.dovecot.org/configuration_manual/fts/lucene/ site we can read about lucene fts plugin: "This plugin is no longer maintained in Dovecot (as of 2.3+) and will be removed in the future." Does anyone know about approximate EOL date for lucene plugin support in Dovecot? -- Pozdrawiam. Jacek Grabowski.

6 8