December 2019 - dovecot

[2.3.8] possible replication issue
by Carsten Rosenberg 06 Dec '19

06 Dec '19

Hi, some of our customers have discovered a replication issue after upgraded from 2.3.7.2 to 2.3.8. Running 2.3.8 several replication connections are hanging until defined timeout. So after some seconds there are $replication_max_conns hanging connections. Other replications are running fast and successful. Also running a doveadm sync tcp:... is working fine for all users. I can't see exactly, but I haven't seen mailboxes timeouting again and again. So I would assume it's not related to the mailbox. >From the logs: server1: Oct 16 08:29:25 server1 dovecot[5715]: dsync-local(username1(a)domain.com)<FXnVDW22pl0tGAAA1cwDxA>: Error: dsync(172.16.0.1): I/O has stalled, no activity for 600 seconds (version not received) Oct 16 08:29:25 server1 dovecot[5715]: dsync-local(username1(a)domain.com)<FXnVDW22pl0tGAAA1cwDxA>: Error: Timeout during state=master_recv_handshake server2: Oct 16 08:29:25 server2 dovecot[8113]: doveadm: Error: read(server1) failed: EOF (last sent=handshake, last recv=handshake) There aren't any additional logs regarding the replication. I have tried increasing vsz_limit or reducing replication_max_conns. Nothing changed. -- Both customers have 10k+ users. Currently I couldn't reproduce this on smaller test systems. Both installation were downgraded to 2.3.7.2 to fix the issue for now -- I've attached a tcpdump showing the client showing the client stops sending any data after the mailbox_guid table headers. Any idea what could be wrong here or the debug this issue? Thanks. Carsten Rosenberg

5 5

Dovecot & OAuth
by mizuki 06 Dec '19

06 Dec '19

Hi all, We'd like to enable OAuth with Keycloak in Dovecot, after enabling 'OAUTHBEARER XOAUTH2' in Dovecot based on online document, I can confirm Dovecot is ready for OAuth using openssl command, however when the auth request comes in, it failed in establishing a SSL connection with Keycloak server on port 443, shown as following in debug logs. I can confirming using commands 'openssl s_client -connect <keycloak_server>:443' or 'curl -v https://<keycloak_server/' all returns normal and no errors. Altering some of the SSL options in dovecot such as 'ssl_ca = </etc/pki/CA/certs/root_ca.pem' or 'ssl_client_ca_file = </etc/pki/CA/certs/root_ca.pem' does not help either. The certificate are NOT self-signed but signed the legit authorities. So I'm not sure why dovecot could not establish the connections. *Debug logs:* ---------------------------------------------------- Dec 5 14:32:07 mktst4 dovecot: auth: Debug: auth client connected (pid=16554) Dec 5 14:32:22 mktst4 dovecot: auth: Debug: client in: AUTH#0111#011OAUTHBEARER#011service=imap#011secured#011session=QPwA/fmY+tqCx5Tr#011lip=130.199.148.187#011rip=10.0.2.1#011lport=993#011rport=56058#011resp=bixhPW1penVraQFob3N0PW1rdHN0NC5zZGNjLmJubC5nb3YBcG9ydD05OTMBYXV0aD1CZWFyZXIgZXlKaGJHY2lPaUpTVXpJMU5pSXNJblI1Y0NJZ09pQWlTbGRVSWl3aWEybGtJaUE2SUNKcmRHeE5TSFZtYkc1NFZUUmlSR1pLTm5kSGRteFBNVVYxWWxWd2FreEZVVmhIY25GTU5UYzNhSEJSSW4wLmV5SnFkR2tpT2lJNE5XSmxZV05tTkMxak1tTmxMVFJtTkRJdFltUTRNUzAxTnpWaU4yRmpZV05sT1RnaUxDSmxlSEFpT2pFMU56VTJNVEF5TURnc0ltNWlaaUk2TUN3aWFXRjBJam94TlRjMU5UYzBNakE0TENKcGMzTWlPaUpvZEhSd2N6b3ZMMnRsZVdOc2IyRnJNaTV6WkdOakxtSnViQzVuYjNZdllYVjBhQzl5WldGc2JYTXZiV0ZwYkNJc0ltRjFaQ0k2SW1GalkyOTFiblFpTENKemRXSWlPaUkyTmpWa05EVTNNeTA0WVdFMUxUUmtPRFl0WW1NelppMHlaVEpqTWpOaU16WTJORFVpTENKMGVYQWlPaUpDWldGeVpYSWlMQ0poZW5BaU9pSmtiM1psWTI5MElpd2lZWFYwYUY5MGFXMWxJam93TENKelpYTnphVzl1WDNOMFlYUmxJam9pT1RNNE5XRTVOakF0TUROa1pDMDBPR05sTFdFeU4yUXRZVFkzWkdZME5tUXhaRGcxSWl3aVlXTnlJam9pTVNJc0luSmxZV3h0WDJGalkyVnpjeUk2ZXlKeWIyeGxjeUk2V3lKdlptWnNhVzVsWDJGalkyVnpjeUlzSW5WdFlWOWhkWFJvYjNKcGVtRjBhVzl1SWwxOUxDSnlaWE52ZFhKalpWOWhZMk5sYzNNaU9uc2lZV05qYjNWdWRDSTZleUp5YjJ4bGN5STZXeUp0WVc1aFoyVXRZV05qYjNWdWRDSXNJbTFoYm1GblpTMWhZMk52ZFc1MExXeHBibXR6SWl3aWRtbGxkeTF3Y205bWFXeGxJbDE5ZlN3aWMyTnZjR1VpT2lKd2NtOW1hV3hsSUdWdFlXbHNJaXdpWlcxaGFXeGZkbVZ5YVdacFpXUWlPbVpoYkhObExDSnVZVzFsSWpvaVRXbDZkV3RwSUV0aGNtRnpZWGRoSWl3aWNISmxabVZ5Y21Wa1gzVnpaWEp1WVcxbElqb2liV2w2ZFd0cElpd2laMmwyWlc1ZmJtRnRaU0k2SWsxcGVuVnJhU0lzSW1aaGJXbHNlVjl1WVcxbElqb2lTMkZ5WVhOaGQyRWlMQ0psYldGcGJDSTZJbTFwZW5WcmFVQmlibXd1WjI5MkluMC5NZTRwNkl0dmx2T0VYRTIxM2pwSnJpa3FhZGNZb0ZsMnlMVlJzQTJvQTdmWDgteEtJYTR2ckZmamwwLXRwb2JodzIyRnBvVDZ2TWoxbXphLXBWWUxzNW1vM0w5Y0xKT2hVVnV2Tm9YSm5nZlRzLXk2TWxuTXVGV1NQemtidEhhOXJTUEVqZGFZQXBDQk52MG9CWEs2bmhIM0U5ZkNvTl9TQlUycWxaSWk2M1dWOUZXSjFrbHVGT2Iwc0xja2JfWGNGZzhUZ0dXOEdVUlRYTlg4bW1VM1dNLWJ5dnJuTkFyNmFjWW1ibXRaVzRhV0NwTk5FSHRZU29LSmgydHZuQjhHNnl5eWptWWJ5Q2ZhTmwwSnVZYU52VFEzTXhCX2FnLV9Pcy04VkwtTGFLdHBGYXBMNEVNWUJaXzFnZmNhSFdUSUV1VS0wSVdjTjlEa2xxcW1MN19jNlEBAQ== (previous base64 data may contain sensitive data) Dec 5 14:32:22 mktst4 dovecot: auth: Debug: http-client: host example.com: Host created Dec 5 14:32:22 mktst4 dovecot: auth: Debug: http-client: host example.com: Performing asynchronous DNS lookup Dec 5 14:32:22 mktst4 dovecot: auth: Debug: http-client: request [Req1: GET https://example.com/auth/realms/mail/protocol/openid-connect/tokeneyJhbGciO…] Submitted Dec 5 14:32:22 mktst4 dovecot: auth: Debug: http-client: host example.com: DNS lookup successful; got 1 IPs Dec 5 14:32:22 mktst4 dovecot: auth: Debug: http-client: peer 10.0.2.2:443: Peer created Dec 5 14:32:22 mktst4 dovecot: auth: Debug: http-client: queue https://example.com"443: Setting up connection to 10.0.2.2:443 (SSL= example.com) (1 requests pending) Dec 5 14:32:22 mktst4 dovecot: auth: Debug: http-client: peer 10.0.2.2:443: Linked queue https://example.com:443 (1 queues linked) Dec 5 14:32:22 mktst4 dovecot: auth: Debug: http-client: queue https://example.com:443: Started new connection to 10.0.2.2:443 (SSL= example.com) Dec 5 14:32:22 mktst4 dovecot: auth: Debug: http-client: peer 10.0.2.2:443: Creating 1 new connections to handle requests (already 0 usable, connecting to 0, closing 0) Dec 5 14:32:22 mktst4 dovecot: auth: Debug: http-client: peer 10.0.2.2:443: Making new connection 1 of 1 Dec 5 14:32:22 mktst4 dovecot: auth: Debug: http-client: conn 10.0.2.2:443 [0]: HTTPS connection created (1 parallel connections exist) Dec 5 14:32:22 mktst4 dovecot: auth: Debug: http-client: conn 10.0.2.2:443 [0]: Connected Dec 5 14:32:22 mktst4 dovecot: auth: Debug: http-client: conn 10.0.2.2:443 [0]: Starting SSL handshake Dec 5 14:32:22 mktst4 dovecot: auth: Received invalid SSL certificate: unable to get issuer certificate: /C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority Dec 5 14:32:22 mktst4 dovecot: auth: Debug: http-client: peer 10.0.2.2:443: Failed to make connection (connections=1, connecting=1) Dec 5 14:32:22 mktst4 dovecot: auth: Debug: http-client: queue https://example.com:443: Failed to set up connection to 10.0.2.2:443 (SSL= example.com) SSL handshaking with 10.0.2.2:443 failed: read(SSL 10.0.2.2:443) failed: Received invalid SSL certificate: unable to get issuer certificate: /C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority (1 peers pending, 1 requests pending) Dec 5 14:32:22 mktst4 dovecot: auth: Debug: http-client: queue https://example.com:443: Failed to set up any connection; failing all queued requests Dec 5 14:32:22 mktst4 dovecot: auth: Debug: http-client: peer 10.0.2.2:443: Unlinked queue https://example.com:443 (0 queues linked) Dec 5 14:32:22 mktst4 dovecot: auth: Debug: http-client: queue https://example.com:443: Dropping request [Req1: GET https://example.com/auth/realms/mail/protocol/openid-connect/tokeneyJhbGciO… ] Dec 5 14:32:22 mktst4 dovecot: auth: Debug: http-client: host example.com: Host is idle (timeout = 1799992 msecs) Dec 5 14:32:22 mktst4 dovecot: auth: Error: oauth2(mizuki,10.0.2.1,<QPwA/fmY+tqCx5Tr>): oauth2 failed: SSL handshaking with 10.0.2.2:443 failed: read(SSL 10.0.2.2:443) failed: Received invalid SSL certificate: unable to get issuer certificate: /C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority Dec 5 14:32:22 mktst4 dovecot: auth: Debug: http-client: request [Req1: GET https://example.com/auth/realms/mail/protocol/openid-connect/tokeneyJhbGciO…] Destroy (requests left=1) Dec 5 14:32:22 mktst4 dovecot: auth: Debug: http-client: request [Req1: GET https://example.com/auth/realms/mail/protocol/openid-connect/tokeneyJhbGciO…] Free (requests left=0) Dec 5 14:32:22 mktst4 dovecot: auth: Debug: http-client: conn 10.0.2.2:443 [0]: SSL handshaking with 10.0.2.2:443 failed: read(SSL 10.0.2.2:443) failed: Received invalid SSL certificate: unable to get issuer certificate: /C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority Dec 5 14:32:22 mktst4 dovecot: auth: Debug: http-client: conn 10.0.2.2:443 [0]: Connection close Dec 5 14:32:22 mktst4 dovecot: auth: Debug: http-client: conn 10.0.2.2:443 [0]: Connection disconnect Dec 5 14:32:22 mktst4 dovecot: auth: Debug: http-client: conn 10.0.2.2:443 [0]: Connection destroy ---------------------------------------------------- *#dovecot -n * ---------------------------------------------------- # 2.2.36 (1f10bfa63): /etc/dovecot/dovecot.conf # OS: Linux 3.10.0-1062.4.3.el7.x86_64 x86_64 Red Hat Enterprise Linux Server release 7.7 (Maipo) # Hostname: mktst4.sdcc.bnl.gov auth_debug = yes auth_debug_passwords = yes auth_mechanisms = oauthbearer xoauth2 auth_verbose = yes auth_verbose_passwords = yes first_valid_uid = 1000 mail_debug = yes mail_location = maildir:~/Maildir mbox_write_locks = fcntl namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = /etc/dovecot/dovecot-oauth2.conf.ext driver = oauth2 mechanisms = oauthbearer xoauth2 } protocols = imap service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } } ssl = required ssl_ca = </etc/pki/CA/certs/2.pem ssl_cert = </etc/pki/dovecot/certs/dovecot.pem ssl_cipher_list = ECDH+AESGCM:ECDH+CHACHA20:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:!aNULL:!MD5:!DSS:!RSA ssl_client_ca_file = </etc/pki/CA/certs/2.pem ssl_key = # hidden, use -P to show it ssl_prefer_server_ciphers = yes ssl_require_crl = no userdb { args = uid=vmail gid=vmail home=/var/vmail/%u driver = static } ---------------------------------------------------- *# cat /etc/dovecot/conf.d/auth-oauth2.conf.ext* ---------------------------------------------------- passdb { driver = oauth2 mechanisms = oauthbearer xoauth2 args = /etc/dovecot/dovecot-oauth2.conf.ext } userdb { driver = static args = uid=vmail gid=vmail home=/var/vmail/%u } ---------------------------------------------------- I wonder if anyone has experienced this possibly know what's going on. Thanks! Mizuki

2 3

Supporting RFC 5466 (IMAP4 Extension for Named Searches (Filters))
by Jesus Cea 06 Dec '19

06 Dec '19

I wonder if Dovecot supports RFC 5466 (IMAP4 Extension for Named Searches (Filters)) or if there is any plan about it. -- Jesús Cea Avión _/_/ _/_/_/ _/_/_/ jcea(a)jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz

4 6

How to rewrite local_lmpt ipc to tcp port 24
by Marc Roos 05 Dec '19

05 Dec '19

How to change this line in sendmail.mc, to the tcp port 24 on eg 127.0.0.1? FEATURE(`local_lmtp',`[IPC]',`FILE /var/run/dovecot/lmtp')dnl

2 1

dovecot expunge
by soumitri＠iitk.ac.in 05 Dec '19

05 Dec '19

Hi, Do we have expunge command by UID. I have some duplicate messages (same Message-ID but different UID) to be deleted. doveadm search gives me folder GUID and file UID, I want to use this specific information to delete/expunge. E.g. Following command deletes all duplicates, but I want to keep just one. doveadm expunge -u user1 mailbox INBOX HEADER Message-ID '5de82b2d.6VWfTzsd/fEAxfV7%cytx(a)iitk.ac.in' The following does not help doveadm deduplicate -u user1 mailbox INBOX -- Sincerely, Soumitri Mishra

2 1

How do I set all my mailboxes to "subscribed"
by Steve Litt 05 Dec '19

05 Dec '19

Hi all, I've had a great deal of trouble accessing my Dovecot IMAP from most supposedly IMAP aware email clients, and have been advised that it might be because some of my email folders are not subscribed. Is there s way I can subscribe all my folders? I see no reason to have a folder not subscribed. Thanks, SteveT Steve Litt December 2019 featured book: Rapid Learning for the 21st Century http://www.troubleshooters.com/rl21

2 1

Email blocking emails
by Peter Nabbefeld 05 Dec '19

05 Dec '19

Hello, I had troubles getting emails, because this one caused a Sig 6: https://p.ip.fi/DGSP It might be important that I'm using SOLR. After I removed this mail, I could fetch emails again. Kind regards Peter

2 1

Dovecot 2.3.9 fails on FreeBSD
by Pascal Christen 04 Dec '19

04 Dec '19

Hi I've just tried to build the latest Dovecot 2.3.9 on FreeBSD 11.3. Without success...It fails on the following commit which was introduced in 2.3.9: https://github.com/dovecot/core/commit/c85f1bc3ce612c736c9d2c468cc08306db1b… Following output is the build log: https://pastebin.com/3nvSeDn8 So I guess it has to do with some changes FreeBSD made: https://reviews.freebsd.org/D18630 What do you guys think? Greetings Pascal

4 10

dovecot sieve duplicates detection
by André Rodier 04 Dec '19

04 Dec '19

Hello, I have tested the sieve duplicate script with success so far, but I have a question. I would like to know if the "duplicate" sieve flag in Dovecot is global to all folders, or specific to one folder only. For instance, if I copy an email from one folder to another, and I have a discard action on duplicate email, is this action will be applied (in this case, discard) or not. If the duplicate is global to all folders, is there a way to restrict the search in one folder only. Thanks for your help. André

3 10

About "received" header when using Dovecot proxy
by ml＠riku22.net 04 Dec '19

04 Dec '19

Hello. My name is Riku. Currently, I use Dovecot as a proxy for another SMTP server. However, this seems to cause the IP address of the "received" header to be that of the proxy server. Is it possible to change this so that the IP address of the sender is entered? The version of Dovecot is "2.3.8 (9df20d2db)". Sorry for the incomprehensible explanation. Thank you. Best regards.

4 6