October 2019 - dovecot

Dovecot v2.3.8 released
by Aki Tuomi 20 Oct '19

20 Oct '19

https://dovecot.org/releases/2.3/dovecot-2.3.8.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.8.tar.gz.sig Binary packages in https://repo.dovecot.org/ Changes + Added mail_delivery_started and mail_delivery_finished events, see https://doc.dovecot.org/admin_manual/list_of_events/ for details. + dsync-replication: Don't replicate users who have "noreplicate" extra field in userdb. + doveadm service status: Show total number of processes created. + When logging to syslog, use instance_name setting's value for the ident. This commonly is added as a log prefix. + Base64 encoding/decoding code was rewritten with additional features. It shouldn't cause any user visible changes. - v2.3.7 regression: If a folder only receives new mails without any other mail access, dovecot.index.log keeps growing forever and dovecot.index keeps being rewritten for every mail delivery. - dsync-replication may lose keywords after syncing mails restored from another replica. This only happened if the mail only had keywords and no system flags. - event filters: Non-textual event fields could not be filtered using wildcards. - auth: Scope parameter was missing from OAuth password grant request. - doveadm client-server communication may hang in some situations. It is also using unnecessarily small TCP/IP packet sizes. - doveadm who and kick did not flush protocol output correctly. - imap: SETMETADATA with literal value would delete the metadata value instead of updating it. - imap: When client issues FETCH PREVIEW (LAZY=FUZZY) command, the caching decisions should be updated so that newly saved mails will have the preview cached. - With mail_nfs_index=yes and/or mail_nfs_storage=yes setuid/setgid permission bits in some files may have become dropped with some NFS servers. Changed NFS flushing to now use chmod() instead of chown(). - quota: warnings did not work if quota root was noenforcing - acl: Global ACL file ignored the last line if it didn't end with LF. - doveadm stats dump: With JSON formatter output numbers using the number type instead of as strings - lmtp_proxy: Ensure that real_* variables are correctly set when using lmtp_proxy. - event exporter: http-post driver had hardcoded timeout and did not support DNS lookups or TLS connections. - auth: Fix user iteration to work with userdb passwd with glibc v2.28. - auth: auth service can crash if auth-policy JSON response is invalid or returned too fast. - In some rare situations "ps" output could have shown a lot of "?" characters after Dovecot process titles. - When dovecot.index.pvt is empty, an unnecessary error is logged: Error: .../dovecot.index.pvt reset, view is now inconsistent - SMTP address encoder duplicated initial double quote character when the localpart of an address ended in '..'. For example "user+..(a)example.com" became ""user+.."@example.com in a sieve redirect. --- Aki Tuomi Open-Xchange Oy

5 9

BUG: Mailbox renaming algorithm got into a potentially infinite loop, aborting
by Alex Ha 20 Oct '19

20 Oct '19

Hi all! I have two dovecot servers with dsync replication over tcp. Replication works fine except for one user. # doveadm replicator status username priority fast sync full sync success sync failed customer(a)example.com none 00:00:33 07:03:23 03:22:31 y If i run dsync manually, i get the following error message: dsync-local(customer(a)example.com): Debug: brain M: -- Mailbox renamed, restart sync -- dsync-local(customer(a)example.com): Error: BUG: Mailbox renaming algorithm got into a potentially infinite loop, aborting dsync-local(customer(a)example.com): Error: Mailbox INBOX.Foldername sync: mailbox_rename failed: Invalid mailbox name 'Foldername-temp-1': Missing namespace prefix 'INBOX.' For more info see the attached sync_loop.log Thanks for your help, Alex

3 5

Coredump v2.3.8 specific msg fetch, corrupted record in index cache, Broken physical size
by Erik de Waard 20 Oct '19

20 Oct '19

Hi, i'm getting a coredump on a specific msg, i've attached the gdb. file on disk i noticed W=<vsize> is missing. 1571209735.M744550P1608.rwvirtual65,S=15886:2,S Best regards, mail.log Oct 18 14:41:39 rwvirtual10 dovecot: imap(johndoe(a)company.nl)<15868><qjTFpy6VPsMKAAok>: Error: Mailbox INBOX.Debug: UID=1041: read(/data/mail/ company.nl/users/johndoe/Maildir/.Debug/cur/1571209735.M744550P1608.rwvirtual65,S=15886:2,S) failed: Cached message size smaller than expected (15886 < 16367, box=INBOX.Debug, UID=1041) (read reason=mail stream) Oct 18 14:41:39 rwvirtual10 dovecot: imap(johndoe(a)company.nl)<15868><qjTFpy6VPsMKAAok>: Error: Corrupted record in index cache file /data/indexes/ johndoe(a)company.nl/.Debug/dovecot.index.cache: UID 1041: Broken physical size in mailbox INBOX.Debug: read(/data/mail/ company.nl/users/johndoe/Maildir/.Debug/cur/1571209735.M744550P1608.rwvirtual65,S=15886:2,S) failed: Cached message size smaller than expected (15886 < 16367, box=INBOX.Debug, UID=1041) Oct 18 14:41:39 rwvirtual10 dovecot: imap(johndoe(a)company.nl)<15868><qjTFpy6VPsMKAAok>: Panic: file istream.c: line 315 (i_stream_read_memarea): assertion failed: (old_size <= _stream->pos - _stream->skip) Oct 18 14:41:39 rwvirtual10 dovecot: imap(johndoe(a)company.nl)<15868><qjTFpy6VPsMKAAok>: Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0xe5c31) [0x7f5237d8cc31] -> /usr/lib/dovecot/libdovecot.so.0(+0xe5cd1) [0x7f5237d8ccd1] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f5237ce4103] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read_memarea+0x300) [0x7f5237d9abd0] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read+0x35) [0x7f5237d9aca5] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read_data+0x3d) [0x7f5237d9b64d] -> /usr/lib/dovecot/libdovecot.so.0(+0xc4da1) [0x7f5237d6bda1] -> /usr/lib/dovecot/libdovecot.so.0(+0xc5d4b) [0x7f5237d6cd4b] -> /usr/lib/dovecot/libdovecot.so.0(message_parser_parse_next_block+0x65) [0x7f5237d6da05] -> /usr/lib/dovecot/libdovecot.so.0(message_parser_parse_body+0x31) [0x7f5237d6dcf1] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xc19d4) [0x7f52381099d4] -> /usr/lib/dovecot/libdovecot-storage.so.0(index_mail_init_stream+0x3aa) [0x7f5238109e9a] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x74b25) [0x7f52380bcb25] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_get_stream_because+0x70) [0x7f52380833a0] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x748d7) [0x7f52380bc8d7] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_get_virtual_size+0x44) [0x7f5238082ea4] -> dovecot/imap [johndoe(a)company.nl 10.0.10.36 FETCH](+0x21523) [0x561f6425f523] -> dovecot/imap [johndoe(a)company.nl 10.0.10.36 FETCH](+0x1f759) [0x561f6425d759] -> dovecot/imap [ johndoe(a)company.nl 10.0.10.36 FETCH](imap_fetch_more+0x39) [0x561f6425eb99] -> dovecot/imap [johndoe(a)company.nl 10.0.10.36 FETCH](cmd_fetch+0x342) [0x561f6424ef32] -> dovecot/imap [johndoe(a)company.nl 10.0.10.36 FETCH](command_exec+0x7c) [0x561f6425bb0c] -> dovecot/imap [ johndoe(a)company.nl 10.0.10.36 FETCH](+0x1bb72) [0x561f64259b72] -> dovecot/imap [johndoe(a)company.nl 10.0.10.36 FETCH](+0x1bc04) [0x561f64259c04] -> dovecot/imap [johndoe(a)company.nl 10.0.10.36 FETCH](client_handle_input+0x1a5) [0x561f6425a025] -> dovecot/imap [ johndoe(a)company.nl 10.0.10.36 FETCH](client_input+0x82) [0x561f6425a6c2] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x75) [0x7f5237da6845] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x11b) [0x7f5237da825b] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x59) [0x7f5237da6959] Oct 18 14:41:39 rwvirtual10 dovecot: imap(johndoe(a)company.nl)<15868><qjTFpy6VPsMKAAok>: Fatal: master: service(imap): child 15868 killed with signal 6 (core dumps disabled - https://dovecot.org/bugreport.html#coredumps)

2 2

dovecot 2.3.7.2-1~bionic: Performance issues caused by excessive IO to ~/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.tmp
by Ralf Hildebrandt 16 Oct '19

16 Oct '19

I set up system copying all mails to a backup system. This used to work without a hitch - now in the last few days mails would pile up in the Postfix Queue, waiting to be delivered using the lmtp transport into dovecot. So dovecot was being slow, but why? After all, nothing changed. After reading some articles on stackoverflow I found a way of finding out which file gets the most IO: % sysdig -c topfiles_bytes; This command quickly pointed to ~/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.tmp That file was written excessively. I then put ~/mdbox/mailboxes/INBOX/dbox-Mails/ into tmpfs and alas, the queue would drain quickly. But why is that? Why would the index file be updated so often? This is dovecot 2.3.7.2-1~bionic # 2.3.7.2 (3c910f64b): /etc/dovecot/dovecot.conf # OS: Linux 5.0.0-29-generic x86_64 Ubuntu 18.04.3 LTS default_vsz_limit = 2 G lmtp_user_concurrency_limit = 10000 mail_attachment_dir = /home/copymail/attachments mail_attachment_hash = %{sha256} mail_fsync = never mail_location = mdbox:~/mdbox mail_plugins = zlib fts fts_lucene mdbox_rotate_size = 128 M namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = username_format=%u /etc/dovecot/passwd driver = passwd-file } plugin { fts = lucene fts_languages = de,en fts_lucene = whitespace_chars=@. } protocols = " imap lmtp" service imap-login { inet_listener imap { address = 127.0.0.1 port = 143 } inet_listener imaps { port = 993 ssl = yes } } service lmtp { inet_listener lmtp { address = 10.0.5.208 port = 1025 } process_min_avail = 5 unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0660 user = postfix } } ssl_ca = /etc/ssl/certs/ca-certificates.crt ssl_cert = </etc/ssl/certs/mail-cbf.pem ssl_cipher_list = EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4 ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it ssl_prefer_server_ciphers = yes userdb { args = username_format=%u /etc/dovecot/passwd driver = passwd-file } protocol lmtp { mail_plugins = zlib fts fts_lucene } -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt(a)charite.de | https://www.charite.de

3 9

Vacation Fcc
by Gregory Heytings 14 Oct '19

14 Oct '19

Hi list, I use Dovecot's Pigeonhole Sieve implementation, and there is one little feature I'm missing: When a vacation message is sent, I'd like to keep a copy of that message in my "sent" mailbox. There is an RFC draft (draft-ietf-extra-sieve-fcc-09) for this, which adds a ":fcc" argument to vacation, but apparently it's not supported by Pigeonhole. I have two questions: Will ":fcc" be supported in the future? In the meantime (or if supporting ":fcc" is not planned), is it possible to achieve the same result with the existing Sieve extensions? (I can imagine a way to do this at the MTA level by checking the presence of the "Auto-Submitted:" header, but did not test it.) Thanks, Gregory

2 3

Sieve replication - Debian 9.11 - Dovecot 2.2.7 - script changes not replicated
by Daniel Botting 14 Oct '19

14 Oct '19

Hi, I am trying to setup Sieve replication, it is almost working: 1/ Create a script - sync'd across 2/ Activate/de-activate a script - change sync'd 3/ Delete a script - change sync'd But if I make a change to a script it is not replicated, unless I de-activate and activate again. I am seeing in the logs the below during this process (de-activating/activating): Oct 9 17:01:39 hostname dovecot: dsync-local(a_user): Debug: doveadm-sieve: Iterating Sieve mailbox attribute: vendor/vendor.dovecot/pvt/server/sieve/files/0910-active-1640 Oct 9 17:01:39 hostname dovecot: dsync-local(a_user): Debug: doveadm-sieve: Value missing for key `vendor/vendor.dovecot/pvt/server/sieve/default' (last change: 2019-10-09 17:01:37) # 2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.16 (fed8554) # OS: Linux 4.9.0-3-amd64 x86_64 Debian 9.11 auth_debug = yes auth_debug_passwords = yes auth_verbose = yes default_vsz_limit = 0 doveadm_password = # hidden, use -P to show it first_valid_gid = 8 first_valid_uid = 8 last_valid_gid = 8 last_valid_uid = 8 lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes mail_debug = yes mail_gid = 8 mail_location = maildir:~/Maildir mail_plugins = " notify replication" mail_uid = 8 managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext editheader imapflags namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { mail_replica = tcps:replication_server_url:port_number sieve = ~/.dovecot.sieve sieve_dir = ~/sieve sieve_extensions = +editheader +imapflags } protocols = " imap sieve pop3" replication_max_conns = 4 service aggregator { fifo_listener replication-notify-fifo { user = mail } unix_listener replication-notify { user = mail } } service auth { unix_listener /var/run/dovecot-exim-bridge { mode = 0660 user = Debian-exim } } service doveadm { inet_listener { port = 12345 ssl = yes } } service imap-login { inet_listener imap { port = 143 } inet_listener imaps { port = 993 ssl = yes } process_limit = 512 process_min_avail = 4 service_count = 1 } service imap { process_limit = 1024 } service managesieve-login { inet_listener sieve { port = 4190 } process_min_avail = 1 service_count = 8 vsz_limit = 256 M } service managesieve { process_limit = 1024 } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0666 } } ssl = required ssl_cert = </path/certs/a_domain/fullchain.pem ssl_client_ca_dir = /etc/ssl/certs ssl_key = # hidden, use -P to show it userdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap override_fields = home=/var/mail/%u uid=mail gid=mail } verbose_ssl = yes protocol lda { mail_plugins = " notify replication sieve" } protocol imap { mail_max_userip_connections = 32 mail_plugins = " notify replication" } protocol sieve { mail_max_userip_connections = 10 managesieve_logout_format = bytes=%i/%o managesieve_max_line_length = 65536 } Kind regards -- Daniel Botting

2 1

Password issue
by ＠lbutlr 13 Oct '19

13 Oct '19

One of my accounts was having login failures when trying to send mail, but was able to check mail. I tried everything I could think of to see what the issue might be, but eventually went in and reset the password in the sql database (I knew the password, so I reset it to the same password). {SHA256-CRYPT}$5$VuS… {SHA256-CRYPT}$5$VI7… So the password was updated properly. Clients can still login to check mail, but are failing to send mail, so the password was not the issue. First, logins in to check mail and succeeds > Oct 09 16:02:16 imap-login: Info: Login: user=<myuser(a)covisp.net>, xx.xx.xx.xx, PLAIN, TLS Sends device MUA info > Oct 09 16:02:16 imap(myuser(a)covisp.net)<84553><bP4acIGUJttJDqGg>: Info: ID sent: name=iPad Mail, version=17A860, os=iOS, os-version=13.1.2 (17A860) Fails to send mail > Oct 09 16:02:50 imap-login: Info: Aborted login (auth failed, 5 attempts in 33 secs): user=<myuser(a)covisp.net>, xx.xx.xx.xx, PLAIN, TLS Postfix logs "Client host rejected: Access denied” but as I said, other accounts can submit and there’s nothing special in the submission service in master.cf. Passwords are all copied/pasted so identical in both IMAP (p993) and SMTP (p587) settings. Settings for the SMTP server are identical to the other accounts using the same server. I don’t know how to login to IMAP via the openssl command line interface, but this is what happens when I login to the submission port.  % openssl s_client -connect mail.covisp.net:587 -starttls smtp … auth login 334 VXNlcm5hbWU6 base64(username)= 334 UGFzc3dvcmQ6 base64(pasword)= DONE (Same results without the terminating ‘=‘) I’m sure I am missing something really obvious? -- Lithium will no longer be available on credit

6 12

Error: SSL_accept() syscall failed
by C. James Ervin 11 Oct '19

11 Oct '19

In setting up my new mail server, I am getting the following in the logs: Oct 11 07:10:59 kumo dovecot[5704]: imap-login: Disconnected (no auth attempts in 0 secs): user=<>, rip=24.53.79.10, lip=172.26.12.90, *TLS handshaking: SSL_accept() syscall failed: Success*, session=<B9OokqCUD+UYNU8K> I have tried various ssl_protocols entries, but for now have defaulted back to ssl_protocols = !SSLv3 (the "out of the box" setting). The certificate (ssl_cert = </etc/ssl/certs/certificate_and_key.crt) is valid and the same one used by postfix. However, I belive the error appeared only after removing the self signed certificate and installing the one I purchased. Warning: though I was a sysadmin a long long time, it has been a while! I have not run a mail server in 15 years, but am moving my mail back "in-house" as I approach retirement! I searched for a forum where I could post this issue, but only found this list. # 2.2.33.2 (d6601f4ec): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.21 (92477967) # OS: Linux 4.15.0-1051-aws x86_64 Ubuntu 18.04.1 LTS auth_mechanisms = plain login mail_location = maildir:~/Maildir mail_privileged_group = mail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = pam } plugin { sieve = ~/.dovecot.sieve sieve_dir = ~/sieve } protocols = " imap sieve pop3" service auth { unix_listener /var/spool/postfix/private/dovecot-auth { group = postfix mode = 0660 user = postfix } } ssl_cert = </etc/ssl/certs/certificate_and_key.crt ssl_cipher_list = ALL:!ADH:!LOW:!EXP:!aNULL:+HIGH:+MEDIUM ssl_client_ca_dir = /etc/ssl/certs ssl_key = # hidden, use -P to show it userdb { driver = passwd } protocol lda { deliver_log_format = msgid=%m: %$ mail_plugins = sieve postmaster_address = postmaster quota_full_tempfail = yes rejection_reason = Your message to <%t> was automatically rejected:%n%r } protocol imap { imap_client_workarounds = delay-newmail mail_max_userip_connections = 10 } protocol pop3 { mail_max_userip_connections = 10 pop3_client_workarounds = outlook-no-nuls oe-ns-eoh }

3 3

Re: Broken link on doc.dovecot.org
by Karen Woodman 11 Oct '19

11 Oct '19

Hi again, I wanted to check in and see if you got my note about the broken link on your site. Thanks! Karen On Monday, October 7, 2019 at 5:05 PM, Karen Woodman <karen(a)getmailbird.co> wrote: > Hi there, > > I noticed that you have a broken link to a website called Qmail.org. That > site was first published 23 years ago (back in 1996!) but unfortunately, it > is no longer a working website. > > You link to Qmail.org on this page: > https://doc.dovecot.org/configuration_manual/authentication/checkpassword/ > > We just published an article that explores what happened to the site. I think > it's an interesting story, and it could be useful to your readers. > > In addition, since all of the original articles are gone, we included a > curated list of the best Qmail resources and mirrors. > > Here's the article if you want to take a look: > https://www.getmailbird.com/what-resources-are-available-now-that-qmail-org… > > Would you consider swapping out the broken link for our article? > > Please let me know if you have any questions and thank you for your time. > > Sincerely, > -Karen > > -- > Karen Woodman, Editor > 5 Ross Rd > Durham, NH 03824 > > By the way, if you didn't like getting this email you can unsubscribe > <http://ec2-52-26-194-35.us-west-2.compute.amazonaws.com/x/u?u=df0298ca-a3e4…> and > I won't email you anymore. :) >

2 1

Using maldet
by Ignacio García 10 Oct '19

10 Oct '19

Hi there. Sometimes I notice that some malware gets through amavis+clamav+postfix and lands in the user's maildir files. I wonder if it would make sense, or even if it would be advisable to scan and clean those files using a program as maldet Thanks for your help. Ignacio

1 0