February 2018 - dovecot

Assertion during dsync receive
by Ian Bobbitt 13 Mar '18

13 Mar '18

Hi, I'm getting an assertion failed on the receiving side, causing syncs to fail for one user. The servers are setup so that only one is receiving any traffic other than replication at any time. The one that's only receiving replications is the one that's failing. I've tried deleting the user's home on the receiving server, but it still crashes during the sync. Oddly, the user's home is 7.4G on the sending server, but ends up at 42G on the receiving side, even after deleting and trying a fresh sync. The mailbox implicated in the backtrace ("Spam") does have a very large number of messages in it. On sender: Spam messages=1217764 recent=0 uidnext=1218103 uidvalidity=1379509105 unseen=16 highestmodseq=744588 vsize=34468460093 guid=090ed93a7a055559abf10200fdf6807a firstsaved=1498744186 On receiver: Spam messages=1217766 recent=352 uidnext=1218105 uidvalidity=1379509105 unseen=16 highestmodseq=744589 vsize=34468496809 guid=090ed93a7a055559abf10200fdf6807a firstsaved=1519396172 Feb 23 14:57:20 dovecot: dsync-local(bgeels): Warning: fscking index file /gnoc/mail/home/bgeels/mail/storage/dovecot.map.index Feb 23 14:57:20 dovecot: dsync-local(bgeels): Warning: mdbox /gnoc/mail/home/bgeels/mail/storage: rebuilding indexes Feb 23 14:57:33 dovecot: dsync-local(bgeels): Panic: file mail-index-util.c: line 10 (mail_index_uint32_to_offset): assertion failed: (offset < 0x40000000) Feb 23 14:57:33 dovecot: dsync-local(bgeels): Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0x9f3de) [0x7feb584143de] -> /usr/lib64/dovecot/libdovecot.so.0(+0x9f4be) [0x7feb584144be] -> /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7feb583a577c] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_index_uint32_to_offset+0xa0) [0x7feb587906d0] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_cache_compress+0x854) [0x7feb58774f34] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_index_sync_commit+0x25f) [0x7feb587884ff] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_storage_rebuild_in_context+0x10de) [0x7feb5870b3ae] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_sync_begin+0x858) [0x7feb5870ccd8] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_sync+0x4c) [0x7feb5870ce7c] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_storage_sync_init+0x4b) [0x7feb5870cf3b] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_sync_init+0x44) [0x7feb586f2834] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_sync+0x37) [0x7feb586f28d7] -> dovecot/doveadm-server(dsync_mailbox_import_deinit+0x475) [0x445495] -> dovecot/doveadm-server() [0x43edc0] -> dovecot/doveadm-server(dsync_brain_sync_mails+0x743) [0x43f653] -> dovecot/doveadm-server(dsync_brain_run+0x541) [0x43acf1] -> dovecot/doveadm-server() [0x43b070] -> dovecot/doveadm-server() [0x44fe5f] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7feb58429cd2] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x10f) [0x7feb5842b3bf] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) [0x7feb58429d6c] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7feb58429f28] -> dovecot/doveadm-server() [0x4209c5] -> dovecot/doveadm-server() [0x422df6] -> dovecot/doveadm-server() [0x4377f4] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7feb58429cd2] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x10f) [0x7feb5842b3bf] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) [0x7feb58429d6c] Feb 23 14:57:33 dsync-local(bgeels): Fatal: master: service(doveadm): child 82098 killed with signal 6 (core dumped) I've attached the output of `doveconf -n` and the full backtrace from a core dump. Dovecot 2.2.33.2 (GhettoForge package) CentOS 7 x86_64 XFS, no NFS. -- Ian

3 12

Unexpected config results with local_name + multiple SSL certs
by David Favor 10 Mar '18

10 Mar '18

Working with SSL on fresh install of latest Ubuntu Artful + Dovecot seems broken somehow. Application is Dovecot listening for many SSL sites... Likely I've missed adding something simple to the config, related to local_name usage. Be great if someone can point out what I've missed, to setup multiple SSL certs for different host.domain entries in config. Thanks. _______ This works as expected... where the SNI server name is returned... #local_name imap.cydec.com { ssl_cert = </etc/letsencrypt/live/imap.cydec.com/fullchain.pem ssl_key = </etc/letsencrypt/live/imap.cydec.com/privkey.pem #} service dovecot restart && echo QUIT | openssl s_client -connect imap.cydec.com:993 -servername imap.cydec.com 2>&1 | egrep ^subject subject=/CN=imap.cydec.com _______ This fails... local_name imap.cydec.com { ssl_cert = </etc/letsencrypt/live/imap.cydec.com/fullchain.pem ssl_key = </etc/letsencrypt/live/imap.cydec.com/privkey.pem } service dovecot restart && echo QUIT | openssl s_client -connect imap.cydec.com:993 -servername imap.cydec.com 2>&1 | egrep ^subject # Empty, so no servername match _______ Full openssl output shows no cert being returned... service dovecot restart && echo QUIT | openssl s_client -connect imap.cydec.com:993 -servername imap.cydec.com CONNECTED(00000004) write:errno=104 --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 0 bytes and written 199 bytes Verification: OK --- New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : 0000 Session-ID: Session-ID-ctx: Master-Key: PSK identity: None PSK identity hint: None SRP username: None Start Time: 1519576210 Timeout : 7200 (sec) Verify return code: 0 (ok) Extended master secret: no --- _______ Config seems correct, with local_name uncommented... dovecot -n # 2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.16 (fed8554) # OS: Linux 4.13.0-36-generic x86_64 Ubuntu 17.10 auth_debug = yes auth_debug_passwords = yes auth_verbose = yes debug_log_path = /var/log/dovecot.log disable_plaintext_auth = no info_log_path = /var/log/dovecot.log log_path = /var/log/dovecot.log mail_debug = yes mail_location = mbox:~/mail:INBOX=/var/mail/%u namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = dovecot driver = pam } protocols = " imap pop3" service imap-login { inet_listener imap { port = 0 } inet_listener imaps { port = 993 ssl = yes } } service pop3-login { inet_listener pop3 { port = 0 } inet_listener pop3s { port = 995 ssl = yes } } userdb { driver = passwd } local_name imap.cydec.com { ssl_cert = </etc/letsencrypt/live/imap.cydec.com/fullchain.pem ssl_key = # hidden, use -P to show it }

3 3

extra IMAP folders: how to make all clients use the same Sent folder?
by Stanislaw Findeisen 07 Mar '18

07 Mar '18

Dear All I have just setup IMAP with Maildir, but unfortunately some clients create their own folders instead of using those specified in 15-mailboxes.conf . What could be the reason? Here's what it looks like: > B list "" * > * LIST (\HasNoChildren \Sent) "." "Sent Messages" > * LIST (\HasNoChildren) "." Gesendet > * LIST (\HasNoChildren) "." Entw&APw-rfe > * LIST (\HasNoChildren \Trash) "." Trash > * LIST (\HasNoChildren) "." Papierkorb > * LIST (\HasNoChildren \Drafts) "." Drafts > * LIST (\HasNoChildren \Sent) "." Sent > * LIST (\HasNoChildren) "." Ausgang > * LIST (\HasNoChildren \Junk) "." Junk > * LIST (\HasNoChildren) "." INBOX > B OK List completed. Here for example "Gesendet" (meaning "Sent" in German) is a folder created by some Android client app. This app uses it to store its sent mail, while another client uses the Sent folder. Also, is it correct that all those folders (both pre-made and extra) get created directly in the user's maildir, alongside cur, new, tmp and Dovecot index files? Shouldn't there be any subdir? > # dovecot --version > 2.2.13 Thanks! Stanisław

2 1

v2.2.34 released
by Timo Sirainen 01 Mar '18

01 Mar '18

https://dovecot.org/releases/2.2/dovecot-2.2.34.tar.gz https://dovecot.org/releases/2.2/dovecot-2.2.34.tar.gz.sig * CVE-2017-15130: TLS SNI config lookups may lead to excessive memory usage, causing imap-login/pop3-login VSZ limit to be reached and the process restarted. This happens only if Dovecot config has local_name { } or local { } configuration blocks and attacker uses randomly generated SNI servernames. * CVE-2017-14461: Parsing invalid email addresses may cause a crash or leak memory contents to attacker. For example, these memory contents might contain parts of an email from another user if the same imap process is reused for multiple users. First discovered by Aleksandar Nikolic of Cisco Talos. Independently also discovered by "flxflndy" via HackerOne. * CVE-2017-15132: Aborted SASL authentication leaks memory in login process. * Linux: Core dumping is no longer enabled by default via PR_SET_DUMPABLE, because this may allow attackers to bypass chroot/group restrictions. Found by cPanel Security Team. Nowadays core dumps can be safely enabled by using "sysctl -w fs.suid_dumpable=2". If the old behaviour is wanted, it can still be enabled by setting: import_environment=$import_environment PR_SET_DUMPABLE=1 * doveconf output now includes the hostname. + mail_attachment_detection_options setting controls when $HasAttachment and $HasNoAttachment keywords are set for mails. + imap: Support fetching body snippets using FETCH (SNIPPET) or (SNIPPET (LAZY=FUZZY)) + fs-compress: Automatically detect whether input is compressed or not. Prefix the compression algorithm with "maybe-" to enable the detection, for example: "compress:maybe-gz:6:..." + Added settings to change dovecot.index* files' optimization behavior. See https://wiki2.dovecot.org/IndexFiles#Settings + Auth cache can now utilize auth workers to do password hash verification by setting auth_cache_verify_password_with_worker=yes. + Added charset_alias plugin. See https://wiki2.dovecot.org/Plugins/CharsetAlias + imap_logout_format and pop3_logout_format settings now support all of the generic variables (e.g. %{rip}, %{session}, etc.) + Added auth_policy_check_before_auth, auth_policy_check_after_auth and auth_policy_report_after_auth settings. - v2.2.33: doveadm-server: Various fixes related to log handling. - v2.2.33: doveadm failed when trying to access UNIX socket that didn't require authentication. - v2.2.33: doveadm log reopen stopped working - v2.2.30+: IMAP stopped advertising SPECIAL-USE capability - v2.2.30+: IMAP stopped sending untagged OK/NO storage notifications - replication: dsync sends unnecessary replication notification for changes it does internally. NOTE: Folder creates, renames, deletes and subscribes still trigger unnecessary replication notifications, but these should be rather rare. - mail_always/never_cache_fields setting changes weren't applied for existing dovecot.index.cache files. - Fix compiling and other problems with OpenSSL v1.1 - auth policy: With master user logins, lookup using login username. - FTS reindexed all mails unnecessarily after loss of dovecot.index.cache file - mdbox rebuild repeatedly fails with "missing map extension" - SSL connections may have been hanging with imapc or doveadm client. - cassandra: Using protocol v3 (Cassandra v2.1) caused memory leaks and also timestamps weren't set to queries. - fs-crypt silently ignored public/private keys specified in configuration (mail_crypt_global_public/private_key) and just emitted plaintext output. - lock_method=dotlock caused crashes - imapc: Reconnection may cause crashes and other errors

7 9

Using virtual folders with younger and index files
by Rob Hoelz 01 Mar '18

01 Mar '18

Hi list, I just encountered a problem while using dovecot's Virtual plugin with 2.3.0. I managed to solve the issue, but I wanted to bring it to the attention of others on the list to see if there exists a better solution, if I found a bug, or if it's just a matter of updating documentation. I have a virtual folder to get the most recent two weeks of mails; it looks something like this: > INBOX > all younger 1209600 I made this folder back in October. Lately, I started to notice that the virtual folder had a surprising amount of mail in it - I don't get 1,000 e-mails per week! After some doveadm commands, I realized that e-mails from October were still present in my virtual folder! Updating the dovecot-virtual would clear away e-mails older than two weeks (I needed to actually introduce a change, even if it was just whitespace - just touching the file didn't update things), and removing the dovecot index file also cleared things away. In then end, I ended up just telling dovecot to disable on-disk indexes for that folder. I created this folder based on the examples on https://wiki.dovecot.org/Plugins/Virtual - I'm wondering if I found a bug or if that page should be changed to recommend disabling on-disk indexes when using certain search query filters such as "younger". If the latter, I can always make the change - just let me know! -Rob

2 2

v2.3.0.1 released
by Timo Sirainen 28 Feb '18

28 Feb '18

https://dovecot.org/releases/2.3/dovecot-2.3.0.1.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.0.1.tar.gz.sig Small patch release to fix the worst bugs in v2.3.0. v2.3.1 is coming in about a month with a lot more changes. * CVE-2017-15130: TLS SNI config lookups may lead to excessive memory usage, causing imap-login/pop3-login VSZ limit to be reached and the process restarted. This happens only if Dovecot config has local_name { } or local { } configuration blocks and attacker uses randomly generated SNI servernames. * CVE-2017-14461: Parsing invalid email addresses may cause a crash or leak memory contents to attacker. For example, these memory contents might contain parts of an email from another user if the same imap process is reused for multiple users. First discovered by Aleksandar Nikolic of Cisco Talos. Independently also discovered by "flxflndy" via HackerOne. * CVE-2017-15132: Aborted SASL authentication leaks memory in login process. * Linux: Core dumping is no longer enabled by default via PR_SET_DUMPABLE, because this may allow attackers to bypass chroot/group restrictions. Found by cPanel Security Team. Nowadays core dumps can be safely enabled by using "sysctl -w fs.suid_dumpable=2". If the old behaviour is wanted, it can still be enabled by setting: import_environment=$import_environment PR_SET_DUMPABLE=1 - imap-login with SSL/TLS connections may end up in infinite loop

1 0

Re: Quota status to postfix in distributed environment (Karol Augustin)
by SAAHIL IFTEKHAR 28 Feb '18

28 Feb '18

Hi Karol Thanks for taking time to go through the email and replying. I'll rewrite the issue again --> For example consider there two nodes. These each having their postfix and dovecot. I am trying to implement quota status to postfix to both the below cases:- case 1) mail sent within the node case 2) mail sent across the nodes In case 1 the postfix will query the dovecot of same node white in case 2 it will query the dovecot of other node. But suppose I am sending mail within the node and dovecot in other node is down. Mail will not go within the node. Even it will not defer also. This is evident from the logs(in last sections). I'll try to work suggestions you have given in mysetup though and post the relevant output. But if you have any other suggestion regarding what i have wrote just now. please let me know. Regards On 28-Feb-2018 5:19 am, <dovecot-request(a)dovecot.org> wrote: Send dovecot mailing list submissions to dovecot(a)dovecot.org To subscribe or unsubscribe via the World Wide Web, visit https://dovecot.org/mailman/listinfo/dovecot or, via email, send a message with subject or body 'help' to dovecot-request(a)dovecot.org You can reach the person managing the list at dovecot-owner(a)dovecot.org When replying, please edit your Subject line so it is more specific than "Re: Contents of dovecot digest..." Today's Topics: 1. Re: Quota status to postfix in distributed environment (@lbutlr) (SAAHIL IFTEKHAR) 2. Re: Quota status to postfix in distributed environment (Karol Augustin) 3. Re: Quota status to postfix in distributed environment (LuKreme) 4. Re: Quota status to postfix in distributed environment (Karol Augustin) ---------------------------------------------------------------------- Message: 1 Date: Tue, 27 Feb 2018 19:29:44 +0530 From: SAAHIL IFTEKHAR <response2saahil(a)gmail.com> To: dovecot(a)dovecot.org Subject: Re: Quota status to postfix in distributed environment (@lbutlr) Message-ID: <CAM5TiAQ2CT+u_BKEn0sKonirOQLD_Q0v=pVCcJBCiJfC08p7Mw(a)mail.gmail.com> Content-Type: text/plain; charset="utf-8" Hi Actually, the requirement is such that we have to work with this version only. I want to point out that the reason from the problem can be found out from logs(the end section). The "check_policy_service" in main.cf of postfix is the problem according to me. This configuration as a part of quota staus to postfix configuration. I hope many have went through this scenario or similar. So I am hoping to get some help for the same. Regards ------------------------------ Message: 2 Date: Tue, 27 Feb 2018 14:18:31 +0000 From: Karol Augustin <karol(a)augustin.pl> To: dovecot(a)dovecot.org Subject: Re: Quota status to postfix in distributed environment Message-ID: <b50c30e1b65c3875565ca2e49efeaafd(a)augustin.pl> Content-Type: text/plain; charset=US-ASCII On 2018-02-26 16:28, SAAHIL IFTEKHAR wrote: > Hi > > I have implemented Quota status to postfix in our setup. I have an imap server (dovecot) and mail server (postfix) in every node. I am able to send quota status to postfix and mails are rejected after 100% mail quota is crossed. This rejection is happening both in across the nodes and within the nodes. > > The problem is if I am sending mails to any node and if any other node's dovecot is down, mails are not going. For example, I am sending an email within the system but if some other node's dovecot is down then email within the system also will not go. > > My dovecot version is 2.2.10. My postfix version is 2.1.10. > > doveconf -n output is below:- > [snap] > > Here "service quota status" is the concerned section in conf file. > ____________________________________________________________ ____________________________________________ > > Postfix configuration is below:- > smtpd_relay_restrictions = > check_policy_service inet:201.123.80.9:54317 > check_policy_service inet:201.123.80.23:54317 > > virtual_transport=lmtp:unix:private/dovecot-lmtp > > Here, I am querying both two nodes. 201.123.80.9 is the other node. 201.123.80.23 is the node within which, email is sent. > ____________________________________________________________ _______________________________________________ > > logs while sending mail is below:- > [snap] > > I am understanding what the logs are trying to say. But I am not able to resolve the issue even after searching solution on internet and trying different hit and trials by myself. I want that if i am sending email to any node or within node, the configuration relating to "check _policy_service" for other node does not interfere and mail goes properly. At the same time I can also fetch quota status from other nodes. > > If I can get any help regarding this it will be really appreciable as I have tried a lot of options already. > Your problem is that if the dovecot is unreachable Postfix can't check the quota for the user. http://www.postfix.org/SMTPD_POLICY_README.html explains how to resolve this problem: smtpd_policy_service_default_action (default: 451 4.3.5 Server configuration problem): The default action when an SMTPD policy service request fails. Available with Postfix 3.0 and later. The default action when an SMTPD policy service request fails. Specify "DUNNO" to behave as if the failed SMTPD policy service request was not sent, and to continue processing other access restrictions, if any. You can configure it per policy also. The issue you might encounter with this setup is that if both of your dovecot nodes are unreachable Postfix will accept the e-mail and try to deliver it. When the nodes come online it will fail if the user is over quota and generate bounce to the envelope sender of the message, which might produce backscatter. I resolved this problem by keeping quota information in Mysql table and using custom policy server to check if user is allowed to receive more e-mail. Also this approach might not work with aliases and other redirections, the e-mail address checked by smtpd policy is the one before alias expansion. You should check if it suits your environment. Best, Karol -- Karol Augustin karol(a)augustin.pl http://karolaugustin.pl/ +353 85 775 5312

1 0

compile-error on pigeonhole
by Jakobus Schürz 28 Feb '18

28 Feb '18

Hi! I try to compile pigeonhole from git-sources and get this error: libtool: link: gcc -std=gnu99 -g -O2 -fstack-protector-strong -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -fno-builtin-strfti me -Wstrict-aliasing=2 -I../.. -pie -Wl,-z -Wl,relro -Wl,-z -Wl,now -o .libs/sievec sievec-sievec.o -Wl,--export-dynamic ../../src/lib-sieve/.libs/libdovecot-sieve.so ../../src/lib-sieve-tool/.libs/libsieve-tool.a /usr/local/lib/dovecot /libdovecot-storage.so /usr/local/lib/dovecot/libdovecot-lda.so -L/usr/local/lib/dovecot /usr/local/lib/dovecot/libdovecot.so -Wl,-rpath -Wl,/usr/local/lib/dovecot ../../src/lib-sieve/.libs/libdovecot-sieve.so: undefined reference to `array_idx_get_space' collect2: error: ld returned 1 exit status I'm not able to compile pigeonhole. Jakob

2 1

dovecot.index.pvt reset, view is now inconsistent
by Rupert Gallagher 28 Feb '18

28 Feb '18

Problem solved by going in manually. The log message appears for empty "public" folders. Say, you have a folder X with subfolder Y, where X does not contain any e-mail. The log message disappears if you drop an email into X, then remove it. Puf, gone! So, there seems to be a baby bug in how dovecot manages the index in this case.

3 2

BUG: Error: dovecot.index.pvt reset, view is now inconsistent when shared folder is new and empty
by Marco Giunta 28 Feb '18

28 Feb '18

Hi, I'm using Dovecot 2.2.33.2 on a RHEL 7, new installation. My log is full of : Error: INDEX_FOLDER/dovecot.index.pvt reset, view is now inconsistent or Error: INDEX_FOLDER/dovecot.index.pvt view is inconsistent when shared folder is never touched and empty. UserA share XXXXX folder with UserB, if XXXXX folder is new (never touched) and empty every time UserB looks in that folder, an error appears in log file. If UserA copy a mail in XXXXX folder, no more errors. If UserA (or UserB) delete all mails in XXXXX folder (the folder is empty again), no more errors. So the errors appear when UserB access a new (never touched) shared empty folder; if the folder is empty, but not new (p.e. UserA has already copied and deleted mails in that folder) error is logged only once. Attached my configuration. Thanks, Marco -- Marco Giunta - ITCS SysAdmin Via Bonomea, 265 34136 - Trieste, Italy Tel: +39-040-3787-503 Fax: +39-040-3787-244

1 0