February 2018 - dovecot

extra IMAP folders: how to make all clients use the same Sent folder?
by Stanislaw Findeisen 07 Mar '18

07 Mar '18

Dear All I have just setup IMAP with Maildir, but unfortunately some clients create their own folders instead of using those specified in 15-mailboxes.conf . What could be the reason? Here's what it looks like: > B list "" * > * LIST (\HasNoChildren \Sent) "." "Sent Messages" > * LIST (\HasNoChildren) "." Gesendet > * LIST (\HasNoChildren) "." Entw&APw-rfe > * LIST (\HasNoChildren \Trash) "." Trash > * LIST (\HasNoChildren) "." Papierkorb > * LIST (\HasNoChildren \Drafts) "." Drafts > * LIST (\HasNoChildren \Sent) "." Sent > * LIST (\HasNoChildren) "." Ausgang > * LIST (\HasNoChildren \Junk) "." Junk > * LIST (\HasNoChildren) "." INBOX > B OK List completed. Here for example "Gesendet" (meaning "Sent" in German) is a folder created by some Android client app. This app uses it to store its sent mail, while another client uses the Sent folder. Also, is it correct that all those folders (both pre-made and extra) get created directly in the user's maildir, alongside cur, new, tmp and Dovecot index files? Shouldn't there be any subdir? > # dovecot --version > 2.2.13 Thanks! Stanisław

2 1

v2.2.34 released
by Timo Sirainen 01 Mar '18

01 Mar '18

https://dovecot.org/releases/2.2/dovecot-2.2.34.tar.gz https://dovecot.org/releases/2.2/dovecot-2.2.34.tar.gz.sig * CVE-2017-15130: TLS SNI config lookups may lead to excessive memory usage, causing imap-login/pop3-login VSZ limit to be reached and the process restarted. This happens only if Dovecot config has local_name { } or local { } configuration blocks and attacker uses randomly generated SNI servernames. * CVE-2017-14461: Parsing invalid email addresses may cause a crash or leak memory contents to attacker. For example, these memory contents might contain parts of an email from another user if the same imap process is reused for multiple users. First discovered by Aleksandar Nikolic of Cisco Talos. Independently also discovered by "flxflndy" via HackerOne. * CVE-2017-15132: Aborted SASL authentication leaks memory in login process. * Linux: Core dumping is no longer enabled by default via PR_SET_DUMPABLE, because this may allow attackers to bypass chroot/group restrictions. Found by cPanel Security Team. Nowadays core dumps can be safely enabled by using "sysctl -w fs.suid_dumpable=2". If the old behaviour is wanted, it can still be enabled by setting: import_environment=$import_environment PR_SET_DUMPABLE=1 * doveconf output now includes the hostname. + mail_attachment_detection_options setting controls when $HasAttachment and $HasNoAttachment keywords are set for mails. + imap: Support fetching body snippets using FETCH (SNIPPET) or (SNIPPET (LAZY=FUZZY)) + fs-compress: Automatically detect whether input is compressed or not. Prefix the compression algorithm with "maybe-" to enable the detection, for example: "compress:maybe-gz:6:..." + Added settings to change dovecot.index* files' optimization behavior. See https://wiki2.dovecot.org/IndexFiles#Settings + Auth cache can now utilize auth workers to do password hash verification by setting auth_cache_verify_password_with_worker=yes. + Added charset_alias plugin. See https://wiki2.dovecot.org/Plugins/CharsetAlias + imap_logout_format and pop3_logout_format settings now support all of the generic variables (e.g. %{rip}, %{session}, etc.) + Added auth_policy_check_before_auth, auth_policy_check_after_auth and auth_policy_report_after_auth settings. - v2.2.33: doveadm-server: Various fixes related to log handling. - v2.2.33: doveadm failed when trying to access UNIX socket that didn't require authentication. - v2.2.33: doveadm log reopen stopped working - v2.2.30+: IMAP stopped advertising SPECIAL-USE capability - v2.2.30+: IMAP stopped sending untagged OK/NO storage notifications - replication: dsync sends unnecessary replication notification for changes it does internally. NOTE: Folder creates, renames, deletes and subscribes still trigger unnecessary replication notifications, but these should be rather rare. - mail_always/never_cache_fields setting changes weren't applied for existing dovecot.index.cache files. - Fix compiling and other problems with OpenSSL v1.1 - auth policy: With master user logins, lookup using login username. - FTS reindexed all mails unnecessarily after loss of dovecot.index.cache file - mdbox rebuild repeatedly fails with "missing map extension" - SSL connections may have been hanging with imapc or doveadm client. - cassandra: Using protocol v3 (Cassandra v2.1) caused memory leaks and also timestamps weren't set to queries. - fs-crypt silently ignored public/private keys specified in configuration (mail_crypt_global_public/private_key) and just emitted plaintext output. - lock_method=dotlock caused crashes - imapc: Reconnection may cause crashes and other errors

7 9

Using virtual folders with younger and index files
by Rob Hoelz 01 Mar '18

01 Mar '18

Hi list, I just encountered a problem while using dovecot's Virtual plugin with 2.3.0. I managed to solve the issue, but I wanted to bring it to the attention of others on the list to see if there exists a better solution, if I found a bug, or if it's just a matter of updating documentation. I have a virtual folder to get the most recent two weeks of mails; it looks something like this: > INBOX > all younger 1209600 I made this folder back in October. Lately, I started to notice that the virtual folder had a surprising amount of mail in it - I don't get 1,000 e-mails per week! After some doveadm commands, I realized that e-mails from October were still present in my virtual folder! Updating the dovecot-virtual would clear away e-mails older than two weeks (I needed to actually introduce a change, even if it was just whitespace - just touching the file didn't update things), and removing the dovecot index file also cleared things away. In then end, I ended up just telling dovecot to disable on-disk indexes for that folder. I created this folder based on the examples on https://wiki.dovecot.org/Plugins/Virtual - I'm wondering if I found a bug or if that page should be changed to recommend disabling on-disk indexes when using certain search query filters such as "younger". If the latter, I can always make the change - just let me know! -Rob

2 2

v2.3.0.1 released
by Timo Sirainen 28 Feb '18

28 Feb '18

https://dovecot.org/releases/2.3/dovecot-2.3.0.1.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.0.1.tar.gz.sig Small patch release to fix the worst bugs in v2.3.0. v2.3.1 is coming in about a month with a lot more changes. * CVE-2017-15130: TLS SNI config lookups may lead to excessive memory usage, causing imap-login/pop3-login VSZ limit to be reached and the process restarted. This happens only if Dovecot config has local_name { } or local { } configuration blocks and attacker uses randomly generated SNI servernames. * CVE-2017-14461: Parsing invalid email addresses may cause a crash or leak memory contents to attacker. For example, these memory contents might contain parts of an email from another user if the same imap process is reused for multiple users. First discovered by Aleksandar Nikolic of Cisco Talos. Independently also discovered by "flxflndy" via HackerOne. * CVE-2017-15132: Aborted SASL authentication leaks memory in login process. * Linux: Core dumping is no longer enabled by default via PR_SET_DUMPABLE, because this may allow attackers to bypass chroot/group restrictions. Found by cPanel Security Team. Nowadays core dumps can be safely enabled by using "sysctl -w fs.suid_dumpable=2". If the old behaviour is wanted, it can still be enabled by setting: import_environment=$import_environment PR_SET_DUMPABLE=1 - imap-login with SSL/TLS connections may end up in infinite loop

1 0

Re: Quota status to postfix in distributed environment (Karol Augustin)
by SAAHIL IFTEKHAR 28 Feb '18

28 Feb '18

Hi Karol Thanks for taking time to go through the email and replying. I'll rewrite the issue again --> For example consider there two nodes. These each having their postfix and dovecot. I am trying to implement quota status to postfix to both the below cases:- case 1) mail sent within the node case 2) mail sent across the nodes In case 1 the postfix will query the dovecot of same node white in case 2 it will query the dovecot of other node. But suppose I am sending mail within the node and dovecot in other node is down. Mail will not go within the node. Even it will not defer also. This is evident from the logs(in last sections). I'll try to work suggestions you have given in mysetup though and post the relevant output. But if you have any other suggestion regarding what i have wrote just now. please let me know. Regards On 28-Feb-2018 5:19 am, <dovecot-request(a)dovecot.org> wrote: Send dovecot mailing list submissions to dovecot(a)dovecot.org To subscribe or unsubscribe via the World Wide Web, visit https://dovecot.org/mailman/listinfo/dovecot or, via email, send a message with subject or body 'help' to dovecot-request(a)dovecot.org You can reach the person managing the list at dovecot-owner(a)dovecot.org When replying, please edit your Subject line so it is more specific than "Re: Contents of dovecot digest..." Today's Topics: 1. Re: Quota status to postfix in distributed environment (@lbutlr) (SAAHIL IFTEKHAR) 2. Re: Quota status to postfix in distributed environment (Karol Augustin) 3. Re: Quota status to postfix in distributed environment (LuKreme) 4. Re: Quota status to postfix in distributed environment (Karol Augustin) ---------------------------------------------------------------------- Message: 1 Date: Tue, 27 Feb 2018 19:29:44 +0530 From: SAAHIL IFTEKHAR <response2saahil(a)gmail.com> To: dovecot(a)dovecot.org Subject: Re: Quota status to postfix in distributed environment (@lbutlr) Message-ID: <CAM5TiAQ2CT+u_BKEn0sKonirOQLD_Q0v=pVCcJBCiJfC08p7Mw(a)mail.gmail.com> Content-Type: text/plain; charset="utf-8" Hi Actually, the requirement is such that we have to work with this version only. I want to point out that the reason from the problem can be found out from logs(the end section). The "check_policy_service" in main.cf of postfix is the problem according to me. This configuration as a part of quota staus to postfix configuration. I hope many have went through this scenario or similar. So I am hoping to get some help for the same. Regards ------------------------------ Message: 2 Date: Tue, 27 Feb 2018 14:18:31 +0000 From: Karol Augustin <karol(a)augustin.pl> To: dovecot(a)dovecot.org Subject: Re: Quota status to postfix in distributed environment Message-ID: <b50c30e1b65c3875565ca2e49efeaafd(a)augustin.pl> Content-Type: text/plain; charset=US-ASCII On 2018-02-26 16:28, SAAHIL IFTEKHAR wrote: > Hi > > I have implemented Quota status to postfix in our setup. I have an imap server (dovecot) and mail server (postfix) in every node. I am able to send quota status to postfix and mails are rejected after 100% mail quota is crossed. This rejection is happening both in across the nodes and within the nodes. > > The problem is if I am sending mails to any node and if any other node's dovecot is down, mails are not going. For example, I am sending an email within the system but if some other node's dovecot is down then email within the system also will not go. > > My dovecot version is 2.2.10. My postfix version is 2.1.10. > > doveconf -n output is below:- > [snap] > > Here "service quota status" is the concerned section in conf file. > ____________________________________________________________ ____________________________________________ > > Postfix configuration is below:- > smtpd_relay_restrictions = > check_policy_service inet:201.123.80.9:54317 > check_policy_service inet:201.123.80.23:54317 > > virtual_transport=lmtp:unix:private/dovecot-lmtp > > Here, I am querying both two nodes. 201.123.80.9 is the other node. 201.123.80.23 is the node within which, email is sent. > ____________________________________________________________ _______________________________________________ > > logs while sending mail is below:- > [snap] > > I am understanding what the logs are trying to say. But I am not able to resolve the issue even after searching solution on internet and trying different hit and trials by myself. I want that if i am sending email to any node or within node, the configuration relating to "check _policy_service" for other node does not interfere and mail goes properly. At the same time I can also fetch quota status from other nodes. > > If I can get any help regarding this it will be really appreciable as I have tried a lot of options already. > Your problem is that if the dovecot is unreachable Postfix can't check the quota for the user. http://www.postfix.org/SMTPD_POLICY_README.html explains how to resolve this problem: smtpd_policy_service_default_action (default: 451 4.3.5 Server configuration problem): The default action when an SMTPD policy service request fails. Available with Postfix 3.0 and later. The default action when an SMTPD policy service request fails. Specify "DUNNO" to behave as if the failed SMTPD policy service request was not sent, and to continue processing other access restrictions, if any. You can configure it per policy also. The issue you might encounter with this setup is that if both of your dovecot nodes are unreachable Postfix will accept the e-mail and try to deliver it. When the nodes come online it will fail if the user is over quota and generate bounce to the envelope sender of the message, which might produce backscatter. I resolved this problem by keeping quota information in Mysql table and using custom policy server to check if user is allowed to receive more e-mail. Also this approach might not work with aliases and other redirections, the e-mail address checked by smtpd policy is the one before alias expansion. You should check if it suits your environment. Best, Karol -- Karol Augustin karol(a)augustin.pl http://karolaugustin.pl/ +353 85 775 5312

1 0

compile-error on pigeonhole
by Jakobus Schürz 28 Feb '18

28 Feb '18

Hi! I try to compile pigeonhole from git-sources and get this error: libtool: link: gcc -std=gnu99 -g -O2 -fstack-protector-strong -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -fno-builtin-strfti me -Wstrict-aliasing=2 -I../.. -pie -Wl,-z -Wl,relro -Wl,-z -Wl,now -o .libs/sievec sievec-sievec.o -Wl,--export-dynamic ../../src/lib-sieve/.libs/libdovecot-sieve.so ../../src/lib-sieve-tool/.libs/libsieve-tool.a /usr/local/lib/dovecot /libdovecot-storage.so /usr/local/lib/dovecot/libdovecot-lda.so -L/usr/local/lib/dovecot /usr/local/lib/dovecot/libdovecot.so -Wl,-rpath -Wl,/usr/local/lib/dovecot ../../src/lib-sieve/.libs/libdovecot-sieve.so: undefined reference to `array_idx_get_space' collect2: error: ld returned 1 exit status I'm not able to compile pigeonhole. Jakob

2 1

dovecot.index.pvt reset, view is now inconsistent
by Rupert Gallagher 28 Feb '18

28 Feb '18

Problem solved by going in manually. The log message appears for empty "public" folders. Say, you have a folder X with subfolder Y, where X does not contain any e-mail. The log message disappears if you drop an email into X, then remove it. Puf, gone! So, there seems to be a baby bug in how dovecot manages the index in this case.

3 2

BUG: Error: dovecot.index.pvt reset, view is now inconsistent when shared folder is new and empty
by Marco Giunta 28 Feb '18

28 Feb '18

Hi, I'm using Dovecot 2.2.33.2 on a RHEL 7, new installation. My log is full of : Error: INDEX_FOLDER/dovecot.index.pvt reset, view is now inconsistent or Error: INDEX_FOLDER/dovecot.index.pvt view is inconsistent when shared folder is never touched and empty. UserA share XXXXX folder with UserB, if XXXXX folder is new (never touched) and empty every time UserB looks in that folder, an error appears in log file. If UserA copy a mail in XXXXX folder, no more errors. If UserA (or UserB) delete all mails in XXXXX folder (the folder is empty again), no more errors. So the errors appear when UserB access a new (never touched) shared empty folder; if the folder is empty, but not new (p.e. UserA has already copied and deleted mails in that folder) error is logged only once. Attached my configuration. Thanks, Marco -- Marco Giunta - ITCS SysAdmin Via Bonomea, 265 34136 - Trieste, Italy Tel: +39-040-3787-503 Fax: +39-040-3787-244

1 0

Out of memory on lmtp vsz_limit
by Terence Lau 28 Feb '18

28 Feb '18

Hi, We've been getting these types or errors for quite a while now ... Fatal: master: service(lmtp): child 63477 returned error 83 (Out of memory (service lmtp { vsz_limit=4096 MB }, you may need to increase it) ... and these errors have been decreasing in occurrence as we increased the default_vsz_limit. Which is good but I would like to get some advice on how I could possibly eliminate the errors. We have an internal smtp server (postfix 3.1.0) that has the config "always_bcc=archive(a)company.com" over lmtp. This mailbox is on a separate dovecot server with the following config (please let me know if the full config is required): # 2.2.22 (fe789d2): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.13 (7b14904) # OS: Linux 4.4.0-109-generic x86_64 Ubuntu 16.04.1 LTS ext4 default_vsz_limit = 4 G mail_location = maildir:/home/vmail/%d/%n protocols = " imap lmtp pop3" service lmtp { inet_listener lmtp { port = 24 } } userdb { args = username_format=%u /etc/dovecot/users default_fields = uid=vmail gid=vmail home=/home/vmail/%d/%n driver = passwd-file } protocol lmtp { mail_plugins = } Since we discovered the errors, we've been increasing the default_vsz_limit to 1G, then 2G and now 4G (Server has 6GB of memory). These errors occur whenever a large number of emails get sent around the same time to our smtp server. This causes the dovecot server to start crunching CPU and Memory. Load average goes through the roof and takes some time to come back down as the smtp queue clears itself. This mailbox is obviously very large but we have a script that runs daily to delete any emails older than a month: find /home/vmail/company.com/archive/new/ -type f -mtime +30 -exec rm {} \; find /home/vmail/company.com/archive/cur/ -type f -mtime +30 -exec rm {} \; Still, the mailbox has on average of just under 300,000 emails. No one accesses this mailbox with an email client, not until we need to dig something up. And this has only happen once. So the emails pretty much never get read/process by a user. Now that we've increased the default_vsz_limit to 4G, the occurrence of these errors has reduced considerably. But they still happen occasionally. Short of increasing the memory further, are there any other options I have? Thanks. [https://www.royhill.com.au/email/Emalisignaturecurrent.jpg]

2 3

use IMAP and POP3 simultaneously (single inbox)
by Stanislaw Findeisen 28 Feb '18

28 Feb '18

Hi Is it safe to use IMAP and POP3 simultaneously to access the same inbox (using Maildir structure)? Thanks! Stanisław

2 1