October 2019 - dovecot

dovecot sieve duplicates detection
by André Rodier 04 Dec '19

04 Dec '19

Hello, I have tested the sieve duplicate script with success so far, but I have a question. I would like to know if the "duplicate" sieve flag in Dovecot is global to all folders, or specific to one folder only. For instance, if I copy an email from one folder to another, and I have a discard action on duplicate email, is this action will be applied (in this case, discard) or not. If the duplicate is global to all folders, is there a way to restrict the search in one folder only. Thanks for your help. André

3 10

IMAP4 extensions for Visual Voicemail (VVM)
by Rajesh Bansal 15 Nov '19

15 Nov '19

Hi Team, I need to develop Visual VoiceMail solution. In this solution I need a IMAP4 server, from which I can get a hit for each mail retrieval. Can anyone help me if dovecot can be used for this purpose. BR, Rajesh Bansal

5 5

Sieve vacation :addresses match only case-sensitive?
by Philipp Faeustlin 11 Nov '19

11 Nov '19

The recipients in the vacation :addresses String list are matched case-sensitive. If the recipient gets a mail with wrong case, the message is discarded with: "discarding vacation response for implicitly delivered message; no known (envelope) recipient address found in message headers" I think the matching should be case-insensitive. Is this a bug or do I miss some option? I'm using Dovecot 2.3.7.2, Pigeonhole version 0.5.7.2. -- Philipp Fäustlin

5 9

User found but password failure
by Mauricio Tavares 11 Nov '19

11 Nov '19

Trying to figure out which step is causing me not to be able to login. I am using a password file, passdb { driver = passwd-file args = scheme=SHA512-CRYPT username_format=%u /etc/dovecot/passwd } We will assume that the pw I created using 'doveadm pw -s SHA512-CRYPT' matches the password I will be using to login below. I crank the debugging mode, auth_debug_passwords = yes auth_debug = yes And then try to login [root@mail ~]# nc -t localhost 143 * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS AUTH=PLAIN AUTH=LOGIN AUTH=GSSAPI] Howdy a login raub(a)example.com password [blank] which from what I gathered from /var/log/dovecot means it found matching username but did not match password: secured session=lyJttvyQ9I0AAAAAAAAAAAAAAAAAAAAB lip=::1 rip=::1 lport=143 rport=36340 resp=base64-reply (previous base64 data may contain sensitive data) Aug 26 03:25:52 auth: Debug: passwd-file(raub@example.com,::1,<lyJttvyQ9I0AAAAAAAAAAAAAAAAAAAAB>): lookup: user=raub(a)example.com file=/etc/dovecot/passwd Aug 26 03:25:52 auth: Debug: client passdb out: OK 1 user=raub(a)example.com Aug 26 03:25:52 auth: Debug: master in: REQUEST 1520959489 26560 1 63ee7f45236f85fd39573a5c8a2eb46a session_pid=26563 request_auth_token Aug 26 03:25:52 auth-worker(26562): Debug: passwd(raub@example.com,::1,<lyJttvyQ9I0AAAAAAAAAAAAAAAAAAAAB>): lookup Aug 26 03:25:52 auth-worker(26562): Info: passwd(raub@example.com,::1,<lyJttvyQ9I0AAAAAAAAAAAAAAAAAAAAB>): unknown user Aug 26 03:25:52 auth: Debug: passwd-file(raub@example.com,::1,<lyJttvyQ9I0AAAAAAAAAAAAAAAAAAAAB>): lookup: user=raub(a)example.com file=/etc/dovecot/passwd Aug 26 03:25:52 auth: Error: plain(raub@example.com,::1,<lyJttvyQ9I0AAAAAAAAAAAAAAAAAAAAB>): user not found from any userdbs Aug 26 03:25:52 auth: Debug: master userdb out: NOTFOUND 1520959489 Aug 26 03:25:52 imap: Error: Authenticated user not found from userdb, auth lookup id=1520959489 (auth connected 1 msecs ago, handshake 0 msecs ago, request took 1 msecs, client-pid=26560 client-id=1) Aug 26 03:25:52 imap-login: Info: Internal login failure (pid=26560 id=1) (internal failure, 1 successful auths): user=<raub(a)example.com>, method=PLAIN, rip=::1, lip=::1, mpid=26563, secured, session=<lyJttvyQ9I0AAAAAAAAAAAAAAAAAAAAB> Per [1] I decided to see what the response (base64-reply) I am submitting to dovecot looks like: echo 'base64-reply' | base64 -d raub(a)example.compassword Which has the right username and password but follows the userpassword format, not useruserpassword as I was led to believe by [1]. Is that to be expected? If it is then the issue is further down the line. [1] https://wiki.dovecot.org/Debugging/Authentication

3 5

Reading config-value from file does not work.
by Jakobus Schürz 08 Nov '19

08 Nov '19

Hi guys! In https://doc.dovecot.org/configuration_manual/config_file/config_file_syntax/ i ca see this: Reading value from file It’s possible to read the value for a setting from a file: key = </path/to/file The value is read exactly as the entire contents of the file. This includes all the whitespace and newlines. The paths are relative to the currently parsed config file’s directory, similar to how !include works. The file is read immediately whenever parsing the configuration file, so if it changes afterwards it requires a configuration reload to see the changes. This functionality is especially useful for reading SSL certificates and keys. So i thought it's a good idea, to write the password for dnpass for my ldap in this notation in my config dnpass = <private/dnpass /etc/dovecot/private/dnpass contains only this myVerYSecretLdApPAssworD no space, no newline. But dovecot cannot authenticate on the ldap-host. I also tried dnpass = </private/dnpass dnpass = </etc/dovecot/private/dnpass Then i tried a file /etc/dovecot/conf.d/00-dnpass.conf with dnpass = myVerYSecretLdApPAssworD It should be included with dovecot.conf !include conf.d/*.conf The only notation that works is the line dnpass = myVerYSecretLdApPAssworD in /etc/dovecot/dovecot-ldap-userdb.conf.ext I thought it's a good idea to write it in an extra file, so i have to manipulate only one small file with ansible, when i renew the password. Is it a bug or does it work as designed, and the doc is wrong? thank you Jakob

1 1

Using attachment_dir with plugin zlib corrupt mails
by MAREN ZUBIZARRETA 06 Nov '19

06 Nov '19

Hello: I have found the same problem reported above by Patrick Cernko affecting our system and corrupting our messages. Even worse, Outlook 2016 will no synchronize and the clients cannot see any message, even if there is only one corrupted mail per mailbox. I cannot figure out a feasible workaround for our system, and I can see that in new version 2.38 the bug is not fixed. Will this issue be treated soon? Thanks a lot Maren Zubizarreta WARNING: using attachment_dir with plugin zlib can corrupt mails Patrick Cernko pcernko at mpi-klsb.mpg.de<mailto:dovecot%40dovecot.org?Subject=Re:%20Re%3A%20WARNING%3A%20using%20attachment_dir%20with%20plugin%20zlib%20can%20corrupt%20mails&In-Reply-To=%3C1ab8edcc-bef4-3b41-d09a-c523674a5031%40mpi-klsb.mpg.de%3E> Fri Jul 19 17:52:37 EEST 2019 * Previous message: index worker 2.3.7 undefined symbol errors<https://dovecot.org/list/dovecot/2019-July/116617.html> * Next message: Address family not supported by protocol<https://dovecot.org/list/dovecot/2019-July/116571.html> * Messages sorted by: [ date ]<https://dovecot.org/list/dovecot/2019-July/date.html#116570> [ thread ]<https://dovecot.org/list/dovecot/2019-July/thread.html#116570> [ subject ]<https://dovecot.org/list/dovecot/2019-July/subject.html#116570> [ author ]<https://dovecot.org/list/dovecot/2019-July/author.html#116570> ________________________________ Hello list, hello Dovecot developers, this week, I discovered a serious bug in Dovecot, that lead to several broken mails on our servers. The bug corrupts the first few characters of the mail header during saving. On our setup, it was almost always only the very first line of text, that was corrupted. Depending on the IMAP client (they seem to request different header fields, ... during mail access), the bug causes the imap process to hang up the TCP connection and log errors like this: > imap(USERNAME)<4767><TeQP4ASOTK5/AAAB>: Error: Corrupted record in index cache file /IMAP/mail/mailboxes/USERNAME/mdbox/mailboxes/Trash/dbox-Mails/dovecot.index.cache: UID 489113: Broken fields in mailbox Trash: read(attachments-connector(zlib(/IMAP/mail/mailboxes/USERNAME/mdbox/storage/m.813))): FETCH BODY[HEADER.FIELDS (RETURN-PATH SUBJECT)] got too little data: 2 vs 122 In our case that finally grabbed my attention, the client was the users iphone that did not display any new messages but his Thunderbird did. The bug seems to be triggered by a bad "interaction" of attachment_dir option and zlib plugin. If you use both, you most likely are affected, too, except you only use zlib plugin for reading previously compressed stored mails. That's also the workaround we use now: zlib plugin only enabled in mail_plugins but no plugin/zlib_save set. The bug occurs on very specific mails. Due to privacy reasons I could not provide sample mails here. Storing such mails seems to trigger the bug reproducible. I attached a very minimal doveconf -n config, that can be used to trigger the bug. If one of the developers is interested, I can try to generate an "anonymized" version of such a specific mail that still causes the issue. I discovered the bug on our productive systems, running latest Dovecot 2.2 release, but the latest 2.3 I used during debugging is affected, too. During debugging, I also found one hint, that might help find the bug: If you store a problematic mail with zlib_save=gz (or zlib_save=bz2) and then disable the zlib plugin in mail_plugins, you can call doveadm fetch -u test hdr all | grep -v ^hdr: | gzip --decompress on test's mailbox with only that one broken mail. This will display the beginning of the rfc822 mail text until gzip terminates with "gzip: stdin: unexpected end of file", approximately after twice the length of the mail HEADER. This might indicate, that dovecot stores the uncompressed size of the header in it's data structures although the mail is stored compressed. I also found a very efficient way to find all affected mails in our setup: doveadm -f flow fetch -A 'user guid mailbox uid seq flags hdr' all | \ grep -a "^[^ ]+ user=" | \ grep -avF ' hdr=Return-path: ' | \ grep -av '.* hdr=[[:print:][:space:]]*$' (runtime for ~6M mails on our servers was 20-30min) This can be even more optimized if you have a powerful storage system with GNU parallel: > doveadm user '*' | parallel "doveadm -f flow fetch -u '{}' 'user guid mailbox uid seq flags hdr' all | grep -a '^user=' | grep -avF ' hdr=Return-path: ' | grep -av '.* hdr=[[:print:][:space:]]*$' || true" (runtime for ~6M mails on our servers was ~4min) The command will give you a list of mails that possibly are affected, check the full output of doveadm fetch -u USERNAME hdr guid GUID | less to verify that the header is really broken. On our systems I found 39 mails within ~12M mails. I was able to recover these mails "manually" by reconstructing the Return-Path header line, importing the fixed mails and expunging the corrupt ones. Before importing, I had to disable zlib_save option obviously. Best regards, -- Patrick Cernko <pcernko at mpi-klsb.mpg.de<https://dovecot.org/mailman/listinfo/dovecot>> +49 681 9325 5815 Joint Administration: Information Services and Technology Max-Planck-Institute fuer Informatik & Softwaresysteme

6 11

warning: NFS hangs with dovecot 2.3.8 on Debian buster
by Jan-Pieter Cornet 01 Nov '19

01 Nov '19

A warning to those considering to upgrade to Debian 10 (buster): we have seen occasional NFS hangs with dovecot when using the stock debian buster kernel (4.19.67-2+deb10u1). When we downgrade to the debian stretch kernel (4.9.189-3+deb9u1), the issue does not occur. Note that we *only* downgraded the kernel, the rest of the OS is still debian buster. Dovecot 2.3.8. A little more info: we have a dovecot cluster, using mdbox for storage, on an NFS server (netapp Cmode version 9.6P2). We use a dovecot director layer, so a user is always connected to the same back-end dovecot server. The NFS hang occurs on the back-end server. Once the process hangs, other processes trying to write to the same mailbox, will get an error like this: Timeout (180s) while waiting for lock for transaction log file /var/mail/.../index/storage/dovecot.map.index.log (WRITE lock held by pid XXXX) The stuck process itself doesn't seem to do anything, is stuck in "D" disk state, "strace" doesn't show anything (and after attaching, strace itself needs a kill -KILL to stop). The only way to unwedge the process seems to be to do a kill -KILL of the stuck process. Reading from the mailbox is still possible. We are in the process of contacting the linux-nfs folks about this, but I'm trying to reproduce this on a test-cluster first, to be able to present a well-documented case. Since this hang doesn't happen immediately, but takes a few hours to a day to occur in the wild, or a few thousand writes to the same mailbox, it's a bit hard to debug. -- Jan-Pieter Cornet <johnpc(a)xs4all.net> Systeembeheer XS4ALL Internet bv www.xs4all.nl

1 1

Dovecot and MySQL aborted connections.
by Benjamin Connelly 01 Nov '19

01 Nov '19

We also spotted these sql connections getting aborted, upon upgrading MySQL from 5.6 to 5.7. (Going back to 5.6 we don't see them!) Turning on mysql general query logging we can see it is Dovecot's mysql connections that inquire about or update quota usage in particular: *** /logs//mysql.log *** 2019-10-30T10:52:22.624690-07:00 2 Connect dovecot@localhost on npomail using Socket 2019-10-30T10:52:40.019780-07:00 2 Query SELECT bytes FROM quota2 WHERE username = 'a@bla' 2019-10-30T10:52:40.020948-07:00 2 Query SELECT messages FROM quota2 WHERE username = 'a@bla' 2019-10-30T10:53:40.113374-07:00 2 [Note] Aborted connection 2 to db: 'npomail' user: 'dovecot' host: 'localhost' (Got an error reading communication packets) Increasing mysqld's net_read_timeout and net_write_timeout values to 3600 does not stop the aborted connections. It seems asymptomatic. If the connection to mysql is still there to do quota operations, Dovecot seems to use it. If it's been aborted by the server, Dovecot seems to just (re)connect. . . But I'd love to help find the root or file a bug report or learn what it is we've configured wrong. . . Benjamin

4 5

dovecot and ldap
by Maciej Milaszewski IQ PL 01 Nov '19

01 Nov '19

Hi Sorry for my question... I use dovecot+ldap How realy works (in dovecot-2.2.x ) lists of LDAP hosts to use ? ------ dovecot.conf ---- hosts = ldap.domain.pl:389 ldap-slave.domain.pl:389 #uris = -------- This is simpe HA ? I mean if ldap.domain.pl have problem another request go to ldap-slave.domain.pl ?

2 1

Broken uids: Director lmtp proxy able to change the destination address?
by Erik de Waard 31 Oct '19

31 Oct '19

Hi, Director -> Backends -> NFS setup. Problem: Broken dovecot-uidlists because of our postfix LMTP delivery using a different address than the IMAP client lookups Error: Mailbox INBOX: Broken file dovecot-uidlist line 169: Invalid data: example: LMTP delivery: lmtp(info#domain.nl(a)mail.local) IMAP lookup: info(a)domain.nl doveadm director status info#domain.nl(a)mail.local Current: 10.0.10.223 (expires 2019-11-01 08:52:42) Hashed: 10.0.10.223 Initial config: 10.0.10.223 doveadm director status info(a)domain.nl Current: 10.0.10.113 (expires 2019-11-01 13:47:27) Hashed: 10.0.10.113 Initial config: 10.0.10.113 On the backends these would write to the same NFS maildir: Now Director would assign these addresses to different backend servers and thus result in the expected corrupted control files. at this time i'm unable to reconfigure the Postfix LMTP delivery in a way that these use the same lookup format. Questions: 1. Disable LMTP updating dovecot-uidlist and indexes. Perhaps using Maildir:CONTROL=/dev/null :INDEX=MEMORY for lmtp-sql.conf? 2. Since i'm using director/proxy is it possible to rewrite the address using SQL in the passdb query? 3. What would be the best settings to avoid corruption in this broken setup.

1 0