November 2019 - dovecot

Using attachment_dir with plugin zlib corrupt mails
by MAREN ZUBIZARRETA 06 Nov '19

06 Nov '19

Hello: I have found the same problem reported above by Patrick Cernko affecting our system and corrupting our messages. Even worse, Outlook 2016 will no synchronize and the clients cannot see any message, even if there is only one corrupted mail per mailbox. I cannot figure out a feasible workaround for our system, and I can see that in new version 2.38 the bug is not fixed. Will this issue be treated soon? Thanks a lot Maren Zubizarreta WARNING: using attachment_dir with plugin zlib can corrupt mails Patrick Cernko pcernko at mpi-klsb.mpg.de<mailto:dovecot%40dovecot.org?Subject=Re:%20Re%3A%20WARNING%3A%20using%20attachment_dir%20with%20plugin%20zlib%20can%20corrupt%20mails&In-Reply-To=%3C1ab8edcc-bef4-3b41-d09a-c523674a5031%40mpi-klsb.mpg.de%3E> Fri Jul 19 17:52:37 EEST 2019 * Previous message: index worker 2.3.7 undefined symbol errors<https://dovecot.org/list/dovecot/2019-July/116617.html> * Next message: Address family not supported by protocol<https://dovecot.org/list/dovecot/2019-July/116571.html> * Messages sorted by: [ date ]<https://dovecot.org/list/dovecot/2019-July/date.html#116570> [ thread ]<https://dovecot.org/list/dovecot/2019-July/thread.html#116570> [ subject ]<https://dovecot.org/list/dovecot/2019-July/subject.html#116570> [ author ]<https://dovecot.org/list/dovecot/2019-July/author.html#116570> ________________________________ Hello list, hello Dovecot developers, this week, I discovered a serious bug in Dovecot, that lead to several broken mails on our servers. The bug corrupts the first few characters of the mail header during saving. On our setup, it was almost always only the very first line of text, that was corrupted. Depending on the IMAP client (they seem to request different header fields, ... during mail access), the bug causes the imap process to hang up the TCP connection and log errors like this: > imap(USERNAME)<4767><TeQP4ASOTK5/AAAB>: Error: Corrupted record in index cache file /IMAP/mail/mailboxes/USERNAME/mdbox/mailboxes/Trash/dbox-Mails/dovecot.index.cache: UID 489113: Broken fields in mailbox Trash: read(attachments-connector(zlib(/IMAP/mail/mailboxes/USERNAME/mdbox/storage/m.813))): FETCH BODY[HEADER.FIELDS (RETURN-PATH SUBJECT)] got too little data: 2 vs 122 In our case that finally grabbed my attention, the client was the users iphone that did not display any new messages but his Thunderbird did. The bug seems to be triggered by a bad "interaction" of attachment_dir option and zlib plugin. If you use both, you most likely are affected, too, except you only use zlib plugin for reading previously compressed stored mails. That's also the workaround we use now: zlib plugin only enabled in mail_plugins but no plugin/zlib_save set. The bug occurs on very specific mails. Due to privacy reasons I could not provide sample mails here. Storing such mails seems to trigger the bug reproducible. I attached a very minimal doveconf -n config, that can be used to trigger the bug. If one of the developers is interested, I can try to generate an "anonymized" version of such a specific mail that still causes the issue. I discovered the bug on our productive systems, running latest Dovecot 2.2 release, but the latest 2.3 I used during debugging is affected, too. During debugging, I also found one hint, that might help find the bug: If you store a problematic mail with zlib_save=gz (or zlib_save=bz2) and then disable the zlib plugin in mail_plugins, you can call doveadm fetch -u test hdr all | grep -v ^hdr: | gzip --decompress on test's mailbox with only that one broken mail. This will display the beginning of the rfc822 mail text until gzip terminates with "gzip: stdin: unexpected end of file", approximately after twice the length of the mail HEADER. This might indicate, that dovecot stores the uncompressed size of the header in it's data structures although the mail is stored compressed. I also found a very efficient way to find all affected mails in our setup: doveadm -f flow fetch -A 'user guid mailbox uid seq flags hdr' all | \ grep -a "^[^ ]+ user=" | \ grep -avF ' hdr=Return-path: ' | \ grep -av '.* hdr=[[:print:][:space:]]*$' (runtime for ~6M mails on our servers was 20-30min) This can be even more optimized if you have a powerful storage system with GNU parallel: > doveadm user '*' | parallel "doveadm -f flow fetch -u '{}' 'user guid mailbox uid seq flags hdr' all | grep -a '^user=' | grep -avF ' hdr=Return-path: ' | grep -av '.* hdr=[[:print:][:space:]]*$' || true" (runtime for ~6M mails on our servers was ~4min) The command will give you a list of mails that possibly are affected, check the full output of doveadm fetch -u USERNAME hdr guid GUID | less to verify that the header is really broken. On our systems I found 39 mails within ~12M mails. I was able to recover these mails "manually" by reconstructing the Return-Path header line, importing the fixed mails and expunging the corrupt ones. Before importing, I had to disable zlib_save option obviously. Best regards, -- Patrick Cernko <pcernko at mpi-klsb.mpg.de<https://dovecot.org/mailman/listinfo/dovecot>> +49 681 9325 5815 Joint Administration: Information Services and Technology Max-Planck-Institute fuer Informatik & Softwaresysteme

6 11

Trying to replace anti spam plugin with IMAPSieve; no joy
by Brad Koehn 06 Nov '19

06 Nov '19

I’m trying to follow the instructions on the wiki to move from antispam to IMAPSieve, but it’s not working at all. A bit of a background: dovecot-core 2:2.3.8-4~buster dovecot-imapd 2:2.3.8-4~buster dovecot-lmtpd 2:2.3.8-4~buster dovecot-managesieved 2:2.3.8-4~buster dovecot-pgsql 2:2.3.8-4~buster dovecot-pop3d 2:2.3.8-4~buster dovecot-sieve 2:2.3.8-4~buster dovecot-solr 2:2.3.8-4~buster My regular sieve scripts work fine for messages arriving through LMTP. I’ve updated my 20-imap.conf thusly: ## ## IMAP specific settings ## protocol imap { mail_max_userip_connections = 30 # Space separated list of plugins to load (default is global mail_plugins). mail_plugins = $mail_plugins imap_acl imap_sieve } plugin { sieve_plugins = sieve_imapsieve sieve_extprograms sieve_trace_dir = /tmp sieve_trace_level = matching # From elsewhere to Spam folder imapsieve_mailbox1_name = Spam imapsieve_mailbox1_causes = COPY imapsieve_mailbox1_before = file:/usr/lib/dovecot/sieve/report-spam.sieve # From Spam folder to elsewhere imapsieve_mailbox2_name = * imapsieve_mailbox2_from = Spam imapsieve_mailbox2_causes = COPY imapsieve_mailbox2_before = file:/usr/lib/dovecot/sieve/report-ham.sieve sieve_pipe_bin_dir = /usr/lib/dovecot/sieve sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment } ======== I created the report-spam.sieve and report-ham.sieve scripts in /usr/lib/dovecot/sieve and compiled them with sievec: require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment", "variables"]; if environment :matches "imap.mailbox" "*" { set "mailbox" "${1}"; } if string "${mailbox}" "Trash" { stop; } pipe :copy "sa-learn-pipe.sh" [ "ham" ]; ======== require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment", "variables"]; pipe :copy "sa-learn-pipe.sh" [ "spam" ]; ======== Lastly, I have the sa-learn-pipe.sh script in /usr/lib/dovecot/sieve: #!/bin/bash echo "`date` learning $*" >> /tmp/sa-learn # We run amavisd-new on another Kubernetes pod and communicate with it via socat on both ends; this worked with the old antispam plugin. if [ "$*" = "ham" ] ; then socat - TCP:amavis:4443 else socat - TCP:amavis:4444 fi exit 0 ======== The /usr/lib/dovecot/sieve directory looks like this: drwxr-xr-x 1 root root 4096 Nov 4 20:49 . drwxr-xr-x 1 root root 4096 Nov 3 20:06 .. -rwxr-xr-x 1 root root 237 Nov 3 15:56 report-ham.sieve -rw-r--r-- 1 root root 384 Nov 4 20:49 report-ham.svbin -rwxr-xr-x 1 root root 122 Nov 3 15:56 report-spam.sieve -rw-r--r-- 1 root root 290 Nov 4 20:49 report-spam.svbin -rwxr-xr-x 1 root root 208 Nov 3 17:58 sa-learn-pipe.sh When the sieve scripts run when mail arrives during LMTP delivery they dutifully log their work in /tmp, so I know sieve works and sieve logging works too. The sieve scripts above never log anything, nor does /tmp/sa-learn ever get created as it would if sa-learn-pipe.sh were ever run. As far as I can tell I’ve duplicated the configuration from the wiki almost exactly, but dovecot is just not running the sieve scripts when messages are moved into/out of Junk. What am I missing?

2 1

newbie question on a dovecot buffer
by Richard Bown 06 Nov '19

06 Nov '19

Hi apologies for what will seem a daft question I'm getting problems with my mail provider who is using dovecot. I've loaded Dovecot on a 8 core ARMHF SBC version 2.2.33.2 on Ubuntu 18.04. The SBC is headless. and the load looks very light and only needs to store/cache 4GB of mail Can I use Dovecot as a buffer between my mail providers Dovecot servers and the several IMAP mail clients on my own network ? ie, so Dovecot would mirror the mail on my account with my mail provider, So my local dovecot server downloads mail and caches/stores on the local SBC and all the local users use IMAP from that, instead of from my mail provider. Can I import the same directory structure as help on my mail providers server ? I'm trying to avoid the timeout problem I get when using IMAP from my mail provider. Can it be done ? and can you withstand the questions on configuration, I haven't used Dovecot before so its a vertical learning curve Thanks -- Best wishes /73 Richard Bown Email : richard(a)g8jvm.com HTTP : http://www.g8jvm.com ###################################################################### Ham Call: G8JVM . QRV: 50-432 MHz + Microwave Maidenhead QRA: IO82SP38, LAT. 52 39.720' N LONG. 2 28.171 W QRV 6mtrs 200W, 4mtrs 150W, 2mtrs 300W, 70cms 200W, Microwave 1296MHz 110W, 2320MHz 100W, 5760MHz 10W & 10368MHz 5W OS: Linux Mint 19.2 x86_64 Tina, on a Dell Inspiron 3580 laptop ###################################################################### Come back Guy Fawkes, all is forgiven

3 3

sievec *.sieve problem.
by Reio Remma 05 Nov '19

05 Nov '19

Hello! For the second time I've tripped onto this banana peel. :) I had 2 sieve files in a directory that I wanted to compile: sievec *.sieve The result of this is that first.sieve is compiled into second.sieve instead of first.svbin, thus destroying the source of second.sieve. Please consider this a bug report. :) Good luck, Reio

2 2

Help with master user
by Marcio Merlone 04 Nov '19

04 Nov '19

Hi, Trying to implement a master user (1) for auditing purposes without luck. Ubuntu 18.04.3, canonical official repos only, no ppa nor self-compiled anything. From the log below I understand the master password succeeds but AD auth fails. I am pretty sure I missed something here. Also, notice the messages "Ignoring unknown passdb extra field: original_user". Log: Nov 1 14:02:32 netuno dovecot: auth: Debug: client in: AUTH#0112#011PLAIN#011service=imap#011secured#011session=H2WM7kuWFKYKCQgI#011lip=10.9.8.8#011rip=10.9.8.8#011lport=143#011rport=42516#011resp=<hidden> Nov 1 14:02:32 netuno dovecot: auth: Debug: passwd-file(master,10.9.8.8,master,<H2WM7kuWFKYKCQgI>): Master user lookup for login: test.account Nov 1 14:02:32 netuno dovecot: auth: Debug: passwd-file(master,10.9.8.8,master,<H2WM7kuWFKYKCQgI>): lookup: user=master file=/etc/dovecot/master-users Nov 1 14:02:32 netuno dovecot: auth: passwd-file(master,10.9.8.8,master,<H2WM7kuWFKYKCQgI>): Master user logging in as test.account Nov 1 14:02:32 netuno dovecot: auth: ldap(test.account,10.9.8.8,<H2WM7kuWFKYKCQgI>): invalid credentials Nov 1 14:02:34 netuno dovecot: auth: Debug: client passdb out: FAIL#0112#011user=test.account#011authz#011original_user=master#011auth_user=master Nov 1 14:02:34 netuno dovecot: imap-login: Debug: Ignoring unknown passdb extra field: original_user Nov 1 14:02:34 netuno dovecot: imap-login: Debug: Ignoring unknown passdb extra field: auth_user Nov 1 14:02:42 netuno dovecot: imap-login: Aborted login (auth failed, 1 attempts in 10 secs): user=<test.account>, method=PLAIN, rip=10.9.8.8, lip=10.9.8.8, secured, session=<H2WM7kuWFKYKCQgI> doveconf -n: https://pastebin.com/3cAvfNqB root@netuno:/etc/dovecot# grep -v "^\s*#\|^\s*$" /etc/dovecot/dovecot-ldap.conf.ext hosts = ad.example.net auth_bind = yes auth_bind_userdn = cn=%Lu,CN=Users,DC=ad,DC=example,DC=net base = DC=ad,DC=example,DC=net scope = base user_attrs = \ =home=/mnt/maildirs/%Lu, \ =uid=vmail,\ =gid=vmail user_filter = (&(objectClass=person)(uid=%Lu)(mail=*(a)example.net)) root@netuno:/etc/dovecot# root@netuno:/etc/dovecot# cat /etc/dovecot/global-acls * user=master lr root@netuno:/etc/dovecot# (1) https://doc.dovecot.org/configuration_manual/authentication/master_users/ Best regards -- *Marcio Merlone*

1 1

Bug report
by Peter Nabbefeld 02 Nov '19

02 Nov '19

Hello, I've got a problem fetching my emails using getmail. First question is, of course, what's causing the problems. I'm going to think that getmail get's an error code from dovecot when trying to connect and just signals the most common error, so it seems to be a problem with dovecot. Sorry but I cannot upload any files with the silly web interface of my provider, so I cannot attach a core dump here. :-( Kind regards Peter ~/.getmail/log ========= 2019-11-01 21:44:20 Delivery error (command deliver 42245 error (127, exec of command deliver failed (change UID/GID to vmail/vmail failed ([Errno 1] Operation not permitted)))) dovecot-info.log ========== Oct 31 20:27:00 lda(mein.name(a)gmx.de)<42409><2+iBJIQ1u12ppQAA7IIYzw>: Panic: file istream-crlf.c: line 24 (i_stream_crlf_read_common): assertion failed: (ret != -2) Oct 31 20:27:00 lda(mein.name(a)gmx.de)<42409><2+iBJIQ1u12ppQAA7IIYzw>: Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0xff784) [0x7f4ec8e62784] -> /usr/lib/dovecot/libdovecot.so.0(+0xff7c1) [0x7f4ec8e627c1] -> /usr/lib/dovecot/libdovecot.so.0(+0x493aa) [0x7f4ec8dac3aa] -> /usr/lib/dovecot/libdovecot.so.0(+0x4bf62) [0x7f4ec8daef62] -> /usr/lib/dovecot/libdovecot.so.0(+0x111076) [0x7f4ec8e74076] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read_memarea+0x81) [0x7f4ec8e70291] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read+0x37) [0x7f4ec8e70477] -> /usr/lib/dovecot/libdovecot.so.0(+0x1164a8) [0x7f4ec8e794a8] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read_memarea+0x81) [0x7f4ec8e70291] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read+0x37) [0x7f4ec8e70477] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read_data+0x4c) [0x7f4ec8e70f3c] -> /usr/lib/dovecot/libdovecot.so.0(io_stream_copy+0x8a) [0x7f4ec8e8bbea] -> /usr/lib/dovecot/libdovecot.so.0(+0x12aa86) [0x7f4ec8e8da86] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_send_istream+0x4f) [0x7f4ec8e8b86f] -> /usr/lib/dovecot/libdovecot-storage.so.0(index_storage_save_continue+0x2e) [0x7f4ec901114e] -> /usr/lib/dovecot/libdovecot-storage.so.0(maildir_save_continue+0x27) [0x7f4ec8fafbe7] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_save_continue+0x47) [0x7f4ec8f815b7] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_storage_copy+0xe8) [0x7f4ec8f72388] -> /usr/lib/dovecot/libdovecot-storage.so.0(maildir_copy+0x6d) [0x7f4ec8fab99d] -> /usr/lib/dovecot/libdovecot-lda.so.0(+0x3c46) [0x7f4ec90a2c46] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0x11004) [0x7f4ec8af0004] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x59b0c) [0x7f4ec8f81b0c] -> /usr/lib/dovecot/libdovecot-sieve.so.0(+0x4edb8) [0x7f4ec67e0db8] -> /usr/lib/dovecot/libdovecot-sieve.so.0(sieve_result_execute+0x54d) [0x7f4ec67d65cd] -> /usr/lib/dovecot/libdovecot-sieve.so.0(+0x57a79) [0x7f4ec67e9a79] -> /usr/lib/dovecot/libdovecot-sieve.so.0(sieve_multiscript_run+0xa9) [0x7f4ec67eac49] -> /usr/lib/dovecot/modules/lib90_sieve_plugin.so(+0x376b) [0x7f4ec685c76b] -> /usr/lib/dovecot/libdovecot-lda.so.0(mail_deliver+0x199) [0x7f4ec90a46d9] dovecot -n ======= # 2.3.8 (9df20d2db): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.8 (b7b03ba2) # OS: Linux 5.3.7-arch1-2-ARCH x86_64 Arch Linux # Hostname: mein.host.name auth_verbose = yes disable_plaintext_auth = no info_log_path = /var/log/dovecot-info.log log_path = /var/log/dovecot.log mail_location = maildir:~/Maildir:LAYOUT=fs mail_plugins = " fts fts_solr" managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext editheader namespace inbox { inbox = yes location = mailbox Entwürfe { special_use = \Drafts } mailbox Gesendet { special_use = \Sent } mailbox Spamverdacht { special_use = \Junk } mailbox Trash { special_use = \Trash } prefix = } passdb { args = username_format=%u /etc/dovecot/passwd driver = passwd-file } plugin { fts = solr fts_solr = url=http://localhost:8983/solr/ sieve = file:/home/vmail/sieve;active=/home/vmail/.dovecot.sieve sieve_extensions = +editheader } service stats { unix_listener stats-reader { group = vmail mode = 0666 user = vmail } unix_listener stats-writer { group = vmail mode = 0666 user = vmail } } ssl = no ssl_dh = # hidden, use -P to show it userdb { args = uid=vmail gid=vmail home=/home/vmail/%d/%n driver = static } protocol lda { mail_plugins = " fts fts_solr sieve" postmaster_address = mein.name(a)gmx.de }

3 2

warning: NFS hangs with dovecot 2.3.8 on Debian buster
by Jan-Pieter Cornet 01 Nov '19

01 Nov '19

A warning to those considering to upgrade to Debian 10 (buster): we have seen occasional NFS hangs with dovecot when using the stock debian buster kernel (4.19.67-2+deb10u1). When we downgrade to the debian stretch kernel (4.9.189-3+deb9u1), the issue does not occur. Note that we *only* downgraded the kernel, the rest of the OS is still debian buster. Dovecot 2.3.8. A little more info: we have a dovecot cluster, using mdbox for storage, on an NFS server (netapp Cmode version 9.6P2). We use a dovecot director layer, so a user is always connected to the same back-end dovecot server. The NFS hang occurs on the back-end server. Once the process hangs, other processes trying to write to the same mailbox, will get an error like this: Timeout (180s) while waiting for lock for transaction log file /var/mail/.../index/storage/dovecot.map.index.log (WRITE lock held by pid XXXX) The stuck process itself doesn't seem to do anything, is stuck in "D" disk state, "strace" doesn't show anything (and after attaching, strace itself needs a kill -KILL to stop). The only way to unwedge the process seems to be to do a kill -KILL of the stuck process. Reading from the mailbox is still possible. We are in the process of contacting the linux-nfs folks about this, but I'm trying to reproduce this on a test-cluster first, to be able to present a well-documented case. Since this hang doesn't happen immediately, but takes a few hours to a day to occur in the wild, or a few thousand writes to the same mailbox, it's a bit hard to debug. -- Jan-Pieter Cornet <johnpc(a)xs4all.net> Systeembeheer XS4ALL Internet bv www.xs4all.nl

1 1

Dovecot and MySQL aborted connections.
by Benjamin Connelly 01 Nov '19

01 Nov '19

We also spotted these sql connections getting aborted, upon upgrading MySQL from 5.6 to 5.7. (Going back to 5.6 we don't see them!) Turning on mysql general query logging we can see it is Dovecot's mysql connections that inquire about or update quota usage in particular: *** /logs//mysql.log *** 2019-10-30T10:52:22.624690-07:00 2 Connect dovecot@localhost on npomail using Socket 2019-10-30T10:52:40.019780-07:00 2 Query SELECT bytes FROM quota2 WHERE username = 'a@bla' 2019-10-30T10:52:40.020948-07:00 2 Query SELECT messages FROM quota2 WHERE username = 'a@bla' 2019-10-30T10:53:40.113374-07:00 2 [Note] Aborted connection 2 to db: 'npomail' user: 'dovecot' host: 'localhost' (Got an error reading communication packets) Increasing mysqld's net_read_timeout and net_write_timeout values to 3600 does not stop the aborted connections. It seems asymptomatic. If the connection to mysql is still there to do quota operations, Dovecot seems to use it. If it's been aborted by the server, Dovecot seems to just (re)connect. . . But I'd love to help find the root or file a bug report or learn what it is we've configured wrong. . . Benjamin

4 5

dovecot and ldap
by Maciej Milaszewski IQ PL 01 Nov '19

01 Nov '19

Hi Sorry for my question... I use dovecot+ldap How realy works (in dovecot-2.2.x ) lists of LDAP hosts to use ? ------ dovecot.conf ---- hosts = ldap.domain.pl:389 ldap-slave.domain.pl:389 #uris = -------- This is simpe HA ? I mean if ldap.domain.pl have problem another request go to ldap-slave.domain.pl ?

2 1