- dovecot - dovecot.org

corrupted mdbox
by sch tel 16 Sep '19

16 Sep '19

Hello list, i run into another problem. It seems there are some null bytes in the middle of an compressed mdbox, if i look into it with hexdump after run doveadm dump. With mdbox-recover.pl i could recover ~2k messages. What should i do to recover this broken mdbox file and dsync can finish? Sep 06 18:47:28 doveadm: Error: zlib.read(/var/spool/mail/<user>/storage/m.770): unexpected EOF at 7947710 Sep 06 18:47:28 doveadm: Error: dsync(server1): read(zlib(/var/spool/mail/<user>/storage/m.770)) failed: read(/var/spool/mail/<user>/storage/m.770) failed: zlib.read(/var/spool/mail/<user>/storage/m.770): unexpected EOF at 7947710 (last sent=mail, last recv=mail_request (EOL)) Sep 06 18:47:29 dsync-local(<user>)<+mJrOaCNcl2eZwAA0H466A>: Error: zlib.read(/var/spool/mail/<user>/storage/m.770): unexpected EOF at 7947710 Sep 06 18:47:29 dsync-local(<user>)<+mJrOaCNcl2eZwAA0H466A>: Error: dsync(server1): read(zlib(/var/spool/mail/<user>/storage/m.770)) failed: read(/var/spool/mail/<user>/storage/m.770) failed: zlib.read(/var/spool/mail/<user>/storage/m.770): unexpected EOF at 7947710 (last sent=mail, last recv=mail_request (EOL))

2 1

question about Mailcrypt plugin
by Daniel Niewerth 14 Sep '19

14 Sep '19

Hi everybody, I have a question about the Dovecot Mailcrypt Plugin. I'm trying to understand what the security of this plugin is based on. The encryption with private and public key and elliptic curves is basically a good thing. But the keys are in the file system together with the encrypted mails. The passwords for the private keys are in the user database. I guess the passwords have to be in plaintext for it to work. Is that true, or did I misunderstand something? Then the encryption would make no sense at all, right? For what purpose was the plugin developed? Can anyone explain this to me? Best regards Daniel

2 2

New to dovecot admin, question about using LDAP for user-specific values
by Gerben Wierda 13 Sep '19

13 Sep '19

I am new to dovecot administration. I’ve read the Wiki but that hasn’t given me the understanding I need. When I query my LDAP (on macOS) on a value for user ‘gerben’, I can get that: dumbledore:~ gerben$ dscl /LDAPv3/127.0.0.1 -read /users/gerben GeneratedUID GeneratedUID: 780D870E-6B00-478E-AB70-3D3307215A82 I would like to use that value in dovecot settings, e.g. something like user_attrs = \ =mail=maildir://Library/Server/Mail/Data/mail/%{ldap:GeneratedUID} Is this possible and if so what do I exactly need to do to get this working? Gerben Wierda Chess and the Art of Enterprise Architecture <https://ea.rna.nl/the-book/> Mastering ArchiMate <https://ea.rna.nl/the-book-edition-iii/> Architecture for Real Enterprises <https://www.infoworld.com/blog/architecture-for-real-enterprises/> at InfoWorld On Slippery Ice <https://eapj.org/on-slippery-ice/> at EAPJ

2 4

Multiple certificate option
by remo＠Mattei.org 13 Sep '19

13 Sep '19

What is the best way to adopt multiple certs? Thanks.

8 9

keywords lost when moving a message in a folder
by wouldsmina 13 Sep '19

13 Sep '19

Hi, I migrate my dovecot server from Debian 9 to Debian 10 (Dovecot 2.3.4.1), since when I move a message in a folder, keywords systematically disappear. I'm using Thunderbird 60.8. Would an option have escaped me? or is there a bug on this version of dovecot? Dovecot 2.3.4.1 OS : Debian 10.0 x64 FS : Ext4 dovecot -n result : # 2.3.4.1 (f79e8e7e4): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.4 () # OS: Linux 4.19.0-5-amd64 x86_64 Debian 10.0 # Hostname: suzuki.domain.tld auth_cache_size = 1 k auth_failure_delay = 3 secs base_dir = /var/run/dovecot/ default_vsz_limit = 4 G first_valid_uid = 100 lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes login_greeting = mail system ready. login_trusted_networks = 172.16.0.0/12 192.168.0.0/16 mail_fsync = always mail_location = mbox:~/mail:INBOX=/var/mail/%u mail_prefetch_count = 20 mail_privileged_group = mail mailbox_idle_check_interval = 1 mins mmap_disable = yes namespace inbox { inbox = yes location = maildir:~/Maildir mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = INBOX. separator = . subscriptions = yes type = private } passdb { driver = pam } plugin { acl_shared_dict = file:/var/lib/dovecot/shared-mailboxes sieve = ~/sieve/.dovecot.sieve sieve_after = /etc/dovecot/sieve.after sieve_before = /etc/dovecot/sieve.before sieve_before2 = ~/sieve/.dovecot.vacation.sieve sieve_dir = ~/sieve sieve_editheader_forbid_add = X-Verified sieve_editheader_forbid_delete = X-Verified X-Seen sieve_editheader_max_header_size = 1k sieve_extensions = +editheader sieve_max_script_size = 2M } postmaster_address = postmaster(a)domain.tld protocols = " imap lmtp pop3" service auth-worker { process_min_avail = 6 service_count = 0 } service imap-login { process_min_avail = 10 service_count = 0 vsz_limit = 4 G } service imap { process_limit = 4096 vsz_limit = 4 G } service pop3 { process_limit = 1024 } ssl_cert = </etc/ssl/certs/suzuki.crt ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it userdb { driver = passwd } verbose_proctitle = yes protocol imap { mail_max_userip_connections = 25 } thanks.

2 2

Outlook 2019 and IMAP
by John Regan 12 Sep '19

12 Sep '19

Hi, We've just upgraded a few of our desktops to use Outlook 2019, and the configuration is different from previous versions. There doesn't appear to be any ability to specify a username in addition to the user's email address. The username is different than the username specified in the email address. I had read about setting auth_username_format, but the user login name is "user" while the email address is user.name(a)company.com. I've included below my dovecot -n config. # 2.3.7.2 (3c910f64b): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.7.2 (7372921a) # OS: Linux 5.1.16-300.fc30.x86_64 x86_64 Fedora release 30 (Thirty) # Hostname: orion.inside.example.com mail_gid = mail mail_home = mail/ mail_location = mbox:~/mail:INBOX=/var/spool/mail/%u mail_max_userip_connections = 500 mail_plugins = " listescape" mail_privileged_group = mail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational rege x imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext mbox_write_locks = fcntl namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = pam } passdb { driver = pam } protocols = imap pop3 service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } } service imap-login { inet_listener imap { port = 0 } inet_listener imaps { port = 993 } process_min_avail = 6 service_count = 0 vsz_limit = 512 M } service pop3-login { inet_listener pop3 { port = 0 } inet_listener pop3s { address = 192.168.1.100 port = 995 } } ssl_cert = </etc/letsencrypt/fullchain.pem ssl_cipher_list = ALL:!LOW:!EXP:!aNULL:!RC4::!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it ssl_prefer_server_ciphers = yes userdb { driver = passwd } userdb { driver = passwd }

2 1

Re: Gnu sieve vs Dovecot sieve-filter - sieve-filter extremely slow at lda (writing emails to local mbox files)
by Zenaan Harkness 12 Sep '19

12 Sep '19

I am wondering why sieve-filter is so slow compared to gnu sieve. I run mpop (like getmail) to download from a pop3 server to a local mbox file: ~/mail/email-incoming-unsorted This step is very fast. The next step, I throw the email-incoming-unsorted mbox file at a sieve processor, to sort the emails from that mbox, into other mboxes, according to the sieve rules file. Up until a couple days ago I was using Gnu sieve. Gnu sieve balks on emails which have no x-message-id (?? something like this) header field, so after a few years, I finally decided to switch "up" to Dovecot/Pigeonhole's "sieve-filter" command. Using Gnu sieve, this mbox sorting step was even faster than mpop (/ getmail) - and mpop and getmail are really fast (compared with fetchmail), since they pipeline the email downloads. Even with 100s of emails, Gnu sieve would take only 10 to 20 seconds at most. Super fast. Using sieve-filter, all emails are being processed - including those without "message id header". This is good. But also, using sieve filter, is really slower - slower than the download step by an order of magnitude or two. See below for details, any ideas appreciated. To add to the below, I added: mbox_very_dirty_syncs = yes to the sieve-filter config, which slightly improves performance, but not by much (in comparison with Gnu sieve). TIA, ----- Forwarded message from Zenaan Harkness <zenaan(a)freedbms.net> ----- From: Zenaan Harkness <zenaan(a)freedbms.net> To: debian-user(a)lists.debian.org Date: Thu, 12 Sep 2019 08:06:12 +1000 Subject: Re: Gnu sieve vs Dovecot sieve-filter - sieve-filter extremely slow at lda (writing emails to local mbox files) On Thu, Sep 12, 2019 at 07:55:23AM +1000, Zenaan Harkness wrote: > Why is Gnu sieve so extremely fast to batch process an mbox file, but > while Dovecot's sieve-filter is an order of magnitude slower? > > Sequence: > > - mpop or getmail to pipeline download emails into temp mbox file > - filter that file > > Gnu sieve just flies through a local mbox file and saving emails to > other local mbox files. > > Gnu sieve rejects too many emails with "malformed" errors, so after a > few years I bit the bullet and upgraded to Dovecot's sieve-filter. > > Dovecot's sieve-filter, at present, is an order of magnitude slower. > > Here's my filter command (one line): > > /usr/bin/sieve-filter -veW -c $HOME/etc/email/sieve-dovecot-config.conf -o mail_location=mbox:~/mail:INBOX=~/mail/Inbox:INDEX=:UTF-8:VOLATILEDIR=/tmp/dovecot-volatile/%2.256Nu/%u:SUBSCRIPTIONS=dovecot_subscriptions ~/etc/email/sieve.rc email-incoming-unsorted > > The sieve script is fine now that I have the correct "require" > clauses (hint: "capability strings"). > > File ~/etc/email/sieve-dovecot-config.conf: > > protocols = pop > lda_mailbox_autocreate = yes > lda_mailbox_autosubscribe = yes > mail_fsync = never > > There's no re-sending of emails into my local Postfix SMTP server - I > checked the system logs and confirmed this (journalctl -f). > > I suspect that Gnu sieve was directly writing each email to the > appropriate sieve-determined mbox file (perhaps with only a sync at > the end of a single batch process - what I've attempted to achieve > above with sieve-filter), and that sieve-filter is instead passing > each email through some (dovecot) lda? > > Here's the output for a sieve-filter batch processing of 11 emails: > > $ /usr/bin/sieve-filter -veW -c /home/zen/etc/email/sieve-dovecot-config.conf -o mail_location=mbox:/home/zen/mail:INBOX=/home/zen/mail/Inbox:INDEX=:UTF-8:VOLATILEDIR=/tmp/dovecot-volatile/%2.256Nu/%u:SUBSCRIPTIONS=dovecot_subscriptions /home/zen/etc/email/sieve.rc email-incoming-unsorted > # PS0 Timestamp: 20190912@07:02:23 > info: filtering: [Tue, 3 Sep 2019 05:17:16 -0500; 10240 bytes] `Re: VentureBeat: The death of disk? H...'. > info: msgid=<CAMjeLr91T9R7APsuxQVuM3WbqDsxAfwn4=OYDeDX4FMcoRdGdQ(a)mail.gmail.com>: stored mail into mailbox 'l/cp/cp'. > info: message expunged from source mailbox upon successful move. > info: filtering: [Tue, 3 Sep 2019 07:29:53 -0400; 12968 bytes] `[zfs-devel] xattr naming format in Zo...'. > info: msgid=<15675101930.d5ba2E.12322(a)composer.zfsonlinux.topicbox.com>: stored mail into mailbox 'l/z/zdev'. > info: message expunged from source mailbox upon successful move. > info: filtering: [Tue, 03 Sep 2019 15:29:09 +0300; 20461 bytes] `Re: [zfs-devel] xattr naming format i...'. > info: msgid=<23955051567513749(a)sas1-02732547ccc0.qloud-c.yandex.net>: stored mail into mailbox 'l/z/zdev'. > info: message expunged from source mailbox upon successful move. > info: filtering: [Tue, 3 Sep 2019 18:20:42 +0530; 18065 bytes] `Re: [Gluster-users] Issues with Geo-r...'. > info: msgid=<CADmkyZMxrfOANrAP+_URAHJcMqCqh=iGdajTSzkfQ5PCZsUfyg(a)mail.gmail.com>: stored mail into mailbox 'l/gl/user'. > info: message expunged from source mailbox upon successful move. > info: filtering: [Tue, 3 Sep 2019 09:34:20 -0400; 13342 bytes] `Re: tasksel'. > info: msgid=<20190903133420.GS6166(a)eeg.ccf.org>: stored mail into mailbox 'l/deb/user'. > info: message expunged from source mailbox upon successful move. > info: filtering: [Tue, 3 Sep 2019 06:56:07 -0700 (PDT); 12390 bytes] `[awx-project] Re: AWX on Kubernetes m...'. > info: msgid=<0715adb7-540f-4cff-9282-e1252c53c2e8(a)googlegroups.com>: stored mail into mailbox 'l/ansible/awx'. > info: message expunged from source mailbox upon successful move. > info: filtering: [Tue, 3 Sep 2019 07:01:27 -0700 (PDT); 12220 bytes] `[awx-project] Re: AWX on Kubernetes m...'. > info: msgid=<949b2c17-4254-49f1-83b4-cd54d15aa17d(a)googlegroups.com>: stored mail into mailbox 'l/ansible/awx'. > info: message expunged from source mailbox upon successful move. > info: filtering: [Tue, 3 Sep 2019 10:14:58 -0400; 25313 bytes] `Re: [zfs-devel] xattr naming format i...'. > info: msgid=<CAB5c7xpHCdFx1w3yA9FyRL-KQ8BUiCr4JbiDQRuFJj9nOgKxTg(a)mail.gmail.com>: stored mail into mailbox 'l/z/zdev'. > info: message expunged from source mailbox upon successful move. > info: filtering: [Tue, 3 Sep 2019 17:10:22 +0200; 7567 bytes] `Re: [asterisk-users] Playing MP3's in...'. > info: msgid=<20190903151022.354xpe6ds2vglher(a)red.localdomain>: stored mail into mailbox 'l/as/users'. > info: message expunged from source mailbox upon successful move. > info: filtering: [Wed, 4 Sep 2019 01:04:49 +0900; 14858 bytes] `Re: [Hyperledger Fabric] a primitive ...'. > info: msgid=<160901d8-b903-9e9a-91ac-267571b0e24d(a)gmx.com>: stored mail into mailbox 'l/hl/fabric'. > info: message expunged from source mailbox upon successful move. > info: filtering: [Tue, 3 Sep 2019 09:55:22 -0700 (PDT); 13337 bytes] `[awx-project] Re: AWX on Kubernetes m...'. > info: msgid=<f9bc4e6a-8445-4b34-927a-35f577ffcc07(a)googlegroups.com>: stored mail into mailbox 'l/ansible/awx'. > info: message expunged from source mailbox upon successful move. > 2 ▶︎️ zen@eye 20190912@07:02:30 ~ $ > > > So about 3/4 of a second is spent by dovecot's sieve-filter, on each > email that it processes - watching it is painful given how fast Gnu > sieve has been for the last few years - it's almost (but not quite) > as slow as my previous fetchmail email download per-email time. > > Attached is a -D debug run of sieve-filter on 20 emails - slightly > longer than the above, and took roughly 15 seconds to run. > > Any help appreciated... On another test run of ~600 emails, sieve-filter is consistently running ~100% of one CPU (for about 4 minutes) to process these emails, which leads to the conclusion that despite what looks like should be a batch process, sieve-filter is perhaps reloading the rules for every single email that it processes, even though I gave it a whole mbox, and not a single email, to process. Can sieve-filter work the way it should / the way I want it / batch process a whole mbox - without reloading the sieve rules for every email? ----- End forwarded message -----

3 2

Re: [Bug] Sieve vacation :addresses match only,> case-sensitive?
by Klaus Steinberger 12 Sep '19

12 Sep '19

> I want to have it case-insensitive again, like in Pigeonhole version 0.4.24 i would also plea for making it case-insensitive again! We were hit by this too, and I never saw any mail system in which the local Part ist Case sensitive! Sincerly, Klaus Steinberger -- Rechnerbetriebsgruppe / IT, Fakultät für Physik Klaus Steinberger FAX: +49 89 28914280 Tel: +49 89 28914287

6 5

WARNING: using attachment_dir with plugin zlib can corrupt mails
by Patrick Cernko 12 Sep '19

12 Sep '19

Hello list, hello Dovecot developers, this week, I discovered a serious bug in Dovecot, that lead to several broken mails on our servers. The bug corrupts the first few characters of the mail header during saving. On our setup, it was almost always only the very first line of text, that was corrupted. Depending on the IMAP client (they seem to request different header fields, ... during mail access), the bug causes the imap process to hang up the TCP connection and log errors like this: > imap(USERNAME)<4767><TeQP4ASOTK5/AAAB>: Error: Corrupted record in index cache file /IMAP/mail/mailboxes/USERNAME/mdbox/mailboxes/Trash/dbox-Mails/dovecot.index.cache: UID 489113: Broken fields in mailbox Trash: read(attachments-connector(zlib(/IMAP/mail/mailboxes/USERNAME/mdbox/storage/m.813))): FETCH BODY[HEADER.FIELDS (RETURN-PATH SUBJECT)] got too little data: 2 vs 122 In our case that finally grabbed my attention, the client was the users iphone that did not display any new messages but his Thunderbird did. The bug seems to be triggered by a bad "interaction" of attachment_dir option and zlib plugin. If you use both, you most likely are affected, too, except you only use zlib plugin for reading previously compressed stored mails. That's also the workaround we use now: zlib plugin only enabled in mail_plugins but no plugin/zlib_save set. The bug occurs on very specific mails. Due to privacy reasons I could not provide sample mails here. Storing such mails seems to trigger the bug reproducible. I attached a very minimal doveconf -n config, that can be used to trigger the bug. If one of the developers is interested, I can try to generate an "anonymized" version of such a specific mail that still causes the issue. I discovered the bug on our productive systems, running latest Dovecot 2.2 release, but the latest 2.3 I used during debugging is affected, too. During debugging, I also found one hint, that might help find the bug: If you store a problematic mail with zlib_save=gz (or zlib_save=bz2) and then disable the zlib plugin in mail_plugins, you can call doveadm fetch -u test hdr all | grep -v ^hdr: | gzip --decompress on test's mailbox with only that one broken mail. This will display the beginning of the rfc822 mail text until gzip terminates with "gzip: stdin: unexpected end of file", approximately after twice the length of the mail HEADER. This might indicate, that dovecot stores the uncompressed size of the header in it's data structures although the mail is stored compressed. I also found a very efficient way to find all affected mails in our setup: doveadm -f flow fetch -A 'user guid mailbox uid seq flags hdr' all | \ grep -a "^[^ ]+ user=" | \ grep -avF ' hdr=Return-path: ' | \ grep -av '.* hdr=[[:print:][:space:]]*$' (runtime for ~6M mails on our servers was 20-30min) This can be even more optimized if you have a powerful storage system with GNU parallel: > doveadm user '*' | parallel "doveadm -f flow fetch -u '{}' 'user guid mailbox uid seq flags hdr' all | grep -a '^user=' | grep -avF ' hdr=Return-path: ' | grep -av '.* hdr=[[:print:][:space:]]*$' || true" (runtime for ~6M mails on our servers was ~4min) The command will give you a list of mails that possibly are affected, check the full output of doveadm fetch -u USERNAME hdr guid GUID | less to verify that the header is really broken. On our systems I found 39 mails within ~12M mails. I was able to recover these mails "manually" by reconstructing the Return-Path header line, importing the fixed mails and expunging the corrupt ones. Before importing, I had to disable zlib_save option obviously. Best regards, -- Patrick Cernko <pcernko(a)mpi-klsb.mpg.de> +49 681 9325 5815 Joint Administration: Information Services and Technology Max-Planck-Institute fuer Informatik & Softwaresysteme

4 4

pigeonhole/lda accessing -m folder
by matthias lay 12 Sep '19

12 Sep '19

Hi all, anybody knows, if there is a way to acces the folder from a call like this deovecot-lda -m destfolder from within a sieve script? thx in advance matze

5 9