September 2019 - dovecot

Re: Gnu sieve vs Dovecot sieve-filter - sieve-filter extremely slow at lda (writing emails to local mbox files)
by Zenaan Harkness 12 Sep '19

12 Sep '19

I am wondering why sieve-filter is so slow compared to gnu sieve. I run mpop (like getmail) to download from a pop3 server to a local mbox file: ~/mail/email-incoming-unsorted This step is very fast. The next step, I throw the email-incoming-unsorted mbox file at a sieve processor, to sort the emails from that mbox, into other mboxes, according to the sieve rules file. Up until a couple days ago I was using Gnu sieve. Gnu sieve balks on emails which have no x-message-id (?? something like this) header field, so after a few years, I finally decided to switch "up" to Dovecot/Pigeonhole's "sieve-filter" command. Using Gnu sieve, this mbox sorting step was even faster than mpop (/ getmail) - and mpop and getmail are really fast (compared with fetchmail), since they pipeline the email downloads. Even with 100s of emails, Gnu sieve would take only 10 to 20 seconds at most. Super fast. Using sieve-filter, all emails are being processed - including those without "message id header". This is good. But also, using sieve filter, is really slower - slower than the download step by an order of magnitude or two. See below for details, any ideas appreciated. To add to the below, I added: mbox_very_dirty_syncs = yes to the sieve-filter config, which slightly improves performance, but not by much (in comparison with Gnu sieve). TIA, ----- Forwarded message from Zenaan Harkness <zenaan(a)freedbms.net> ----- From: Zenaan Harkness <zenaan(a)freedbms.net> To: debian-user(a)lists.debian.org Date: Thu, 12 Sep 2019 08:06:12 +1000 Subject: Re: Gnu sieve vs Dovecot sieve-filter - sieve-filter extremely slow at lda (writing emails to local mbox files) On Thu, Sep 12, 2019 at 07:55:23AM +1000, Zenaan Harkness wrote: > Why is Gnu sieve so extremely fast to batch process an mbox file, but > while Dovecot's sieve-filter is an order of magnitude slower? > > Sequence: > > - mpop or getmail to pipeline download emails into temp mbox file > - filter that file > > Gnu sieve just flies through a local mbox file and saving emails to > other local mbox files. > > Gnu sieve rejects too many emails with "malformed" errors, so after a > few years I bit the bullet and upgraded to Dovecot's sieve-filter. > > Dovecot's sieve-filter, at present, is an order of magnitude slower. > > Here's my filter command (one line): > > /usr/bin/sieve-filter -veW -c $HOME/etc/email/sieve-dovecot-config.conf -o mail_location=mbox:~/mail:INBOX=~/mail/Inbox:INDEX=:UTF-8:VOLATILEDIR=/tmp/dovecot-volatile/%2.256Nu/%u:SUBSCRIPTIONS=dovecot_subscriptions ~/etc/email/sieve.rc email-incoming-unsorted > > The sieve script is fine now that I have the correct "require" > clauses (hint: "capability strings"). > > File ~/etc/email/sieve-dovecot-config.conf: > > protocols = pop > lda_mailbox_autocreate = yes > lda_mailbox_autosubscribe = yes > mail_fsync = never > > There's no re-sending of emails into my local Postfix SMTP server - I > checked the system logs and confirmed this (journalctl -f). > > I suspect that Gnu sieve was directly writing each email to the > appropriate sieve-determined mbox file (perhaps with only a sync at > the end of a single batch process - what I've attempted to achieve > above with sieve-filter), and that sieve-filter is instead passing > each email through some (dovecot) lda? > > Here's the output for a sieve-filter batch processing of 11 emails: > > $ /usr/bin/sieve-filter -veW -c /home/zen/etc/email/sieve-dovecot-config.conf -o mail_location=mbox:/home/zen/mail:INBOX=/home/zen/mail/Inbox:INDEX=:UTF-8:VOLATILEDIR=/tmp/dovecot-volatile/%2.256Nu/%u:SUBSCRIPTIONS=dovecot_subscriptions /home/zen/etc/email/sieve.rc email-incoming-unsorted > # PS0 Timestamp: 20190912@07:02:23 > info: filtering: [Tue, 3 Sep 2019 05:17:16 -0500; 10240 bytes] `Re: VentureBeat: The death of disk? H...'. > info: msgid=<CAMjeLr91T9R7APsuxQVuM3WbqDsxAfwn4=OYDeDX4FMcoRdGdQ(a)mail.gmail.com>: stored mail into mailbox 'l/cp/cp'. > info: message expunged from source mailbox upon successful move. > info: filtering: [Tue, 3 Sep 2019 07:29:53 -0400; 12968 bytes] `[zfs-devel] xattr naming format in Zo...'. > info: msgid=<15675101930.d5ba2E.12322(a)composer.zfsonlinux.topicbox.com>: stored mail into mailbox 'l/z/zdev'. > info: message expunged from source mailbox upon successful move. > info: filtering: [Tue, 03 Sep 2019 15:29:09 +0300; 20461 bytes] `Re: [zfs-devel] xattr naming format i...'. > info: msgid=<23955051567513749(a)sas1-02732547ccc0.qloud-c.yandex.net>: stored mail into mailbox 'l/z/zdev'. > info: message expunged from source mailbox upon successful move. > info: filtering: [Tue, 3 Sep 2019 18:20:42 +0530; 18065 bytes] `Re: [Gluster-users] Issues with Geo-r...'. > info: msgid=<CADmkyZMxrfOANrAP+_URAHJcMqCqh=iGdajTSzkfQ5PCZsUfyg(a)mail.gmail.com>: stored mail into mailbox 'l/gl/user'. > info: message expunged from source mailbox upon successful move. > info: filtering: [Tue, 3 Sep 2019 09:34:20 -0400; 13342 bytes] `Re: tasksel'. > info: msgid=<20190903133420.GS6166(a)eeg.ccf.org>: stored mail into mailbox 'l/deb/user'. > info: message expunged from source mailbox upon successful move. > info: filtering: [Tue, 3 Sep 2019 06:56:07 -0700 (PDT); 12390 bytes] `[awx-project] Re: AWX on Kubernetes m...'. > info: msgid=<0715adb7-540f-4cff-9282-e1252c53c2e8(a)googlegroups.com>: stored mail into mailbox 'l/ansible/awx'. > info: message expunged from source mailbox upon successful move. > info: filtering: [Tue, 3 Sep 2019 07:01:27 -0700 (PDT); 12220 bytes] `[awx-project] Re: AWX on Kubernetes m...'. > info: msgid=<949b2c17-4254-49f1-83b4-cd54d15aa17d(a)googlegroups.com>: stored mail into mailbox 'l/ansible/awx'. > info: message expunged from source mailbox upon successful move. > info: filtering: [Tue, 3 Sep 2019 10:14:58 -0400; 25313 bytes] `Re: [zfs-devel] xattr naming format i...'. > info: msgid=<CAB5c7xpHCdFx1w3yA9FyRL-KQ8BUiCr4JbiDQRuFJj9nOgKxTg(a)mail.gmail.com>: stored mail into mailbox 'l/z/zdev'. > info: message expunged from source mailbox upon successful move. > info: filtering: [Tue, 3 Sep 2019 17:10:22 +0200; 7567 bytes] `Re: [asterisk-users] Playing MP3's in...'. > info: msgid=<20190903151022.354xpe6ds2vglher(a)red.localdomain>: stored mail into mailbox 'l/as/users'. > info: message expunged from source mailbox upon successful move. > info: filtering: [Wed, 4 Sep 2019 01:04:49 +0900; 14858 bytes] `Re: [Hyperledger Fabric] a primitive ...'. > info: msgid=<160901d8-b903-9e9a-91ac-267571b0e24d(a)gmx.com>: stored mail into mailbox 'l/hl/fabric'. > info: message expunged from source mailbox upon successful move. > info: filtering: [Tue, 3 Sep 2019 09:55:22 -0700 (PDT); 13337 bytes] `[awx-project] Re: AWX on Kubernetes m...'. > info: msgid=<f9bc4e6a-8445-4b34-927a-35f577ffcc07(a)googlegroups.com>: stored mail into mailbox 'l/ansible/awx'. > info: message expunged from source mailbox upon successful move. > 2 ▶︎️ zen@eye 20190912@07:02:30 ~ $ > > > So about 3/4 of a second is spent by dovecot's sieve-filter, on each > email that it processes - watching it is painful given how fast Gnu > sieve has been for the last few years - it's almost (but not quite) > as slow as my previous fetchmail email download per-email time. > > Attached is a -D debug run of sieve-filter on 20 emails - slightly > longer than the above, and took roughly 15 seconds to run. > > Any help appreciated... On another test run of ~600 emails, sieve-filter is consistently running ~100% of one CPU (for about 4 minutes) to process these emails, which leads to the conclusion that despite what looks like should be a batch process, sieve-filter is perhaps reloading the rules for every single email that it processes, even though I gave it a whole mbox, and not a single email, to process. Can sieve-filter work the way it should / the way I want it / batch process a whole mbox - without reloading the sieve rules for every email? ----- End forwarded message -----

3 2

Re: [Bug] Sieve vacation :addresses match only,> case-sensitive?
by Klaus Steinberger 12 Sep '19

12 Sep '19

> I want to have it case-insensitive again, like in Pigeonhole version 0.4.24 i would also plea for making it case-insensitive again! We were hit by this too, and I never saw any mail system in which the local Part ist Case sensitive! Sincerly, Klaus Steinberger -- Rechnerbetriebsgruppe / IT, Fakultät für Physik Klaus Steinberger FAX: +49 89 28914280 Tel: +49 89 28914287

6 5

WARNING: using attachment_dir with plugin zlib can corrupt mails
by Patrick Cernko 12 Sep '19

12 Sep '19

Hello list, hello Dovecot developers, this week, I discovered a serious bug in Dovecot, that lead to several broken mails on our servers. The bug corrupts the first few characters of the mail header during saving. On our setup, it was almost always only the very first line of text, that was corrupted. Depending on the IMAP client (they seem to request different header fields, ... during mail access), the bug causes the imap process to hang up the TCP connection and log errors like this: > imap(USERNAME)<4767><TeQP4ASOTK5/AAAB>: Error: Corrupted record in index cache file /IMAP/mail/mailboxes/USERNAME/mdbox/mailboxes/Trash/dbox-Mails/dovecot.index.cache: UID 489113: Broken fields in mailbox Trash: read(attachments-connector(zlib(/IMAP/mail/mailboxes/USERNAME/mdbox/storage/m.813))): FETCH BODY[HEADER.FIELDS (RETURN-PATH SUBJECT)] got too little data: 2 vs 122 In our case that finally grabbed my attention, the client was the users iphone that did not display any new messages but his Thunderbird did. The bug seems to be triggered by a bad "interaction" of attachment_dir option and zlib plugin. If you use both, you most likely are affected, too, except you only use zlib plugin for reading previously compressed stored mails. That's also the workaround we use now: zlib plugin only enabled in mail_plugins but no plugin/zlib_save set. The bug occurs on very specific mails. Due to privacy reasons I could not provide sample mails here. Storing such mails seems to trigger the bug reproducible. I attached a very minimal doveconf -n config, that can be used to trigger the bug. If one of the developers is interested, I can try to generate an "anonymized" version of such a specific mail that still causes the issue. I discovered the bug on our productive systems, running latest Dovecot 2.2 release, but the latest 2.3 I used during debugging is affected, too. During debugging, I also found one hint, that might help find the bug: If you store a problematic mail with zlib_save=gz (or zlib_save=bz2) and then disable the zlib plugin in mail_plugins, you can call doveadm fetch -u test hdr all | grep -v ^hdr: | gzip --decompress on test's mailbox with only that one broken mail. This will display the beginning of the rfc822 mail text until gzip terminates with "gzip: stdin: unexpected end of file", approximately after twice the length of the mail HEADER. This might indicate, that dovecot stores the uncompressed size of the header in it's data structures although the mail is stored compressed. I also found a very efficient way to find all affected mails in our setup: doveadm -f flow fetch -A 'user guid mailbox uid seq flags hdr' all | \ grep -a "^[^ ]+ user=" | \ grep -avF ' hdr=Return-path: ' | \ grep -av '.* hdr=[[:print:][:space:]]*$' (runtime for ~6M mails on our servers was 20-30min) This can be even more optimized if you have a powerful storage system with GNU parallel: > doveadm user '*' | parallel "doveadm -f flow fetch -u '{}' 'user guid mailbox uid seq flags hdr' all | grep -a '^user=' | grep -avF ' hdr=Return-path: ' | grep -av '.* hdr=[[:print:][:space:]]*$' || true" (runtime for ~6M mails on our servers was ~4min) The command will give you a list of mails that possibly are affected, check the full output of doveadm fetch -u USERNAME hdr guid GUID | less to verify that the header is really broken. On our systems I found 39 mails within ~12M mails. I was able to recover these mails "manually" by reconstructing the Return-Path header line, importing the fixed mails and expunging the corrupt ones. Before importing, I had to disable zlib_save option obviously. Best regards, -- Patrick Cernko <pcernko(a)mpi-klsb.mpg.de> +49 681 9325 5815 Joint Administration: Information Services and Technology Max-Planck-Institute fuer Informatik & Softwaresysteme

4 4

pigeonhole/lda accessing -m folder
by matthias lay 12 Sep '19

12 Sep '19

Hi all, anybody knows, if there is a way to acces the folder from a call like this deovecot-lda -m destfolder from within a sieve script? thx in advance matze

5 9

Re: Gnu sieve vs Dovecot sieve-filter - sieve-filter extremely slow at lda (writing emails to local mbox files)
by Zenaan Harkness 12 Sep '19

12 Sep '19

(I did subscribe to this mailing list, albeit with zen at freedbms.net, so either way I'm getting all your emails - thank you -so- much for replying...) MUA is mutt, reading email in a terminal (sorry, forgot to mention this before). For many years now my email folder (mbox files) collection has grown to many GiB, mostly mailing lists. If I am to change email storage format, it should be mutt compatible; looking at https://wiki2.dovecot.org/MailboxFormat I see that only DJB's Maildir is compatible with both Dovecot ("a reliable choice" says the wiki), and mutt. I can imagine that sdbox or mdbox could be made "mutt compatible" so to speak, by running some sort of local IMAP server, and accessing my email from mutt that way; this is undesirable to my mind because this would require: 1) a new learning curve wrt mutt and reading email on IMAP servers 2) a new learning curve to set up a local IMAP server (securely) 3) the inability to use mutt without a local IMAP server to read my local email but such a setup would also have some quite desirable benefits: 1) once set up, multiple MUAs could be used, and I'd have a beginning grasp on setting up an IMAP server and front ends (this is something on my bucket list, to assist my local church with) 2) simpler remote "online" access to my local "offline" email store (e.g. using my mobile phone when on the road) by setting up a webmail server (much simpler (read "possible" to use on a mobile phone) than using a vpn and mutt...), thus freeing me up from the behemoth web email providers... Next, I do not know how to "pipe the messages to the dovecot lda". After downloading from my POP3 provider into a local mbox file (this is my step 1), then I sort the emails (this is my step 2): the following should be on a single line: /usr/bin/sieve-filter -veW -c $HOME/etc/email/sieve-dovecot-config.conf -o mail_location=mbox:~/mail:INBOX=~/mail/Inbox:INDEX=:UTF-8:VOLATILEDIR=/tmp/dovecot-volatile/%2.256Nu/%u:SUBSCRIPTIONS=dovecot_subscriptions ~/etc/email/sieve.rc email-incoming-unsorted As you can see from the above command, sieve-filter is given the name of the mbox ("mail folder") to sort, as its very last argument on the command line - so in this instance, sieve-filter really has no excuse, and should be not be re-reading the sieve rules script for each email - now perhaps that's not happening, I only made an assumption because of a CPU hitting 100% for a minute or two just to process a few 100 emails... What could also be happening (again, an assumption), is that sieve-filter is written to assume dovecot index files to be in existence. I disabled those with the "INDEX=" clause you see in the command above, which obviously has been given no value. The reason I figured out how to disable the creation of the indexes in the .imap directories, is that for my setup, Gnu sieve has proven that I should not need such indexes - with mbox files, just append each email to the end of the target "mailbox folder" mbox file, and we're done! This literally should not cost 100% CPU, even for one millisecond! But more importantly, because my working email folder is ~30GiB, without disabling this index creation step, sieve-filter forced the creation of indexes, which "took so long I gave up and hit CTRL-C, which did not work, so I kill -9'ed the sieve-filter and whatever other process was not stopping". Last year someone on debian-user recommended I upgrade to using Dovecot/Pigeonhole's sieve-filter (rather than Gnu sieve) due to the issues with Gnu sieve. I am starting to think that I should perhaps try to figure out if it's possible to (re)process the emails Gnu sieve has a problem with, to massage them into a shape that Gnu sieve accepts - then my immediate problem would certainly be solved... Thank you all again.. Zenaan

1 1

Subscribe to a fileinto :create mailbox?
by Larry Rosenman 11 Sep '19

11 Sep '19

Is it possible in pigeonhole? I.E. I do a fileinto :create "some/mail/box" How can I make it autosubscribe? doveconf -n: # 2.3.7.2 (3c910f64b): /usr/local/etc/dovecot/dovecot.conf # Pigeonhole version 0.5.7.2 (7372921a) # OS: FreeBSD 12.0-STABLE amd64 # Hostname: thebighonker.lerctr.org auth_default_realm = lerctr.org auth_mechanisms = plain login auth_realms = lerctr.org thebighonker.lerctr.org tbh.lerctr.org thejonesonair.com thejonesonair.net why.net default_vsz_limit = 1 G deliver_log_format = msgid=%m: %$ (subject=%s from=%f size=%w) doveadm_password = # hidden, use -P to show it first_valid_gid = 0 first_valid_uid = 0 lda_mailbox_autocreate = yes listen = 192.147.25.65, :: login_access_sockets = tcpwrap mail_attribute_dict = file:%h/mail/.imap/dovecot-mail-attributes mail_location = mbox:~/mail:INBOX=~/mail/INBOX mail_log_prefix = "%s(%u/%p): " mail_plugins = " fts fts_solr notify virtual" mail_privileged_group = mail mail_server_admin = mailto:ler@lerctr.org mail_server_comment = LERCTR Mail Server managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext vacation-seconds editheader mboxmetadata servermetadata vnd.dovecot.debug imapsieve vnd.dovecot.imapsieve namespace archive { hidden = no list = no location = mbox:~/MAIL-ARCHIVE prefix = ARCHIVE/ separator = / } namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox INBOX { auto = create } mailbox SENT { special_use = \Sent } mailbox SPAM { special_use = \Junk } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } mailbox virtual/Flagged { special_use = \Flagged } mailbox virtual/all { special_use = \All } prefix = separator = / } namespace virtual { hidden = no list = yes location = virtual:~/MAIL-VIRTUAL prefix = Virtual/ separator = / } passdb { args = /usr/local/etc/dovecot/dovecot-sql.conf.ext driver = sql } passdb { args = user=%Ln noauthenticate driver = static skip = authenticated } passdb { args = failure_show_msg=yes session=yes max_requests=20 driver = pam override_fields = domain=lerctr.org skip = authenticated } plugin { fts = solr fts_autoindex = yes fts_solr = url=http://thebighonker.lerctr.org:8983/solr/dovecot/ fts_tika = http://localhost:9998/tika/ imapsieve_mailbox1_before = file:/usr/local/share/dovecot-pigeonhole/sieve/report-spam.sieve imapsieve_mailbox1_causes = COPY imapsieve_mailbox1_name = SPAM imapsieve_mailbox2_before = file:/usr/local/share/dovecot-pigeonhole/sieve/report-ham.sieve imapsieve_mailbox2_causes = COPY imapsieve_mailbox2_from = SPAM imapsieve_mailbox2_name = * imapsieve_url = sieve://thebighonker.lerctr.org mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename flag_change append mail_log_fields = uid box msgid size from subject vsize flags recipient_delimiter = +-_ sieve = ~/.dovecot.sieve sieve_dir = ~/sieve sieve_execute_bin_dir = /usr/local/share/dovecot-pigeonhole/sieve sieve_extensions = +editheader +vacation-seconds +mboxmetadata +servermetadata +vnd.dovecot.debug sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.execute sieve_pipe_bin_dir = /usr/local/share/dovecot-pigeonhole/sieve sieve_plugins = sieve_imapsieve sieve_extprograms } postmaster_address = postmaster(a)lerctr.org protocols = imap pop3 lmtp sieve recipient_delimiter = +-_ service anvil { unix_listener anvil { group = mail mode = 0666 } } service auth { unix_listener auth-client { mode = 0666 } unix_listener auth-master { mode = 0666 } } service doveadm { inet_listener http { port = 8080 ssl = yes } } service indexer-worker { drop_priv_before_exec = yes } service lmtp { inet_listener lmtp { address = 127.0.0.1 port = 24 } } service managesieve-login { inet_listener sieve { port = 4190 } inet_listener sieve_deprecated { port = 2000 } } service stats { unix_listener stats-reader { group = mail mode = 0666 user = } unix_listener stats-writer { group = mail mode = 0666 user = } } service tcpwrap { unix_listener login/tcpwrap { group = $default_login_user mode = 0600 user = $default_login_user } } ssl_cert = </home/ler/letsencrypt-home/*.lerctr.org/fullchain.cer ssl_cipher_list = EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+AESGCM:EECDH:EDH+AESGCM:EDH+aRSA:HIGH:!MEDIUM:!LOW:!aNULL:!eNULL:!LOW:!RC4:!MD5:!EXP:!PSK:!SRP:!DSS ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it userdb { args = /usr/local/etc/dovecot/dovecot-sql.conf.ext driver = sql } userdb { args = username_format=%Ln /etc/passwd driver = passwd-file override_fields = domain=lerctr.org } verbose_proctitle = yes protocol lmtp { mail_plugins = " fts fts_solr notify virtual sieve mail_log" } protocol lda { mail_plugins = " fts fts_solr notify virtual sieve mail_log" } protocol pop3 { mail_plugins = " fts fts_solr notify virtual mail_log" } protocol !doveadm { mail_plugins = " fts fts_solr notify virtual mail_log" } protocol imap { imap_capability = +SPECIAL-USE imap_client_workarounds = tb-extra-mailbox-sep tb-lsub-flags imap_logout_format = in=%i out=%o fhc=%{fetch_hdr_count} fhb=%{fetch_hdr_bytes} fbc=%{fetch_body_count} fbb=%{fetch_body_bytes} del=%{deleted} exp=%{expunged} trash=%{trashed} imap_metadata = yes mail_max_userip_connections = 50 mail_plugins = " fts fts_solr notify virtual mail_log imap_sieve imap_zlib" } -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx(a)gmail.com US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

2 1

Case sensitive problem for vacation with dovecot lmtp pigeonhole
by kadafax＠gmail.com 11 Sep '19

11 Sep '19

Hi list, This issue was raised here by John Fawcett on 12/02/19 but got no response. So, as I'm facing the same issue, I would like to try again: When using the vacation sieve extension, if I send an email to user.name(a)mydomain.com it will work, but not to User.Name(a)mydomain.com. Is there a way to make the user part case insensitive so that the vacation auto-reply is fired whatever the way user part is written ? doveconf -n output: # 2.3.5.1 (7ec6d0ade): /usr/local/etc/dovecot/dovecot.conf # Pigeonhole version 0.5.5 (2483b085) # OS: Linux 3.10.0-957.5.1.el7.x86_64 x86_64 CentOS Linux release 7.6.1810 (Core) ext4 # Hostname: myserver.mydomain.com auth_master_user_separator = * auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@* auth_verbose = yes debug_log_path = /var/log/dovecot-debug.log default_client_limit = 10640 default_process_limit = 2128 default_vsz_limit = 0 hostname = mydomain.com info_log_path = /var/log/dovecot.log lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes listen = * log_path = /var/log/dovecot-error.log login_greeting = Welcome mail_gid = vmail mail_location = maildir:/var/vmail/%Ln mail_plugins = quota fts fts_lucene mail_uid = vmail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext namespace inbox { inbox = yes location = maildir:/var/vmail/%Ln mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } prefix = type = private } passdb { args = /usr/local/etc/dovecot/deny-users deny = yes driver = passwd-file } passdb { args = /usr/local/etc/dovecot/master-users driver = passwd-file master = yes pass = yes } passdb { args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { fts = lucene fts_autoindex = yes fts_lucene = whitespace_chars=@. quota = maildir:User quota quota_rule = *:storage=30GB quota_rule2 = Trash:storage=+200M quota_warning = storage=95%% quota-warning 95 %u quota_warning2 = storage=80%% quota-warning 80 %u sieve = /var/sieve-scripts/%Ln.sieve sieve_dir = /var/vmail/%Ln/home/sieve } postmaster_address = postmaster(a)mydomain.com protocols = imap pop3 lmtp sieve sieve quota_full_tempfail = yes service auth-worker { user = $default_internal_user } service auth { unix_listener /var/spool/postfix/private/auth { mode = 0777 user = vmail } } service imap-login { inet_listener imaps { port = 993 ssl = yes } process_min_avail = 10 } service imap { process_limit = 2128 } service lmtp { executable = lmtp -L process_min_avail = 15 unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } } service managesieve-login { inet_listener sieve { port = 4190 } } service pop3-login { inet_listener pop3s { port = 995 ssl = yes } } service quota-warning { executable = script /usr/local/bin/quota-warning.sh user = nobody } ssl_cert = </etc/letsencrypt/live/myserver.mydomain.com/fullchain.pem ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it userdb { args = /usr/local/etc/dovecot/dovecot-ldap-userdb.conf.ext driver = ldap override_fields = uid=vmail gid=vmail home=/var/vmail/%Ln/home } protocol imap { mail_max_userip_connections = 30 mail_plugins = quota fts fts_lucene imap_quota } protocol lmtp { info_log_path = /var/log/dovecot-lmtp.log log_path = /var/log/dovecot-lmtp-errors.log mail_plugins = quota fts fts_lucene sieve } protocol sieve { mail_max_userip_connections = 20 }

2 2

Quota and maildir does not work with subfolders of INBOX
by Niels Kobschätzki 11 Sep '19

11 Sep '19

Hi, we just upgraded on FreeBSD (11.2-p14) from Dovecot 2.2 to Dovecot 2.3.7.2 and now we have kind of a big problem: the quota-plugin does not work correctly anymore. If a user has a subfolder of the INBOX (like .INBOX.foobar) the quota-calculation does not work anymore. It seems to be arbitrary but users get counted like double and more than they actually have in mails (the number of mails is correct, but the size gets calculated wrongly). The moment I remove those folders, the size gets calculated correctly. Unfortunately those folders are generated by some clients automatically afaik (like .INBOX.Trash) What can we do about that? There were already messages to the list about this in July 2018 but I didn't find any solution https://dovecot.org/pipermail/dovecot/2018-July/112439.html https://dovecot.org/pipermail/dovecot/2018-July/112440.html Niels

3 4

Off-site cloud backup (eg Amazon S3, Wasabi)
by James Brown 11 Sep '19

11 Sep '19

I’d like to synchronise our mail store to an Amazon S3 bucket or similar, for protection from ransomware. Server is running on macOS with maildir. Should I use a Dovecot process, rsync, a Mac app like Arq or Jungle Disk, or something else? Anyone have experience or opinions on this? Thanks, James.

3 2

Random duplicated emails
by Francis 11 Sep '19

11 Sep '19

Hello, I migrated our mail infrastructure to Dovecot on Ubuntu 18.04 some months ago. It works fine, but recently some users told me that they sometime receive duplicated emails. Same email content, same headers including message-id. I'm using two dovecot servers on two sites. Both server are in cluster. We don't use shared folders. All users that reported this issue so far are using the same server instance. The problematic emails are coming from local users on that instance too. The examples they given to me was emails with many recipients (To/CC). A specific message can be received twice (or more) by recipient A but only once by recipient B. I didn't see anything in the logs about sieve rules that redirect emails to others recipients. Where should I look to diagnostic this issue? Thanks. Server config: # 2.2.33.2 (d6601f4ec): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.21 (92477967) doveconf: Warning: service auth { client_limit=1000 } is lower than required under max. load (2500) doveconf: Warning: service anvil { client_limit=1000 } is lower than required under max. load (1503) # OS: Linux 4.15.0-60-generic x86_64 Ubuntu 18.04.3 LTS auth_default_realm = **edited** auth_master_user_separator = * auth_mechanisms = plain login default_process_limit = 500 first_valid_uid = 200 last_valid_uid = 200 mail_location = mbox:~/mail:INBOX=/var/mail/%u mail_plugins = " notify replication zlib quota" mail_privileged_group = mail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext vnd.dovecot.debug mdbox_rotate_size = 10 M namespace inbox { inbox = yes location = mailbox "Éléments envoyés" { special_use = \Sent } mailbox "Éléments supprimés" { special_use = \Trash } mailbox Brouillon { special_use = \Drafts } mailbox "Courrier indésirable" { special_use = \Junk } mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } prefix = INBOX/ separator = / } passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } passdb { args = /etc/dovecot/passwd.masterusers driver = passwd-file master = yes } plugin { mail_replica = tcps:**edited**:53683 quota = count:User quota quota_grace = 10%% quota_rule = *:storage=10G quota_rule2 = INBOX/Trash:storage=+100M quota_status_nouser = DUNNO quota_status_overquota = 552 5.2.2 La boite courriel du destinataire est pleine - Mailbox is full quota_status_success = DUNNO quota_vsizes = yes quota_warning = storage=100%% quota-warning 100 %u quota_warning2 = storage=95%% quota-warning 95 %u quota_warning3 = storage=90%% quota-warning 90 %u quota_warning4 = storage=85%% quota-warning 85 %u quota_warning5 = storage=75%% quota-warning 75 %u quota_warning6 = -storage=80%% quota-warning '-80' %u quota_warning7 = -storage=100%% quota-warning '-100' %u sieve = /var/vmail/domains/%Ld/%Ln/.dovecot.sieve sieve_before = /var/vmail/sieve/before.sieve sieve_default = /var/vmail/sieve/default.sieve sieve_extensions = +vnd.dovecot.debug sieve_global = /var/vmail/sieve/global zlib_save = lz4 } protocols = " imap lmtp sieve" service auth { unix_listener /var/spool/postfix/private/auth { mode = 0666 } unix_listener auth-userdb { group = vmail mode = 0666 user = vmail } } service imap-login { inet_listener imap { port = 0 } process_min_avail = 10 } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0660 user = postfix } } service quota-status { client_limit = 1 executable = quota-status -p postfix inet_listener { address = **edited** port = 53684 } } service quota-warning { executable = script /usr/local/bin/quota-warning.sh unix_listener quota-warning { user = vmail } } ssl = required ssl_cert = </etc/ssl/**edited**/**edited**.crtbundle ssl_client_ca_dir = /etc/ssl/certs ssl_key = # hidden, use -P to show it userdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } protocol lmtp { mail_plugins = " notify replication zlib quota sieve" postmaster_address = postmaster@**edited** } protocol imap { mail_plugins = " notify replication zlib quota imap_zlib imap_quota" } -- Francis

4 4