- dovecot - dovecot.org

correct usage for Pigeonhole sieve message *redirect* to external server?
by PGNet Dev 16 Sep '20

16 Sep '20

i've installed rpm -qa | grep dovecot dovecot-2.3.10.1-1.fc32.x86_64 dovecot-pigeonhole-2.3.10.1-1.fc32.x86_64 i'm setting up imapsieve spam reporting; following, https://doc.dovecot.org/configuration_manual/spam_reporting/ https://wiki.dovecot.org/HowTo/AntispamWithSieve https://wiki.dovecot.org/Pigeonhole/Sieve/Examples my config currrently includes protocol imap { mail_plugins = $mail_plugins acl imap_acl quota imap_quota imap_sieve mail_log notify mail_max_userip_connections = 20 imap_client_workarounds = tb-extra-mailbox-sep } protocol sieve { mail_plugins = $mail_plugins mail_max_userip_connections = 10 } plugin { # From elsewhere to Spam folder imapsieve_mailbox1_name = Spam imapsieve_mailbox1_causes = COPY imapsieve_mailbox1_before = file:/etc/dovecot/scripts/report-spam.sieve # From Spam folder to elsewhere imapsieve_mailbox2_name = * imapsieve_mailbox2_from = Spam imapsieve_mailbox2_causes = COPY imapsieve_mailbox2_before = file:/etc/dovecot/scripts/report-ham.sieve sieve_plugins = sieve_imapsieve sieve_pipe_bin_dir = /etc/dovecot/scripts } service submission-login { inet_listener submission { address = 10.0.1.70, 127.0.0.1 port = 60465 ssl = yes } } examples @ wiki include report-spam.sieve require "vnd.dovecot.report"; report "abuse" "User added this message to the Spam folder." "spam-report(a)example.com"; instead of a 'report', i want to *redirect* the _unmodified_ Spam message to an _external_ email that's a feed for a bayes spam learner. to that end i modified /etc/dovecot/scripts/report-spam.sieve require ["vnd.dovecot.report", "vnd.dovecot.pipe", "copy", "imapsieve", "environment", "variables"]; if environment "imap.mailbox" "Trash" { stop; } if environment "imap.mailbox" "Spam" { stop; } redirect "spam.learn(a)othermx.example.net"; with that^, when I drag a spam message into my user account's Spam folder, I see in logs only 2020-09-15 08:52:16 imap(myuser(a)example.com)<bCdiIOuhPImsDgsd>: Debug: Mailbox Junk: Mailbox opened because: UID move 2020-09-15 08:52:16 imap(myuser(a)example.com)<bCdiIOuhPImsDgsd>: Debug: acl vfile: file /home/vmail/conf.d/example.com/acls/Junk not found 2020-09-15 08:52:16 imap(myuser(a)example.com)<bCdiIOuhPImsDgsd>: Debug: acl vfile: file /data/vmail/example.com/myuser/Maildir/Junk/dovecot-acl not found 2020-09-15 08:52:16 imap(myuser(a)example.com)<bCdiIOuhPImsDgsd>: Debug: imapsieve: mailbox Junk: MOVE event 2020-09-15 08:52:16 imap(myuser(a)example.com)<bCdiIOuhPImsDgsd>: Debug: sieve: Pigeonhole version 0.5.10 (bf8ef1c2) initializing 2020-09-15 08:52:16 imap(myuser(a)example.com)<bCdiIOuhPImsDgsd>: Debug: sieve: include: sieve_global is not set; it is currently not possible to include `:global' scripts. 2020-09-15 08:52:16 imap(myuser(a)example.com)<bCdiIOuhPImsDgsd>: Debug: sieve: Sieve imapsieve plugin for Pigeonhole version 0.5.10 (bf8ef1c2) loaded 2020-09-15 08:52:16 imap(myuser(a)example.com)<bCdiIOuhPImsDgsd>: Debug: imapsieve: Static mailbox rule [1]: mailbox=`Spam' from=`*' causes=(COPY) => before=`file:/etc/dovecot/scripts/report-spam.sieve' after=(none) 2020-09-15 08:52:16 imap(myuser(a)example.com)<bCdiIOuhPImsDgsd>: Debug: imapsieve: Static mailbox rule [2]: mailbox=`*' from=`Spam' causes=(COPY) => before=`file:/etc/dovecot/scripts/report-ham.sieve' after=(none) ==> /var/log/dovecot/dovecot-info.log <== 2020-09-15 08:52:16 imap(myuser(a)example.com)<bCdiIOuhPImsDgsd>: Info: copy from INBOX: box=Junk, uid=4, msgid=<0e9fc21a-c046-46bd-a4ae-e7a1b6424ca7.maildroid@localhost>, size=5647 2020-09-15 08:52:16 imap(myuser(a)example.com)<bCdiIOuhPImsDgsd>: Info: expunge: box=INBOX, uid=50, msgid=<0e9fc21a-c046-46bd-a4ae-e7a1b6424ca7.maildroid@localhost>, size=5647 but no redirect occurs; nothing's sent. what additional/different config is required to get the sieve processor to *send* that email? in sieve redirect, _is_ the dovecot submission port implicitly used for sending/redirect? or does the report-spam script need additional capability to exec the send?

1 1

Auth Panic hmac.c while Local Validation
by Mrinal Sharma 16 Sep '20

16 Sep '20

I am trying to use a newly added Local Validation functionality in dovecot version 2.3.11. I am running dovecot inside a Docker container With base image " debian:buster-slim". When I try to login through below command , a crash is seen. ''' a1 login admin eyJhbGci44444zUxMiIsInR5cCI6IkpXVCJ9.eyJ1c2fyX25hbWUiOiJhZG1pbiIsInNjb3BlIjpbIm9wZW5pZCJdLCJleffiOjE3MDAxODAwNTksInN1YiI6ImFkbWluIiwiaWF0IjoxNjAwMTc5NzU5LCJhdXRob3JpdGllcyI6WyJST0xFX0FETUlOIiwiUk9MRV9VU0VSIl0sImp0aSI6ImRyOUV0MVVJWkdJZkZ0emFVZW5VRzRzcmQtQSIsImNsaWVudF9pZCI6IndlYl9hcHAifQ.T9BTZYW52p0VG9gxmTb-cf5GXF5jTOjdkqMaUMAGX-tffffft7YfdPptphGKE8FO9opxcnL--Bjy9ip-XYuWqA '''' Crash: dovecot_1 | Sep 15 16:11:50 auth: Panic: file hmac.c: line 26 (hmac_init): assertion failed: (meth->context_size <= HMAC_MAX_CONTEXT_SIZE) dovecot_1 | Sep 15 16:11:50 auth: Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(backtrace_append+0x3d) [0x7ff77b36beed] -> /usr/lib/dovecot/libdovecot.so.0(backtrace_get+0x1e) [0x7ff77b36c00e] -> /usr/lib/dovecot/libdovecot.so.0(+0xf465b) [0x7ff77b37565b] -> /usr/lib/dovecot/libdovecot.so.0(+0xf46f1) [0x7ff77b3756f1] -> /usr/lib/dovecot/libdovecot.so.0(+0x514a6) [0x7ff77b2d24a6] -> /usr/lib/dovecot/libdovecot.so.0(+0x52415) [0x7ff77b2d3415] -> /usr/lib/dovecot/libdovecot.so.0(+0x59c09) [0x7ff77b2dac09] -> /usr/lib/dovecot/libdovecot.so.0(oauth2_try_parse_jwt+0x7bb) [0x7ff77b2db60b] -> dovecot/auth(+0x3bf71) [0x55576d3cdf71] -> dovecot/auth(db_oauth2_lookup+0x350) [0x55576d3cf030] -> dovecot/auth(auth_request_default_verify_plain_continue+0x2d6) [0x55576d3b3386] -> dovecot/auth(auth_request_verify_plain_callback_finish+0x5c) [0x55576d3b214c] -> dovecot/auth(auth_request_verify_plain_callback+0x51) [0x55576d3b2281] -> dovecot/auth(+0x30171) [0x55576d3c2171] -> dovecot/auth(+0x27ebb) [0x55576d3b9ebb] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x69) [0x7ff77b38b989] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x131) [0x7ff77b38ced1] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x4c) [0x7ff77b38ba2c] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x40) [0x7ff77b38bba0] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7ff77b3059c3] -> dovecot/auth(main+0x3e1) [0x55576d3a6fa1] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb) [0x7ff77b03809b] -> dovecot/auth(_start+0x2a) [0x55576d3a713a] dovecot_1 | Sep 15 16:11:50 auth-worker(25): Debug: conn unix:auth-worker (pid=24,uid=101): Disconnected: Connection closed (fd=-1) dovecot_1 | Sep 15 16:11:50 imap-login: Debug: Ignoring unknown passdb extra field: temp dovecot --version 2.3.11.3 (502c39af9) dovecot -n # 2.3.11.3 (502c39af9): /etc/dovecot/dovecot.conf # OS: Linux 4.19.76-linuxkit x86_64 Debian 10.5 fuse.grpcfuse # Hostname: 45e39b46f6ab auth_debug = yes auth_mechanisms = plain oauthbearer xoauth2 auth_verbose = yes disable_plaintext_auth = no doveadm_password = # hidden, use -P to show it log_path = /dev/stdout mail_debug = yes mail_location = maildir:/data/imap_store/%n mail_plugins = " quota" namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Greeting { auto = create } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { auto = no special_use = \Trash } prefix = } passdb { driver = pam } passdb { args = /etc/dovecot/dovecot-oauth2.plain.conf.ext driver = oauth2 mechanisms = plain login } plugin { quota = maildir:User quota quota_rule = *:storage=5MB quota_status_nouser = DUNNO quota_status_overquota = 552 5.2.2 Mailbox is full quota_status_success = DUNNO } protocols = imap lmtp service doveadm { inet_listener http { port = 80 } } service lmtp { inet_listener lmtp { address = * port = 24 } process_min_avail = 5 } ssl = no userdb { driver = passwd } userdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } protocol lmtp { mail_plugins = quota postmaster_address = postmaster@domainname } protocol lda { mail_plugins = " quota notify push_notification" } protocol imap { imap_metadata = yes mail_plugins = " quota imap_quota quota" } Any help would be appreciated. Thanks, Mrinal

2 2

Shared mailboxes setups and dictionaries
by Matej Tyc 16 Sep '20

16 Sep '20

Hello, I am relatively new to the world of MTAs and MDAs, and I try to set up shared mailboxes. So far I have somehow succeeded - I have defined a shared namespace and I have managed to create per-mailbox ACL files thanks to the doveadm command. However, I have been following these resources and there were bits that have puzzled me: When learning about how ACL work in e.g. https://doc.dovecot.org/settings/plugin/acl/ - when one wishes to use the Global ACL file, how does one link it to a particular user's mailboxes? Examples that are listed in the documentation are far too generic. For example does "* user=foo lrw" imply that all mailboxes of all accounts are shared to the user foo? The doveadm command works only if dovecot is set up with per-mailbox ACL files, so I can't use it to reverse-engineer the correct syntax. An interesting aspect to ACLs are dictionaries. I understood it as some kind of cache - if there is no dictionary or it is empty, then shared mailboxes don't work. Conversely, dictionary itself is not enough, one needs actual ACLs set up correctly. Is this a correct understanding? Next what https://wiki.dovecot.org/SharedMailboxes/Shared and https://wiki.dovecot.org/Dictionary describe is a possibility to reference LDAP data to define an ACL dictionary. Do I understand it correctly that if a LDAP database is the single source of truth, then I don't have to worry about updating dictionaries as long as LDAP itself is up-to-date, but I have to keep ACLs and LDAP in sync manually (or using an application)? Thanks, Matej

2 3

Btrfs RAID-10 performance
by Miloslav Hůla 15 Sep '20

15 Sep '20

Hello, I sent this into the Linux Kernel Btrfs mailing list and I got reply: "RAID-1 would be preferable" (https://lore.kernel.org/linux-btrfs/7b364356-7041-7d18-bd77-f60e0e2e2112@le…) May I ask you for the comments as from people around the Dovecot? We are using btrfs RAID-10 (/data, 4.7TB) on a physical Supermicro server with Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz and 125GB of RAM. We run 'btrfs scrub start -B -d /data' every Sunday as a cron task. It takes about 50 minutes to finish. # uname -a Linux imap 4.9.0-12-amd64 #1 SMP Debian 4.9.210-1 (2020-01-20) x86_64 GNU/Linux RAID is a composition of 16 harddrives. Harddrives are connected via AVAGO MegaRAID SAS 9361-8i as a RAID-0 devices. All harddrives are SAS 2.5" 15k drives. Server serves as a IMAP with Dovecot 2.2.27-3+deb9u6, 4104 accounts, Mailbox format, LMTP delivery. We run 'rsync' to remote NAS daily. It takes about 6.5 hours to finish, 12'265'387 files last night. Last half year, we encoutered into performace troubles. Server load grows up to 30 in rush hours, due to IO waits. We tried to attach next harddrives (the 838G ones in a list below) and increase a free space by rebalace. I think, it helped a little bit, not not so rapidly. Is this a reasonable setup and use case for btrfs RAID-10? If so, are there some recommendations to achieve better performance? Thank you. With kind regards Milo # megaclisas-status -- Controller information -- -- ID | H/W Model | RAM | Temp | BBU | Firmware c0 | AVAGO MegaRAID SAS 9361-8i | 1024MB | 72C | Good | FW: 24.16.0-0082 -- Array information -- -- ID | Type | Size | Strpsz | Flags | DskCache | Status | OS Path | CacheCade |InProgress c0u0 | RAID-0 | 838G | 256 KB | RA,WB | Enabled | Optimal | /dev/sdq | None |None c0u1 | RAID-0 | 558G | 256 KB | RA,WB | Enabled | Optimal | /dev/sda | None |None c0u2 | RAID-0 | 558G | 256 KB | RA,WB | Enabled | Optimal | /dev/sdb | None |None c0u3 | RAID-0 | 558G | 256 KB | RA,WB | Enabled | Optimal | /dev/sdc | None |None c0u4 | RAID-0 | 558G | 256 KB | RA,WB | Enabled | Optimal | /dev/sdd | None |None c0u5 | RAID-0 | 558G | 256 KB | RA,WB | Enabled | Optimal | /dev/sde | None |None c0u6 | RAID-0 | 558G | 256 KB | RA,WB | Enabled | Optimal | /dev/sdf | None |None c0u7 | RAID-0 | 558G | 256 KB | RA,WB | Enabled | Optimal | /dev/sdg | None |None c0u8 | RAID-0 | 558G | 256 KB | RA,WB | Enabled | Optimal | /dev/sdh | None |None c0u9 | RAID-0 | 558G | 256 KB | RA,WB | Enabled | Optimal | /dev/sdi | None |None c0u10 | RAID-0 | 558G | 256 KB | RA,WB | Enabled | Optimal | /dev/sdj | None |None c0u11 | RAID-0 | 558G | 256 KB | RA,WB | Enabled | Optimal | /dev/sdk | None |None c0u12 | RAID-0 | 558G | 256 KB | RA,WB | Enabled | Optimal | /dev/sdl | None |None c0u13 | RAID-0 | 558G | 256 KB | RA,WB | Enabled | Optimal | /dev/sdm | None |None c0u14 | RAID-0 | 558G | 256 KB | RA,WB | Enabled | Optimal | /dev/sdn | None |None c0u15 | RAID-0 | 838G | 256 KB | RA,WB | Enabled | Optimal | /dev/sdr | None |None -- Disk information -- -- ID | Type | Drive Model | Size | Status | Speed | Temp | Slot ID | LSI ID c0u0p0 | HDD | SEAGATE ST900MP0006 N003WAG0Q3S3 | 837.8 Gb | Online, Spun Up | 12.0Gb/s | 53C | [8:14] | 32 c0u1p0 | HDD | HGST HUC156060CSS200 A3800XV250TJ | 558.4 Gb | Online, Spun Up | 12.0Gb/s | 38C | [8:0] | 12 c0u2p0 | HDD | HGST HUC156060CSS200 A3800XV3XT4J | 558.4 Gb | Online, Spun Up | 12.0Gb/s | 43C | [8:1] | 11 c0u3p0 | HDD | HGST HUC156060CSS200 ADB05ZG4XLZU | 558.4 Gb | Online, Spun Up | 12.0Gb/s | 46C | [8:2] | 25 c0u4p0 | HDD | HGST HUC156060CSS200 A3800XV3DWRL | 558.4 Gb | Online, Spun Up | 12.0Gb/s | 48C | [8:3] | 14 c0u5p0 | HDD | HGST HUC156060CSS200 A3800XV3XZTL | 558.4 Gb | Online, Spun Up | 12.0Gb/s | 52C | [8:4] | 18 c0u6p0 | HDD | HGST HUC156060CSS200 A3800XV3VSKJ | 558.4 Gb | Online, Spun Up | 12.0Gb/s | 55C | [8:5] | 15 c0u7p0 | HDD | SEAGATE ST600MP0006 N003WAF1LWKE | 558.4 Gb | Online, Spun Up | 12.0Gb/s | 56C | [8:6] | 28 c0u8p0 | HDD | HGST HUC156060CSS200 A3800XV3XTDJ | 558.4 Gb | Online, Spun Up | 12.0Gb/s | 55C | [8:7] | 20 c0u9p0 | HDD | HGST HUC156060CSS200 A3800XV3T8XL | 558.4 Gb | Online, Spun Up | 12.0Gb/s | 57C | [8:8] | 19 c0u10p0 | HDD | HGST HUC156060CSS200 A7030XHL0ZYP | 558.4 Gb | Online, Spun Up | 12.0Gb/s | 61C | [8:9] | 23 c0u11p0 | HDD | HGST HUC156060CSS200 ADB05ZG4VR3P | 558.4 Gb | Online, Spun Up | 12.0Gb/s | 60C | [8:10] | 24 c0u12p0 | HDD | SEAGATE ST600MP0006 N003WAF195KA | 558.4 Gb | Online, Spun Up | 12.0Gb/s | 60C | [8:11] | 29 c0u13p0 | HDD | SEAGATE ST600MP0006 N003WAF1LTZW | 558.4 Gb | Online, Spun Up | 12.0Gb/s | 56C | [8:12] | 26 c0u14p0 | HDD | SEAGATE ST600MP0006 N003WAF1LWH6 | 558.4 Gb | Online, Spun Up | 12.0Gb/s | 55C | [8:13] | 27 c0u15p0 | HDD | SEAGATE ST900MP0006 N003WAG0Q414 | 837.8 Gb | Online, Spun Up | 12.0Gb/s | 47C | [8:15] | 33 # btrfs --version btrfs-progs v4.7.3 # btrfs fi show Label: 'DATA' uuid: 5b285a46-e55d-4191-924f-0884fa06edd8 Total devices 16 FS bytes used 3.49TiB devid 1 size 558.41GiB used 448.66GiB path /dev/sda devid 2 size 558.41GiB used 448.66GiB path /dev/sdb devid 4 size 558.41GiB used 448.66GiB path /dev/sdd devid 5 size 558.41GiB used 448.66GiB path /dev/sde devid 7 size 558.41GiB used 448.66GiB path /dev/sdg devid 8 size 558.41GiB used 448.66GiB path /dev/sdh devid 9 size 558.41GiB used 448.66GiB path /dev/sdf devid 10 size 558.41GiB used 448.66GiB path /dev/sdi devid 11 size 558.41GiB used 448.66GiB path /dev/sdj devid 13 size 558.41GiB used 448.66GiB path /dev/sdk devid 14 size 558.41GiB used 448.66GiB path /dev/sdc devid 15 size 558.41GiB used 448.66GiB path /dev/sdl devid 16 size 558.41GiB used 448.66GiB path /dev/sdm devid 17 size 558.41GiB used 448.66GiB path /dev/sdn devid 18 size 837.84GiB used 448.66GiB path /dev/sdr devid 19 size 837.84GiB used 448.66GiB path /dev/sdq # btrfs fi df /data/ Data, RAID10: total=3.48TiB, used=3.47TiB System, RAID10: total=256.00MiB, used=320.00KiB Metadata, RAID10: total=21.00GiB, used=18.17GiB GlobalReserve, single: total=512.00MiB, used=0.00B I do not attach whole dmesg log. It is almost empty, without errors. Only lines about BTRFS are about relocations, like: BTRFS info (device sda): relocating block group 29435663220736 flags 65 BTRFS info (device sda): found 54460 extents BTRFS info (device sda): found 54459 extents

6 18

[BUG] GSSAPI auth doesn't work on 2.3.11
by Carlos Mogas da Silva 15 Sep '20

15 Sep '20

Hi list! I have a CentOS 7 server running dovecot with GSSAPI auth enabled. Version 2.3.10 was working fine but after the upgrade to 2.3.11 it stopped working. On thing I see on the logs is that the user field comes up empty when compared to the 2.3.10. Example: 2.3.10: Sep 14 13:39:20 mail.r3pek.org dovecot[10189]: imap-login: Login: user=<r3pek(a)r3pek.org>, method=GSSAPI, rip=xxxx, lip=2a01:4f8:1c17:61c0::1, mpid=10195, TLS, session=<PaDSUkWvAOkgAQgY3LZuALKwbzMw3BW/> 2.3.11: Sep 14 12:19:24 mail.r3pek.org dovecot[9264]: imap-login: Disconnected (auth failed, 1 attempts in 6 secs): user=<>, method=GSSAPI, rip=2001:818:dcb6:6e00:b2b0:6f33:30dc:15bf, lip=2a01:4f8:1c17:61c0::1, TLS, session=<pRubNESvMuQgAQgY3LZuALKwbzMw3BW/> Reverting back to 2.3.10 confirmed it's a 2.3.11 thing. Is there any change necessary after the upgrade to 2.3.11 or is this really a bug?

2 2

Is it possible to only replicate indexes and not the mail data
by Marc Roos 14 Sep '20

14 Sep '20

Is it possible to only replicate indexes and not the mail data? (Because the data is already on distributed storage)

2 2

logging LMTP activity
by Janis 11 Sep '20

11 Sep '20

Hello, I've been trying to set up Postfix + Dovecot LMTP setup. Basic idea taken from https://wiki2.dovecot.org/HowTo/PostfixDovecotLMTP, but since Postfix is on different host inet listener is used instead of unix socket. Message delivery works. Messages are delivered to correct mailboxes, if they exist. But the problem is with logging on Dovecot side; i can not get any lmtp logs at all. I kind of expected that i could get logs about: 1) Postfix has connected and delivering mail to mail(a)address.com, 2) If mailbox does not exists, would expect to see in logs the reason for rejection - mailbox does not exist 3) any other useful info would be nice as well. How do you get LMTP to log something? My Dovecot version is 2.3.11.3. Basic IMAP logging is working. I see when users have connected, etc. I tried as suggested at https://doc.dovecot.org/configuration_manual/protocols/lmtp_server/#logging If the given logging example block is used in configuration, i get no logs in file /var/log/dovecot-lmtp.log. I thought maybe file is not created automatically, so i tried to create it myself as well. Does not work. chmod 0777 does not either, chown to dovecot:dovecot neither. In a way i am curious about "executable = lmtp -L" part. What is -L flag? Are there any more flags? Where could i learn about them? I have read all the docs that come with the buld and mention something of lmtp, it's basically wiki, i guess... but i could not find any helpful info. I found that file lmtp is at /usr/local/libexec/dovecot/lmtp. When i execute it with -L flag it works. Other flags don't. The problem is that there are no -h or --help flags. Then i tried raw logging as described here: https://doc.dovecot.org/admin_manual/debugging/debugging_rawlog/#debugging-… As i understand, only 4th possibility is applicable to LMTP, am i right? 2nd kind of seems maybe somehow could be usable, but i don't see how... It is not clearly stated in wiki where "lmtp_rawlog_dir" should be added, but i add it within "protocol lmtp { .. }" block, similar as imap. Does not work for me - no logs while messages arrive. The catchy part in wiki is that for imap special arguments are used "-R rawlog" in service imap-login, postlogin. But i don't see any lmtp-login or such service available. So should it work only with "lmtp_rawlog_dir" option? What could i be doing wrong? Does anyone have working lmtp configuration with logging available? Thank you very much! I am sorry that my message is so long, but i tried even more to get it working... :) Any help or idea of how to better debug the situation or search for solution is kindly appreciated.

2 2

Re: Doveadm error since 2.3.11.3 when run as unprivileged user
by Olav Seyfarth 11 Sep '20

11 Sep '20

On Thu, Aug 13, 2020, Arjen de Korte wrote: > include optional config file and suppress errors if it doesn't exist I followed this workaround, which is nicely summarized at https://github.com/postfixadmin/postfixadmin/issues/381#issuecomment-678269… Yet, not all errors are suppressed this way: $ doveadm pw doveadm(olav): Error: net_connect_unix(/run/dovecot/stats-writer) failed: Permission denied $ ls -l /run/dovecot/stats-writer srw-rw---- 1 root dovecot 0 11. Sep 00:27 stats-writer Olav

2 2

Re: Btrfs RAID-10 performance
by Robert Nowotny 11 Sep '20

11 Sep '20

"Miloslav" == Miloslav Hůla <miloslav.hula(a)gmail.com> <miloslav.hula(a)gmail.com> writes: Miloslav> Dne 09.09.2020 v 17:52 John Stoffel napsal(a): Miloslav> There is a one PCIe RAID controller in a chasis. AVAGO Miloslav> MegaRAID SAS 9361-8i. And 16x SAS 15k drives conneced to Miloslav> it. Because the controller does not support pass-through for Miloslav> the drives, we use 16x RAID-0 on controller. So, we get Miloslav> /dev/sda ... /dev/sdp (roughly) in OS. And over that we have Miloslav> single btrfs RAID-10, composed of 16 devices, mounted as Miloslav> /data. I will bet that this is one of your bottlenecks as well. Get a secord or third controller and split your disks across them evenly. Miloslav> That's plan for a next step. Miloslav> We run 'rsync' to remote NAS daily. It takes about 6.5 hours to finish, Miloslav> 12'265'387 files last night. That's.... sucky. So basically you're hitting the drives hard with random IOPs and you're probably running out of performance. How much space are you using on the filesystem? Miloslav> It's not so sucky how it seems. rsync runs during the Miloslav> night. And even reading is high, server load stays low. We Miloslav> have problems with writes. Ok. So putting in an SSD pair to cache things should help. And why not use brtfs send to ship off snapshots instead of using rsync? I'm sure that would be an improvement... Miloslav> We run backup to external NAS (NetApp) for a disaster Miloslav> recovery scenario. Moreover NAS is spreaded across multiple Miloslav> locations. Then we create NAS snapshot, tens days Miloslav> backward. All snapshots easily available via NFS mount. And Miloslav> NAS capacity is cheaper. So why not run the backend storage on the Netapp, and just keep the indexes and such local to the system? I've run Netapps for many years and they work really well. And then you'd get automatic backups using schedule snapshots. Keep the index files local on disk/SSDs and put the maildirs out to NFSv3 volume(s) on the Netapp(s). Should do wonders. And you'll stop needing to do rsync at night. Miloslav> It's the option we have in minds. As you wrote, NetApp is very solid. Miloslav> The main reason for local storage is, that IMAP server is completely Miloslav> isolated from network. But maybe one day will use it. It's not completely isolated, it can rsync data to another host that has access to the Netapp. **grin** Miloslav> Unfortunately, to quickly fix the problem and make server Miloslav> usable again, we already added SSD and moved indexes on Miloslav> it. So we have no measurements in old state. That's ok, if it's better, then its better. How is the load now? Looking at the output of 'iostat -x 30' might be a good thing. Miloslav> Situation is better, but I guess, problem still exists. I Miloslav> takes some time to load be growing. We will see. Hmm... how did you setup the new indexes volume? Did you just use btrfs again? Did you mirror your SSDs as well? Do the indexes fill the SSD, or is there 20-30% free space? When an SSD gets fragmented, it's performance can drop quite a bit. Did you put the SSDs onto a seperate controller? Probably not. So now you've just increased the load on the single controller, when you really should be spreading it out more to improve things. Another possible hack would be to move some stuff to a RAM disk, assuming your server is on a UPS/Generator incase of power loss. But that's an unsafe hack. Also, do you have quotas turned on? That's a performance hit for sure. Miloslav> Thank you for the fio tip. Definetly I'll try that. It's a good way to test and measure how the system will react. Unfortunately, you will need to do your testing outside of normal work hours so as to not impact your users too much. Good luck! Please post some numbers if you get them. If you see only a few disks are 75% or more busy, then **maybe** you have a bad disk in the system, and moving off that disk or replacing it might help. Again, hard to know. Rebalancing btrfs might also help, especially now that you've moved the indexes off that volume. Robert: Fio is an acronym for *Flexible IO Tester* and describes a tool for measuring IO performance. With Fio, devices such as hard drives or SSDs can be tested for their speed by executing a *user-defined **workload* and collecting performance data. Therefore it might be difficult to really simulate the load with that, because You need to define the workload Yourself. But, at least, You might use it to get an idea of maximum transfer rates, random I/O etc. *iotop* shows the *current I/O transfer **rates* for the currently running processes / threads. It uses the I/O usage information of the Linux kernel, so it might be a good tool for You. *htop *might also be Your friend. and of course, as someone else pointed out : > Looking at the output of 'iostat -x 30' might be a good thing. my default setup for such diagnosis is to run iotop, htop, ntop to get an idea whats going on and measure ... for the index-SSD (on a different Channel ? NvMe ?) I strongly suggest to compare EXT4 vs BTRFS - some Benchmarks suggest that on fast SSD´s EXT4 is much (factor 4-5) faster then btrfs. You might also try ZSTD transparent compression on the remaining (non-SSD) btrfs disks - this can boost the transfer rate (since the data is compressed) if the Server CPU´s have enough juice. But from my guts feeling, its not the transfer rate - its the random I/O (in that case the situation will get worse with compression), but You really need to measure that. In general I dont understand that (old fashioned) configuration - using a bunch of 2TB or even 4TB TLC SSD in Raid10 is nowadays really affordable. You can keep the old disks for (local) rsync before You rsync from there to the NAS ... Robert

1 0

Bug in sieve script compilation
by Иннокентий Сенновский 10 Sep '20

10 Sep '20

Hi, this is Rumata888 from hackerone. This is the other bug I found in sievec. If you try to compile the following script, it will result in a panic. The reason is: export expects its arguments to be either strings or lists of strings. When we issue an export command with a number immediately followed by an export command with a string argument, on parsing the second command sievec tries to merge the commands by merging the arguments. When it encounters an argument of unexpected type, it panics. Once again, I would advise unlinking invalid commands from ast, so that parsing of following commands was not dependent on previous irregularities.

1 0

dbox alternate storage and archived namespace
by Daniel Miller 10 Sep '20

10 Sep '20

This may (and probably does) come under the heading of "really dumb ideas", but... Before I develop this further I need to ask - is it possible to have a "primary" mail_location using single-dbox with an alternate storage using multi-dbox? This is *not* the same as different storages for different namespaces (already have that). -- Daniel

1 0

Re: zlib errors after upgrading to 2.3.11.3
by Robert Nowotny 09 Sep '20

09 Sep '20

On 8. Sep 2020, at 12.35, Robert Nowotny <rnowotny1966(a)gmail.com> wrote: Dear Aki, I switched to "gz" now, since "zstd" also gave some errors on writing to files. What kind of errors? Probably this? : Panic: file ostream.c: line 287 (o_stream_sendv_int): assertion failed: (!stream->blocking) can be - unfortunately I do not have the log files anymore... Sep 3 08:33:25 lxc-imap dovecot: imap(mpaul)<48684><2/9E5mKuAezAqKjk>: Error: Mailbox Sent: UID=2171: read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe (FETCH BODY[]) Also this way you can see if the broken mail is actually xz or zstd or zlib. It would be nice to know if there are any zstd or zlib compressed mails that have problems. We did a lot of stress testing with zstd and also with xz, but haven't been able to reproduce any problems. It's also strange that it says taht the error is "Broken pipe" - that doesn't indicate that the mail is corrupted but that there is something more strange going on. So perhaps you don't actually have any mails written as corrupted, but Dovecot is just somehow having trouble reading the mails. I managed to reproduce this. The files aren't corrupted, it's just that reading is failing. The attached patch should fix the xz code and should make your files readable again. Yes, it works ! I switched now to "zstd" compression and currently re-compressing all user imap folders (one by one) thank You so much ! Robert

2 1

Leaked Events
by bobby 08 Sep '20

08 Sep '20

I am going through my maillog file, and noticed these events: Sep 8 15:56:17 mail dovecot[294462]: lmtp(391257): Warning: Event 0x55db5bc41200 leaked (parent=0x55db5bc3b830): mail-storage-service.c:1325 Sep 8 15:56:17 mail dovecot[294462]: lmtp(391257): Warning: Event 0x55db5bc3b830 leaked (parent=0x55db5bc34450): smtp-server-recipient.c:38 Sep 8 15:56:17 mail dovecot[294462]: lmtp(391257): Warning: Event 0x55db5bc34450 leaked (parent=0x55db5bc34130): connection.c:523 Sep 8 15:56:17 mail dovecot[294462]: lmtp(391257): Warning: Event 0x55db5bc34130 leaked (parent=0x55db5bc20ce0): smtp-server-connection.c:773 Sep 8 15:56:17 mail dovecot[294462]: lmtp(391257): Warning: Event 0x55db5bc20ce0 leaked (parent=(nil)): lmtp-client.c:164 Is this something to be concerned about?

2 1

zlib errors after upgrading to 2.3.11.3
by Robert Nowotny 08 Sep '20

08 Sep '20

Dear Aki, I switched to "gz" now, since "zstd" also gave some errors on writing to files. I dont know if "xz" compression or "zstd" shreddered my MDBOX Files, but I lost 4 days of mail. (a couple of thousand mails). After restoring the backup (what was made after switching to version 2.3.11.3) I still have some broken mdfiles, but not too many. Interestingly always in the "Sent" Mailbox on a number of Users. I just can not go back to 20.08.2020 before I updated to 2.3.11.3 - too many emails would have been lost. So - the current status is 2.3.11.3, with "gz" compression. I force-synced and re-indexed all the mdbox files (250 GB), but still have some broken - please how can I fix those ? You stated : (In theory you could leave the existing mails xz-compressed, but best would be to re-compress everything via dsync so old mails can be read when we eventually remove xz support.) what is the optimal way to do that, expecially not loosing the index (?) for outlook/thunderbird, for third party tools that rely on some index/hash (dunno how exactly that works) ? Sep 3 08:33:25 lxc-imap dovecot: imap(mpaul)<48684><2/9E5mKuAezAqKjk>: Error: Mailbox Sent: UID=2171: read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe (FETCH BODY[]) Sep 3 08:35:26 lxc-imap dovecot: imap(mpaul)<48089><YRio1GKuEdHAqKi3>: Error: Mailbox Sent: UID=2171: read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe (FETCH BODY[]) Sep 3 08:35:26 lxc-imap dovecot: imap(mpaul)<49228><p/0g9WKuVdHAqKi3>: Error: Mailbox Sent: UID=2171: read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe (FETCH BODY[]) Sep 3 08:35:26 lxc-imap dovecot: imap(mpaul)<49229><GMAm9WKuVtHAqKi3>: Error: Mailbox Sent: UID=2171: read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe (FETCH BODY[]) Sep 3 08:35:27 lxc-imap dovecot: imap(mpaul)<49230><YYot9WKuV9HAqKi3>: Error: Mailbox Sent: UID=2171: read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe (FETCH BODY[]) ... Sep 3 09:04:44 lxc-imap dovecot: imap(cpotzinger)<49040></X+P72KuUNHAqKi3>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:04:51 lxc-imap dovecot: imap(cpotzinger)<54382><eYT1XWOux9HAqKi3>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:04:59 lxc-imap dovecot: imap(cpotzinger)<54396><nKllXmOuyNHAqKi3>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:05:06 lxc-imap dovecot: imap(cpotzinger)<54409><m2TOXmOuydHAqKi3>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:05:12 lxc-imap dovecot: imap(cpotzinger)<54422><IP86X2OuytHAqKi3>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) ... Sep 3 09:08:58 lxc-imap dovecot: imap(cpotzinger)<54867><BtGtbGOu49HAqKi3>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:19:15 lxc-imap dovecot: imap(mpaul)<56030></kHNkWOuAO3AqKjk>: Error: Mailbox Sent: UID=2171: read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe (FETCH BODY[]) Sep 3 09:29:41 lxc-imap dovecot: imap(cpotzinger)<57274><c4rCtWOuKuzAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:29:47 lxc-imap dovecot: imap(cpotzinger)<57320><K3Uqt2OuLOzAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:29:54 lxc-imap dovecot: imap(cpotzinger)<57333><e4aMt2OuLezAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:30:01 lxc-imap dovecot: imap(cpotzinger)<57346><JEfzt2OuLuzAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:30:08 lxc-imap dovecot: imap(cpotzinger)<57360><mH1YuGOuMOzAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:30:14 lxc-imap dovecot: imap(cpotzinger)<57382><IzLBuGOuMezAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:30:21 lxc-imap dovecot: imap(cpotzinger)<57397><q10kuWOuM+zAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:30:27 lxc-imap dovecot: imap(cpotzinger)<57410><CSuJuWOuNOzAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:31:22 lxc-imap dovecot: imap(mpaul)<57525><TfoOvWOuWO3AqKjk>: Error: Mailbox Sent: UID=2171: read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe (FETCH BODY[]) Sep 3 09:31:49 lxc-imap dovecot: imap(cpotzinger)<57423><Be/ruWOuNuzAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:31:55 lxc-imap dovecot: imap(cpotzinger)<57584><yvXOvmOuQuzAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:32:02 lxc-imap dovecot: imap(cpotzinger)<57597><e/8tv2OuQ+zAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:32:08 lxc-imap dovecot: imap(cpotzinger)<57611><BLSOv2OuROzAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:32:19 lxc-imap dovecot: imap(cpotzinger)<57633><Ky0twGOuR+zAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:32:26 lxc-imap dovecot: imap(cpotzinger)<57646><ljSRwGOuS+zAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:32:33 lxc-imap dovecot: imap(cpotzinger)<57659><rn75wGOuT+zAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:32:39 lxc-imap dovecot: imap(cpotzinger)<57672><GP9jwWOuUuzAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:33:45 lxc-imap dovecot: imap(mpaul)<57795><2CEwxWOubu3AqKjk>: Error: Mailbox Sent: UID=2171: read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe (FETCH BODY[]) Sep 3 09:38:29 lxc-imap dovecot: imap(mpaul)<58347><CKyX1mOuie3AqKjk>: Error: Mailbox Sent: UID=2171: read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe (FETCH BODY[]) Sep 3 09:38:32 lxc-imap dovecot: imap(mpaul)<58352><osHU1mOuiu3AqKjk>: Error: Mailbox Sent: UID=2171: read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe (FETCH BODY[]) Sep 3 09:38:44 lxc-imap dovecot: imap(cpotzinger)<54527><5ZpiYmOuBOfAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:38:51 lxc-imap dovecot: imap(cpotzinger)<58378><kAuN12OumuzAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:38:57 lxc-imap dovecot: imap(cpotzinger)<58392><xJLx12Oum+zAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) ... Am 26.08.2020 um 09:42 schrieb Robert Nowotny: I had 4 broken mdbox files for 4 different users. After deleting them, dovecot re-indexed, now everything is ok again. I can try to update tonight again to 2.3.11.3 and see if some broken files will appear again tomorrow. Am 25.08.2020 um 14:51 schrieb Aki Tuomi: On 25/08/2020 14:35 Robert Nowotny <rnowotny(a)rotek.at> <rnowotny(a)rotek.at> wrote: I get ZLIB Errors after dovecot upgrade from 2.3.10.1 to 2.3.11.3 Aug 21 15:27:34 lxc-imap dovecot: imap(acsida)<63870><jZk...>: Error: Mailbox Sent: UID=40826: read(zlib(/home/vmail/virtualmailboxes/acsida/storage/m.2409)) failed: read(/home/vmail/virtualmailboxes/acsida/storage/m.2409) failed: Broken pipe (FETCH BODY[]).... many of those entries.... after installing2.3.10.1 those files are still broken - but new files are working again. anything I can tweak or do ? Hi! Is it possible your mdbox file is corrupted? Can you share doveconf -n? Aki

2 3

Re: Re: Outlook problems on Apple systems(Mac os)
by hfh＠cndns.com 08 Sep '20

08 Sep '20

Outlook version is 16.40 hfh(a)cndns.com From: Jerry Date: 2020-09-08 13:22 To: Dovecot Mailing List Subject: Re: Outlook problems on Apple systems(Mac os) On Tue, 8 Sep 2020 09:57:59 +0800, hfh(a)cndns.com <hfh(a)cndns.com> stated: >Dovecot is on version 2.2.36. I tested it. Outlook on the Apple system >successfully logged in with the imap protocol, and then sent the LIST >command and outlook did not send any further. Dovecot returns the >results as follows: > >A003 OK LIST completed (0.001 + 0.000 secs). > > >There is no problem with version 2.1.17 using dovecot before. The >information returned by dovecot in version 2.1.17 is as follows: > >A003 OK LIST completed. > > >I suspect that the feedback of the following information causes >Outlook to not recognize it normally: > >(0.001 + 0.000 secs) What version of Outlook? -- Jerry

2 1

Outlook problems on Apple systems(Mac os)
by hfh＠cndns.com 08 Sep '20

08 Sep '20

Dovecot is on version 2.2.36. I tested it. Outlook on the Apple system successfully logged in with the imap protocol, and then sent the LIST command and outlook did not send any further. Dovecot returns the results as follows: A003 OK LIST completed (0.001 + 0.000 secs). There is no problem with version 2.1.17 using dovecot before. The information returned by dovecot in version 2.1.17 is as follows: A003 OK LIST completed. I suspect that the feedback of the following information causes Outlook to not recognize it normally: (0.001 + 0.000 secs) hfh(a)cndns.com

2 1

Search in Mailbox via doveadm REST-API
by rudolf＠padaru.de 07 Sep '20

07 Sep '20

Hey, i need some help with the http-api from doveadm, i try to search a string in the mail-text from all mails in my inbox. My curl-post looks like these: curl -v -X POST -u <user>:<password> -H "Content-Type: application/json" -d '[["search",{"query":["mailbox","INBOX","TEXT","8887"],"user":"<mailUser>"}, "tag1"]]' http://<server>:8080/doveadm/v1 | python -mjson.tool > out.json I receive only empty results, do have anyone an idea? Mit freundlichen Grüßen Yours sincerely <https://padaru.de/> Pascal Rudolf | Geschäftsführer PADARU-IT | Bielebohstraße 10 | 02736 Beiersdorf @: rudolf(a)padaru.de | www: <http://padaru.de/> padaru.de mobil: +49 (0) 17 772 595 32

1 0

LMTP-Process stays in RCPT TO state
by Pascal Christen 07 Sep '20

07 Sep '20

Hi We tried to update our Dovecot Director-Ring to 2.3 (latest version) and had one problem/bug. When we try to deliver an email from Exim to a receiving Dovecot (2.3 latest) via the LMTP-Proxy the LMTP process stays open forever and never closes. So after a few hours, we hit the LMPT-Process limit. The problem only exists if the receiving Dovecot reports "Quota exceeded (mailbox for user is full)" back to the Exim. If this is the case the state on the LMPT process stays at "lmtp: [_IP_ RCPT TO] (lmtp) and never closes. The TCP-Connection to the Exim is then in CLOSE_WAIT state on the Proxy. On the receiving Dovecot we set the "lmtp_rcpt_check_quota=yes" option. For me, it sounds similar to this bug: https://dovecot.org/list/dovecot/2018-January/110697.html but without the SEGFAULT which makes it harder to debug it. May you guys know why it doesn't change the state to IDLE or READY after reporting the quota issue or see something fishy in the commit above. If not I try to get a trace of the hanging process. Greetings Pascal

2 6

Backup questions - errors
by Mark Patruck 07 Sep '20

07 Sep '20

Hi, although i've search the archives, there are questions left regarding backup strategies and errors that occur. I'm using dovecot 2.3.11.3 with mdbox (mailbox size 50MB - 30GB). Every night, i ran the following command: doveadm -o mail_fsync=never -o plugin/quota= -o plugin/zlib_save=gz \ backup -u account mdbox:/nfs/storage/account/mdbox ### incremental backup Thanks to zlib compression, i save a lot of storage space, nonetheless i'd go with an incremental backup, but it seems to only merge changes, comparing to rsyncs' --link-dest option where you can put all "changes of the day" in an extra folder. I'd like to prevent loosing mail that gets deleted by accident/on purpose. How can i achieve this with dsync and w/o doing a full backup every day? ### backup errors After a few days w/o any errors, doveadm prints the following lines (only for the biggest, 30GB mailbox) two days in a row. As dovecot has self- healing capabilities, i think these errors/warnings can be ignored and should go away...? Error: Transaction log ....mdbox/dovecot.list.index.log: duplicate transaction log sequence (38) Error: Index ....mdbox/dovecot.list.index: Lost log for seq=38 offset=63312: Missing middle file seq=38 (between 38..4294967295, we have seqs 37,39): .log.2 contains file_seq=37 (initial_mapped=1, reason=Index mapped) Warning: fscking index file ....mdbox/dovecot.list.index Error: Fixed index file ....mdbox/dovecot.list.index: log_file_seq 38 -> 39 Error: Mailbox list index was marked as fsck'd ....mdbox/dovecot.list.index Error: Mailbox list index was marked as fsck'd ....mdbox/dovecot.list.index Error: Mailbox list index was marked as fsck'd ....mdbox/dovecot.list.index Thanks, -Mark -- Mark Patruck ( mark at wrapped.cx ) GPG key 0xF2865E51 / 187F F6D3 EE04 1DCE 1C74 F644 0D3C F66F F286 5E51 https://www.wrapped.cx

1 1

sieve_max_script_size is ignored
by Zdeněk Zámečník 06 Sep '20

06 Sep '20

I am in troubles with compiling sieve scripts larger than 1MB. I see in logs following errors: Aug 19 13:10:26 mail dovecot: lmtp(z.z(a)xxx.xxx)<22117><uNBGHKIIPV9lVgAA5ldI4A>: Error: sieve: autoreply: line 16818: quoted string started at line 3 is too long (longer than 1048576 bytes) Aug 19 13:10:26 mail dovecot: lmtp(z.z(a)xxx.xxx)<22117><uNBGHKIIPV9lVgAA5ldI4A>: Error: sieve: autoreply: parse failed Aug 19 13:10:26 mail dovecot: lmtp(z.z(a)xxx.xxx)<22117><uNBGHKIIPV9lVgAA5ldI4A>: Error: sieve: Failed to compile script `/var/vmail/xxx.xxx/z.z/sieve/autoreply.sieve' I tried to set sieve_max_script_size = 2M or 0 (unlimited). I also tried to run "sievec -o sieve_max_script_size=2M /var/vmail/xxx.xxx/z.z/sieve/autoreply.sieve" but the result is still same. Below is my config. Can you please advice where can be the problem? # 2.3.10.1 (a3d0e1171): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.10 (67bf5bd7) # OS: Linux 5.4.44-2-pve x86_64 Debian 10.4 # Hostname: mail.xxx.xxx auth_cache_size = 2 M auth_cache_ttl = 5 mins auth_master_user_separator = * auth_mechanisms = plain login auth_policy_check_before_auth = no auth_policy_hash_nonce = # hidden, use -P to show it auth_policy_report_after_auth = no auth_policy_server_timeout_msecs = 1500 auth_policy_server_url = http://127.0.0.1:8090/ dict { acl = pgsql:/etc/dovecot/dovecot-dict-sql.conf.ext quota = pgsql:/etc/dovecot/dovecot-dict-sql.conf.ext } disable_plaintext_auth = no imap_client_workarounds = delay-newmail imap_id_log = * imap_id_retain = yes login_trusted_networks = 127.0.0.0/8 mail_gid = 2000 mail_home = /var/vmail/%d/%n mail_location = mdbox:~/mdbox:ALT=/var/vmail-archive/%d/%n/mdbox mail_max_userip_connections = 60 mail_plugins = acl zlib fts quota mail_uid = 2000 managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext vacation-seconds editheader imapsieve vnd.dovecot.imapsieve mdbox_rotate_interval = 1 days mdbox_rotate_size = 16 M namespace { inbox = yes location = mailbox "Deleted Items" { auto = no special_use = \Trash } mailbox "Deleted Messages" { auto = no special_use = \Trash } mailbox Drafts { auto = no special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk } mailbox "Junk E-mail" { auto = no special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Items" { auto = no special_use = \Sent } mailbox "Sent Messages" { auto = no special_use = \Sent } mailbox Spam { auto = no special_use = \Junk } mailbox Trash { special_use = \Trash } mailbox virtual/All { auto = no special_use = \All } prefix = separator = / type = private } namespace { inbox = no list = children location = mdbox:/var/vmail/%%d/%%n/mdbox:ALT=/var/vmail-archive/%%d/%%n/mdbox prefix = shared/%%u/ separator = / subscriptions = no type = shared } passdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } passdb { args = /etc/dovecot/masters.db driver = passwd-file master = yes pass = yes } plugin { acl = vfile acl_shared_dict = proxy::acl imapsieve_mailbox1_before = file:/var/vmail/sieve/report-spam.sieve imapsieve_mailbox1_causes = COPY imapsieve_mailbox1_name = Spam imapsieve_mailbox2_before = file:/var/vmail/sieve/report-ham.sieve imapsieve_mailbox2_causes = COPY imapsieve_mailbox2_from = Spam imapsieve_mailbox2_name = * imapsieve_mailbox3_before = file:/var/vmail/sieve/report-spam.sieve imapsieve_mailbox3_causes = COPY imapsieve_mailbox3_name = Junk imapsieve_mailbox4_before = file:/var/vmail/sieve/report-ham.sieve imapsieve_mailbox4_causes = COPY imapsieve_mailbox4_from = Junk imapsieve_mailbox4_name = * mailbox_alias_new = Sent Messages mailbox_alias_new2 = Sent Items mailbox_alias_new3 = Deleted Items mailbox_alias_old = Sent mailbox_alias_old2 = Sent mailbox_alias_old3 = Trash quota = dict:User quota::proxy::quota quota_grace = 10%% quota_rule2 = Trash:ignore quota_rule3 = Junk:ignore quota_warning = storage=80%% quota-warning 90 %u quota_warning2 = storage=85%% quota-warning 95 %u quota_warning3 = storage=95%% quota-warning 105 %u sieve = /var/vmail/%d/%n/sieve/.sieve sieve_after = /var/vmail/%d/%n/sieve/autoreply.sieve sieve_before = /var/vmail/sieve/global.sieve sieve_dir = /var/vmail/%d/%n/sieve sieve_extensions = +editheader +vacation-seconds sieve_global_dir = /var/vmail/sieve/ sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment sieve_max_redirects = 20 sieve_max_script_size = 2M sieve_pipe_bin_dir = /usr/lib/dovecot sieve_plugins = sieve_imapsieve sieve_extprograms sieve_vacation_send_from_recipient = yes } protocols = imap sieve lmtp pop3 service auth-worker { unix_listener auth-worker { user = vmail } user = $default_internal_user } service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-userdb { mode = 0600 user = vmail } user = vmail } service dict { unix_listener dict { mode = 0600 user = vmail } } service imap-login { process_min_avail = 10 service_count = 0 vsz_limit = 512 M } service imap { executable = imap process_limit = 3500 unix_listener imap-master { user = $default_internal_user } vsz_limit = 2 G } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } } service managesieve-login { inet_listener sieve { port = 4190 } process_min_avail = 0 service_count = 1 vsz_limit = 64 M } service managesieve { process_limit = 400 } service pop3-login { process_min_avail = 3 service_count = 0 vsz_limit = 320 M } service pop3 { process_limit = 200 vsz_limit = 320 M } service quota-warning { executable = script /etc/dovecot/quota_warning.sh unix_listener quota-warning { mode = 0666 user = vmail } user = vmail } ssl_cert = </etc/ssl/private/multi.z-technics.cz.dovecot.pem ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it stats_writer_socket_path = userdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } protocol sieve { managesieve_implementation_string = Dovecot Pigeonhole managesieve_max_compile_errors = 5 } protocol imap { mail_plugins = quota imap_quota fts mailbox_alias imap_acl acl imap_zlib imap_sieve } protocol lda { mail_fsync = optimized } protocol lmtp { mail_fsync = optimized mail_plugins = quota sieve acl }

2 4

imapc master user & shared mailboxes
by Toma 04 Sep '20

04 Sep '20

Hello, I am trying to configure dovecot like this: https://wiki.dovecot.org/SharedMailboxes/ClusterSetup imapc_host = 127.0.0.1 imapc_master_user = %u #imapc_user = imapc_password = XXXXXXXX namespace { type = shared separator = / prefix = shared/%%u/ location = imapc:~/shared/%%u } plugin { acl_shared_dict = proxy::acl } However, contrary to the article claims, imapc master user is set to the destination username. This makes it impossible to check the ACLs against the logged user. If user.b shares mailbox with user.a, accessing /shared/user.b as user.a a will log in as "user.b*user.b". Can we tell dovecot to use the currently authenticated user for imapc_master_user and the remote user for imapc_user, so it logs in as "user.b*user.a"? I'm using dovecot 2.3.11.3 (502c39af9). Thanks! Toma

1 0

syslog logging + 2.3.10 + centos 7
by Matt Bryant 04 Sep '20

04 Sep '20

All, Running dovecot 2.3.10 on a CentOS 7.x and logging via syslog seems to stop when i restart rsyslog process. Reload doesnt get it working again only a dovecot restart. Tested this with dovecot 2.2.36 and this did not exhibit the same behaviour. systemctl restart dovecot rsyslog ----- this gets a working state systemctl restart rsyslog ------------- this restarts syslog other processes like exim continue to log but dovecot stops systemctl reload dovecot -------------- this has no impact logging still does not work systemctl restart dovecot ------------- this fixes the issue. Are there any changes to the log process between 2.2 and 2.3 that might explain this ? rgds Matt

1 1

How to backup maildir
by Robert Moskowitz 04 Sep '20

04 Sep '20

I have been thinking, and reading, on how to back up my mailserver. I have not found any approach that seems ready to use. I have run years without any backup, but would really like to have something in place. I figure I can attach a USB drive and backup to that, then from there rsync to something elsewhere. Further if that USB drive is a full mailserver image, I actually have a 'hot backup' where I only have to put the backup drive into a system and boot up at the last backup. But this means properly copying all of /home/vmail and probably /home/sieve plus the /var/lib/mysql Are there good tools that nicely does this? Or do I choose a time late at night (only I am sometimes in non-US timezones) to shut down all services and just use rsync? And stopping services itself is thought provoking. What if Dovecot, amavis, mysql, or whatelse is in the middle of writing out a mail file what happens to that file and restart. Just scary stuff and, in part, why I have never tackled this in the past. thanks for all feedback

4 6

Deleted mail reappearing
by John S 04 Sep '20

04 Sep '20

Recently I upgraded dovecot quite a few major versions. Since then some mailboxes are experiencing deleted mail reappearing. It seems to be happening roughly once a day, but not for every mail client or mailbox, I have at least two clients with this issue: Thunderbird and Outlook. Storage is local under two partitions, one as alternative storage. # 2.3.7.2 (3c910f64b): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.7.2 () # OS: Linux 5.4.0-29-generic x86_64 Ubuntu 20.04 LTS ext4 auth_mechanisms = plain login digest-md5 base_dir = /var/run/dovecot/ default_client_limit = 4000 default_vsz_limit = 512 M dict { expire = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext quota = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext } disable_plaintext_auth = no lda_mailbox_autocreate = yes log_timestamp = "%Y-%m-%d %H:%M:%S " mail_location = mdbox:/home/vmail/%d/%n mail_max_userip_connections = 240 mail_plugins = quota mail_privileged_group = mail mdbox_rotate_size = 64 M namespace { inbox = yes location = prefix = separator = . } passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } plugin { quota = dict:user_quota::proxy:/var/run/dovecot/dict:quota quota_warning = storage=95%% quota-warning 95 %u quota_warning2 = storage=80%% quota-warning 80 %u sieve = file:~/sieve;active=~/.dovecot.sieve sieve_dir = ~/sieve sieve_global_dir = /etc/dovecot/sieve/ } protocols = imap pop3 service anvil { client_limit = 2615 } service auth { client_limit = 5536 unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-master { group = root mode = 0600 user = vmail } } service dict { unix_listener dict { mode = 0600 user = vmail } } service imap-login { client_limit = 2115 executable = /usr/lib/dovecot/imap-login inet_listener imap { address = * port = 143 } inet_listener imaps { address = * port = 993 } process_limit = 2000 } service imap { executable = /usr/lib/dovecot/imap process_limit = 2000 } service pop3-login { client_limit = 1024 executable = /usr/lib/dovecot/pop3-login inet_listener pop3 { address = * port = 110 } inet_listener pop3s { address = * port = 995 } process_limit = 512 } service pop3 { executable = /usr/lib/dovecot/pop3 } service stats { unix_listener stats-reader { group = vmail mode = 0660 user = vmail } unix_listener stats-writer { group = vmail mode = 0660 user = vmail } } ssl_cert = < /etc/dovecot/ssl/crt.pem ssl_cipher_list = ALL:!LOW:!SSLv2:!EXP:!aNULL ssl_client_ca_dir = /etc/ssl/certs ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it ssl_prefer_server_ciphers = yes syslog_facility = local0 userdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } protocol imap { imap_max_line_length = 64 k mail_plugins = quota imap_quota } protocol pop3 { mail_plugins = quota pop3_uidl_format = %08Xu%08Xv } protocol lda { auth_socket_path = /var/run/dovecot/auth-master mail_plugins = quota sieve postmaster_address = postmaster@localhost sendmail_path = /usr/lib/sendmail } dovecot-sql.conf: driver = mysql connect = host=127.0.0.1 dbname=mail user=mail password=#### password_query = SELECT IF('%m' = 'DIGEST-MD5', users.password_digest_md5, NULL) as password, IF('%m' = 'DIGEST-MD5', NULL, 'Y') as nopassword, username FROM users LEFT JOIN domains ON users.domain = domains.domain WHERE users.username = '%u' AND users.domain = '%d' AND users.active = 'yes' AND domains.state = '1' AND ((users.password = encrypt('%w', LEFT(users.password, 11)) AND ('%m' = 'PLAIN' OR '%m' = 'LOGIN')) OR ('%m' = 'DIGEST-MD5')) user_query = SELECT maildir as home, uid, gid, CONCAT(CONCAT(type, ':', maildir), ':ALT=/home/altmail/%d/%n') AS mail FROM users LEFT JOIN domains ON users.domain = domains.domain WHERE users.username = '%u' AND domains.state = '1' iterate_query = SELECT SUBSTRING_INDEX(email, '@', 1) AS username, domain FROM users;

3 3

Sieve: deleteheader now working with duplicate filter for implicit keep
by Alec Moskvin 04 Sep '20

04 Sep '20

Hello, I have a rule to always delete a header. If the message gets fileinto'd, the header is gone, but if it's delivered into the INBOX through implicit keep, the header does not get deleted. Interestingly, if I remove the "if duplicate" filter, the header does get deleted as expected. I'm using the latest Dovecot 2.3.11.3 / Pigeonhole 0.5.11. Any idea why this is happening? Thanks, Alec === Sieve filter require ["fileinto", "editheader", "duplicate", "mailbox"]; deleteheader "X-Bad-Header"; if duplicate { fileinto :create "Trash"; stop; } if header :matches "subject" "*filter*" { fileinto :create "Filtered"; } === Example (note ${RANDOM} to bypass duplicate filter) $ /usr/libexec/dovecot/deliver << -end- To: me(a)example.com From: someone(a)example.com Subject: Test Message Message-Id: <${RANDOM}(a)example.net> Date: Thu, 20 Aug 2020 00:21:24 +0000 (UTC) X-Bad-Header: DELETE ME Hello world -end-

2 4