- dovecot - dovecot.org

Re: Doveadm error since 2.3.11.3 when run as unprivileged user
by Olav Seyfarth 11 Sep '20

11 Sep '20

On Thu, Aug 13, 2020, Arjen de Korte wrote: > include optional config file and suppress errors if it doesn't exist I followed this workaround, which is nicely summarized at https://github.com/postfixadmin/postfixadmin/issues/381#issuecomment-678269… Yet, not all errors are suppressed this way: $ doveadm pw doveadm(olav): Error: net_connect_unix(/run/dovecot/stats-writer) failed: Permission denied $ ls -l /run/dovecot/stats-writer srw-rw---- 1 root dovecot 0 11. Sep 00:27 stats-writer Olav

2 2

Re: Btrfs RAID-10 performance
by Robert Nowotny 11 Sep '20

11 Sep '20

"Miloslav" == Miloslav Hůla <miloslav.hula(a)gmail.com> <miloslav.hula(a)gmail.com> writes: Miloslav> Dne 09.09.2020 v 17:52 John Stoffel napsal(a): Miloslav> There is a one PCIe RAID controller in a chasis. AVAGO Miloslav> MegaRAID SAS 9361-8i. And 16x SAS 15k drives conneced to Miloslav> it. Because the controller does not support pass-through for Miloslav> the drives, we use 16x RAID-0 on controller. So, we get Miloslav> /dev/sda ... /dev/sdp (roughly) in OS. And over that we have Miloslav> single btrfs RAID-10, composed of 16 devices, mounted as Miloslav> /data. I will bet that this is one of your bottlenecks as well. Get a secord or third controller and split your disks across them evenly. Miloslav> That's plan for a next step. Miloslav> We run 'rsync' to remote NAS daily. It takes about 6.5 hours to finish, Miloslav> 12'265'387 files last night. That's.... sucky. So basically you're hitting the drives hard with random IOPs and you're probably running out of performance. How much space are you using on the filesystem? Miloslav> It's not so sucky how it seems. rsync runs during the Miloslav> night. And even reading is high, server load stays low. We Miloslav> have problems with writes. Ok. So putting in an SSD pair to cache things should help. And why not use brtfs send to ship off snapshots instead of using rsync? I'm sure that would be an improvement... Miloslav> We run backup to external NAS (NetApp) for a disaster Miloslav> recovery scenario. Moreover NAS is spreaded across multiple Miloslav> locations. Then we create NAS snapshot, tens days Miloslav> backward. All snapshots easily available via NFS mount. And Miloslav> NAS capacity is cheaper. So why not run the backend storage on the Netapp, and just keep the indexes and such local to the system? I've run Netapps for many years and they work really well. And then you'd get automatic backups using schedule snapshots. Keep the index files local on disk/SSDs and put the maildirs out to NFSv3 volume(s) on the Netapp(s). Should do wonders. And you'll stop needing to do rsync at night. Miloslav> It's the option we have in minds. As you wrote, NetApp is very solid. Miloslav> The main reason for local storage is, that IMAP server is completely Miloslav> isolated from network. But maybe one day will use it. It's not completely isolated, it can rsync data to another host that has access to the Netapp. **grin** Miloslav> Unfortunately, to quickly fix the problem and make server Miloslav> usable again, we already added SSD and moved indexes on Miloslav> it. So we have no measurements in old state. That's ok, if it's better, then its better. How is the load now? Looking at the output of 'iostat -x 30' might be a good thing. Miloslav> Situation is better, but I guess, problem still exists. I Miloslav> takes some time to load be growing. We will see. Hmm... how did you setup the new indexes volume? Did you just use btrfs again? Did you mirror your SSDs as well? Do the indexes fill the SSD, or is there 20-30% free space? When an SSD gets fragmented, it's performance can drop quite a bit. Did you put the SSDs onto a seperate controller? Probably not. So now you've just increased the load on the single controller, when you really should be spreading it out more to improve things. Another possible hack would be to move some stuff to a RAM disk, assuming your server is on a UPS/Generator incase of power loss. But that's an unsafe hack. Also, do you have quotas turned on? That's a performance hit for sure. Miloslav> Thank you for the fio tip. Definetly I'll try that. It's a good way to test and measure how the system will react. Unfortunately, you will need to do your testing outside of normal work hours so as to not impact your users too much. Good luck! Please post some numbers if you get them. If you see only a few disks are 75% or more busy, then **maybe** you have a bad disk in the system, and moving off that disk or replacing it might help. Again, hard to know. Rebalancing btrfs might also help, especially now that you've moved the indexes off that volume. Robert: Fio is an acronym for *Flexible IO Tester* and describes a tool for measuring IO performance. With Fio, devices such as hard drives or SSDs can be tested for their speed by executing a *user-defined **workload* and collecting performance data. Therefore it might be difficult to really simulate the load with that, because You need to define the workload Yourself. But, at least, You might use it to get an idea of maximum transfer rates, random I/O etc. *iotop* shows the *current I/O transfer **rates* for the currently running processes / threads. It uses the I/O usage information of the Linux kernel, so it might be a good tool for You. *htop *might also be Your friend. and of course, as someone else pointed out : > Looking at the output of 'iostat -x 30' might be a good thing. my default setup for such diagnosis is to run iotop, htop, ntop to get an idea whats going on and measure ... for the index-SSD (on a different Channel ? NvMe ?) I strongly suggest to compare EXT4 vs BTRFS - some Benchmarks suggest that on fast SSD´s EXT4 is much (factor 4-5) faster then btrfs. You might also try ZSTD transparent compression on the remaining (non-SSD) btrfs disks - this can boost the transfer rate (since the data is compressed) if the Server CPU´s have enough juice. But from my guts feeling, its not the transfer rate - its the random I/O (in that case the situation will get worse with compression), but You really need to measure that. In general I dont understand that (old fashioned) configuration - using a bunch of 2TB or even 4TB TLC SSD in Raid10 is nowadays really affordable. You can keep the old disks for (local) rsync before You rsync from there to the NAS ... Robert

1 0

Bug in sieve script compilation
by Иннокентий Сенновский 10 Sep '20

10 Sep '20

Hi, this is Rumata888 from hackerone. This is the other bug I found in sievec. If you try to compile the following script, it will result in a panic. The reason is: export expects its arguments to be either strings or lists of strings. When we issue an export command with a number immediately followed by an export command with a string argument, on parsing the second command sievec tries to merge the commands by merging the arguments. When it encounters an argument of unexpected type, it panics. Once again, I would advise unlinking invalid commands from ast, so that parsing of following commands was not dependent on previous irregularities.

1 0

dbox alternate storage and archived namespace
by Daniel Miller 10 Sep '20

10 Sep '20

This may (and probably does) come under the heading of "really dumb ideas", but... Before I develop this further I need to ask - is it possible to have a "primary" mail_location using single-dbox with an alternate storage using multi-dbox? This is *not* the same as different storages for different namespaces (already have that). -- Daniel

1 0

Re: zlib errors after upgrading to 2.3.11.3
by Robert Nowotny 09 Sep '20

09 Sep '20

On 8. Sep 2020, at 12.35, Robert Nowotny <rnowotny1966(a)gmail.com> wrote: Dear Aki, I switched to "gz" now, since "zstd" also gave some errors on writing to files. What kind of errors? Probably this? : Panic: file ostream.c: line 287 (o_stream_sendv_int): assertion failed: (!stream->blocking) can be - unfortunately I do not have the log files anymore... Sep 3 08:33:25 lxc-imap dovecot: imap(mpaul)<48684><2/9E5mKuAezAqKjk>: Error: Mailbox Sent: UID=2171: read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe (FETCH BODY[]) Also this way you can see if the broken mail is actually xz or zstd or zlib. It would be nice to know if there are any zstd or zlib compressed mails that have problems. We did a lot of stress testing with zstd and also with xz, but haven't been able to reproduce any problems. It's also strange that it says taht the error is "Broken pipe" - that doesn't indicate that the mail is corrupted but that there is something more strange going on. So perhaps you don't actually have any mails written as corrupted, but Dovecot is just somehow having trouble reading the mails. I managed to reproduce this. The files aren't corrupted, it's just that reading is failing. The attached patch should fix the xz code and should make your files readable again. Yes, it works ! I switched now to "zstd" compression and currently re-compressing all user imap folders (one by one) thank You so much ! Robert

2 1

Leaked Events
by bobby 08 Sep '20

08 Sep '20

I am going through my maillog file, and noticed these events: Sep 8 15:56:17 mail dovecot[294462]: lmtp(391257): Warning: Event 0x55db5bc41200 leaked (parent=0x55db5bc3b830): mail-storage-service.c:1325 Sep 8 15:56:17 mail dovecot[294462]: lmtp(391257): Warning: Event 0x55db5bc3b830 leaked (parent=0x55db5bc34450): smtp-server-recipient.c:38 Sep 8 15:56:17 mail dovecot[294462]: lmtp(391257): Warning: Event 0x55db5bc34450 leaked (parent=0x55db5bc34130): connection.c:523 Sep 8 15:56:17 mail dovecot[294462]: lmtp(391257): Warning: Event 0x55db5bc34130 leaked (parent=0x55db5bc20ce0): smtp-server-connection.c:773 Sep 8 15:56:17 mail dovecot[294462]: lmtp(391257): Warning: Event 0x55db5bc20ce0 leaked (parent=(nil)): lmtp-client.c:164 Is this something to be concerned about?

2 1

zlib errors after upgrading to 2.3.11.3
by Robert Nowotny 08 Sep '20

08 Sep '20

Dear Aki, I switched to "gz" now, since "zstd" also gave some errors on writing to files. I dont know if "xz" compression or "zstd" shreddered my MDBOX Files, but I lost 4 days of mail. (a couple of thousand mails). After restoring the backup (what was made after switching to version 2.3.11.3) I still have some broken mdfiles, but not too many. Interestingly always in the "Sent" Mailbox on a number of Users. I just can not go back to 20.08.2020 before I updated to 2.3.11.3 - too many emails would have been lost. So - the current status is 2.3.11.3, with "gz" compression. I force-synced and re-indexed all the mdbox files (250 GB), but still have some broken - please how can I fix those ? You stated : (In theory you could leave the existing mails xz-compressed, but best would be to re-compress everything via dsync so old mails can be read when we eventually remove xz support.) what is the optimal way to do that, expecially not loosing the index (?) for outlook/thunderbird, for third party tools that rely on some index/hash (dunno how exactly that works) ? Sep 3 08:33:25 lxc-imap dovecot: imap(mpaul)<48684><2/9E5mKuAezAqKjk>: Error: Mailbox Sent: UID=2171: read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe (FETCH BODY[]) Sep 3 08:35:26 lxc-imap dovecot: imap(mpaul)<48089><YRio1GKuEdHAqKi3>: Error: Mailbox Sent: UID=2171: read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe (FETCH BODY[]) Sep 3 08:35:26 lxc-imap dovecot: imap(mpaul)<49228><p/0g9WKuVdHAqKi3>: Error: Mailbox Sent: UID=2171: read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe (FETCH BODY[]) Sep 3 08:35:26 lxc-imap dovecot: imap(mpaul)<49229><GMAm9WKuVtHAqKi3>: Error: Mailbox Sent: UID=2171: read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe (FETCH BODY[]) Sep 3 08:35:27 lxc-imap dovecot: imap(mpaul)<49230><YYot9WKuV9HAqKi3>: Error: Mailbox Sent: UID=2171: read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe (FETCH BODY[]) ... Sep 3 09:04:44 lxc-imap dovecot: imap(cpotzinger)<49040></X+P72KuUNHAqKi3>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:04:51 lxc-imap dovecot: imap(cpotzinger)<54382><eYT1XWOux9HAqKi3>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:04:59 lxc-imap dovecot: imap(cpotzinger)<54396><nKllXmOuyNHAqKi3>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:05:06 lxc-imap dovecot: imap(cpotzinger)<54409><m2TOXmOuydHAqKi3>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:05:12 lxc-imap dovecot: imap(cpotzinger)<54422><IP86X2OuytHAqKi3>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) ... Sep 3 09:08:58 lxc-imap dovecot: imap(cpotzinger)<54867><BtGtbGOu49HAqKi3>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:19:15 lxc-imap dovecot: imap(mpaul)<56030></kHNkWOuAO3AqKjk>: Error: Mailbox Sent: UID=2171: read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe (FETCH BODY[]) Sep 3 09:29:41 lxc-imap dovecot: imap(cpotzinger)<57274><c4rCtWOuKuzAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:29:47 lxc-imap dovecot: imap(cpotzinger)<57320><K3Uqt2OuLOzAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:29:54 lxc-imap dovecot: imap(cpotzinger)<57333><e4aMt2OuLezAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:30:01 lxc-imap dovecot: imap(cpotzinger)<57346><JEfzt2OuLuzAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:30:08 lxc-imap dovecot: imap(cpotzinger)<57360><mH1YuGOuMOzAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:30:14 lxc-imap dovecot: imap(cpotzinger)<57382><IzLBuGOuMezAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:30:21 lxc-imap dovecot: imap(cpotzinger)<57397><q10kuWOuM+zAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:30:27 lxc-imap dovecot: imap(cpotzinger)<57410><CSuJuWOuNOzAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:31:22 lxc-imap dovecot: imap(mpaul)<57525><TfoOvWOuWO3AqKjk>: Error: Mailbox Sent: UID=2171: read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe (FETCH BODY[]) Sep 3 09:31:49 lxc-imap dovecot: imap(cpotzinger)<57423><Be/ruWOuNuzAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:31:55 lxc-imap dovecot: imap(cpotzinger)<57584><yvXOvmOuQuzAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:32:02 lxc-imap dovecot: imap(cpotzinger)<57597><e/8tv2OuQ+zAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:32:08 lxc-imap dovecot: imap(cpotzinger)<57611><BLSOv2OuROzAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:32:19 lxc-imap dovecot: imap(cpotzinger)<57633><Ky0twGOuR+zAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:32:26 lxc-imap dovecot: imap(cpotzinger)<57646><ljSRwGOuS+zAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:32:33 lxc-imap dovecot: imap(cpotzinger)<57659><rn75wGOuT+zAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:32:39 lxc-imap dovecot: imap(cpotzinger)<57672><GP9jwWOuUuzAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:33:45 lxc-imap dovecot: imap(mpaul)<57795><2CEwxWOubu3AqKjk>: Error: Mailbox Sent: UID=2171: read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe (FETCH BODY[]) Sep 3 09:38:29 lxc-imap dovecot: imap(mpaul)<58347><CKyX1mOuie3AqKjk>: Error: Mailbox Sent: UID=2171: read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe (FETCH BODY[]) Sep 3 09:38:32 lxc-imap dovecot: imap(mpaul)<58352><osHU1mOuiu3AqKjk>: Error: Mailbox Sent: UID=2171: read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe (FETCH BODY[]) Sep 3 09:38:44 lxc-imap dovecot: imap(cpotzinger)<54527><5ZpiYmOuBOfAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:38:51 lxc-imap dovecot: imap(cpotzinger)<58378><kAuN12OumuzAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:38:57 lxc-imap dovecot: imap(cpotzinger)<58392><xJLx12Oum+zAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) ... Am 26.08.2020 um 09:42 schrieb Robert Nowotny: I had 4 broken mdbox files for 4 different users. After deleting them, dovecot re-indexed, now everything is ok again. I can try to update tonight again to 2.3.11.3 and see if some broken files will appear again tomorrow. Am 25.08.2020 um 14:51 schrieb Aki Tuomi: On 25/08/2020 14:35 Robert Nowotny <rnowotny(a)rotek.at> <rnowotny(a)rotek.at> wrote: I get ZLIB Errors after dovecot upgrade from 2.3.10.1 to 2.3.11.3 Aug 21 15:27:34 lxc-imap dovecot: imap(acsida)<63870><jZk...>: Error: Mailbox Sent: UID=40826: read(zlib(/home/vmail/virtualmailboxes/acsida/storage/m.2409)) failed: read(/home/vmail/virtualmailboxes/acsida/storage/m.2409) failed: Broken pipe (FETCH BODY[]).... many of those entries.... after installing2.3.10.1 those files are still broken - but new files are working again. anything I can tweak or do ? Hi! Is it possible your mdbox file is corrupted? Can you share doveconf -n? Aki

2 3

Re: Re: Outlook problems on Apple systems(Mac os)
by hfh＠cndns.com 08 Sep '20

08 Sep '20

Outlook version is 16.40 hfh(a)cndns.com From: Jerry Date: 2020-09-08 13:22 To: Dovecot Mailing List Subject: Re: Outlook problems on Apple systems(Mac os) On Tue, 8 Sep 2020 09:57:59 +0800, hfh(a)cndns.com <hfh(a)cndns.com> stated: >Dovecot is on version 2.2.36. I tested it. Outlook on the Apple system >successfully logged in with the imap protocol, and then sent the LIST >command and outlook did not send any further. Dovecot returns the >results as follows: > >A003 OK LIST completed (0.001 + 0.000 secs). > > >There is no problem with version 2.1.17 using dovecot before. The >information returned by dovecot in version 2.1.17 is as follows: > >A003 OK LIST completed. > > >I suspect that the feedback of the following information causes >Outlook to not recognize it normally: > >(0.001 + 0.000 secs) What version of Outlook? -- Jerry

2 1

Outlook problems on Apple systems(Mac os)
by hfh＠cndns.com 08 Sep '20

08 Sep '20

Dovecot is on version 2.2.36. I tested it. Outlook on the Apple system successfully logged in with the imap protocol, and then sent the LIST command and outlook did not send any further. Dovecot returns the results as follows: A003 OK LIST completed (0.001 + 0.000 secs). There is no problem with version 2.1.17 using dovecot before. The information returned by dovecot in version 2.1.17 is as follows: A003 OK LIST completed. I suspect that the feedback of the following information causes Outlook to not recognize it normally: (0.001 + 0.000 secs) hfh(a)cndns.com

2 1

Search in Mailbox via doveadm REST-API
by rudolf＠padaru.de 07 Sep '20

07 Sep '20

Hey, i need some help with the http-api from doveadm, i try to search a string in the mail-text from all mails in my inbox. My curl-post looks like these: curl -v -X POST -u <user>:<password> -H "Content-Type: application/json" -d '[["search",{"query":["mailbox","INBOX","TEXT","8887"],"user":"<mailUser>"}, "tag1"]]' http://<server>:8080/doveadm/v1 | python -mjson.tool > out.json I receive only empty results, do have anyone an idea? Mit freundlichen Grüßen Yours sincerely <https://padaru.de/> Pascal Rudolf | Geschäftsführer PADARU-IT | Bielebohstraße 10 | 02736 Beiersdorf @: rudolf(a)padaru.de | www: <http://padaru.de/> padaru.de mobil: +49 (0) 17 772 595 32

1 0

LMTP-Process stays in RCPT TO state
by Pascal Christen 07 Sep '20

07 Sep '20

Hi We tried to update our Dovecot Director-Ring to 2.3 (latest version) and had one problem/bug. When we try to deliver an email from Exim to a receiving Dovecot (2.3 latest) via the LMTP-Proxy the LMTP process stays open forever and never closes. So after a few hours, we hit the LMPT-Process limit. The problem only exists if the receiving Dovecot reports "Quota exceeded (mailbox for user is full)" back to the Exim. If this is the case the state on the LMPT process stays at "lmtp: [_IP_ RCPT TO] (lmtp) and never closes. The TCP-Connection to the Exim is then in CLOSE_WAIT state on the Proxy. On the receiving Dovecot we set the "lmtp_rcpt_check_quota=yes" option. For me, it sounds similar to this bug: https://dovecot.org/list/dovecot/2018-January/110697.html but without the SEGFAULT which makes it harder to debug it. May you guys know why it doesn't change the state to IDLE or READY after reporting the quota issue or see something fishy in the commit above. If not I try to get a trace of the hanging process. Greetings Pascal

2 6

Backup questions - errors
by Mark Patruck 07 Sep '20

07 Sep '20

Hi, although i've search the archives, there are questions left regarding backup strategies and errors that occur. I'm using dovecot 2.3.11.3 with mdbox (mailbox size 50MB - 30GB). Every night, i ran the following command: doveadm -o mail_fsync=never -o plugin/quota= -o plugin/zlib_save=gz \ backup -u account mdbox:/nfs/storage/account/mdbox ### incremental backup Thanks to zlib compression, i save a lot of storage space, nonetheless i'd go with an incremental backup, but it seems to only merge changes, comparing to rsyncs' --link-dest option where you can put all "changes of the day" in an extra folder. I'd like to prevent loosing mail that gets deleted by accident/on purpose. How can i achieve this with dsync and w/o doing a full backup every day? ### backup errors After a few days w/o any errors, doveadm prints the following lines (only for the biggest, 30GB mailbox) two days in a row. As dovecot has self- healing capabilities, i think these errors/warnings can be ignored and should go away...? Error: Transaction log ....mdbox/dovecot.list.index.log: duplicate transaction log sequence (38) Error: Index ....mdbox/dovecot.list.index: Lost log for seq=38 offset=63312: Missing middle file seq=38 (between 38..4294967295, we have seqs 37,39): .log.2 contains file_seq=37 (initial_mapped=1, reason=Index mapped) Warning: fscking index file ....mdbox/dovecot.list.index Error: Fixed index file ....mdbox/dovecot.list.index: log_file_seq 38 -> 39 Error: Mailbox list index was marked as fsck'd ....mdbox/dovecot.list.index Error: Mailbox list index was marked as fsck'd ....mdbox/dovecot.list.index Error: Mailbox list index was marked as fsck'd ....mdbox/dovecot.list.index Thanks, -Mark -- Mark Patruck ( mark at wrapped.cx ) GPG key 0xF2865E51 / 187F F6D3 EE04 1DCE 1C74 F644 0D3C F66F F286 5E51 https://www.wrapped.cx

1 1

sieve_max_script_size is ignored
by Zdeněk Zámečník 06 Sep '20

06 Sep '20

I am in troubles with compiling sieve scripts larger than 1MB. I see in logs following errors: Aug 19 13:10:26 mail dovecot: lmtp(z.z(a)xxx.xxx)<22117><uNBGHKIIPV9lVgAA5ldI4A>: Error: sieve: autoreply: line 16818: quoted string started at line 3 is too long (longer than 1048576 bytes) Aug 19 13:10:26 mail dovecot: lmtp(z.z(a)xxx.xxx)<22117><uNBGHKIIPV9lVgAA5ldI4A>: Error: sieve: autoreply: parse failed Aug 19 13:10:26 mail dovecot: lmtp(z.z(a)xxx.xxx)<22117><uNBGHKIIPV9lVgAA5ldI4A>: Error: sieve: Failed to compile script `/var/vmail/xxx.xxx/z.z/sieve/autoreply.sieve' I tried to set sieve_max_script_size = 2M or 0 (unlimited). I also tried to run "sievec -o sieve_max_script_size=2M /var/vmail/xxx.xxx/z.z/sieve/autoreply.sieve" but the result is still same. Below is my config. Can you please advice where can be the problem? # 2.3.10.1 (a3d0e1171): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.10 (67bf5bd7) # OS: Linux 5.4.44-2-pve x86_64 Debian 10.4 # Hostname: mail.xxx.xxx auth_cache_size = 2 M auth_cache_ttl = 5 mins auth_master_user_separator = * auth_mechanisms = plain login auth_policy_check_before_auth = no auth_policy_hash_nonce = # hidden, use -P to show it auth_policy_report_after_auth = no auth_policy_server_timeout_msecs = 1500 auth_policy_server_url = http://127.0.0.1:8090/ dict { acl = pgsql:/etc/dovecot/dovecot-dict-sql.conf.ext quota = pgsql:/etc/dovecot/dovecot-dict-sql.conf.ext } disable_plaintext_auth = no imap_client_workarounds = delay-newmail imap_id_log = * imap_id_retain = yes login_trusted_networks = 127.0.0.0/8 mail_gid = 2000 mail_home = /var/vmail/%d/%n mail_location = mdbox:~/mdbox:ALT=/var/vmail-archive/%d/%n/mdbox mail_max_userip_connections = 60 mail_plugins = acl zlib fts quota mail_uid = 2000 managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext vacation-seconds editheader imapsieve vnd.dovecot.imapsieve mdbox_rotate_interval = 1 days mdbox_rotate_size = 16 M namespace { inbox = yes location = mailbox "Deleted Items" { auto = no special_use = \Trash } mailbox "Deleted Messages" { auto = no special_use = \Trash } mailbox Drafts { auto = no special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk } mailbox "Junk E-mail" { auto = no special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Items" { auto = no special_use = \Sent } mailbox "Sent Messages" { auto = no special_use = \Sent } mailbox Spam { auto = no special_use = \Junk } mailbox Trash { special_use = \Trash } mailbox virtual/All { auto = no special_use = \All } prefix = separator = / type = private } namespace { inbox = no list = children location = mdbox:/var/vmail/%%d/%%n/mdbox:ALT=/var/vmail-archive/%%d/%%n/mdbox prefix = shared/%%u/ separator = / subscriptions = no type = shared } passdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } passdb { args = /etc/dovecot/masters.db driver = passwd-file master = yes pass = yes } plugin { acl = vfile acl_shared_dict = proxy::acl imapsieve_mailbox1_before = file:/var/vmail/sieve/report-spam.sieve imapsieve_mailbox1_causes = COPY imapsieve_mailbox1_name = Spam imapsieve_mailbox2_before = file:/var/vmail/sieve/report-ham.sieve imapsieve_mailbox2_causes = COPY imapsieve_mailbox2_from = Spam imapsieve_mailbox2_name = * imapsieve_mailbox3_before = file:/var/vmail/sieve/report-spam.sieve imapsieve_mailbox3_causes = COPY imapsieve_mailbox3_name = Junk imapsieve_mailbox4_before = file:/var/vmail/sieve/report-ham.sieve imapsieve_mailbox4_causes = COPY imapsieve_mailbox4_from = Junk imapsieve_mailbox4_name = * mailbox_alias_new = Sent Messages mailbox_alias_new2 = Sent Items mailbox_alias_new3 = Deleted Items mailbox_alias_old = Sent mailbox_alias_old2 = Sent mailbox_alias_old3 = Trash quota = dict:User quota::proxy::quota quota_grace = 10%% quota_rule2 = Trash:ignore quota_rule3 = Junk:ignore quota_warning = storage=80%% quota-warning 90 %u quota_warning2 = storage=85%% quota-warning 95 %u quota_warning3 = storage=95%% quota-warning 105 %u sieve = /var/vmail/%d/%n/sieve/.sieve sieve_after = /var/vmail/%d/%n/sieve/autoreply.sieve sieve_before = /var/vmail/sieve/global.sieve sieve_dir = /var/vmail/%d/%n/sieve sieve_extensions = +editheader +vacation-seconds sieve_global_dir = /var/vmail/sieve/ sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment sieve_max_redirects = 20 sieve_max_script_size = 2M sieve_pipe_bin_dir = /usr/lib/dovecot sieve_plugins = sieve_imapsieve sieve_extprograms sieve_vacation_send_from_recipient = yes } protocols = imap sieve lmtp pop3 service auth-worker { unix_listener auth-worker { user = vmail } user = $default_internal_user } service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-userdb { mode = 0600 user = vmail } user = vmail } service dict { unix_listener dict { mode = 0600 user = vmail } } service imap-login { process_min_avail = 10 service_count = 0 vsz_limit = 512 M } service imap { executable = imap process_limit = 3500 unix_listener imap-master { user = $default_internal_user } vsz_limit = 2 G } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } } service managesieve-login { inet_listener sieve { port = 4190 } process_min_avail = 0 service_count = 1 vsz_limit = 64 M } service managesieve { process_limit = 400 } service pop3-login { process_min_avail = 3 service_count = 0 vsz_limit = 320 M } service pop3 { process_limit = 200 vsz_limit = 320 M } service quota-warning { executable = script /etc/dovecot/quota_warning.sh unix_listener quota-warning { mode = 0666 user = vmail } user = vmail } ssl_cert = </etc/ssl/private/multi.z-technics.cz.dovecot.pem ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it stats_writer_socket_path = userdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } protocol sieve { managesieve_implementation_string = Dovecot Pigeonhole managesieve_max_compile_errors = 5 } protocol imap { mail_plugins = quota imap_quota fts mailbox_alias imap_acl acl imap_zlib imap_sieve } protocol lda { mail_fsync = optimized } protocol lmtp { mail_fsync = optimized mail_plugins = quota sieve acl }

2 4

imapc master user & shared mailboxes
by Toma 04 Sep '20

04 Sep '20

Hello, I am trying to configure dovecot like this: https://wiki.dovecot.org/SharedMailboxes/ClusterSetup imapc_host = 127.0.0.1 imapc_master_user = %u #imapc_user = imapc_password = XXXXXXXX namespace { type = shared separator = / prefix = shared/%%u/ location = imapc:~/shared/%%u } plugin { acl_shared_dict = proxy::acl } However, contrary to the article claims, imapc master user is set to the destination username. This makes it impossible to check the ACLs against the logged user. If user.b shares mailbox with user.a, accessing /shared/user.b as user.a a will log in as "user.b*user.b". Can we tell dovecot to use the currently authenticated user for imapc_master_user and the remote user for imapc_user, so it logs in as "user.b*user.a"? I'm using dovecot 2.3.11.3 (502c39af9). Thanks! Toma

1 0

syslog logging + 2.3.10 + centos 7
by Matt Bryant 04 Sep '20

04 Sep '20

All, Running dovecot 2.3.10 on a CentOS 7.x and logging via syslog seems to stop when i restart rsyslog process. Reload doesnt get it working again only a dovecot restart. Tested this with dovecot 2.2.36 and this did not exhibit the same behaviour. systemctl restart dovecot rsyslog ----- this gets a working state systemctl restart rsyslog ------------- this restarts syslog other processes like exim continue to log but dovecot stops systemctl reload dovecot -------------- this has no impact logging still does not work systemctl restart dovecot ------------- this fixes the issue. Are there any changes to the log process between 2.2 and 2.3 that might explain this ? rgds Matt

1 1

How to backup maildir
by Robert Moskowitz 04 Sep '20

04 Sep '20

I have been thinking, and reading, on how to back up my mailserver. I have not found any approach that seems ready to use. I have run years without any backup, but would really like to have something in place. I figure I can attach a USB drive and backup to that, then from there rsync to something elsewhere. Further if that USB drive is a full mailserver image, I actually have a 'hot backup' where I only have to put the backup drive into a system and boot up at the last backup. But this means properly copying all of /home/vmail and probably /home/sieve plus the /var/lib/mysql Are there good tools that nicely does this? Or do I choose a time late at night (only I am sometimes in non-US timezones) to shut down all services and just use rsync? And stopping services itself is thought provoking. What if Dovecot, amavis, mysql, or whatelse is in the middle of writing out a mail file what happens to that file and restart. Just scary stuff and, in part, why I have never tackled this in the past. thanks for all feedback

4 6

Deleted mail reappearing
by John S 04 Sep '20

04 Sep '20

Recently I upgraded dovecot quite a few major versions. Since then some mailboxes are experiencing deleted mail reappearing. It seems to be happening roughly once a day, but not for every mail client or mailbox, I have at least two clients with this issue: Thunderbird and Outlook. Storage is local under two partitions, one as alternative storage. # 2.3.7.2 (3c910f64b): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.7.2 () # OS: Linux 5.4.0-29-generic x86_64 Ubuntu 20.04 LTS ext4 auth_mechanisms = plain login digest-md5 base_dir = /var/run/dovecot/ default_client_limit = 4000 default_vsz_limit = 512 M dict { expire = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext quota = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext } disable_plaintext_auth = no lda_mailbox_autocreate = yes log_timestamp = "%Y-%m-%d %H:%M:%S " mail_location = mdbox:/home/vmail/%d/%n mail_max_userip_connections = 240 mail_plugins = quota mail_privileged_group = mail mdbox_rotate_size = 64 M namespace { inbox = yes location = prefix = separator = . } passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } plugin { quota = dict:user_quota::proxy:/var/run/dovecot/dict:quota quota_warning = storage=95%% quota-warning 95 %u quota_warning2 = storage=80%% quota-warning 80 %u sieve = file:~/sieve;active=~/.dovecot.sieve sieve_dir = ~/sieve sieve_global_dir = /etc/dovecot/sieve/ } protocols = imap pop3 service anvil { client_limit = 2615 } service auth { client_limit = 5536 unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-master { group = root mode = 0600 user = vmail } } service dict { unix_listener dict { mode = 0600 user = vmail } } service imap-login { client_limit = 2115 executable = /usr/lib/dovecot/imap-login inet_listener imap { address = * port = 143 } inet_listener imaps { address = * port = 993 } process_limit = 2000 } service imap { executable = /usr/lib/dovecot/imap process_limit = 2000 } service pop3-login { client_limit = 1024 executable = /usr/lib/dovecot/pop3-login inet_listener pop3 { address = * port = 110 } inet_listener pop3s { address = * port = 995 } process_limit = 512 } service pop3 { executable = /usr/lib/dovecot/pop3 } service stats { unix_listener stats-reader { group = vmail mode = 0660 user = vmail } unix_listener stats-writer { group = vmail mode = 0660 user = vmail } } ssl_cert = < /etc/dovecot/ssl/crt.pem ssl_cipher_list = ALL:!LOW:!SSLv2:!EXP:!aNULL ssl_client_ca_dir = /etc/ssl/certs ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it ssl_prefer_server_ciphers = yes syslog_facility = local0 userdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } protocol imap { imap_max_line_length = 64 k mail_plugins = quota imap_quota } protocol pop3 { mail_plugins = quota pop3_uidl_format = %08Xu%08Xv } protocol lda { auth_socket_path = /var/run/dovecot/auth-master mail_plugins = quota sieve postmaster_address = postmaster@localhost sendmail_path = /usr/lib/sendmail } dovecot-sql.conf: driver = mysql connect = host=127.0.0.1 dbname=mail user=mail password=#### password_query = SELECT IF('%m' = 'DIGEST-MD5', users.password_digest_md5, NULL) as password, IF('%m' = 'DIGEST-MD5', NULL, 'Y') as nopassword, username FROM users LEFT JOIN domains ON users.domain = domains.domain WHERE users.username = '%u' AND users.domain = '%d' AND users.active = 'yes' AND domains.state = '1' AND ((users.password = encrypt('%w', LEFT(users.password, 11)) AND ('%m' = 'PLAIN' OR '%m' = 'LOGIN')) OR ('%m' = 'DIGEST-MD5')) user_query = SELECT maildir as home, uid, gid, CONCAT(CONCAT(type, ':', maildir), ':ALT=/home/altmail/%d/%n') AS mail FROM users LEFT JOIN domains ON users.domain = domains.domain WHERE users.username = '%u' AND domains.state = '1' iterate_query = SELECT SUBSTRING_INDEX(email, '@', 1) AS username, domain FROM users;

3 3

Sieve: deleteheader now working with duplicate filter for implicit keep
by Alec Moskvin 04 Sep '20

04 Sep '20

Hello, I have a rule to always delete a header. If the message gets fileinto'd, the header is gone, but if it's delivered into the INBOX through implicit keep, the header does not get deleted. Interestingly, if I remove the "if duplicate" filter, the header does get deleted as expected. I'm using the latest Dovecot 2.3.11.3 / Pigeonhole 0.5.11. Any idea why this is happening? Thanks, Alec === Sieve filter require ["fileinto", "editheader", "duplicate", "mailbox"]; deleteheader "X-Bad-Header"; if duplicate { fileinto :create "Trash"; stop; } if header :matches "subject" "*filter*" { fileinto :create "Filtered"; } === Example (note ${RANDOM} to bypass duplicate filter) $ /usr/libexec/dovecot/deliver << -end- To: me(a)example.com From: someone(a)example.com Subject: Test Message Message-Id: <${RANDOM}(a)example.net> Date: Thu, 20 Aug 2020 00:21:24 +0000 (UTC) X-Bad-Header: DELETE ME Hello world -end-

2 4

ltmp delivery with ldap and prefetch
by Marc Roos 04 Sep '20

04 Sep '20

I am able to authorize and connect via imap. But I am not able to deliver a message via lmtp I am getting this error message. passdb didn't return userdb entries, trying the next userdb [1] dovecot-ldap.conf.ext pass_attrs = uid=user,userPassword=password,host=host,homeDirectory=userdb_home,uidNu mber=userdb_uid,gidNumber=userdb_gid user_attrs = uid=%{ldap:uidNumber},gid=%{ldap:gidNumber},home=%{ldap:homeDirectory} [2] pass / user db's passdb { args = /etc/dovecot/dovecot-ldap.conf.ext auth_verbose = default default_fields = deny = no driver = ldap master = no mechanisms = name = override_fields = pass = no result_failure = continue result_internalfail = continue result_success = return-ok skip = never username_filter = } userdb { args = /etc/dovecot/special-userdb auth_verbose = default default_fields = driver = passwd-file name = override_fields = result_failure = continue result_internalfail = continue result_success = return-ok skip = never } userdb { args = auth_verbose = default default_fields = driver = prefetch name = override_fields = result_failure = continue result_internalfail = continue result_success = return-ok skip = never }

1 0

Moving Spam to Junk Folder
by bobby 03 Sep '20

03 Sep '20

I am following this tutorial: https://www.linuxbabe.com/redhat/spamassassin-centos-rhel-block-email-spam. I followed the steps in "Move Spam into the Junk Folder". When I send an email from a blacklisted e-mail address, I get a bounce e-mail from my e-mail server. Here is what is in my spamass-milter file: EXTRA_FLAGS="-m -r 8 -R NO_SPAM -i 127.0.0.1 -g sa-milt -- --max-size=5120000" I would prefer it to go into my Junk folder. How can I make this happen?

3 3

Fwd: zlib errors after upgrading to 2.3.11.3
by Robert Nowotny 03 Sep '20

03 Sep '20

Dear Aki, I switched to "gz" now, since "zstd" also gave some errors on writing to files. I dont know if "xz" compression or "zstd" shreddered my MDBOX Files, but I lost 4 days of mail. (a couple of thousand mails). After restoring the backup (what was made after switching to version 2.3.11.3) I still have some broken mdfiles, but not too many. Interestingly always in the "Sent" Mailbox on a number of Users. I just can not go back to 20.08.2020 before I updated to 2.3.11.3 - too many emails would have been lost. So - the current status is 2.3.11.3, with "gz" compression. I force-synced and re-indexed all the mdbox files (250 GB), but still have some broken - please how can I fix those ? You stated : (In theory you could leave the existing mails xz-compressed, but best would be to re-compress everything via dsync so old mails can be read when we eventually remove xz support.) what is the optimal way to do that, expecially not loosing the index (?) for outlook/thunderbird, for third party tools that rely on some index/hash (dunno how exactly that works) ? Sep 3 08:33:25 lxc-imap dovecot: imap(mpaul)<48684><2/9E5mKuAezAqKjk>: Error: Mailbox Sent: UID=2171: read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe (FETCH BODY[]) Sep 3 08:35:26 lxc-imap dovecot: imap(mpaul)<48089><YRio1GKuEdHAqKi3>: Error: Mailbox Sent: UID=2171: read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe (FETCH BODY[]) Sep 3 08:35:26 lxc-imap dovecot: imap(mpaul)<49228><p/0g9WKuVdHAqKi3>: Error: Mailbox Sent: UID=2171: read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe (FETCH BODY[]) Sep 3 08:35:26 lxc-imap dovecot: imap(mpaul)<49229><GMAm9WKuVtHAqKi3>: Error: Mailbox Sent: UID=2171: read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe (FETCH BODY[]) Sep 3 08:35:27 lxc-imap dovecot: imap(mpaul)<49230><YYot9WKuV9HAqKi3>: Error: Mailbox Sent: UID=2171: read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe (FETCH BODY[]) ... Sep 3 09:04:44 lxc-imap dovecot: imap(cpotzinger)<49040></X+P72KuUNHAqKi3>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:04:51 lxc-imap dovecot: imap(cpotzinger)<54382><eYT1XWOux9HAqKi3>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:04:59 lxc-imap dovecot: imap(cpotzinger)<54396><nKllXmOuyNHAqKi3>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:05:06 lxc-imap dovecot: imap(cpotzinger)<54409><m2TOXmOuydHAqKi3>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:05:12 lxc-imap dovecot: imap(cpotzinger)<54422><IP86X2OuytHAqKi3>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) ... Sep 3 09:08:58 lxc-imap dovecot: imap(cpotzinger)<54867><BtGtbGOu49HAqKi3>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:19:15 lxc-imap dovecot: imap(mpaul)<56030></kHNkWOuAO3AqKjk>: Error: Mailbox Sent: UID=2171: read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe (FETCH BODY[]) Sep 3 09:29:41 lxc-imap dovecot: imap(cpotzinger)<57274><c4rCtWOuKuzAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:29:47 lxc-imap dovecot: imap(cpotzinger)<57320><K3Uqt2OuLOzAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:29:54 lxc-imap dovecot: imap(cpotzinger)<57333><e4aMt2OuLezAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:30:01 lxc-imap dovecot: imap(cpotzinger)<57346><JEfzt2OuLuzAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:30:08 lxc-imap dovecot: imap(cpotzinger)<57360><mH1YuGOuMOzAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:30:14 lxc-imap dovecot: imap(cpotzinger)<57382><IzLBuGOuMezAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:30:21 lxc-imap dovecot: imap(cpotzinger)<57397><q10kuWOuM+zAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:30:27 lxc-imap dovecot: imap(cpotzinger)<57410><CSuJuWOuNOzAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:31:22 lxc-imap dovecot: imap(mpaul)<57525><TfoOvWOuWO3AqKjk>: Error: Mailbox Sent: UID=2171: read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe (FETCH BODY[]) Sep 3 09:31:49 lxc-imap dovecot: imap(cpotzinger)<57423><Be/ruWOuNuzAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:31:55 lxc-imap dovecot: imap(cpotzinger)<57584><yvXOvmOuQuzAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:32:02 lxc-imap dovecot: imap(cpotzinger)<57597><e/8tv2OuQ+zAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:32:08 lxc-imap dovecot: imap(cpotzinger)<57611><BLSOv2OuROzAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:32:19 lxc-imap dovecot: imap(cpotzinger)<57633><Ky0twGOuR+zAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:32:26 lxc-imap dovecot: imap(cpotzinger)<57646><ljSRwGOuS+zAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:32:33 lxc-imap dovecot: imap(cpotzinger)<57659><rn75wGOuT+zAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:32:39 lxc-imap dovecot: imap(cpotzinger)<57672><GP9jwWOuUuzAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:33:45 lxc-imap dovecot: imap(mpaul)<57795><2CEwxWOubu3AqKjk>: Error: Mailbox Sent: UID=2171: read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe (FETCH BODY[]) Sep 3 09:38:29 lxc-imap dovecot: imap(mpaul)<58347><CKyX1mOuie3AqKjk>: Error: Mailbox Sent: UID=2171: read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe (FETCH BODY[]) Sep 3 09:38:32 lxc-imap dovecot: imap(mpaul)<58352><osHU1mOuiu3AqKjk>: Error: Mailbox Sent: UID=2171: read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe (FETCH BODY[]) Sep 3 09:38:44 lxc-imap dovecot: imap(cpotzinger)<54527><5ZpiYmOuBOfAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:38:51 lxc-imap dovecot: imap(cpotzinger)<58378><kAuN12OumuzAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) Sep 3 09:38:57 lxc-imap dovecot: imap(cpotzinger)<58392><xJLx12Oum+zAqKiU>: Error: Mailbox Sent: UID=29534: read(zlib(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703)) failed: read(/home/vmail/virtualmailboxes/cpotzinger/storage/m.4703) failed: Broken pipe (FETCH BODY[]) ... Am 26.08.2020 um 09:42 schrieb Robert Nowotny: > I had 4 broken mdbox files for 4 different users. > After deleting them, dovecot re-indexed, now everything is ok again. > > I can try to update tonight again to 2.3.11.3 and see if some broken > files will appear again tomorrow. > > Am 25.08.2020 um 14:51 schrieb Aki Tuomi: >>> On 25/08/2020 14:35 Robert Nowotny<rnowotny(a)rotek.at> wrote: >>> >>> >>> I get ZLIB Errors after dovecot upgrade from 2.3.10.1 to 2.3.11.3 >>> >>> >>> Aug 21 15:27:34 lxc-imap dovecot: imap(acsida)<63870><jZk...>: Error: Mailbox Sent: UID=40826: read(zlib(/home/vmail/virtualmailboxes/acsida/storage/m.2409)) failed: read(/home/vmail/virtualmailboxes/acsida/storage/m.2409) failed: Broken pipe (FETCH BODY[]).... >>> >>> many of those entries.... >>> >>> after installing2.3.10.1 those files are still broken - but new files are working again. >>> >>> anything I can tweak or do ? >>> >>> >> Hi! >> >> Is it possible your mdbox file is corrupted? Can you share doveconf -n? >> >> Aki >

1 0

Submission proxy warning connecting to Exim
by Alec Moskvin 03 Sep '20

03 Sep '20

Hello, When connecting to Exim 4.94, dovecot submission logs this warning: dovecot[25235]: submission(alec)<25347><QNydV8+t+oMKAgAF>: Warning: smtp-client: conn localhost:25 ([::1]:25) [1]: Received invalid EHLO response line: Unexpected character in EHLO keyword Looks like it doesn't like the underscore in "X_PIPE_CONNECT", since it's neither "isalnum" nor a dash [1]: 1598480923.401085 220 myhostname ESMTP Exim 4.94 Wed, 26 Aug 2020 18:28:43 -0400 1598480923.401906 250-myhostname Hello localhost [::1] 1598480923.401906 250-SIZE 52428800 1598480923.401906 250-8BITMIME 1598480923.401906 250-PIPELINING 1598480923.401906 250-X_PIPE_CONNECT 1598480923.401906 250-CHUNKING 1598480923.401906 250-STARTTLS 1598480923.401906 250 HELP 1598480923.409242 250 OK 1598480923.434743 250 Accepted 1598480923.578335 250- 511 byte chunk, total 511 1598480923.578335 250 OK id=1kB3uN-0006ar-HU Adding the line "pipelining_connect_advertise_hosts =" to Exim's config fixes the issue by disabling X_PIPE_CONNECT. Is this a bug in Exim or is Dovecot being too strict? Thanks, Alec [1] https://github.com/dovecot/core/blob/2.3.11.3/src/lib-smtp/smtp-syntax.c#L2…

3 2

Sumbission crashes when relaying over TLS
by Alec Moskvin 03 Sep '20

03 Sep '20

Hello, I'm trying to set up the submission proxy, but if I set submission_relay_ssl = starttls, it crashes. Without it, it works. Please find the details below. Thanks, Alec dovecot[256855]: submission-login: Login: user=<alec>, method=PLAIN, rip=::1, lip=::1, mpid=257033, secured, session=<kMd1B9uthLUAAAAAAAAAAAAAAAAAAAAB> dovecot[256855]: submission(alec)<257033><kMd1B9uthLUAAAAAAAAAAAAAAAAAAAAB>: Panic: file smtp-client-connection.c: line 290 (smtp_client_connection_fail): assertion failed: (conn->to_connect == NULL) dovecot[256855]: submission(alec)<257033><kMd1B9uthLUAAAAAAAAAAAAAAAAAAAAB>: Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(backtrace_append+0x42) [0x7fbbfb2bcaf2] -> /usr/lib64/dovecot/libdovecot.so.0(backtrace_get+0x1e) [0x7fbbfb2bcc0e] -> /usr/lib64/dovecot/libdovecot.so.0(+0xf5f09) [0x7fbbfb2c5f09] -> /usr/lib64/dovecot/libdovecot.so.0(+0xf5fa1) [0x7fbbfb2c5fa1] -> /usr/lib64/dovecot/libdovecot.so.0(+0x51eec) [0x7fbbfb221eec] -> /usr/lib64/dovecot/libdovecot.so.0(+0x44d5e) [0x7fbbfb214d5e] -> /usr/lib64/dovecot/libdovecot.so.0(+0x6c431) [0x7fbbfb23c431] -> /usr/lib64/dovecot/libdovecot.so.0(smtp_client_command_input_reply+0x201) [0x7fbbfb235261] -> /usr/lib64/dovecot/libdovecot.so.0(+0x6ccfd) [0x7fbbfb23ccfd] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x68) [0x7fbbfb2dc048] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x132) [0x7fbbfb2dd662] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x51) [0x7fbbfb2dc0f1] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x40) [0x7fbbfb2dc2b0] -> /usr/lib64/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7fbbfb255ee3] -> dovecot/submission(main+0x4df) [0x561d486b3daf] -> /lib64/libc.so.6(__libc_start_main+0xea) [0x7fbbfb054b8a] -> dovecot/submission(_start+0x2a) [0x561d486b3e3a] dovecot[256855]: submission(alec)<257033><kMd1B9uthLUAAAAAAAAAAAAAAAAAAAAB>: Fatal: master: service(submission): child 257033 killed with signal 6 (core dumped) (gdb) bt #0 0x00007fbbfb069a31 in raise () from /lib64/libc.so.6 #1 0x00007fbbfb0532b6 in abort () from /lib64/libc.so.6 #2 0x00007fbbfb22223c in default_fatal_finish (status=0, type=LOG_TYPE_PANIC) at failures.c:459 #3 fatal_handler_real (ctx=<optimized out>, format=<optimized out>, args=<optimized out>) at failures.c:471 #4 0x00007fbbfb2c5fa1 in i_internal_fatal_handler (ctx=<optimized out>, format=<optimized out>, args=<optimized out>) at failures.c:848 #5 0x00007fbbfb221eec in i_panic (format=format@entry=0x7fbbfb30fd08 "file %s: line %d (%s): assertion failed: (%s)") at failures.c:523 #6 0x00007fbbfb214d5e in smtp_client_connection_fail (status=9002, error=<optimized out>, conn=0x561d491296a8) at smtp-client-connection.c:290 #7 smtp_client_connection_fail (conn=0x561d491296a8, status=<optimized out>, error=<optimized out>) at smtp-client-connection.c:282 #8 0x00007fbbfb23c431 in smtp_client_connection_starttls_cb (reply=0x561d4912d048, conn=0x561d491296a8) at smtp-client-connection.c:788 #9 0x00007fbbfb235261 in smtp_client_command_input_reply (cmd=<optimized out>, reply=reply@entry=0x561d4912d048) at smtp-client-command.c:971 #10 0x00007fbbfb23ccfd in smtp_client_connection_input_reply (reply=0x561d4912d048, conn=0x561d491296a8) at smtp-client-connection.c:1045 #11 smtp_client_connection_input (_conn=<optimized out>) at smtp-client-connection.c:1116 #12 0x00007fbbfb2dc048 in io_loop_call_io (io=0x561d49109cb0) at ioloop.c:713 #13 0x00007fbbfb2dd662 in io_loop_handler_run_internal (ioloop=ioloop@entry=0x561d490fd050) at ioloop-epoll.c:222 #14 0x00007fbbfb2dc0f1 in io_loop_handler_run (ioloop=0x561d490fd050) at ioloop.c:765 #15 0x00007fbbfb2dc2b0 in io_loop_run (ioloop=0x561d490fd050) at ioloop.c:738 #16 0x00007fbbfb255ee3 in master_service_run (service=0x561d490fceb0, callback=<optimized out>) at master-service.c:851 #17 0x0000561d486b3daf in main (argc=<optimized out>, argv=<optimized out>) at main.c:396 $ openssl version OpenSSL 1.1.1g 21 Apr 2020 # doveconf -n # 2.3.11.3 (502c39af9): /etc/dovecot/dovecot.conf # OS: Linux 5.8.4 x86_64 Gentoo Base System release 2.7 # Hostname: haswell listen = localhost mail_location = maildir:~/.maildir namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = scheme=CRYPT username_format=%u /etc/dovecot/users driver = passwd-file } protocols = submission ssl_cert = </etc/ssl/dovecot/server.pem ssl_client_ca_file = </etc/dovecot/ca.crt ssl_key = # hidden, use -P to show it submission_relay_host = localhost submission_relay_port = 2525 submission_relay_rawlog_dir = %h/rawlog submission_relay_ssl = starttls userdb { args = username_format=%u /etc/dovecot/users driver = passwd-file } protocol submission { rawlog_dir = %h/rawlog }

2 2

submission message quota
by Gerry 03 Sep '20

03 Sep '20

Hello, I am trying to come up with a way to have individual quotas per user for the submission service. Similar to what I could achieve with Postfix and policyd. More specifically, the quota I am most interested in, is limiting the number of messages a single account can send within a given timeframe. Ideally, I'd also like to limit the number of total recipients within a given timeframe, to mitigate the loophole of adding multiple recipients to a single message. Example: account Y is allowed to send 500 messages per 60 minutes, with a maximum of 2000 recipients overall. What would be the best path to take? Thanks! Gerry

4 5

Metric label values truncated when using OpenMetrics endpoint
by Daan van Gorkum 02 Sep '20

02 Sep '20

Hi, Recently we upgraded to Dovecot 2.3.11.3 and configured an example metric like this: metric imap_command { event_name = imap_command_finished group_by = cmd_name tagged_reply_state user remote_ip } And enabled the OpenMetrics listener like this: service stats { inet_listener http { port = 5000 } } While the result is great, I noticed that some metrics are being truncated (three dots at the end of the label value): dovecot_imap_command_duration_usecs_sum{cmd_name="FETCH",tagged_reply_state="OK",user="xxxxxxxxxxx.inbox@xxxxxxxxxxxxx…",remote_ip="XXXX:XXXX:XXXX:XXXX:cc6b:cc7a:d…"} 2346 1598055662473 I couldn't find this information in the documentation so I'm wondering if this is a bug. Thank you for your time. Regards, Daan

2 5