September 2020 - dovecot

sieve-extdata plugin breaks on pigeonhole 0.5 ish
by Ed W 01 Nov '20

01 Nov '20

Hi, I wonder if someone could give me a little help bringing the sieve-extdata plugin up to date to be usable with latest Dovecot/pigeonhole Trying to use both together at present I get an error: managesieve: Fatal: Couldn't load required plugin /usr/lib/dovecot/sieve/lib90_sieve_extdata_plugin.so: dlopen() failed: /usr/lib/dovecot/sieve/lib90_sieve_extdata_plugin.so: undefined symbol: sieve_sys_error This appears to be due to the change in error handling in pigeonhole 0.5 which appears to have removes the "sieve_sys_error()" and "sieve_sys_warning()" functions I'm unclear how this code should be change (I've commented it out for now - eek). I find this extension useful to pass in user specific parameters to my sieve scripts that are stored elsewhere in an sql database, eg spam score, size threshold for attachments, perhaps there is some other way to achieve this? Any hints on how to fixing this would be appreciated Thanks Ed W

3 2

Re: zlib errors after upgrading to 2.3.11.3
by Robert Nowotny 14 Oct '20

14 Oct '20

On 9. Sep 2020, at 11.14, Robert Nowotny <rnowotny1966(a)gmail.com> wrote: Sep 3 08:33:25 lxc-imap dovecot: imap(mpaul)<48684><2/9E5mKuAezAqKjk>: Error: Mailbox Sent: UID=2171: read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe (FETCH BODY[]) Also this way you can see if the broken mail is actually xz or zstd or zlib. It would be nice to know if there are any zstd or zlib compressed mails that have problems. We did a lot of stress testing with zstd and also with xz, but haven't been able to reproduce any problems. It's also strange that it says taht the error is "Broken pipe" - that doesn't indicate that the mail is corrupted but that there is something more strange going on. So perhaps you don't actually have any mails written as corrupted, but Dovecot is just somehow having trouble reading the mails. I managed to reproduce this. The files aren't corrupted, it's just that reading is failing. The attached patch should fix the xz code and should make your files readable again. Yes, it works ! I switched now to "zstd" compression and currently re-compressing all user imap folders (one by one) Actually, with further testing it looks like the mails were written truncated. My patch simply hides the truncation when reading the mail. But it's anyway best that can be done about them. zstd also has a bug in writing compressed output, but it should always crash instead of writing broken output. We'll try to get fixes to these merged to git soon. currently I am re-compressing the mailboxes and so far i can see only one (1) Error, after about 120GB of recomresses emails : Sep 10 11:32:24 lxc-imap dovecot: imap(ebay)<37070><1aDvPvKus+/AqKjk>: Panic: file ostream.c: line 287 (o_stream_sendv_int): assertion failed: (!stream->blocking) Sep 10 11:32:24 lxc-imap dovecot: imap(ebay)<37070><1aDvPvKus+/AqKjk>: Error: Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(backtrace_append+0x41) [0x7f5b00de6da1] -> /usr/local/lib/dovecot/libdovecot.so.0(backtrace_get+0x22) [0x7f5b00de6ec2] -> /usr/local/lib/dovecot/libdovecot.so.0(+0xfe5eb) [0x7f5b00df05eb] -> /usr/local/lib/dovecot/libdovecot.so.0(+0xfe687) [0x7f5b00df0687] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x598e1) [0x7f5b00d4b8e1] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x5e2c1) [0x7f5b00d502c1] -> /usr/local/lib/dovecot/libdovecot.so.0(o_stream_sendv+0x32) [0x7f5b00e14a92] -> /usr/local/lib/dovecot/libdovecot.so.0(io_stream_copy+0x5a) [0x7f5b00e1535a] -> /usr/local/lib/dovecot/libdovecot.so.0(o_stream_send_istream+0x50) [0x7f5b00e15060] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(index_storage_save_continue+0x2d) [0x7f5b00f9659d] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(dbox_save_continue+0x30) [0x7f5b00f3d3b0] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(mailbox_save_continue+0x3e) [0x7f5b00f1a34e] -> dovecot/imap [ebay 192.168.168.228 APPEND](+0x13801) [0x55d6905bf801] -> dovecot/imap [ebay 192.168.168.228 APPEND](command_exec+0x6c) [0x55d6905ce3ec] -> dovecot/imap [ebay 192.168.168.228 APPEND](+0x12fdb) [0x55d6905befdb] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x6d) [0x7f5b00e06ced] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x145) [0x7f5b00e08305] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x54) [0x7f5b00e06d94] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x48) [0x7f5b00e06f08] -> /usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x17) [0x7f5b00d7f8d7] -> dovecot/imap [ebay 192.168.168.228 APPEND](main+0x459) [0x55d6905bec99] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f5b00b0e0b3] -> dovecot/imap [ebay 192.168.168.228 APPEND](_start+0x2e) [0x55d6905bed2e] Sep 10 11:32:24 lxc-imap dovecot: imap-login: Panic: epoll_ctl(del, 21) failed: Bad file descriptor Sep 10 11:32:24 lxc-imap dovecot: imap(ebay)<37070><1aDvPvKus+/AqKjk>: Fatal: master: service(imap): child 37070 killed with signal 6 (core dumps disabled - https://dovecot.org/bugreport.html#coredumps) until now I only have one broken mailbox, I need to repair (I will try that later, just finish all the other users before): Sep 10 12:13:13 lxc-imap dovecot: imap(mtrenner)<41872><KVTf0PKu7+7AqKi0>: Error: lzma.read(/home/vmail/virtualmailboxes/mtrenner/storage/m.349): corrupted data at 3763912 Sep 10 12:13:13 lxc-imap dovecot: imap(mtrenner)<41872><KVTf0PKu7+7AqKi0>: Error: Mailbox Sent: UID=1480: read(zlib(/home/vmail/virtualmailboxes/mtrenner/storage/m.349)) failed: read(/home/vmail/virtualmailboxes/mtrenner/storage/m.349) failed: lzma.read(/home/vmail/virtualmailboxes/mtrenner/storage/m.349): corrupted data at 3763912 (FETCH BODY[]) > zstd also has a bug in writing compressed output, but it should always crash instead of writing broken output. > We'll try to get fixes to these merged to git soon. Uh-Oh, that would be good ... pls notify ! how such thing can slip through the tests ? I just can not remember, the last error with compression was like 10 years ago (dont know if it was me who reported that) However, thank You for Your fast response and taking care !!! Robert

3 2

Providers running dovecot?
by Olivier Cailloux 11 Oct '20

11 Oct '20

Dear list, I am looking for providers of free e-mail addresses known to run Dovecot (or a variant thereof) for IMAP access. I need only a few MB storage space and no particular features beyond SMTP and IMAP. The reason I ask is that Dovecot is known to implement the IMAP spec quite respectfully, and I am writing a software which uses IMAP search (so I would suggest my users to register an e-mail to a provider implementing correctly the IMAP Search specifications, to reduce the probability of bugs). (More details here: https://www.webhostingtalk.com/showthread.php?t=1821627.) Do you know of any such provider? -- Olivier

7 12

How do I make "vnd.dovecot.environment" available?
by Graham Leggett 08 Oct '20

08 Oct '20

Hi all, According to the docs at https://wiki.dovecot.org/Pigeonhole/Sieve, some extensions are not available by default. The docs tell me the sieve_plugins, sieve_extensions and/or sieve_global_extensions configs are involved, but I can find no authoritative documentation as to what the values should be. What values must I use to make “vnd.dovecot.environment” available? Adding "sieve_global_extensions = +vnd.dovecot.environment” to 90-sieve.conf seems to not be enough. [root@gatekeeper ~]# sievec -D /var/lib/dovecot-sieve/default.sieve sievec(root): Debug: sieve: Pigeonhole version 0.5.8 (b7b03ba2) initializing sievec(root): Debug: sieve: include: sieve_global is not set; it is currently not possible to include `:global' scripts. sievec(root): Debug: sieve: file storage: Using Sieve script path: /var/lib/dovecot-sieve/default.sieve sievec(root): Debug: sieve: file script: Opened script `default' from `/var/lib/dovecot-sieve/default.sieve' default: line 5: error: require command: unknown Sieve capability `vnd.dovecot.environment'. default: line 19: error: referring to variable in unknown namespace 'vnd'. default: error: validation failed. sievec(root): Fatal: failed to compile sieve script '/var/lib/dovecot-sieve/default.sieve' Regards, Graham —

3 5

dovecot: lmtp: Error: fatal error: failed to reserve page summary memory
by Roman Vasilyev 08 Oct '20

08 Oct '20

Sorry for offtopic, don't know where to ask it after updating from `ubuntu` 18 to 20 started getting this on running golang binary from sieve rules file ``` Sep 17 08:54:00 lonjemail spamd[3231]: spamd: connection from ::1 [::1]:57844 to port 783, fd 5 Sep 17 08:54:00 lonjemail spamd[3231]: spamd: setuid to Debian-exim succeeded Sep 17 08:54:00 lonjemail spamd[3231]: spamd: processing message <CANcP0tbws-r=L8KrTug_NKLk0EuPwo1vKotdpV=Pf95Pxusbww(a)mail.gmail.com> for Debian-exim:114 Sep 17 08:54:00 lonjemail systemd-resolved[527]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP. Sep 17 08:54:00 lonjemail systemd-resolved[527]: message repeated 8 times: [ Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.] Sep 17 08:54:00 lonjemail spamd[3231]: spamd: clean message (1.1/5.0) for Debian-exim:114 in 0.3 seconds, 2760 bytes. Sep 17 08:54:00 lonjemail spamd[3231]: spamd: result: . 1 - DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,RDNS_NONE,SPF_HELO_NONE,SPF_PASS scantime=0.3,size=2760,user=Debian-exim,uid=114,required_score=5.0,rhost=::1,raddr=::1,rport=57844,mid=<CANcP0tbws-r=L8KrTug_NKLk0EuPwo1vKotdpV= Pf95Pxusbww(a)mail.gmail.com>,autolearn=no autolearn_force=no Sep 17 08:54:00 lonjemail dovecot: lmtp(4254): Connect from local Sep 17 08:54:00 lonjemail dovecot: lmtp: Error: fatal error: failed to reserve page summary memory Sep 17 08:54:00 lonjemail dovecot: lmtp: Error: Sep 17 08:54:00 lonjemail dovecot: lmtp: Error: runtime stack: Sep 17 08:54:00 lonjemail dovecot: lmtp: Error: runtime.throw(0x1a48db5, 0x25) Sep 17 08:54:00 lonjemail dovecot: lmtp: Error: #011/usr/local/go/src/runtime/panic.go:1116 +0x72 fp=0x7ffe160d5860 sp=0x7ffe160d5830 pc=0x4ce5d2 Sep 17 08:54:00 lonjemail dovecot: lmtp: Error: runtime.(*pageAlloc).sysInit(0x232c268) Sep 17 08:54:00 lonjemail dovecot: lmtp: Error: #011/usr/local/go/src/runtime/mpagealloc_64bit.go:80 +0x185 fp=0x7ffe160d58f0 sp=0x7ffe160d5860 pc=0x4c4e25 Sep 17 08:54:00 lonjemail dovecot: lmtp: Error: runtime.(*pageAlloc).init(0x232c268, 0x232c260, 0x25cb8d8) Sep 17 08:54:00 lonjemail dovecot: lmtp: Error: #011/usr/local/go/src/runtime/mpagealloc.go:317 +0x75 fp=0x7ffe160d5918 sp=0x7ffe160d58f0 pc=0x4c2815 Sep 17 08:54:00 lonjemail dovecot: lmtp: Error: runtime.(*mheap).init(0x232c260) Sep 17 08:54:00 lonjemail dovecot: lmtp: Error: #011/usr/local/go/src/runtime/mheap.go:743 +0x24b fp=0x7ffe160d5940 sp=0x7ffe160d5918 pc=0x4bf84b Sep 17 08:54:00 lonjemail dovecot: lmtp: Error: runtime.mallocinit() Sep 17 08:54:00 lonjemail dovecot: lmtp: Error: #011/usr/local/go/src/runtime/malloc.go:480 +0x109 fp=0x7ffe160d5968 sp=0x7ffe160d5940 pc=0x4a4c49 Sep 17 08:54:00 lonjemail dovecot: lmtp: Error: runtime.schedinit() Sep 17 08:54:00 lonjemail dovecot: lmtp: Error: #011/usr/local/go/src/runtime/proc.go:563 +0x65 fp=0x7ffe160d59c0 sp=0x7ffe160d5968 pc=0x4d1f25 Sep 17 08:54:00 lonjemail dovecot: lmtp: Error: runtime.rt0_go(0x7ffe160d5ad8, 0x2, 0x7ffe160d5ad8, 0x1813580, 0x7fd3cc8a80b3, 0x7fd3cca6cb80, 0x7ffe160d5ad8, 0x200011c00, 0x501080, 0x1813580, ...) Sep 17 08:54:00 lonjemail dovecot: lmtp: Error: #011/usr/local/go/src/runtime/asm_amd64.s:214 +0x125 fp=0x7ffe160d59c8 sp=0x7ffe160d59c0 pc=0x5011c5 Sep 17 08:54:00 lonjemail dovecot: lmtp(roman)<4254></X0wOCgkY1+eEAAAgcArzw>: program exec:/usr/lib/dovecot/sieve-execute/smtp (4255): Terminated with non-zero exit code 2 Sep 17 08:54:00 lonjemail dovecot: lmtp(roman)<4254></X0wOCgkY1+eEAAAgcArzw>: sieve: msgid=<CANcP0tbws-r=L8KrTug_NKLk0EuPwo1vKotdpV=Pf95Pxusbww(a)mail.gmail.com>: stored mail into mailbox 'Unsubscribe.Neighbors' Sep 17 08:54:00 lonjemail dovecot: lmtp(4254): Disconnect from local: Client has quit the connection (state=READY) Sep 17 08:55:01 lonjemail CRON[4257]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1) ``` here is sieve rules: ``` require ["variables", "fileinto", "mailbox", "vnd.dovecot.execute"]; if address :matches "from" "*" { set "from" "${1}"; } if anyof ( header :contains "subject" "unsubscribe neighbours", header :contains "subject" "unsubscribe neighbors") { execute "smtp" "-neighbors=${from}"; fileinto :create "Unsubscribe.Neighbors"; } elsif address :contains "from" ["mailer-daemon", "postmaster", "internet mail delivery"] { if header :matches "X-Failed-Recipients" "*" { execute "smtp" "-email=${1}"; fileinto :create "Mail Delivery System"; } else { fileinto :create "Mail Delivery System.Warning"; } } else { # The rest goes into INBOX # default is "implicit keep", we do it explicitly here keep; } ``` Attempted in `/etc/dovecot/conf.d/90-sieve.conf` vsz_limit = 256M with no luck Can somebody help me in fixing it?

2 2

custom userdb server, Exim, and proxying
by Felipe Gasper 08 Oct '20

08 Oct '20

Hi all, We have Exim using Dovecot for authentication. Dovecot, in turn, consults a custom internal server that answers Dovecot’s userdb queries. When IMAP connections arrive, for some users we want to forward those connections--without authentication--to an external IMAP server. For these users, we return “proxy_maybe” and “nopassword” in the authn response from our userdb server. This tells Dovecot to proxy the connection to a new server without trying to authenticate. Exim, though, doesn’t grok “proxy_maybe”, so it just sees “nopassword”. In response, it just skips SMTP authentication entirely. We could address this if our custom service could know whether the authn is for Exim or not. Exim does send a “service=smtp” in its request to /var/run/dovecot/auth-client, but thus far I don’t see that that’s being passed on to the userdb server. Is there any way I can configure Dovecot so that our userdb server can detect when it’s Exim doing the authentication? Alternatively we could just authenticate on the proxy (i.e., stop sending “nopassword”), but that will entail a new requirements for passwords to be in sync; ideally we’d like to avoid that. Any insight from this group would be much appreciated. Thank you! cheers, -Felipe Gasper

2 1

Dsync issues since 2.3.11.3.
by Reio Remma 02 Oct '20

02 Oct '20

Hello! I've started seeing intermittent dsync failures since upgrading Dovecot to 2.3.11.3-3 (CentOS 8 using the official Dovecot repo). Sep 27 17:49:04 host dovecot[130]: dsync-local(user@host)<5GWFD2CmcF+TJwAAsNnMGQ>: Warning: Failed to do incremental sync for mailbox INBOX, retry with a full sync (Modseq 66543 no longer in transaction log (highest=66547, last_common_uid=46346, nextuid=46347)) Sep 27 17:49:04 host dovecot[130]: dsync-local(user@host)<5GWFD2CmcF+TJwAAsNnMGQ>: Error: Remote command returned error 75: ssh <options> doveadm dsync-server -u user@host It seems to happen at random times to random users. My setup has 2 servers - users are connecting to the main server and the secondary server is just for backup - noone is actually connecting to it. Any suggestions? Thanks! Reio

1 2

debugging TLS with wireshark and a custom application ?
by Kurt Jaeger 01 Oct '20

01 Oct '20

Hi! My question is: can dovecot be used to debug/decrypt TLS sessions ? The reason I'm asking: A custom application wants to speak IMAP with TLS with a dovecot instance. It fails, and the error message is, unfortunatly, not very helpful. tcpdump shows that the session is established, but fails. The custom application says error 60000, not much more. There is a way to decode TLS sessions in some cases: https://wiki.wireshark.org/TLS using firefox and the enviroment variable SSLKEYLOGFILE. Basically, the TLS client (firefox), while negotiating keys with the other side, writes those logfiles to a logfile specified by SSLKEYLOGFILE and at the same time, wireshark is used to capture / write the pcap of the session. Then wireshark can be used to decode the contents of the TLS-encrypted session. Is something similiar possible with dovecot as the server and from the server side ? Is, by chance, the session=<...> value already such a key so that a tcpdump pcap file would be able to descrypt that session ? Sep 29 08:51:30 imap-login: Info: Login: user=<someuser>, method=PLAIN, rip=<someip>, lip=<someotherip>, mpid=63667, TLS, session=<GGF0Nm6wxy0qAQHoAAMAAwAAAAAAAAAB> -- pi(a)opsec.eu +49 171 3101372 Now what ?

3 2

auth_username_format effective only within passdb/userdb
by Bob 30 Sep '20

30 Sep '20

Hello dovecot list. I have multiple passdb/userdb, some use %Ln some use %Lu. How to make auth_username_format effective only within passdb/userdb?

2 2

Leaked files in maildir "tmp" after vsz_limit crashes
by Peter Mogensen 30 Sep '20

30 Sep '20

Hi, Lately I've seen a few examples of users hitting the vsz_limit (usually trying to "delete" mails i Spam/Junk by moving them to Trash with a large dovecot.index.cache - which resulted in mails left/leaked in the tmp directory of Trash. Sometimes it seems the client gets into a state were it repeatedly tried to sync the client and server state so it does it again and again, building up the number of files/links in tmp. It seems the default 1 week interval to "unlink_old_files()" is not enough to prevent this from blowing up inode wise. However, ... lowering it, - or increasing vsz_limit feels a bit like kicking the can down the road. PS: This on dovecot 2.2.36 /Peter

2 1