December 2023 - dovecot

Unconfigurable 500 ms debounce on IMAP IDLE notifications
by Alex 14 Jan '24

14 Jan '24

Hello all, I have been investigating the reason for high end-to-end email delivery delay (>1 second to send a message to echo bot and receive a reply back) on a Postfix+Dovecot email server setup used to run the tests for Delta Chat core [1]. It appeared that most of the time was spent between the submission SMTP service accepting the mail for delivery and receiving an IDLE wakeup on the other side. Looking into the Dovecot source code, I found that all notifications from the storage to callbacks set up by IDLE are delayed by a debounce constant NOTIFY_DELAY_MSECS [2]. It appears to have been added in 2009 [3], but even before that Dovecot was only delivering notifications when a second-resolution timer is changed by at least 1, so IDLE notifications probably were delayed by half a second on average even before that. Commit message says "If the notification is done immediately, IDLE may not notice the change because it's not finished yet." This sounds like a workaround for some internal Dovecot bug. I have nevertheless tried to compile Dovecot with this constant reduced from 500 ms to 1 ms [4] and it resulted in exceptionally fast delivery without any problems caused to the Delta Chat core online test suite which is using this Postfix+Dovecot server accounts. We are running all the tests against this server and it appears to work fine. Round trip time for sending a message to an echo bot and receiving a reply back is reduced from 1.1 s to 0.2 s, which is easily visible in a chat app [5]. I wonder if this debounce delay was added as a workaround for an internal Dovecot bug or a client bug and is not necessary anymore. It does not seem to be useful for well-behaving clients because if necessary the client may debounce notifications on its side and postpone interrupting IDLE with "DONE". I would actually like to disable this delay completely on our server setup [6], as for chat it is common to receive multiple messages in a short period of time, e.g. when sending a message to a group and receiving multiple read receipts from currently active users. [1] https://github.com/deltachat/deltachat-core-rust/ [2] https://github.com/dovecot/core/blob/93a53a9d590f0220de28600f36a969cc38c314… [3] https://github.com/dovecot/core/commit/56fb5d09955b6097f77b341717fd9b70e9f1… [4] https://github.com/deltachat/chatmail/issues/72#issuecomment-1806645153 [5] https://delta.chat/assets/blog/nine-echo.mp4 [6] https://github.com/deltachat/chatmail Best regards, Alex

2 6

Solr/Tika patches for dovecot 2.3.21
by John Fawcett 03 Jan '24

03 Jan '24

Hi I've just made updated versions of 3 patches against the latest release 2.3.21 in case they are useful to someone or might get considered for official inclusion. John *dovecot-2.3.21-tika-http-auth.patch* Allows specification of username and password in the fts_tika setting for basic auth against tika server. For example fts_tika = https://user:password@tika_server:443/tika *dovecot-2.3.21-solr-max-size.patch * This is a simplified version of my previous patch. Sets a size limit (configuration fts_max_size) on message bodies that are to be indexed. Message bodies for messages larger than fts_max_size are not sent to solr. Defaults to zero which means no limit. For example fts_max_size = 10M *dovecot-2.3.21-solr-max-rows.patch * When dovecot sends a search to solr it uses the rows parameter. For multiple mailbox search the value used is SOLR_MAX_MULTI_ROWS , hardcoded to 100000. For single mailbox search the value is uidnext. This patch introduces an upper limit for single mailbox search using the same value as SOLR_MAX_MULTI_ROWS, while leaving the existing functionality of sending the uidnext value if it is smaller. This is just to place a more reasonable upper bound since uidnext can get much larger. Hi I've just made updated versions of 3 patches against the latest release 2.3.21 in case they are useful to someone or might get considered for official inclusion. John dovecot-2.3.21-tika-http-auth.patch Allows specification of username and password in the fts_tika setting for basic auth against tika server. For example fts_tika = https://user:password@tika_server:443/tika dovecot-2.3.21-solr-max-size.patch This is a simplified version of my previous patch. Sets a size limit (configuration fts_max_size) on message bodies that are to be indexed. Message bodies for messages larger than fts_max_size are not sent to solr. Defaults to zero which means no limit. For example fts_max_size = 10M dovecot-2.3.21-solr-max-rows.patch When dovecot sends a search to solr it uses the rows parameter. For multiple mailbox search the value used is SOLR_MAX_MULTI_ROWS , hardcoded to 100000. For single mailbox search the value is uidnext. This patch introduces an upper limit for single mailbox search using the same value as SOLR_MAX_MULTI_ROWS, while leaving the existing functionality of sending the uidnext value if it is smaller. This is just to place a more reasonable upper bound since uidnext can get much larger.

2 2

how to take mdbox extract of time interval and restore that again?
by lutz.niederer＠gmx.net 03 Jan '24

03 Jan '24

Hi! Due to a fatal hardware error we have to extract data of all existing imap accounts and all their boxes/folders etc. from an mdbox store starting with the 5th of december until today. Then we have to restore the old system and import the data back into it. Mail before 5th will come from the system backup, newer mail from that export. After that it may be possible that some/few mails may exist twice in the boxes, but that's ok. 1. What would be the command line for doveadm to extract everything starting the 5th? I find that the command line for doveadm is pretty complicated. I tried but I am not sure if I really got all mails exported. So I would like to ask if someone with much more experience can give me a command line that surely does what we want. The command will be run from a remote machine. 2. And what would be the command line to restore all the mails that we have exported back into the system? (The folder and account structure is the same.) 3. After that some of the mails may exist twice. Is that a problem, maybe because of internal IDs? 4. Are there any other things we need to be aware of when doing that? I thank you very much for your help! Cheers! -lutzn

2 1

Authentication failure due to address space limit
by Max Nikulin 26 Dec '23

26 Dec '23

Hi, I faced an error that appeared rather obscure at first, so I am trying to figure out if something can be improved. I have managed to solve the issue. I am running a Dovecot IMAP server to keep my personal mail archive. This server does not receive or send mail, just IMAP access. There was no reason to tune its performance, so its configuration is close to what Debian package has out of the box (actually doc/example-config). At certain moment I started to get authentication failures: dovecot[72165]: auth-worker: Error: fatal error: failed to reserve page summary memory dovecot[72165]: auth-worker: Error: dovecot[72165]: auth-worker: Error: runtime stack: dovecot[72165]: auth-worker: Error: runtime.throw({0x7f552c418194?, 0x7f552c1feb10?}) dovecot[72165]: auth-worker: Error: runtime/panic.go:1047 +0x5f fp=0x7f552c1feac0 sp=0x7f552c1fea90 pc=0x7f552c28a53f dovecot[72165]: auth-worker: Error: runtime.(*pageAlloc).sysInit(0x7f552c5f6fd0) dovecot[72165]: auth-worker: Error: runtime/mpagealloc_64bit.go:82 +0x195 fp=0x7f552c1feb48 sp=0x7f552c1feac0 pc=0x7f552c280ef5 dovecot[72165]: auth-worker: Error: runtime.(*pageAlloc).init(0x7f552c5f6fd0, 0x7f552c5f6fc0, 0x0?) dovecot[72165]: auth-worker: Error: runtime/mpagealloc.go:324 +0x70 fp=0x7f552c1feb70 sp=0x7f552c1feb48 pc=0x7f552c27eb50 dovecot[72165]: auth-worker: Error: runtime.(*mheap).init(0x7f552c5f6fc0) dovecot[72165]: auth-worker: Error: runtime/mheap.go:729 +0x13f fp=0x7f552c1feba8 sp=0x7f552c1feb70 pc=0x7f552c27bf5f dovecot[72165]: auth-worker: Error: runtime.mallocinit() dovecot[72165]: auth-worker: Error: runtime/malloc.go:407 +0xb2 fp=0x7f552c1febd0 sp=0x7f552c1feba8 pc=0x7f552c260e72 dovecot[72165]: auth-worker: Error: runtime.schedinit() dovecot[72165]: auth-worker: Error: runtime/proc.go:693 +0xab fp=0x7f552c1fec30 sp=0x7f552c1febd0 pc=0x7f552c28df0b dovecot[72165]: auth-worker: Error: runtime.rt0_go() dovecot[72165]: auth-worker: Error: runtime/asm_amd64.s:345 +0x120 fp=0x7f552c1fec38 sp=0x7f552c1fec30 pc=0x7f552c2b7c20 dovecot[72165]: auth: Error: auth-worker: Aborted PASSV request for mailuser: Worker process died unexpectedly dovecot[72165]: auth-worker: Fatal: master: service(auth-worker): child 72211 returned error 2 Such errors happen even for invalid users curl -v 'imap://bad:bad@localhost/' I have realized that it may be related to the pam_fscrypt plugin I had installed before. It is intended for another system user, so I did not expect any negative consequence on Dovecot. Disabling the module confirmed my guess. This PAM module is written in Go and language runtime requires almost 1G of address space (RLIMIT_AS) even when actual memory allocation is several times lower (the user has no configured login protector). https://github.com/golang/go/issues/38010 "runtime: high startup address space usage (RLIMIT_AS) on Linux AMD64" The code of the fscrypt PAM module has Go panic handlers, but this case error happens early during runtime initialization, so error handlers have not set yet and can not provide a more instructive error. I do not like that Go has no knobs like compile or link flags to prevent early allocation of rather large address space. At the same time I admit that doing garbage collection efficiently is tricky and unreasonable efforts may be required to allow modest address space at startup. Default Dovecot configuration has vsz_limit = $default_vsz_limit = 256M The following configuration snippet solves the issue: service auth-worker { # still crash, but later # vsz_limit = 800M vsz_limit = 1024M } I am realizing that my use case is rather specific. Mail and encrypted home directories are hardly compatible. However some other PAM modules may cause similar issues. Cgroup Linux kernel feature should allow to limit real memory usage and it should be better than per-process limits imposed on address space size. However assigning proper control groups for Dovecot processes may be not so trivial. I am unsure if auth-worker vsz_limit or default_vsz_limit should be raised in the example configuration. Do you think it is reasonable to add warnings concerning PAM plugins and vsz_limit to docs? I mean - https://doc.dovecot.org/configuration_manual/authentication/pam/ - https://doc.dovecot.org/configuration_manual/service_configuration/#auth-wo… in a similar way as it is done for Argon2 in https://doc.dovecot.org/configuration_manual/authentication/password_scheme… P.S. Debian 12 bookworm, Dovecot 1:2.3.19.1+dfsg1-2.1

2 4

Elevate Your Storytelling Experience with Start Motion Media
by petersthomas383＠gmail.com 19 Dec '23

19 Dec '23

Startmotionmedia.com Media, we specialize in crafting captivating visual narratives through expert video production services. Our dedicated team focuses on delivering compelling storytelling that resonates across diverse industries, creating impactful videos that leave a lasting impression. At Start Motion Media, our mission is to bring creativity to life. From conceptualization to production, we ensure each project reflects our commitment to excellence. Our portfolio showcases the artistry and skill behind Start Motion Media's captivating visual storytelling. Experience storytelling prowess firsthand by exploring <a href="https://www.startmotionmedia.com/can-web-development-be-a-side-hustle-for-s…">startmotionmedia</a>. Immerse yourself in our gallery of projects, witnessing the transformative power of visual narratives. Start Motion Media invites you to witness how storytelling unfolds through our lens, shaping brands and captivating audiences worldwide. For businesses seeking to amplify their narratives and engage audiences effectively, Start Motion Media is the answer. Harness the storytelling expertise of Start Motion Media to elevate your brand's story and make a lasting impact in the digital sphere.

1 0

Re-Enabling TLS1.1 in dovecot 2.3.16
by Nils 14 Dec '23

14 Dec '23

Hi, I'd like to re-enable TLS1.1 (at least temporary) to see if it fixes a problem with one of my clients. When I add "ssl_min_protocol = TLSv1.1" to either dovecot.conf or conf.d/10-ssl.conf it doesn't seem to have an effect. Still the client is not working. I used testssl.sh to check the available ssl versions and it says that tls1.1 is not offered: $ ./testssl.sh --starttls imap SERVERNAME.TLD:143 ... Service set: STARTTLS via IMAP Testing protocols via sockets SSLv2 not offered (OK) SSLv3 not offered (OK) TLS 1 not offered TLS 1.1 not offered TLS 1.2 offered (OK) TLS 1.3 offered (OK): final Kind regards, Nils

2 2

dovecot submission max line length
by Kees van Vloten 13 Dec '23

13 Dec '23

Hi all, After upgrading Debian Bullseye (Dovecot 2.3.13) to Bookworm (Dovcecot 2.3.19), submission-login returns an error on Kerberos authentication: "Client sent invalid command: Command line is too long". When I initially set this up on Bullseye I have seen the same issue, but adding "imap_max_line_length = 2M" solved it. Now with 2.3.19 kerberos authentication still works (with this setting) for imap connections but it stopped working for submission. Is there a new config setting for submission-login that I missed? Or is this a bug in 2.3.19? - Kees.

3 3

Username used by dovecot to access database certificate and key files
by John Fawcett 08 Dec '23

08 Dec '23

Hi does anyone happen to know the username under which dovecot accesses the certificate and key files when connecting to a mariadb database with ssl? I can't seem to get dovecot to read the files without a "other" readable bit set. I have dovecot users in mysql with the following connect string in dovecot-sql.conf.ext connect = host=x.x.x host=y.y.y host=z.z.z dbname=maildb user=mailuser password=xxxxxx \ ssl_ca=/etc/ssl/mysql/ca-cert.pem \ ssl_cert=/etc/ssl/mysql/client-cert.pem \ ssl_key=/etc/ssl/mysql/client-key.pem \ ssl_verify_server_cert=yes The certificate and key files in /etc/ssl/mysql/ are owned by root with group users: ls -l /etc/ssl/mysql total 12 -r--r--r--. 1 root root 2041 Jun 22 15:32 ca-cert.pem -r--r--r--. 1 root users 1907 Jun 22 15:32 client-cert.pem -r--r--r--. 1 root users 3272 Jun 22 15:32 client-key.pem For my test dovecot, dovenull and vmail are all members of group users: grep users /etc/group users:x:100:postfix,dovecot,dovenull,vmail If I remove the "other" readable attribute on the certificate and key files, dovecot complains it can't connect to the db server: auth: Error: mysql(x.x.x): Connect failed to database (maildb): TLS/SSL error: Permission denied (13) - waiting for 1 seconds before retry I realize it only adds marginal security to not have these files world readable, but still it is puzzling why dovecot cannot read them in that case. Can anyone see what I am missing? Thanks John

3 5

Unlocking Entertainment Excellence: Explore jGuru's Latest Netflix Series Guide
by Angelique Medina 08 Dec '23

08 Dec '23

Discover a digital haven at jgurucom, where entertainment expertise meets insightful curation. JGuru, a premier online destination, stands as a beacon for enthusiasts seeking the latest trends and in-depth knowledge. As a dynamic platform, jGuru transcends conventional boundaries, offering a plethora of resources for tech aficionados, gamers, and pop culture enthusiasts alike. Navigate through jGuru's diverse content to unearth gems like their comprehensive guide to the most anticipated Netflix series of 2022. Delve into the intricacies of storytelling, cinematography, and exclusive insights that make each series a must-watch. JGuru's commitment to excellence shines through in its meticulous coverage, establishing it as a go-to hub for all things entertainment. For an unparalleled experience, visit http://www.jguru.com/the-most-anticipated-netflix-series-2022/ – where entertainment aficionados converge to elevate their viewing journey. JGuru: Your Gateway to Entertainment Brilliance.

1 0

Compiler error
by Goetz Schultz 06 Dec '23

06 Dec '23

Hi, I have upgraded my system to FreeBSD14 and tried to recompile dovecot. The following error is thrown now (It did compile on FBSD13): <quote> test-mail-index-transaction-update.c:650:36: error: incompatible pointer to integer conversion passing 'char *(*)(int, int)' to parameter of type 'time_t' (aka 'long') [-Wint-conversion] mail_index_update_day_headers(t, tests[i].now + timezone); ^~~~~~~~~~~~~~~~~~~~~~~ </quote> (t, tests[i].now + timezone); ^~~~~~~~~~~~~~~~~~~~~~~ Is there a patch or something like that to compensate? -- Thanks and regards Goetz R Schultz ---------------->8---------------- Quis custodiet ipsos custodes? /"\ \ / ASCII Ribbon Campaign X against HTML e-mail / \ ----------------8<---------------- ---------------------------->8------------------------------ /"\ \ / ASCII Ribbon Campaign X against HTML e-mail / \ This message is transmitted on 100% recycled electrons. ---------------------------->8------------------------------ Unsigned message - no responsibillity that content is not altered

4 9