Hi,
I faced an error that appeared rather obscure at first, so I am trying to figure out if something can be improved. I have managed to solve the issue.
I am running a Dovecot IMAP server to keep my personal mail archive. This server does not receive or send mail, just IMAP access. There was no reason to tune its performance, so its configuration is close to what Debian package has out of the box (actually doc/example-config).
At certain moment I started to get authentication failures:
dovecot[72165]: auth-worker: Error: fatal error: failed to reserve page summary memory dovecot[72165]: auth-worker: Error: dovecot[72165]: auth-worker: Error: runtime stack: dovecot[72165]: auth-worker: Error: runtime.throw({0x7f552c418194?, 0x7f552c1feb10?}) dovecot[72165]: auth-worker: Error: runtime/panic.go:1047 +0x5f fp=0x7f552c1feac0 sp=0x7f552c1fea90 pc=0x7f552c28a53f dovecot[72165]: auth-worker: Error: runtime.(*pageAlloc).sysInit(0x7f552c5f6fd0) dovecot[72165]: auth-worker: Error: runtime/mpagealloc_64bit.go:82 +0x195 fp=0x7f552c1feb48 sp=0x7f552c1feac0 pc=0x7f552c280ef5 dovecot[72165]: auth-worker: Error: runtime.(*pageAlloc).init(0x7f552c5f6fd0, 0x7f552c5f6fc0, 0x0?) dovecot[72165]: auth-worker: Error: runtime/mpagealloc.go:324 +0x70 fp=0x7f552c1feb70 sp=0x7f552c1feb48 pc=0x7f552c27eb50 dovecot[72165]: auth-worker: Error: runtime.(*mheap).init(0x7f552c5f6fc0) dovecot[72165]: auth-worker: Error: runtime/mheap.go:729 +0x13f fp=0x7f552c1feba8 sp=0x7f552c1feb70 pc=0x7f552c27bf5f dovecot[72165]: auth-worker: Error: runtime.mallocinit() dovecot[72165]: auth-worker: Error: runtime/malloc.go:407 +0xb2 fp=0x7f552c1febd0 sp=0x7f552c1feba8 pc=0x7f552c260e72 dovecot[72165]: auth-worker: Error: runtime.schedinit() dovecot[72165]: auth-worker: Error: runtime/proc.go:693 +0xab fp=0x7f552c1fec30 sp=0x7f552c1febd0 pc=0x7f552c28df0b dovecot[72165]: auth-worker: Error: runtime.rt0_go() dovecot[72165]: auth-worker: Error: runtime/asm_amd64.s:345 +0x120 fp=0x7f552c1fec38 sp=0x7f552c1fec30 pc=0x7f552c2b7c20 dovecot[72165]: auth: Error: auth-worker: Aborted PASSV request for mailuser: Worker process died unexpectedly dovecot[72165]: auth-worker: Fatal: master: service(auth-worker): child 72211 returned error 2
Such errors happen even for invalid users
curl -v 'imap://bad:bad@localhost/'I have realized that it may be related to the pam_fscrypt plugin I had installed before. It is intended for another system user, so I did not expect any negative consequence on Dovecot. Disabling the module confirmed my guess.
This PAM module is written in Go and language runtime requires almost 1G of address space (RLIMIT_AS) even when actual memory allocation is several times lower (the user has no configured login protector).
https://github.com/golang/go/issues/38010 "runtime: high startup address space usage (RLIMIT_AS) on Linux AMD64"
The code of the fscrypt PAM module has Go panic handlers, but this case error happens early during runtime initialization, so error handlers have not set yet and can not provide a more instructive error.
I do not like that Go has no knobs like compile or link flags to prevent early allocation of rather large address space. At the same time I admit that doing garbage collection efficiently is tricky and unreasonable efforts may be required to allow modest address space at startup.
Default Dovecot configuration has vsz_limit = $default_vsz_limit = 256M The following configuration snippet solves the issue:
service auth-worker { # still crash, but later # vsz_limit = 800M vsz_limit = 1024M }
I am realizing that my use case is rather specific. Mail and encrypted home directories are hardly compatible. However some other PAM modules may cause similar issues.
Cgroup Linux kernel feature should allow to limit real memory usage and it should be better than per-process limits imposed on address space size. However assigning proper control groups for Dovecot processes may be not so trivial.
I am unsure if auth-worker vsz_limit or default_vsz_limit should be raised in the example configuration.
Do you think it is reasonable to add warnings concerning PAM plugins and vsz_limit to docs? I mean
- https://doc.dovecot.org/configuration_manual/authentication/pam/
- https://doc.dovecot.org/configuration_manual/service_configuration/#auth-wor... in a similar way as it is done for Argon2 in https://doc.dovecot.org/configuration_manual/authentication/password_schemes...
P.S. Debian 12 bookworm, Dovecot 1:2.3.19.1+dfsg1-2.1