Authentication failure due to address space limit
Hi,
I faced an error that appeared rather obscure at first, so I am trying to figure out if something can be improved. I have managed to solve the issue.
I am running a Dovecot IMAP server to keep my personal mail archive. This server does not receive or send mail, just IMAP access. There was no reason to tune its performance, so its configuration is close to what Debian package has out of the box (actually doc/example-config).
At certain moment I started to get authentication failures:
dovecot[72165]: auth-worker: Error: fatal error: failed to reserve page summary memory dovecot[72165]: auth-worker: Error: dovecot[72165]: auth-worker: Error: runtime stack: dovecot[72165]: auth-worker: Error: runtime.throw({0x7f552c418194?, 0x7f552c1feb10?}) dovecot[72165]: auth-worker: Error: runtime/panic.go:1047 +0x5f fp=0x7f552c1feac0 sp=0x7f552c1fea90 pc=0x7f552c28a53f dovecot[72165]: auth-worker: Error: runtime.(*pageAlloc).sysInit(0x7f552c5f6fd0) dovecot[72165]: auth-worker: Error: runtime/mpagealloc_64bit.go:82 +0x195 fp=0x7f552c1feb48 sp=0x7f552c1feac0 pc=0x7f552c280ef5 dovecot[72165]: auth-worker: Error: runtime.(*pageAlloc).init(0x7f552c5f6fd0, 0x7f552c5f6fc0, 0x0?) dovecot[72165]: auth-worker: Error: runtime/mpagealloc.go:324 +0x70 fp=0x7f552c1feb70 sp=0x7f552c1feb48 pc=0x7f552c27eb50 dovecot[72165]: auth-worker: Error: runtime.(*mheap).init(0x7f552c5f6fc0) dovecot[72165]: auth-worker: Error: runtime/mheap.go:729 +0x13f fp=0x7f552c1feba8 sp=0x7f552c1feb70 pc=0x7f552c27bf5f dovecot[72165]: auth-worker: Error: runtime.mallocinit() dovecot[72165]: auth-worker: Error: runtime/malloc.go:407 +0xb2 fp=0x7f552c1febd0 sp=0x7f552c1feba8 pc=0x7f552c260e72 dovecot[72165]: auth-worker: Error: runtime.schedinit() dovecot[72165]: auth-worker: Error: runtime/proc.go:693 +0xab fp=0x7f552c1fec30 sp=0x7f552c1febd0 pc=0x7f552c28df0b dovecot[72165]: auth-worker: Error: runtime.rt0_go() dovecot[72165]: auth-worker: Error: runtime/asm_amd64.s:345 +0x120 fp=0x7f552c1fec38 sp=0x7f552c1fec30 pc=0x7f552c2b7c20 dovecot[72165]: auth: Error: auth-worker: Aborted PASSV request for mailuser: Worker process died unexpectedly dovecot[72165]: auth-worker: Fatal: master: service(auth-worker): child 72211 returned error 2
Such errors happen even for invalid users
curl -v 'imap://bad:bad@localhost/'I have realized that it may be related to the pam_fscrypt plugin I had installed before. It is intended for another system user, so I did not expect any negative consequence on Dovecot. Disabling the module confirmed my guess.
This PAM module is written in Go and language runtime requires almost 1G of address space (RLIMIT_AS) even when actual memory allocation is several times lower (the user has no configured login protector).
https://github.com/golang/go/issues/38010 "runtime: high startup address space usage (RLIMIT_AS) on Linux AMD64"
The code of the fscrypt PAM module has Go panic handlers, but this case error happens early during runtime initialization, so error handlers have not set yet and can not provide a more instructive error.
I do not like that Go has no knobs like compile or link flags to prevent early allocation of rather large address space. At the same time I admit that doing garbage collection efficiently is tricky and unreasonable efforts may be required to allow modest address space at startup.
Default Dovecot configuration has vsz_limit = $default_vsz_limit = 256M The following configuration snippet solves the issue:
service auth-worker { # still crash, but later # vsz_limit = 800M vsz_limit = 1024M }
I am realizing that my use case is rather specific. Mail and encrypted home directories are hardly compatible. However some other PAM modules may cause similar issues.
Cgroup Linux kernel feature should allow to limit real memory usage and it should be better than per-process limits imposed on address space size. However assigning proper control groups for Dovecot processes may be not so trivial.
I am unsure if auth-worker vsz_limit or default_vsz_limit should be raised in the example configuration.
Do you think it is reasonable to add warnings concerning PAM plugins and vsz_limit to docs? I mean
- https://doc.dovecot.org/configuration_manual/authentication/pam/
- https://doc.dovecot.org/configuration_manual/service_configuration/#auth-wor... in a similar way as it is done for Argon2 in https://doc.dovecot.org/configuration_manual/authentication/password_schemes...
P.S. Debian 12 bookworm, Dovecot 1:2.3.19.1+dfsg1-2.1
On 06/12/2023 18:06, Max Nikulin wrote:
dovecot[72165]: auth-worker: Error: fatal error: failed to reserve page summary memory dovecot[72165]: auth-worker: Error: dovecot[72165]: auth-worker: Error: runtime stack: dovecot[72165]: auth-worker: Error: runtime.throw({0x7f552c418194?, 0x7f552c1feb10?}) dovecot[72165]: auth-worker: Error: runtime/panic.go:1047 +0x5f fp=0x7f552c1feac0 sp=0x7f552c1fea90 pc=0x7f552c28a53f dovecot[72165]: auth-worker: Error: runtime.(*pageAlloc).sysInit(0x7f552c5f6fd0)
For the case that somebody will get this thread in search engine results:
I have found another thread describing similar errors due to a custom PAM module for 2 factor authentication:
dovecot: lmtp: Error: fatal error: failed to reserve page summary memory https://dovecot.org/pipermail/dovecot/2020-September/119952.html Thu Sep 17 12:20:12 EEST 2020 https://dovecot.org/pipermail/dovecot/2020-October/120108.html Thu Oct 8 18:04:26 EEST 2020
Perhaps Go is not the best language to implement PAM modules if memory footprint is not expected to be large. On the other hand e.g. argon2 hashes may be really memory hungry, so vsz_limit must be set to appropriate value.
On 14/12/2023 12:38 EET Max Nikulin <manikulin@gmail.com> wrote:
On 06/12/2023 18:06, Max Nikulin wrote:
dovecot[72165]: auth-worker: Error: fatal error: failed to reserve page summary memory dovecot[72165]: auth-worker: Error: dovecot[72165]: auth-worker: Error: runtime stack: dovecot[72165]: auth-worker: Error: runtime.throw({0x7f552c418194?, 0x7f552c1feb10?}) dovecot[72165]: auth-worker: Error: runtime/panic.go:1047 +0x5f fp=0x7f552c1feac0 sp=0x7f552c1fea90 pc=0x7f552c28a53f dovecot[72165]: auth-worker: Error: runtime.(*pageAlloc).sysInit(0x7f552c5f6fd0)
For the case that somebody will get this thread in search engine results:
I have found another thread describing similar errors due to a custom PAM module for 2 factor authentication:
dovecot: lmtp: Error: fatal error: failed to reserve page summary memory https://dovecot.org/pipermail/dovecot/2020-September/119952.html Thu Sep 17 12:20:12 EEST 2020 https://dovecot.org/pipermail/dovecot/2020-October/120108.html Thu Oct 8 18:04:26 EEST 2020
Perhaps Go is not the best language to implement PAM modules if memory footprint is not expected to be large. On the other hand e.g. argon2 hashes may be really memory hungry, so vsz_limit must be set to appropriate value.
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
You can set service auth and service auth-worker vsz_limit to 2G, that is usually enough for PAM and Argon2.
Aki
On 14/12/2023 18:01, Aki Tuomi via dovecot wrote:
You can set service auth and service auth-worker vsz_limit to 2G, that is usually enough for PAM and Argon2.
Thank you. In my first message I wrote that 1G was enough in my case. It is rather disappointing that pam_fscrypt does nothing for a mail user. Amount of rally used memory is negligible. Configuration adjustment is necessary to avoid Go runtime initialization only.
I am realizing that implementation of cgroup based resource limits require significant efforts.
I asked if this pitfall should be documented in the manual. Nobody is interested however. It may be reasonable since the case is quite specific. At least those who might face a similar issue in future have more chances to notice this thread in search results.
On 15/12/2023 18:56, Max Nikulin wrote:
On 14/12/2023 18:01, Aki Tuomi via dovecot wrote:
You can set service auth and service auth-worker vsz_limit to 2G, that is usually enough for PAM and Argon2.
Thank you. In my first message I wrote that 1G was enough in my case.
Actually 1G sometimes is not enough. Thunderbird with its multiple parallel connections created simultaneously may be quite successful in making authentication failures rather annoying.
I have not seen failure with "vsz_limit = 2G" yet.
It appears just like incorrect passwords in logs mixed with successful authentications.
12:33:23 dovecot[4299]: auth-worker(4449): Debug: conn unix:auth-worker (pid=4307,uid=114): auth-worker<1>: pam(test,127.0.0.1,<JPhaBWMN8Jp/AAAB>): #1/1 style=1 msg=Password: 12:33:23 auth[4449]: pam_unix(dovecot:auth): authentication failure; logname= uid=0 euid=0 tty=dovecot ruser=test rhost=127.0.0.1 user=test 12:33:24 dovecot[4299]: auth-worker(4449): conn unix:auth-worker (pid=4307,uid=114): auth-worker<1>: pam(test,127.0.0.1,<JPhaBWMN8Jp/AAAB>): pam_authenticate() failed: Authentication failure (Password mismatch?)
Earlier I believed that I know another (inconvenient) workaround: edit /etc/pam.d/dovecot to put content of common-auth and common-session literally and disabling pam_fscrypt.so specifically for dovecot. Unfortunately it does not work. Auth worker still loads pam_fscrypt.so from common-password through some other config file. Even if debug is enabled for pam_fscrypt.so, this library generates no output for both successful and failed authentication attempts. On the other hand I can get debug output from "auth optional pam_fscrypt.so debug".
I have no idea what piece of code (Go runtime, Dovecot, https://github.com/google/fscrypt) causes silent errors leading to authentication failures. It is disappointing since Go+fscrypt should be no-op in this case.
participants (2)
-
Aki Tuomi
-
Max Nikulin