Dovecot 2.4 doveadm_worker_count corrupts tab-formatted output
Hi there
On a Dovecot 2.4.2 (Debian Bookworm) server with over 1300 users, I am using the following config to retrieve quota usage information with doveadm:
quota "User quota" { }
mail_plugins = quota acl mail_log notify
userdb sql { iterate_query = SELECT username AS user, CONCAT(quota, 'M') AS quota_storage_size FROM mailaccounts WHERE active = 1
query =
SELECT '/var/vmail/%{user | domain}/%{user | username}' AS home, 'vmail' AS uid, 'vmail' AS gid, CONCAT(quota, 'M') AS quota_storage_size
FROM mailaccounts
WHERE username = '%{user}' AND (active = 1 OR transport_suspended = 0)
}
Initially, I had to run doveadm quota recalc -A to recalc quotas and populate the count quota driver which is enabled by default for "User quota".
But still, it took roughly 20ms per single user or 15-20s for all users:
$ time doveadm -f tab quota get -u demo@example.com real 0m0.020s $ time doveadm -f tab quota get -A real 0m16.800s
Now, to greatly speed this up, I have introduced parallel processing:
doveadm_worker_count = 4
Like this, doveadm -f tab quota get -A runs approx 10x faster, done in 1.5 - 2.5s
But there is one major drawback which I assume to be a bug:
sane output without doveadm_worker_count = 4 => always 6 fields
demo@example.com User quota STORAGE 29 204800 0 demo@example.com User quota MESSAGE 1 - 0
broken output with doveadm_worker_count = 4 => sometimes 7 fields
demo@example.com User quota STORAGE 29 2048 00 0 demo@example.com User quota MESSAGE 1 - 0
Somehow, the tab formatter does not play nicely together with parallel processing. On some output, the value gets split apart by an extra tab, e.g. '2024<TAB>00' in above example. Can you explain this corrupted data? Some kind of "output interleaving" seems to happen when running parallel workers.
If I switch to json output (doveadm -f json quota get -A), this does not seem to happen and I always get valid JSON output, even with doveadm_worker_count = 4.
Cheers, Philip
Some more insights about this issue... It happens every time (on different users) on quota lookup for all users:
$ doveadm quota get -A > /dev/null; echo $? doveadm(demo@example.com): Error: auth-master: userdb list: User listing returned failure doveadm: Error: cmd quota get: Failed to iterate through some users 75
Full debug output see [1].
In mail.log, I then find this line:
dovecot: auth: Error: auth-worker: Aborted LIST request for *: Shutting down
The weird thing is that this problem happens always one one server (2200 users), while never happening on another (1200 users). Both running 100% same configuration / same versions (Dovecot 2.4.2 / Debian Bookworm, Percona for MySQL 8.4.7), and both now with disabled IMAPSieve Plugin (imap-sieve) which is broken in 2.4.2.
Above problem occurs both with or without doveadm_worker_count = 4, so that's not the culprit.
The whole doveadm quota get -A always runs in less than 3s, so I cannot imagine any MySQL connection issues with the iterate_query.
I did not override anything in service auth or service auth-worker sections.
Any recommendations?
Thanks, Philip
[1] $ doveadm -Dv quota get -A 2>&1 | tee /tmp/debug.log (...) Feb 26 13:06:17 doveadm(demo@example.com)<358389><>: Debug: auth-master: userdb lookup(demo@example.com): Started userdb lookup Feb 26 13:06:17 doveadm(demo@example.com)<358389><>: Debug: auth-master: request [4211869176]: Created Feb 26 13:06:17 doveadm(demo@example.com)<358389><>: Debug: auth-master: userdb lookup(demo@example.com): request [4211869176]: Waiting for request to complete Feb 26 13:06:17 doveadm(demo@example.com)<358389><>: Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): Sending requests Feb 26 13:06:17 doveadm(demo@example.com)<358389><>: Debug: auth-master: userdb lookup(demo@example.com): request [4211869176]: Sent Feb 26 13:06:17 doveadm(demo@example.com)<358389><>: Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): auth input: USER 4211869176 demo@example.com home=/var/vmail/example.com/demo uid=998 gid=998 quota_storage_size=300M Feb 26 13:06:17 doveadm(demo@example.com)<358389><>: Debug: auth-master: userdb lookup(demo@example.com): request [4211869176]: Got reply: USER demo@example.com home=/var/vmail/example.com/demo uid=998 gid=998 quota_storage_size=300M Feb 26 13:06:17 doveadm(demo@example.com)<358389><>: Debug: auth-master: userdb lookup(demo@example.com): auth USER input: demo@example.com home=/var/vmail/example.com/demo uid=998 gid=998 quota_storage_size=300M Feb 26 13:06:17 doveadm(demo@example.com)<358389><>: Debug: auth-master: userdb lookup(demo@example.com): Finished userdb lookup (username=demo@example.com home=/var/vmail/example.com/demo uid=998 gid=998 quota_storage_size=300M) Feb 26 13:06:17 doveadm(demo@example.com)<358389><>: Debug: auth-master: userdb lookup(demo@example.com): request [4211869176]: Remove Feb 26 13:06:17 doveadm(demo@example.com)<358389><>: Debug: auth-master: userdb lookup(demo@example.com): request [4211869176]: Finished waiting for request Feb 26 13:06:17 doveadm(demo@example.com)<358389><>: Debug: auth-master: userdb lookup(demo@example.com): request [4211869176]: Destroy Feb 26 13:06:17 doveadm(demo@example.com)<358389><>: Debug: Added setting via userdb: quota_storage_size=300M Feb 26 13:06:17 doveadm(demo@example.com): Debug: Effective uid=998, gid=998, home=/var/vmail/example.com/demo Feb 26 13:06:17 doveadm(demo@example.com): Debug: acl: Shared mailbox listing disabled: dict { .. } named list filter is missing Feb 26 13:06:17 doveadm(demo@example.com): Debug: Namespace inbox: type=private, prefix=INBOX/, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes Feb 26 13:06:17 doveadm(demo@example.com): Debug: fs: root=/var/vmail/example.com/demo/sdbox, index=, indexpvt=, control=, inbox=, alt= Feb 26 13:06:17 doveadm(demo@example.com): Debug: acl: initializing backend vfile Feb 26 13:06:17 doveadm(demo@example.com): Debug: acl: acl username = demo@example.com Feb 26 13:06:17 doveadm(demo@example.com): Debug: acl: owner = yes Feb 26 13:06:17 doveadm(demo@example.com): Debug: acl: ignore = no Feb 26 13:06:17 doveadm(demo@example.com): Debug: acl: vfile: Deprecated Global ACL file: /etc/dovecot/dovecot-acl Feb 26 13:06:17 doveadm(demo@example.com): Debug: Namespace : type=private, prefix=, sep=, inbox=no, hidden=yes, list=no, subscriptions=no Feb 26 13:06:17 doveadm(demo@example.com): Debug: none: root=/var/vmail/example.com/demo/sdbox, index=, indexpvt=, control=, inbox=, alt= Feb 26 13:06:17 doveadm(demo@example.com): Debug: quota-count: quota_over_status check: quota_over_mask unset - skipping Feb 26 13:06:17 doveadm(demo@example.com): Debug: Mailbox INBOX: Mailbox opened Feb 26 13:06:17 doveadm(demo@example.com): Debug: User session is finished Feb 26 13:06:17 doveadm(demo@example.com): Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): auth input: DONE 2802712577 fail Feb 26 13:06:17 doveadm(demo@example.com): Debug: auth-master: userdb list: request [2802712577]: Got reply: DONE fail Feb 26 13:06:17 doveadm(demo@example.com): Error: auth-master: userdb list: User listing returned failure Feb 26 13:06:17 doveadm(demo@example.com): Debug: auth-master: userdb list: request [2802712577]: Remove Feb 26 13:06:17 doveadm(demo@example.com): Debug: auth-master: userdb list: request [2802712577]: Destroy Feb 26 13:06:17 doveadm(demo@example.com): Debug: auth-master: userdb list: Listing users failed Feb 26 13:06:17 doveadm(demo@example.com): Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): Disconnected: Connection closed (fd=9) Feb 26 13:06:17 doveadm(demo@example.com): Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): Disconnected from auth service Feb 26 13:06:17 doveadm: Error: cmd quota get: Failed to iterate through some users Feb 26 13:06:17 doveadm: Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): Disconnected: Connection closed (fd=10) Feb 26 13:06:17 doveadm: Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): Disconnected from auth service
On 25 Feb 2026, at 15:48, Philip Iezzi via dovecot <dovecot@dovecot.org> wrote:
Hi there
On a Dovecot 2.4.2 (Debian Bookworm) server with over 1300 users, I am using the following config to retrieve quota usage information with doveadm:
quota "User quota" { }
mail_plugins = quota acl mail_log notify
userdb sql { iterate_query = SELECT username AS user, CONCAT(quota, 'M') AS quota_storage_size FROM mailaccounts WHERE active = 1
query =
SELECT '/var/vmail/%{user | domain}/%{user | username}' AS home, 'vmail' AS uid, 'vmail' AS gid, CONCAT(quota, 'M') AS quota_storage_size
FROM mailaccounts
WHERE username = '%{user}' AND (active = 1 OR transport_suspended = 0) }
Initially, I had to run
doveadm quota recalc -Ato recalc quotas and populate thecountquota driver which is enabled by default for "User quota". But still, it took roughly 20ms per single user or 15-20s for all users:$ time doveadm -f tab quota get -u demo@example.com real 0m0.020s $ time doveadm -f tab quota get -A real 0m16.800s
Now, to greatly speed this up, I have introduced parallel processing:
doveadm_worker_count = 4
Like this,
doveadm -f tab quota get -Aruns approx 10x faster, done in 1.5 - 2.5s But there is one major drawback which I assume to be a bug:sane output without
doveadm_worker_count = 4=> always 6 fieldsdemo@example.com User quota STORAGE 29 204800 0 demo@example.com User quota MESSAGE 1 - 0
broken output with
doveadm_worker_count = 4=> sometimes 7 fieldsdemo@example.com User quota STORAGE 29 2048 00 0 demo@example.com User quota MESSAGE 1 - 0
Somehow, the tab formatter does not play nicely together with parallel processing. On some output, the value gets split apart by an extra tab, e.g. '2024<TAB>00' in above example. Can you explain this corrupted data? Some kind of "output interleaving" seems to happen when running parallel workers. If I switch to json output (
doveadm -f json quota get -A), this does not seem to happen and I always get valid JSON output, even withdoveadm_worker_count = 4.Cheers, Philip
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
On 26/02/2026 15:42, Philip Iezzi via dovecot wrote:
Some more insights about this issue... It happens every time (on different users) on quota lookup for all users:
$ doveadm quota get -A > /dev/null; echo $? doveadm(demo@example.com): Error: auth-master: userdb list: User listing returned failure doveadm: Error: cmd quota get: Failed to iterate through some users 75
Full debug output see [1].
... $ doveadm -Dv quota get -A 2>&1 | tee /tmp/debug.log (...) Feb 26 13:06:17 doveadm(demo@example.com)<358389><>: Debug: auth-master: userdb lookup(demo@example.com): Started userdb lookup Feb 26 13:06:17 doveadm(demo@example.com)<358389><>: Debug: auth-master: request [4211869176]: Created Feb 26 13:06:17 doveadm(demo@example.com)<358389><>: Debug: auth-master: userdb lookup(demo@example.com): request [4211869176]: Waiting for request to complete Feb 26 13:06:17 doveadm(demo@example.com)<358389><>: Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): Sending requests Feb 26 13:06:17 doveadm(demo@example.com)<358389><>: Debug: auth-master: userdb lookup(demo@example.com): request [4211869176]: Sent Feb 26 13:06:17 doveadm(demo@example.com)<358389><>: Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): auth input: USER 4211869176 demo@example.com home=/var/vmail/example.com/demo uid=998 gid=998 quota_storage_size=300M Feb 26 13:06:17 doveadm(demo@example.com)<358389><>: Debug: auth-master: userdb lookup(demo@example.com): request [4211869176]: Got reply: USERdemo@example.com home=/var/vmail/example.com/demo uid=998 gid=998 quota_storage_size=300M Feb 26 13:06:17 doveadm(demo@example.com)<358389><>: Debug: auth-master: userdb lookup(demo@example.com): auth USER input:demo@example.com home=/var/vmail/example.com/demo uid=998 gid=998 quota_storage_size=300M Feb 26 13:06:17 doveadm(demo@example.com)<358389><>: Debug: auth-master: userdb lookup(demo@example.com): Finished userdb lookup (username=demo@example.com home=/var/vmail/example.com/demo uid=998 gid=998 quota_storage_size=300M) Feb 26 13:06:17 doveadm(demo@example.com)<358389><>: Debug: auth-master: userdb lookup(demo@example.com): request [4211869176]: Remove Feb 26 13:06:17 doveadm(demo@example.com)<358389><>: Debug: auth-master: userdb lookup(demo@example.com): request [4211869176]: Finished waiting for request Feb 26 13:06:17 doveadm(demo@example.com)<358389><>: Debug: auth-master: userdb lookup(demo@example.com): request [4211869176]: Destroy Feb 26 13:06:17 doveadm(demo@example.com)<358389><>: Debug: Added setting via userdb: quota_storage_size=300M Feb 26 13:06:17 doveadm(demo@example.com): Debug: Effective uid=998, gid=998, home=/var/vmail/example.com/demo Feb 26 13:06:17 doveadm(demo@example.com): Debug: acl: Shared mailbox listing disabled: dict { .. } named list filter is missing Feb 26 13:06:17 doveadm(demo@example.com): Debug: Namespace inbox: type=private, prefix=INBOX/, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes Feb 26 13:06:17 doveadm(demo@example.com): Debug: fs: root=/var/vmail/example.com/demo/sdbox, index=, indexpvt=, control=, inbox=, alt= Feb 26 13:06:17 doveadm(demo@example.com): Debug: acl: initializing backend vfile Feb 26 13:06:17 doveadm(demo@example.com): Debug: acl: acl username =demo@example.com Feb 26 13:06:17 doveadm(demo@example.com): Debug: acl: owner = yes Feb 26 13:06:17 doveadm(demo@example.com): Debug: acl: ignore = no Feb 26 13:06:17 doveadm(demo@example.com): Debug: acl: vfile: Deprecated Global ACL file: /etc/dovecot/dovecot-acl Feb 26 13:06:17 doveadm(demo@example.com): Debug: Namespace : type=private, prefix=, sep=, inbox=no, hidden=yes, list=no, subscriptions=no Feb 26 13:06:17 doveadm(demo@example.com): Debug: none: root=/var/vmail/example.com/demo/sdbox, index=, indexpvt=, control=, inbox=, alt= Feb 26 13:06:17 doveadm(demo@example.com): Debug: quota-count: quota_over_status check: quota_over_mask unset - skipping Feb 26 13:06:17 doveadm(demo@example.com): Debug: Mailbox INBOX: Mailbox opened Feb 26 13:06:17 doveadm(demo@example.com): Debug: User session is finished Feb 26 13:06:17 doveadm(demo@example.com): Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): auth input: DONE 2802712577 fail Feb 26 13:06:17 doveadm(demo@example.com): Debug: auth-master: userdb list: request [2802712577]: Got reply: DONE fail Feb 26 13:06:17 doveadm(demo@example.com): Error: auth-master: userdb list: User listing returned failure Feb 26 13:06:17 doveadm(demo@example.com): Debug: auth-master: userdb list: request [2802712577]: Remove Feb 26 13:06:17 doveadm(demo@example.com): Debug: auth-master: userdb list: request [2802712577]: Destroy Feb 26 13:06:17 doveadm(demo@example.com): Debug: auth-master: userdb list: Listing users failed Feb 26 13:06:17 doveadm(demo@example.com): Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): Disconnected: Connection closed (fd=9) Feb 26 13:06:17 doveadm(demo@example.com): Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): Disconnected from auth service Feb 26 13:06:17 doveadm: Error: cmd quota get: Failed to iterate through some users Feb 26 13:06:17 doveadm: Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): Disconnected: Connection closed (fd=10) Feb 26 13:06:17 doveadm: Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): Disconnected from auth service
Hi
I could be wrong but is this debug output complete? It looks like request 2802712577 is failing, but the previous logging relating to demo@example.com seems to be from request 4211869176.
Also, though I don't believe either it is related to the issue:
What is the reason for returning quota_storage_size in the iterate_query? It shouldn't matter if you return extra fields, but why do it?
Your iterate query can potentially return a subset of the users compared to your user query due to differing where conditions.
John
On 26/02/2026 15:42, Philip Iezzi via dovecot wrote:
Some more insights about this issue... It happens every time (on different users) on quota lookup for all users:
$ doveadm quota get -A > /dev/null; echo $? doveadm([1]demo@example.com): Error: auth-master: userdb list: User listing returned failure doveadm: Error: cmd quota get: Failed to iterate through some users 75
Full debug output see [1].
... $ doveadm -Dv quota get -A 2>&1 | tee /tmp/debug.log (...) Feb 26 13:06:17 doveadm([2]demo@example.com)<358389><>: Debug: auth-master: userdb lookup([3]demo@example.com): Started userdb lookup Feb 26 13:06:17 doveadm([4]demo@example.com)<358389><>: Debug: auth-master: request [4211869176]: Created Feb 26 13:06:17 doveadm([5]demo@example.com)<358389><>: Debug: auth-master: userdb lookup([6]demo@example.com): request [4211869176]: Waiting for request to complete Feb 26 13:06:17 doveadm([7]demo@example.com)<358389><>: Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): Sending requests Feb 26 13:06:17 doveadm([8]demo@example.com)<358389><>: Debug: auth-master: userdb lookup([9]demo@example.com): request [4211869176]: Sent Feb 26 13:06:17 doveadm([10]demo@example.com)<358389><>: Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): auth input: USER 4211869176 [11]demo@example.com home=/var/vmail/example.com/demo uid=998 gid=998 quota_storage_size=300M Feb 26 13:06:17 doveadm([12]demo@example.com)<358389><>: Debug: auth-master: userdb lookup([13]demo@example.com): request [4211869176]: Got reply: USER [14]demo@example.com home=/var/vmail/example.com/demo uid=998 gid=998 quota_storage_size=300M Feb 26 13:06:17 doveadm([15]demo@example.com)<358389><>: Debug: auth-master: userdb lookup([16]demo@example.com): auth USER input: [17]demo@example.com home=/var/vmail/example.com/demo uid=998 gid=998 quota_storage_size=300M Feb 26 13:06:17 doveadm([18]demo@example.com)<358389><>: Debug: auth-master: userdb lookup([19]demo@example.com): Finished userdb lookup ([20]username=demo@example.com home=/var/vmail/example.com/demo uid=998 gid=998 quota_storage_size=300M) Feb 26 13:06:17 doveadm([21]demo@example.com)<358389><>: Debug: auth-master: userdb lookup([22]demo@example.com): request [4211869176]: Remove Feb 26 13:06:17 doveadm([23]demo@example.com)<358389><>: Debug: auth-master: userdb lookup([24]demo@example.com): request [4211869176]: Finished waiting for request Feb 26 13:06:17 doveadm([25]demo@example.com)<358389><>: Debug: auth-master: userdb lookup([26]demo@example.com): request [4211869176]: Destroy Feb 26 13:06:17 doveadm([27]demo@example.com)<358389><>: Debug: Added setting via userdb: quota_storage_size=300M Feb 26 13:06:17 doveadm([28]demo@example.com): Debug: Effective uid=998, gid=998, home=/var/vmail/example.com/demo Feb 26 13:06:17 doveadm([29]demo@example.com): Debug: acl: Shared mailbox listing disabled: dict { .. } named list filter is missing Feb 26 13:06:17 doveadm([30]demo@example.com): Debug: Namespace inbox: type=private, prefix=INBOX/, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes Feb 26 13:06:17 doveadm([31]demo@example.com): Debug: fs: root=/var/vmail/example.com/demo/sdbox, index=, indexpvt=, control=, inbox=, alt= Feb 26 13:06:17 doveadm([32]demo@example.com): Debug: acl: initializing backend vfile Feb 26 13:06:17 doveadm([33]demo@example.com): Debug: acl: acl username = [34]demo@example.com Feb 26 13:06:17 doveadm([35]demo@example.com): Debug: acl: owner = yes Feb 26 13:06:17 doveadm([36]demo@example.com): Debug: acl: ignore = no Feb 26 13:06:17 doveadm([37]demo@example.com): Debug: acl: vfile: Deprecated Global ACL file: /etc/dovecot/dovecot-acl Feb 26 13:06:17 doveadm([38]demo@example.com): Debug: Namespace : type=private, prefix=, sep=, inbox=no, hidden=yes, list=no, subscriptions=no Feb 26 13:06:17 doveadm([39]demo@example.com): Debug: none: root=/var/vmail/example.com/demo/sdbox, index=, indexpvt=, control=, inbox=, alt= Feb 26 13:06:17 doveadm([40]demo@example.com): Debug: quota-count: quota_over_status check: quota_over_mask unset - skipping Feb 26 13:06:17 doveadm([41]demo@example.com): Debug: Mailbox INBOX: Mailbox opened Feb 26 13:06:17 doveadm([42]demo@example.com): Debug: User session is finished Feb 26 13:06:17 doveadm([43]demo@example.com): Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): auth input: DONE 2802712577 fail Feb 26 13:06:17 doveadm([44]demo@example.com): Debug: auth-master: userdb list: request [2802712577]: Got reply: DONE fail Feb 26 13:06:17 doveadm([45]demo@example.com): Error: auth-master: userdb list: User listing returned failure Feb 26 13:06:17 doveadm([46]demo@example.com): Debug: auth-master: userdb list: request [2802712577]: Remove Feb 26 13:06:17 doveadm([47]demo@example.com): Debug: auth-master: userdb list: request [2802712577]: Destroy Feb 26 13:06:17 doveadm([48]demo@example.com): Debug: auth-master: userdb list: Listing users failed Feb 26 13:06:17 doveadm([49]demo@example.com): Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): Disconnected: Connection closed (fd=9) Feb 26 13:06:17 doveadm([50]demo@example.com): Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): Disconnected from auth service Feb 26 13:06:17 doveadm: Error: cmd quota get: Failed to iterate through some users Feb 26 13:06:17 doveadm: Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): Disconnected: Connection closed (fd=10) Feb 26 13:06:17 doveadm: Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): Disconnected from auth service
Hi
I could be wrong but is this debug output complete? It looks like request 2802712577 is failing, but the previous logging relating to [51]demo@example.com seems to be from request 4211869176.
Also, though I don't believe either it is related to the issue:
What is the reason for returning quota_storage_size in the iterate_query? It shouldn't matter if you return extra fields, but why do it?
Your iterate query can potentially return a subset of the users compared to your user query due to differing where conditions.
John
References
Visible links
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:username=demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
- mailto:demo@example.com
On 26 Feb 2026, at 18:45, John Fawcett via dovecot <dovecot@dovecot.org> wrote:
On 26/02/2026 15:42, Philip Iezzi via dovecot wrote:
Some more insights about this issue... It happens every time (on different users) on quota lookup for all users:
$ doveadm quota get -A > /dev/null; echo $? doveadm([1]demo@example.com): Error: auth-master: userdb list: User listing returned failure doveadm: Error: cmd quota get: Failed to iterate through some users 75
Full debug output see [1].
... $ doveadm -Dv quota get -A 2>&1 | tee /tmp/debug.log (...) Feb 26 13:06:17 doveadm([2]demo@example.com)<358389><>: Debug: auth-master: userdb lookup([3]demo@example.com): Started userdb lookup Feb 26 13:06:17 doveadm([4]demo@example.com)<358389><>: Debug: auth-master: request [4211869176]: Created Feb 26 13:06:17 doveadm([5]demo@example.com)<358389><>: Debug: auth-master: userdb lookup([6]demo@example.com): request [4211869176]: Waiting for request to complete Feb 26 13:06:17 doveadm([7]demo@example.com)<358389><>: Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): Sending requests Feb 26 13:06:17 doveadm([8]demo@example.com)<358389><>: Debug: auth-master: userdb lookup([9]demo@example.com): request [4211869176]: Sent Feb 26 13:06:17 doveadm([10]demo@example.com)<358389><>: Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): auth input: USER 4211869176 [11]demo@example.com home=/var/vmail/example.com/demo uid=998 gid=998 quota_storage_size=300M Feb 26 13:06:17 doveadm([12]demo@example.com)<358389><>: Debug: auth-master: userdb lookup([13]demo@example.com): request [4211869176]: Got reply: USER [14]demo@example.com home=/var/vmail/example.com/demo uid=998 gid=998 quota_storage_size=300M Feb 26 13:06:17 doveadm([15]demo@example.com)<358389><>: Debug: auth-master: userdb lookup([16]demo@example.com): auth USER input: [17]demo@example.com home=/var/vmail/example.com/demo uid=998 gid=998 quota_storage_size=300M Feb 26 13:06:17 doveadm([18]demo@example.com)<358389><>: Debug: auth-master: userdb lookup([19]demo@example.com): Finished userdb lookup ([20]username=demo@example.com home=/var/vmail/example.com/demo uid=998 gid=998 quota_storage_size=300M) Feb 26 13:06:17 doveadm([21]demo@example.com)<358389><>: Debug: auth-master: userdb lookup([22]demo@example.com): request [4211869176]: Remove Feb 26 13:06:17 doveadm([23]demo@example.com)<358389><>: Debug: auth-master: userdb lookup([24]demo@example.com): request [4211869176]: Finished waiting for request Feb 26 13:06:17 doveadm([25]demo@example.com)<358389><>: Debug: auth-master: userdb lookup([26]demo@example.com): request [4211869176]: Destroy Feb 26 13:06:17 doveadm([27]demo@example.com)<358389><>: Debug: Added setting via userdb: quota_storage_size=300M Feb 26 13:06:17 doveadm([28]demo@example.com): Debug: Effective uid=998, gid=998, home=/var/vmail/example.com/demo Feb 26 13:06:17 doveadm([29]demo@example.com): Debug: acl: Shared mailbox listing disabled: dict { .. } named list filter is missing Feb 26 13:06:17 doveadm([30]demo@example.com): Debug: Namespace inbox: type=private, prefix=INBOX/, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes Feb 26 13:06:17 doveadm([31]demo@example.com): Debug: fs: root=/var/vmail/example.com/demo/sdbox, index=, indexpvt=, control=, inbox=, alt= Feb 26 13:06:17 doveadm([32]demo@example.com): Debug: acl: initializing backend vfile Feb 26 13:06:17 doveadm([33]demo@example.com): Debug: acl: acl username = [34]demo@example.com Feb 26 13:06:17 doveadm([35]demo@example.com): Debug: acl: owner = yes Feb 26 13:06:17 doveadm([36]demo@example.com): Debug: acl: ignore = no Feb 26 13:06:17 doveadm([37]demo@example.com): Debug: acl: vfile: Deprecated Global ACL file: /etc/dovecot/dovecot-acl Feb 26 13:06:17 doveadm([38]demo@example.com): Debug: Namespace : type=private, prefix=, sep=, inbox=no, hidden=yes, list=no, subscriptions=no Feb 26 13:06:17 doveadm([39]demo@example.com): Debug: none: root=/var/vmail/example.com/demo/sdbox, index=, indexpvt=, control=, inbox=, alt= Feb 26 13:06:17 doveadm([40]demo@example.com): Debug: quota-count: quota_over_status check: quota_over_mask unset - skipping Feb 26 13:06:17 doveadm([41]demo@example.com): Debug: Mailbox INBOX: Mailbox opened Feb 26 13:06:17 doveadm([42]demo@example.com): Debug: User session is finished Feb 26 13:06:17 doveadm([43]demo@example.com): Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): auth input: DONE 2802712577 fail Feb 26 13:06:17 doveadm([44]demo@example.com): Debug: auth-master: userdb list: request [2802712577]: Got reply: DONE fail Feb 26 13:06:17 doveadm([45]demo@example.com): Error: auth-master: userdb list: User listing returned failure Feb 26 13:06:17 doveadm([46]demo@example.com): Debug: auth-master: userdb list: request [2802712577]: Remove Feb 26 13:06:17 doveadm([47]demo@example.com): Debug: auth-master: userdb list: request [2802712577]: Destroy Feb 26 13:06:17 doveadm([48]demo@example.com): Debug: auth-master: userdb list: Listing users failed Feb 26 13:06:17 doveadm([49]demo@example.com): Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): Disconnected: Connection closed (fd=9) Feb 26 13:06:17 doveadm([50]demo@example.com): Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): Disconnected from auth service Feb 26 13:06:17 doveadm: Error: cmd quota get: Failed to iterate through some users Feb 26 13:06:17 doveadm: Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): Disconnected: Connection closed (fd=10) Feb 26 13:06:17 doveadm: Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): Disconnected from auth service
Hi
I could be wrong but is this debug output complete? It looks like request 2802712577 is failing, but the previous logging relating to [51]demo@example.com seems to be from request 4211869176.
Also, though I don't believe either it is related to the issue:
What is the reason for returning quota_storage_size in the iterate_query? It shouldn't matter if you return extra fields, but why do it?
Your iterate query can potentially return a subset of the users compared to your user query due to differing where conditions.
John
Hi John
Thanks for looking into this. Yes, those were the last lines of doveadm -Dv quota get -A debug output, all related to the same user - I have simply redacted the user with demo@example.com, but did not change anything else / did not remove any lines. And yes, the problem always occurs on another user and is definitely not related to a specific user.
I don't really understand the logic here - so maybe it's just some kind of chained request that triggers the next one (2802712577) via userdb list.
I also don't quite remember why I added quota_storage_size to the iterate_query. Somehow I struggled to get quotas working during Dovecot 2.3 => 2.4 config migration. I have removed it now and all still seems to be working with:
iterate_query = SELECT username AS user FROM mailaccounts WHERE active = 1
Thanks for this input. But this doesn't make a difference. I always get the same Error:
$ doveadm -f json quota get -A >/dev/null; echo $? doveadm(demo@example.com): Error: auth-master: userdb list: User listing returned failure doveadm: Error: cmd quota get: Failed to iterate through some users 75
correlating with exactly 1 such line in mail.log:
dovecot: auth: Error: auth-worker: Aborted LIST request for *: Shutting down
Here's the thing: This only happens on my mail server with 2200 users, not on the other with "only" 1200 users, both running the exact same config/versions.
I have now invented this nice funky workaround to completely do without depending on the iterate_query. I don't even need to grab a subset of the users / do any sharding, just pipe all users into -F:
$ mysql --quick -srN maildb -e 'SELECT username FROM mailaccounts WHERE active=1' | doveadm -f json quota get -F -
It always works, always gives me the full data (checked with wc -c, while -A always broke in the middle and never returned the quotas for all users), and it never provokes a crashed auth-worker, above Error no longer showing up.
But I'd like to emphasize, that on Dovecot 2.3, I was running doveadm quota get -A for years, without ever running into this issue. It ran as a cache warmup job from my application and pulled all quota usages from Dovecot server, every 4 minutes! I know this is probably not recommended and I could easily reduce it to run only every 15mins or even less frequently. But no matter how often I run it on that server with 2200 users, it always runs fast enough, but always breaks since upgrading to Dovecot 2.4
Cheers, Philip
On 26/02/2026 21:20, Philip Iezzi via dovecot wrote:
... Hi John
Thanks for looking into this. Yes, those were the last lines of
doveadm -Dv quota get -Adebug output, all related to the same user - I have simply redacted the user with demo@example.com, but did not change anything else / did not remove any lines. And yes, the problem always occurs on another user and is definitely not related to a specific user. I don't really understand the logic here - so maybe it's just some kind of chained request that triggers the next one (2802712577) viauserdb list.I also don't quite remember why I added
quota_storage_sizeto the iterate_query. Somehow I struggled to get quotas working during Dovecot 2.3 => 2.4 config migration. I have removed it now and all still seems to be working with:iterate_query = SELECT username AS user FROM mailaccounts WHERE active = 1
Thanks for this input. But this doesn't make a difference. I always get the same Error:
$ doveadm -f json quota get -A >/dev/null; echo $? doveadm(demo@example.com): Error: auth-master: userdb list: User listing returned failure doveadm: Error: cmd quota get: Failed to iterate through some users 75
correlating with exactly 1 such line in mail.log:
dovecot: auth: Error: auth-worker: Aborted LIST request for *: Shutting down
Here's the thing: This only happens on my mail server with 2200 users, not on the other with "only" 1200 users, both running the exact same config/versions. I have now invented this nice funky workaround to completely do without depending on the iterate_query. I don't even need to grab a subset of the users / do any sharding, just pipe all users into
-F:$ mysql --quick -srN maildb -e 'SELECT username FROM mailaccounts WHERE active=1' | doveadm -f json quota get -F -
It always works, always gives me the full data (checked with
wc -c, while-Aalways broke in the middle and never returned the quotas for all users), and it never provokes a crashed auth-worker, above Error no longer showing up.But I'd like to emphasize, that on Dovecot 2.3, I was running
doveadm quota get -Afor years, without ever running into this issue. It ran as a cache warmup job from my application and pulled all quota usages from Dovecot server, every 4 minutes! I know this is probably not recommended and I could easily reduce it to run only every 15mins or even less frequently. But no matter how often I run it on that server with 2200 users, it always runs fast enough, but always breaks since upgrading to Dovecot 2.4Cheers, Philip
Hi Philip, it really does look like you're hitting some kind of limit. Potentially it could be a mysql timeout but seems unlikely you would have changed mysql_read_timeout from its default 30s. I wonder if it's an issue only for quota or whether it happens on other iteration queries, for example doveadm user '*'. John
On 27 Feb 2026, at 01:24, John Fawcett via dovecot <dovecot@dovecot.org> wrote:
On 26/02/2026 21:20, Philip Iezzi via dovecot wrote:
... Hi John
Thanks for looking into this. Yes, those were the last lines of
doveadm -Dv quota get -Adebug output, all related to the same user - I have simply redacted the user with demo@example.com, but did not change anything else / did not remove any lines. And yes, the problem always occurs on another user and is definitely not related to a specific user. I don't really understand the logic here - so maybe it's just some kind of chained request that triggers the next one (2802712577) viauserdb list.I also don't quite remember why I added
quota_storage_sizeto the iterate_query. Somehow I struggled to get quotas working during Dovecot 2.3 => 2.4 config migration. I have removed it now and all still seems to be working with:iterate_query = SELECT username AS user FROM mailaccounts WHERE active = 1
Thanks for this input. But this doesn't make a difference. I always get the same Error:
$ doveadm -f json quota get -A >/dev/null; echo $? doveadm(demo@example.com): Error: auth-master: userdb list: User listing returned failure doveadm: Error: cmd quota get: Failed to iterate through some users 75
correlating with exactly 1 such line in mail.log:
dovecot: auth: Error: auth-worker: Aborted LIST request for *: Shutting down
Here's the thing: This only happens on my mail server with 2200 users, not on the other with "only" 1200 users, both running the exact same config/versions. I have now invented this nice funky workaround to completely do without depending on the iterate_query. I don't even need to grab a subset of the users / do any sharding, just pipe all users into
-F:$ mysql --quick -srN maildb -e 'SELECT username FROM mailaccounts WHERE active=1' | doveadm -f json quota get -F -
It always works, always gives me the full data (checked with
wc -c, while-Aalways broke in the middle and never returned the quotas for all users), and it never provokes a crashed auth-worker, above Error no longer showing up.But I'd like to emphasize, that on Dovecot 2.3, I was running
doveadm quota get -Afor years, without ever running into this issue. It ran as a cache warmup job from my application and pulled all quota usages from Dovecot server, every 4 minutes! I know this is probably not recommended and I could easily reduce it to run only every 15mins or even less frequently. But no matter how often I run it on that server with 2200 users, it always runs fast enough, but always breaks since upgrading to Dovecot 2.4Cheers, Philip
Hi Philip, it really does look like you're hitting some kind of limit. Potentially it could be a mysql timeout but seems unlikely you would have changed mysql_read_timeout from its default 30s. I wonder if it's an issue only for quota or whether it happens on other iteration queries, for example doveadm user '*'. John
Yes, I am definitely running into some kind of limit. It's not on MySQL level - tuned it a lot for high performance and connection hammering, and never found any error in MySQL log.
Above workaround is now running for a full day, pulling all user quotas every 4mins, without running into this issue again.
As doveadm quota get -A was always running fine on Dovecot 2.3 and I can now (under Dovecot 2.4.2) retrieve all quotas by piping all users into -F, it should definitely be possible to fix/optimize the -A iterate_query for mailservers with larger user base.
@Aki did you guys experience this issue and can I provide you any more information to help you get this fixed? I can definitely live with my workaround, just hope others are not struggling with it and wasting so much time as I did.
Best regards, Philip
On 27 Feb 2026, at 01:24, John Fawcett via dovecot <dovecot@dovecot.org>
wrote:
On 26/02/2026 21:20, Philip Iezzi via dovecot wrote:
...
Hi John
Thanks for looking into this. Yes, those were the last lines of
`doveadm -Dv quota get -A` debug output, all related to the same user
- I have simply redacted the user with demo@example.com, but did not
change anything else / did not remove any lines. And yes, the problem
always occurs on another user and is definitely not related to a
specific user.
I don't really understand the logic here - so maybe it's just some
kind of chained request that triggers the next one (2802712577) via
`userdb list`.
I also don't quite remember why I added `quota_storage_size` to the
iterate_query. Somehow I struggled to get quotas working during
Dovecot 2.3 => 2.4 config migration. I have removed it now and all
still seems to be working with:
iterate_query = SELECT username AS user FROM mailaccounts WHERE active
= 1
Thanks for this input. But this doesn't make a difference. I always
get the same Error:
$ doveadm -f json quota get -A >/dev/null; echo $?
doveadm(demo@example.com): Error: auth-master: userdb list: User
listing returned failure
doveadm: Error: cmd quota get: Failed to iterate through some users
75
correlating with exactly 1 such line in mail.log:
dovecot: auth: Error: auth-worker: Aborted LIST request for *:
Shutting down
Here's the thing: This only happens on my mail server with 2200 users,
not on the other with "only" 1200 users, both running the exact same
config/versions.
I have now invented this nice funky workaround to completely do
without depending on the iterate_query. I don't even need to grab a
subset of the users / do any sharding, just pipe all users into `-F`:
$ mysql --quick -srN maildb -e 'SELECT username FROM mailaccounts
WHERE active=1' | doveadm -f json quota get -F -
It always works, always gives me the full data (checked with `wc -c`,
while `-A` always broke in the middle and never returned the quotas
for all users), and it never provokes a crashed auth-worker, above
Error no longer showing up.
But I'd like to emphasize, that on Dovecot 2.3, I was running `doveadm
quota get -A` for years, without ever running into this issue. It ran
as a cache warmup job from my application and pulled all quota usages
from Dovecot server, every 4 minutes! I know this is probably not
recommended and I could easily reduce it to run only every 15mins or
even less frequently. But no matter how often I run it on that server
with 2200 users, it always runs fast enough, but always breaks since
upgrading to Dovecot 2.4
Cheers,
Philip
Hi Philip, it really does look like you're hitting some kind of limit.
Potentially it could be a mysql timeout but seems unlikely you would
have changed mysql_read_timeout from its default 30s.
I wonder if it's an issue only for quota or whether it happens on other
iteration queries, for example doveadm user '*'.
John
Yes, I am definitely running into some kind of limit. It's not on MySQL
level - tuned it a lot for high performance and connection hammering, and
never found any error in MySQL log.
Above workaround is now running for a full day, pulling all user quotas
every 4mins, without running into this issue again.
As doveadm quota get -A was always running fine on Dovecot 2.3 and I can
now (under Dovecot 2.4.2) retrieve all quotas by piping all users into
-F, it should definitely be possible to fix/optimize the -A
iterate_query for mailservers with larger user base.
@Aki did you guys experience this issue and can I provide you any more
information to help you get this fixed?
I can definitely live with my workaround, just hope others are not
struggling with it and wasting so much time as I did.
Best regards,
Philip
On 27/02/2026 17:56, Philip Iezzi via dovecot wrote:
...
Yes, I am definitely running into some kind of limit. It's not on MySQL level - tuned it a lot for high performance and connection hammering, and never found any error in MySQL log. Above workaround is now running for a full day, pulling all user quotas every 4mins, without running into this issue again. As `doveadm quota get -A` was always running fine on Dovecot 2.3 and I can now (under Dovecot 2.4.2) retrieve all quotas by piping all users into `-F`, it should definitely be possible to fix/optimize the `-A` iterate_query for mailservers with larger user base. @Aki did you guys experience this issue and can I provide you any more information to help you get this fixed? I can definitely live with my workaround, just hope others are not struggling with it and wasting so much time as I did. Best regards, Philip
Hi Philip
this is reproducible by filling the usertable with around 2000 users and running doveadm quota get -A. For me it also fails on
doveadm quota recalc -A
/usr/bin/doveadm index -A '*'
As you already saw the process never completes and gives an error on a different user each time though roughly somewhere between 2-3 seconds into the execution for quota recalc.
quota get seems to present slightly more variable behaviour in terms of how long before it fails but that could be influenced by terminal output time.
The log entry I saw was "auth: Error: auth-worker: Aborted LIST request for *: Shutting down"
doveadm user '*' does complete ok, but that only does the iterate query, whereas the other commands do the iterate and then lookup the single users too.
I don't see any configurable timeout limits that seem relevant to this issue, so it looks like a bug when running commands needing to iterate through 1000s of users with the iterate_query and then doing lookups for the single users user the userdb query.
John
Hi
I can confirm the issue of these iteration operation not completing for 1000's of users is linked to:
MAX_OUTBUF_SIZE defined in auth/auth-master-connection.c
I made it 1024000 and managed to complete quota recalc for 10000 users. But that's not going to be a solution, since whatever it is set to provides a hard coded limit to how many users can be iterated for quota get, quota recalc and index operations. Surprisingly it is not a completely deterministic limit, since sometimes more users get processed than other times. I was looking for a time based limit before I thought of looking at the buffer size.
The strange thing is that the current value hasn't changed in more than 20 years if ever, so that does not explain of itself why this issue arose now in 2.4.
#defineMAX_OUTBUF_SIZE(1024*50)
The likely explanation is that exceeding the buffer size is now being handled differently to before.
John
On 28/02/2026 15:35, John Fawcett via dovecot wrote:
On 27/02/2026 17:56, Philip Iezzi via dovecot wrote:
...
Yes, I am definitely running into some kind of limit. It's not on MySQL level - tuned it a lot for high performance and connection hammering, and never found any error in MySQL log. Above workaround is now running for a full day, pulling all user quotas every 4mins, without running into this issue again. As
doveadm quota get -Awas always running fine on Dovecot 2.3 and I can now (under Dovecot 2.4.2) retrieve all quotas by piping all users into-F, it should definitely be possible to fix/optimize the-Aiterate_query for mailservers with larger user base. @Aki did you guys experience this issue and can I provide you any more information to help you get this fixed? I can definitely live with my workaround, just hope others are not struggling with it and wasting so much time as I did. Best regards, PhilipHi Philip
this is reproducible by filling the usertable with around 2000 users and running doveadm quota get -A. For me it also fails on
doveadm quota recalc -A
/usr/bin/doveadm index -A '*'
As you already saw the process never completes and gives an error on a different user each time though roughly somewhere between 2-3 seconds into the execution for quota recalc.
quota get seems to present slightly more variable behaviour in terms of how long before it fails but that could be influenced by terminal output time.
The log entry I saw was "auth: Error: auth-worker: Aborted LIST request for *: Shutting down"
doveadm user '*' does complete ok, but that only does the iterate query, whereas the other commands do the iterate and then lookup the single users too.
I don't see any configurable timeout limits that seem relevant to this issue, so it looks like a bug when running commands needing to iterate through 1000s of users with the iterate_query and then doing lookups for the single users user the userdb query.
John
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
Hi
I can confirm the issue of these iteration operation not completing for 1000's of users is linked to:
MAX_OUTBUF_SIZE defined in auth/auth-master-connection.c
I made it 1024000 and managed to complete quota recalc for 10000 users. But that's not going to be a solution, since whatever it is set to provides a hard coded limit to how many users can be iterated for quota get, quota recalc and index operations. Surprisingly it is not a completely deterministic limit, since sometimes more users get processed than other times. I was looking for a time based limit before I thought of looking at the buffer size.
The strange thing is that the current value hasn't changed in more than 20 years if ever, so that does not explain of itself why this issue arose now in 2.4.
#define MAX_OUTBUF_SIZE (1024*50)
The likely explanation is that exceeding the buffer size is now being handled differently to before.
John
On 28/02/2026 15:35, John Fawcett via dovecot wrote:
On 27/02/2026 17:56, Philip Iezzi via dovecot wrote:
...
Yes, I am definitely running into some kind of limit. It's not on
MySQL
level - tuned it a lot for high performance and connection
hammering, and
never found any error in MySQL log.
Above workaround is now running for a full day, pulling all user
quotas
every 4mins, without running into this issue again.
As `doveadm quota get -A` was always running fine on Dovecot 2.3
and I can
now (under Dovecot 2.4.2) retrieve all quotas by piping all users
into
`-F`, it should definitely be possible to fix/optimize the `-A`
iterate_query for mailservers with larger user base.
@Aki did you guys experience this issue and can I provide you any
more
information to help you get this fixed?
I can definitely live with my workaround, just hope others are not
struggling with it and wasting so much time as I did.
Best regards,
Philip
Hi Philip
this is reproducible by filling the usertable with around 2000 users and
running doveadm quota get -A. For me it also fails on
doveadm quota recalc -A
/usr/bin/doveadm index -A '*'
As you already saw the process never completes and gives an error on a
different user each time though roughly somewhere between 2-3 seconds
into the execution for quota recalc.
quota get seems to present slightly more variable behaviour in terms of
how long before it fails but that could be influenced by terminal output
time.
The log entry I saw was "auth: Error: auth-worker: Aborted LIST request
for *: Shutting down"
doveadm user '*' does complete ok, but that only does the iterate query,
whereas the other commands do the iterate and then lookup the single
users too.
I don't see any configurable timeout limits that seem relevant to this
issue, so it looks like a bug when running commands needing to iterate
through 1000s of users with the iterate_query and then doing lookups for
the single users user the userdb query.
John
_______________________________________________
dovecot mailing list -- [1]dovecot@dovecot.org
To unsubscribe send an email to [2]dovecot-leave@dovecot.org
References
Visible links
- mailto:dovecot@dovecot.org
- mailto:dovecot-leave@dovecot.org
Hi Aki, Dovecot developers so this is a bug in dovecot 2.4 which prevents iteration over 1000s of users with doveadm and -A flag. The workaround as identified by Philip is to do iteration outside dovecot and then feed that into doveadm with -u flag. Maybe it's not such a big issue for Dovecot CE where user bases may be low enough not to run into it. Could be of more of interest for large users bases and Dovecot Pro. The effect is that doveadm quota recalc and index does not terminate correctly. It was working on dovecot 2.3 and the code hasn't really undergone that much change. My assumption is that in auth/auth-master-connection.c, in the master_input_list_callback() function where there is a call o_stream_flush(ctx->conn->output), that this call no longer appears to reduce the o_stream_get_buffer_used_size under the MAX_OUTBUF_SIZE limit and so get into a case with the code did not handle correctly. This patch seems to fix it in Dovecot 2.4 by always calling userdb_blocking_iter_next(ctx->iter) regardless of used buffer size. I admit though that this may not be the right fix, if the difference in the way o_stream_flush works is the root cause. --- auth/auth-master-connection.c.orig 2026-03-08 18:06:28.025317016 +0000 +++ auth/auth-master-connection.c 2026-03-08 18:07:08.590631101 +0000 @@ -570,10 +570,9 @@ master_input_list_finish(ctx); return; } - if (o_stream_get_buffer_used_size(ctx->conn->conn.output) < MAX_OUTBUF_SIZE) - userdb_blocking_iter_next(ctx->iter); - else + if (o_stream_get_buffer_used_size(ctx->conn->conn.output) >= MAX_OUTBUF_SIZE) o_stream_uncork(ctx->conn->conn.output); + userdb_blocking_iter_next(ctx->iter); } static int master_input_list(struct auth_master_connection *conn, John On 01/03/2026 16:36, John Fawcett via dovecot wrote:
Hi
I can confirm the issue of these iteration operation not completing for 1000's of users is linked to:
MAX_OUTBUF_SIZE defined in auth/auth-master-connection.c
I made it 1024000 and managed to complete quota recalc for 10000 users. But that's not going to be a solution, since whatever it is set to provides a hard coded limit to how many users can be iterated for quota get, quota recalc and index operations. Surprisingly it is not a completely deterministic limit, since sometimes more users get processed than other times. I was looking for a time based limit before I thought of looking at the buffer size.
The strange thing is that the current value hasn't changed in more than 20 years if ever, so that does not explain of itself why this issue arose now in 2.4.
#define MAX_OUTBUF_SIZE (1024*50)
The likely explanation is that exceeding the buffer size is now being handled differently to before.
John
participants (2)
-
John Fawcett
-
Philip Iezzi