On 26 Feb 2026, at 18:45, John Fawcett via dovecot <dovecot@dovecot.org> wrote:

On 26/02/2026 15:42, Philip Iezzi via dovecot wrote:

Some more insights about this issue... It happens every time (on different users) on quota lookup for all users:

$ doveadm quota get -A > /dev/null; echo $? doveadm([1]demo@example.com): Error: auth-master: userdb list: User listing returned failure doveadm: Error: cmd quota get: Failed to iterate through some users 75

Full debug output see [1].

... $ doveadm -Dv quota get -A 2>&1 | tee /tmp/debug.log (...) Feb 26 13:06:17 doveadm([2]demo@example.com)<358389><>: Debug: auth-master: userdb lookup([3]demo@example.com): Started userdb lookup Feb 26 13:06:17 doveadm([4]demo@example.com)<358389><>: Debug: auth-master: request [4211869176]: Created Feb 26 13:06:17 doveadm([5]demo@example.com)<358389><>: Debug: auth-master: userdb lookup([6]demo@example.com): request [4211869176]: Waiting for request to complete Feb 26 13:06:17 doveadm([7]demo@example.com)<358389><>: Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): Sending requests Feb 26 13:06:17 doveadm([8]demo@example.com)<358389><>: Debug: auth-master: userdb lookup([9]demo@example.com): request [4211869176]: Sent Feb 26 13:06:17 doveadm([10]demo@example.com)<358389><>: Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): auth input: USER 4211869176 [11]demo@example.com home=/var/vmail/example.com/demo uid=998 gid=998 quota_storage_size=300M Feb 26 13:06:17 doveadm([12]demo@example.com)<358389><>: Debug: auth-master: userdb lookup([13]demo@example.com): request [4211869176]: Got reply: USER [14]demo@example.com home=/var/vmail/example.com/demo uid=998 gid=998 quota_storage_size=300M Feb 26 13:06:17 doveadm([15]demo@example.com)<358389><>: Debug: auth-master: userdb lookup([16]demo@example.com): auth USER input: [17]demo@example.com home=/var/vmail/example.com/demo uid=998 gid=998 quota_storage_size=300M Feb 26 13:06:17 doveadm([18]demo@example.com)<358389><>: Debug: auth-master: userdb lookup([19]demo@example.com): Finished userdb lookup ([20]username=demo@example.com home=/var/vmail/example.com/demo uid=998 gid=998 quota_storage_size=300M) Feb 26 13:06:17 doveadm([21]demo@example.com)<358389><>: Debug: auth-master: userdb lookup([22]demo@example.com): request [4211869176]: Remove Feb 26 13:06:17 doveadm([23]demo@example.com)<358389><>: Debug: auth-master: userdb lookup([24]demo@example.com): request [4211869176]: Finished waiting for request Feb 26 13:06:17 doveadm([25]demo@example.com)<358389><>: Debug: auth-master: userdb lookup([26]demo@example.com): request [4211869176]: Destroy Feb 26 13:06:17 doveadm([27]demo@example.com)<358389><>: Debug: Added setting via userdb: quota_storage_size=300M Feb 26 13:06:17 doveadm([28]demo@example.com): Debug: Effective uid=998, gid=998, home=/var/vmail/example.com/demo Feb 26 13:06:17 doveadm([29]demo@example.com): Debug: acl: Shared mailbox listing disabled: dict { .. } named list filter is missing Feb 26 13:06:17 doveadm([30]demo@example.com): Debug: Namespace inbox: type=private, prefix=INBOX/, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes Feb 26 13:06:17 doveadm([31]demo@example.com): Debug: fs: root=/var/vmail/example.com/demo/sdbox, index=, indexpvt=, control=, inbox=, alt= Feb 26 13:06:17 doveadm([32]demo@example.com): Debug: acl: initializing backend vfile Feb 26 13:06:17 doveadm([33]demo@example.com): Debug: acl: acl username = [34]demo@example.com Feb 26 13:06:17 doveadm([35]demo@example.com): Debug: acl: owner = yes Feb 26 13:06:17 doveadm([36]demo@example.com): Debug: acl: ignore = no Feb 26 13:06:17 doveadm([37]demo@example.com): Debug: acl: vfile: Deprecated Global ACL file: /etc/dovecot/dovecot-acl Feb 26 13:06:17 doveadm([38]demo@example.com): Debug: Namespace : type=private, prefix=, sep=, inbox=no, hidden=yes, list=no, subscriptions=no Feb 26 13:06:17 doveadm([39]demo@example.com): Debug: none: root=/var/vmail/example.com/demo/sdbox, index=, indexpvt=, control=, inbox=, alt= Feb 26 13:06:17 doveadm([40]demo@example.com): Debug: quota-count: quota_over_status check: quota_over_mask unset - skipping Feb 26 13:06:17 doveadm([41]demo@example.com): Debug: Mailbox INBOX: Mailbox opened Feb 26 13:06:17 doveadm([42]demo@example.com): Debug: User session is finished Feb 26 13:06:17 doveadm([43]demo@example.com): Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): auth input: DONE 2802712577 fail Feb 26 13:06:17 doveadm([44]demo@example.com): Debug: auth-master: userdb list: request [2802712577]: Got reply: DONE fail Feb 26 13:06:17 doveadm([45]demo@example.com): Error: auth-master: userdb list: User listing returned failure Feb 26 13:06:17 doveadm([46]demo@example.com): Debug: auth-master: userdb list: request [2802712577]: Remove Feb 26 13:06:17 doveadm([47]demo@example.com): Debug: auth-master: userdb list: request [2802712577]: Destroy Feb 26 13:06:17 doveadm([48]demo@example.com): Debug: auth-master: userdb list: Listing users failed Feb 26 13:06:17 doveadm([49]demo@example.com): Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): Disconnected: Connection closed (fd=9) Feb 26 13:06:17 doveadm([50]demo@example.com): Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): Disconnected from auth service Feb 26 13:06:17 doveadm: Error: cmd quota get: Failed to iterate through some users Feb 26 13:06:17 doveadm: Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): Disconnected: Connection closed (fd=10) Feb 26 13:06:17 doveadm: Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=353974,uid=0): Disconnected from auth service

Hi

I could be wrong but is this debug output complete? It looks like request 2802712577 is failing, but the previous logging relating to [51]demo@example.com seems to be from request 4211869176.

Also, though I don't believe either it is related to the issue:

What is the reason for returning quota_storage_size in the iterate_query? It shouldn't matter if you return extra fields, but why do it?

Your iterate query can potentially return a subset of the users compared to your user query due to differing where conditions.

John

Hi John

Thanks for looking into this. Yes, those were the last lines of doveadm -Dv quota get -A debug output, all related to the same user - I have simply redacted the user with demo@example.com, but did not change anything else / did not remove any lines. And yes, the problem always occurs on another user and is definitely not related to a specific user. I don't really understand the logic here - so maybe it's just some kind of chained request that triggers the next one (2802712577) via userdb list.

I also don't quite remember why I added quota_storage_size to the iterate_query. Somehow I struggled to get quotas working during Dovecot 2.3 => 2.4 config migration. I have removed it now and all still seems to be working with:

iterate_query = SELECT username AS user FROM mailaccounts WHERE active = 1

Thanks for this input. But this doesn't make a difference. I always get the same Error:

$ doveadm -f json quota get -A >/dev/null; echo $? doveadm(demo@example.com): Error: auth-master: userdb list: User listing returned failure doveadm: Error: cmd quota get: Failed to iterate through some users 75

correlating with exactly 1 such line in mail.log:

dovecot: auth: Error: auth-worker: Aborted LIST request for *: Shutting down

Here's the thing: This only happens on my mail server with 2200 users, not on the other with "only" 1200 users, both running the exact same config/versions. I have now invented this nice funky workaround to completely do without depending on the iterate_query. I don't even need to grab a subset of the users / do any sharding, just pipe all users into -F:

$ mysql --quick -srN maildb -e 'SELECT username FROM mailaccounts WHERE active=1' | doveadm -f json quota get -F -

It always works, always gives me the full data (checked with wc -c, while -A always broke in the middle and never returned the quotas for all users), and it never provokes a crashed auth-worker, above Error no longer showing up.

But I'd like to emphasize, that on Dovecot 2.3, I was running doveadm quota get -A for years, without ever running into this issue. It ran as a cache warmup job from my application and pulled all quota usages from Dovecot server, every 4 minutes! I know this is probably not recommended and I could easily reduce it to run only every 15mins or even less frequently. But no matter how often I run it on that server with 2200 users, it always runs fast enough, but always breaks since upgrading to Dovecot 2.4

Cheers, Philip

On 27 Feb 2026, at 01:24, John Fawcett via dovecot <dovecot@dovecot.org> wrote:

On 26/02/2026 21:20, Philip Iezzi via dovecot wrote:

...

... Hi John

Thanks for looking into this. Yes, those were the last lines of doveadm -Dv quota get -A debug output, all related to the same user - I have simply redacted the user with demo@example.com, but did not change anything else / did not remove any lines. And yes, the problem always occurs on another user and is definitely not related to a specific user. I don't really understand the logic here - so maybe it's just some kind of chained request that triggers the next one (2802712577) via userdb list.

I also don't quite remember why I added quota_storage_size to the iterate_query. Somehow I struggled to get quotas working during Dovecot 2.3 => 2.4 config migration. I have removed it now and all still seems to be working with:

iterate_query = SELECT username AS user FROM mailaccounts WHERE active = 1

Thanks for this input. But this doesn't make a difference. I always get the same Error:

$ doveadm -f json quota get -A >/dev/null; echo $? doveadm(demo@example.com): Error: auth-master: userdb list: User listing returned failure doveadm: Error: cmd quota get: Failed to iterate through some users 75

correlating with exactly 1 such line in mail.log:

dovecot: auth: Error: auth-worker: Aborted LIST request for *: Shutting down

Here's the thing: This only happens on my mail server with 2200 users, not on the other with "only" 1200 users, both running the exact same config/versions. I have now invented this nice funky workaround to completely do without depending on the iterate_query. I don't even need to grab a subset of the users / do any sharding, just pipe all users into -F:

$ mysql --quick -srN maildb -e 'SELECT username FROM mailaccounts WHERE active=1' | doveadm -f json quota get -F -

It always works, always gives me the full data (checked with wc -c, while -A always broke in the middle and never returned the quotas for all users), and it never provokes a crashed auth-worker, above Error no longer showing up.

But I'd like to emphasize, that on Dovecot 2.3, I was running doveadm quota get -A for years, without ever running into this issue. It ran as a cache warmup job from my application and pulled all quota usages from Dovecot server, every 4 minutes! I know this is probably not recommended and I could easily reduce it to run only every 15mins or even less frequently. But no matter how often I run it on that server with 2200 users, it always runs fast enough, but always breaks since upgrading to Dovecot 2.4

Cheers, Philip

Hi Philip, it really does look like you're hitting some kind of limit. Potentially it could be a mysql timeout but seems unlikely you would have changed mysql_read_timeout from its default 30s. I wonder if it's an issue only for quota or whether it happens on other iteration queries, for example doveadm user '*'. John

Yes, I am definitely running into some kind of limit. It's not on MySQL level - tuned it a lot for high performance and connection hammering, and never found any error in MySQL log.

Above workaround is now running for a full day, pulling all user quotas every 4mins, without running into this issue again. As doveadm quota get -A was always running fine on Dovecot 2.3 and I can now (under Dovecot 2.4.2) retrieve all quotas by piping all users into -F, it should definitely be possible to fix/optimize the -A iterate_query for mailservers with larger user base.

@Aki did you guys experience this issue and can I provide you any more information to help you get this fixed? I can definitely live with my workaround, just hope others are not struggling with it and wasting so much time as I did.

Best regards, Philip

 On 27 Feb 2026, at 01:24, John Fawcett via dovecot <dovecot@dovecot.org>
 wrote:
 On 26/02/2026 21:20, Philip Iezzi via dovecot wrote:

   ...
   Hi John

   Thanks for looking into this. Yes, those were the last lines of
   `doveadm -Dv quota get -A` debug output, all related to the same user
   - I have simply redacted the user with demo@example.com, but did not
   change anything else / did not remove any lines. And yes, the problem
   always occurs on another user and is definitely not related to a
   specific user.
   I don't really understand the logic here - so maybe it's just some
   kind of chained request that triggers the next one (2802712577) via
   `userdb list`.

   I also don't quite remember why I added `quota_storage_size` to the
   iterate_query. Somehow I struggled to get quotas working during
   Dovecot 2.3 => 2.4 config migration. I have removed it now and all
   still seems to be working with:

   iterate_query = SELECT username AS user FROM mailaccounts WHERE active
   = 1

   Thanks for this input. But this doesn't make a difference. I always
   get the same Error:

   $ doveadm -f json quota get -A >/dev/null; echo $?
   doveadm(demo@example.com): Error: auth-master: userdb list: User
   listing returned failure
   doveadm: Error: cmd quota get: Failed to iterate through some users
   75

   correlating with exactly 1 such line in mail.log:

   dovecot: auth: Error: auth-worker: Aborted LIST request for *:
   Shutting down

   Here's the thing: This only happens on my mail server with 2200 users,
   not on the other with "only" 1200 users, both running the exact same
   config/versions.
   I have now invented this nice funky workaround to completely do
   without depending on the iterate_query. I don't even need to grab a
   subset of the users / do any sharding, just pipe all users into `-F`:

   $ mysql --quick -srN maildb -e 'SELECT username FROM mailaccounts
   WHERE active=1' | doveadm -f json quota get -F -

   It always works, always gives me the full data (checked with `wc -c`,
   while `-A` always broke in the middle and never returned the quotas
   for all users), and it never provokes a crashed auth-worker, above
   Error no longer showing up.

   But I'd like to emphasize, that on Dovecot 2.3, I was running `doveadm
   quota get -A` for years, without ever running into this issue. It ran
   as a cache warmup job from my application and pulled all quota usages
   from Dovecot server, every 4 minutes! I know this is probably not
   recommended and I could easily reduce it to run only every 15mins or
   even less frequently. But no matter how often I run it on that server
   with 2200 users, it always runs fast enough, but always breaks since
   upgrading to Dovecot 2.4

   Cheers,
   Philip

 Hi Philip, it really does look like you're hitting some kind of limit.
 Potentially it could be a mysql timeout but seems unlikely you would
 have changed mysql_read_timeout from its default 30s.
 I wonder if it's an issue only for quota or whether it happens on other
 iteration queries, for example doveadm user '*'.
 John

Yes, I am definitely running into some kind of limit. It's not on MySQL level - tuned it a lot for high performance and connection hammering, and never found any error in MySQL log. Above workaround is now running for a full day, pulling all user quotas every 4mins, without running into this issue again. As doveadm quota get -A was always running fine on Dovecot 2.3 and I can now (under Dovecot 2.4.2) retrieve all quotas by piping all users into -F, it should definitely be possible to fix/optimize the -A iterate_query for mailservers with larger user base. @Aki did you guys experience this issue and can I provide you any more information to help you get this fixed? I can definitely live with my workaround, just hope others are not struggling with it and wasting so much time as I did. Best regards, Philip

Hi

I can confirm the issue of these iteration operation not completing for 1000's of users is linked to:

MAX_OUTBUF_SIZE defined in auth/auth-master-connection.c

I made it 1024000 and managed to complete quota recalc for 10000 users. But that's not going to be a solution, since whatever it is set to provides a hard coded limit to how many users can be iterated for quota get, quota recalc and index operations. Surprisingly it is not a completely deterministic limit, since sometimes more users get processed than other times. I was looking for a time based limit before I thought of looking at the buffer size.

The strange thing is that the current value hasn't changed in more than 20 years if ever, so that does not explain of itself why this issue arose now in 2.4.

#defineMAX_OUTBUF_SIZE(1024*50)

The likely explanation is that exceeding the buffer size is now being handled differently to before.

John

On 28/02/2026 15:35, John Fawcett via dovecot wrote:

On 27/02/2026 17:56, Philip Iezzi via dovecot wrote:

...

  ...

    Yes, I am definitely running into some kind of limit. It's not on MySQL level - tuned it a lot for high performance and connection hammering, and never found any error in MySQL log. Above workaround is now running for a full day, pulling all user quotas every 4mins, without running into this issue again. As doveadm quota get -A was always running fine on Dovecot 2.3 and I can now (under Dovecot 2.4.2) retrieve all quotas by piping all users into -F, it should definitely be possible to fix/optimize the -A iterate_query for mailservers with larger user base. @Aki did you guys experience this issue and can I provide you any more information to help you get this fixed? I can definitely live with my workaround, just hope others are not struggling with it and wasting so much time as I did. Best regards, Philip

Hi Philip

this is reproducible by filling the usertable with around 2000 users and running doveadm quota get -A. For me it also fails on

     doveadm quota recalc -A

     /usr/bin/doveadm index -A '*'

As you already saw the process never completes and gives an error on a different user each time though roughly somewhere between 2-3 seconds into the execution for quota recalc.

quota get seems to present slightly more variable behaviour in terms of how long before it fails but that could be influenced by terminal output time.

The log entry I saw was "auth: Error: auth-worker: Aborted LIST request for *: Shutting down"

doveadm user '*' does complete ok, but that only does the iterate query, whereas the other commands do the iterate and then lookup the single users too.

I don't see any configurable timeout limits that seem relevant to this issue, so it looks like a bug when running commands needing to iterate through 1000s of users with the iterate_query and then doing lookups for the single users user the userdb query.

John

---

dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org

Hi

I can confirm the issue of these iteration operation not completing for 1000's of users is linked to:

MAX_OUTBUF_SIZE defined in auth/auth-master-connection.c

I made it 1024000 and managed to complete quota recalc for 10000 users. But that's not going to be a solution, since whatever it is set to provides a hard coded limit to how many users can be iterated for quota get, quota recalc and index operations. Surprisingly it is not a completely deterministic limit, since sometimes more users get processed than other times. I was looking for a time based limit before I thought of looking at the buffer size.

The strange thing is that the current value hasn't changed in more than 20 years if ever, so that does not explain of itself why this issue arose now in 2.4.

#define MAX_OUTBUF_SIZE (1024*50)

The likely explanation is that exceeding the buffer size is now being handled differently to before.

John

On 28/02/2026 15:35, John Fawcett via dovecot wrote:

 On 27/02/2026 17:56, Philip Iezzi via dovecot wrote:

     ...

       Yes, I am definitely running into some kind of limit. It's not on
   MySQL
       level - tuned it a lot for high performance and connection
   hammering, and
       never found any error in MySQL log.
       Above workaround is now running for a full day, pulling all user
   quotas
       every 4mins, without running into this issue again.
       As `doveadm quota get -A` was always running fine on Dovecot 2.3
   and I can
       now (under Dovecot 2.4.2) retrieve all quotas by piping all users
   into
       `-F`, it should definitely be possible to fix/optimize the `-A`
       iterate_query for mailservers with larger user base.
       @Aki did you guys experience this issue and can I provide you any
   more
       information to help you get this fixed?
       I can definitely live with my workaround, just hope others are not
       struggling with it and wasting so much time as I did.
       Best regards,
       Philip

 Hi Philip

 this is reproducible by filling the usertable with around 2000 users and
 running doveadm quota get -A. For me it also fails on

      doveadm quota recalc -A

      /usr/bin/doveadm index -A '*'

 As you already saw the process never completes and gives an error on a
 different user each time though roughly somewhere between 2-3 seconds
 into the execution for quota recalc.

 quota get seems to present slightly more variable behaviour in terms of
 how long before it fails but that could be influenced by terminal output
 time.

 The log entry I saw was "auth: Error: auth-worker: Aborted LIST request
 for *: Shutting down"

 doveadm user '*' does complete ok, but that only does the iterate query,
 whereas the other commands do the iterate and then lookup the single
 users too.

 I don't see any configurable timeout limits that seem relevant to this
 issue, so it looks like a bug when running commands needing to iterate
 through 1000s of users with the iterate_query and then doing lookups for
 the single users user the userdb query.

 John

 _______________________________________________
 dovecot mailing list -- [1]dovecot@dovecot.org
 To unsubscribe send an email to [2]dovecot-leave@dovecot.org

References

Visible links

1. mailto:dovecot@dovecot.org
2. mailto:dovecot-leave@dovecot.org