[Dovecot] Re: Dovecot memory use and settings
We're continuing to have dovecot auth-worker failures once or more
times per day. The symptoms are these in the log:
pop1 dovecot: auth-worker(default): Out of memory (Needed 8164 bytes)
or
auth-worker(default): sql(lewis1,12.43.59.134): Password query
failed: MySQL client ran out of memory
Timo suggested that we need to up the auth_process_size = 256
However, we've tried increasing this to 512 and then 768. The
comment on the entry says
# Set max. process size in megabytes.
These changes don't seem to make any difference. On both of our
servers (RH ES4, 1MB RAM), we never seem to have more than 300MB of
RAM used.
However, the auth-worker process continues to fail occasionally. It
does respawn itself after a couple of minutes, but in that time all
authentication for dovecot fails.
Am I missing something else in the config?
-- Roger J. Weeks Systems & Network Administrator Mendocino Community Network
On Jan 11, 2006, at 12:51 PM, Timo Sirainen wrote:
On Wed, 2006-01-11 at 12:40 -0800, Roger Weeks wrote:
pop1 dovecot: auth-worker(default): Out of memory (Needed 8164 bytes)
or
auth-worker(default): sql(lewis1,12.43.59.134): Password query failed: MySQL client ran out of memory
These are different from the system running out of memory. It's about authentication process size hitting its configured limit. Default is:
# Set max. process size in megabytes. #auth_process_size = 256
So it appears that it's trying to grow above 256MB at which point it gets killed. This could mean that something is leaking memory, but I don't remember other people having this problem, so it might be
somehow specific to your configuration.In any case those auth-workers are restarted automatically whenever
they die so it shouldn't be that big of a problem?There are also ways to debug this if you want to..
Okay, it appears to me that increasing auth_process_size above 256
just doesn't make any difference to this problem. It seems that
someone else has reported this same issue with 1.0alpha5:
http://permalink.gmane.org/gmane.mail.imap.dovecot/9423
Does anyone else think this might be a mysql issue? Or is this
specifically a dovecot problem?
Even on really busy servers, I never see more than three auth-worker
processes on either of our servers. The dovecot.conf has this:
auth_worker_max_count = 150
But I don't see anywhere I can tell it a minimum number of auth
worker processes to start when dovecot starts. Is there an
auth_worker_min_count or something similar?
-- Roger J. Weeks Systems & Network Administrator Mendocino Community Network
On Jan 12, 2006, at 3:43 PM, Roger Weeks wrote:
We're continuing to have dovecot auth-worker failures once or more
times per day. The symptoms are these in the log:pop1 dovecot: auth-worker(default): Out of memory (Needed 8164 bytes)
or
auth-worker(default): sql(lewis1,12.43.59.134): Password query
failed: MySQL client ran out of memoryTimo suggested that we need to up the auth_process_size = 256
However, we've tried increasing this to 512 and then 768. The
comment on the entry says # Set max. process size in megabytes.These changes don't seem to make any difference. On both of our
servers (RH ES4, 1MB RAM), we never seem to have more than 300MB of
RAM used.However, the auth-worker process continues to fail occasionally.
It does respawn itself after a couple of minutes, but in that time
all authentication for dovecot fails.Am I missing something else in the config?
-- Roger J. Weeks Systems & Network Administrator Mendocino Community Network
On Jan 11, 2006, at 12:51 PM, Timo Sirainen wrote:
On Wed, 2006-01-11 at 12:40 -0800, Roger Weeks wrote:
pop1 dovecot: auth-worker(default): Out of memory (Needed 8164
bytes)or
auth-worker(default): sql(lewis1,12.43.59.134): Password query failed: MySQL client ran out of memory
These are different from the system running out of memory. It's about authentication process size hitting its configured limit. Default is:
# Set max. process size in megabytes. #auth_process_size = 256
So it appears that it's trying to grow above 256MB at which point it gets killed. This could mean that something is leaking memory, but I don't remember other people having this problem, so it might be
somehow specific to your configuration.In any case those auth-workers are restarted automatically
whenever they die so it shouldn't be that big of a problem?There are also ways to debug this if you want to..
Roger Weeks wrote:
Okay, it appears to me that increasing auth_process_size above 256 just doesn't make any difference to this problem. It seems that someone else has reported this same issue with 1.0alpha5:
http://permalink.gmane.org/gmane.mail.imap.dovecot/9423
Does anyone else think this might be a mysql issue? Or is this specifically a dovecot problem?
I think it is a Dovecot problem if your system is 64bit.. I have seen something similar.
Even on really busy servers, I never see more than three auth-worker processes on either of our servers. The dovecot.conf has this: auth_worker_max_count = 150
But I don't see anywhere I can tell it a minimum number of auth worker processes to start when dovecot starts. Is there an auth_worker_min_count or something similar?
There is no such option, more workers are spawned automatically if all current workers are busy.
Tomi Hakala
-- Roger J. Weeks Systems & Network Administrator Mendocino Community Network
On Jan 12, 2006, at 3:43 PM, Roger Weeks wrote:
We're continuing to have dovecot auth-worker failures once or more times per day. The symptoms are these in the log:
pop1 dovecot: auth-worker(default): Out of memory (Needed 8164 bytes)
or
auth-worker(default): sql(lewis1,12.43.59.134): Password query failed: MySQL client ran out of memory
Timo suggested that we need to up the auth_process_size = 256
However, we've tried increasing this to 512 and then 768. The comment on the entry says # Set max. process size in megabytes.
These changes don't seem to make any difference. On both of our servers (RH ES4, 1MB RAM), we never seem to have more than 300MB of RAM used.
However, the auth-worker process continues to fail occasionally. It does respawn itself after a couple of minutes, but in that time all authentication for dovecot fails.
Am I missing something else in the config?
-- Roger J. Weeks Systems & Network Administrator Mendocino Community Network
On Jan 11, 2006, at 12:51 PM, Timo Sirainen wrote:
On Wed, 2006-01-11 at 12:40 -0800, Roger Weeks wrote:
pop1 dovecot: auth-worker(default): Out of memory (Needed 8164 bytes)
or
auth-worker(default): sql(lewis1,12.43.59.134): Password query failed: MySQL client ran out of memory
These are different from the system running out of memory. It's about authentication process size hitting its configured limit. Default is:
# Set max. process size in megabytes. #auth_process_size = 256
So it appears that it's trying to grow above 256MB at which point it gets killed. This could mean that something is leaking memory, but I don't remember other people having this problem, so it might be somehow specific to your configuration.
In any case those auth-workers are restarted automatically whenever they die so it shouldn't be that big of a problem?
There are also ways to debug this if you want to..
On Jan 14, 2006, at 12:40 AM, Tomi Hakala wrote:
Even on really busy servers, I never see more than three auth- worker processes on either of our servers. The dovecot.conf has
this: auth_worker_max_count = 150 But I don't see anywhere I can tell it a minimum number of auth
worker processes to start when dovecot starts. Is there an
auth_worker_min_count or something similar?There is no such option, more workers are spawned automatically if all current workers are busy.
Actually, it seems to depend on your type of authentication. Digging
through dovecot.conf yesterday, I finally found this:
# SQL database userdb sql { args = /usr/local/etc/dovecot-sql.conf }
# Number of authentication processes to create #count = 1
I have changed this to count=20 on my servers and since then I have
not experienced any out of memory issues with auth-workers or the
MySQL client.
Leaving this at the default of 1 for authentication against MySQL
does not work for servers with lots of clients. 20 may be too high -
but I'd rather err on the side of not having 2 minutes of refused
authentication while auth-workers restart.
I might suggest that this default be changed to something other than
- With the default settings in the conf file, using MySQL
authentication, no more than a maximum of 3 auth-worker processes are
ever created.
Roger Weeks
On Sat, 2006-01-14 at 10:43 -0800, Roger Weeks wrote:
# SQL database userdb sql { args = /usr/local/etc/dovecot-sql.conf }
# Number of authentication processes to create #count = 1
I have changed this to count=20 on my servers and since then I have
not experienced any out of memory issues with auth-workers or the
MySQL client.
Did you check if the dovecot-auth processes keep constantly growing? Maybe they're still growing, but since you have 20 of them, they're just not reaching the maximum limit as fast as with fewer processes?
On Jan 14, 2006, at 10:59 AM, Timo Sirainen wrote:
On Sat, 2006-01-14 at 10:43 -0800, Roger Weeks wrote:
# SQL database userdb sql { args = /usr/local/etc/dovecot-sql.conf }
# Number of authentication processes to create #count = 1
I have changed this to count=20 on my servers and since then I have not experienced any out of memory issues with auth-workers or the MySQL client.
Did you check if the dovecot-auth processes keep constantly growing? Maybe they're still growing, but since you have 20 of them, they're
just not reaching the maximum limit as fast as with fewer processes?
They don't seem to be constantly growing, not as far as I can tell by
watching the memory usage of a few individual processes.
Also, I notice that with this setting, my machines are actually using
less RAM overall than they were with the default count=1.
Roger Weeks
participants (3)
-
Roger Weeks
-
Timo Sirainen
-
Tomi Hakala