[Dovecot] Auth Issues - Urgent - Help!
After a few hours of running, I get tons of the following errors in my logs:
dovecot: Oct 08 07:41:50 Error: auth(default):
ldap(user@domain,x.x.x.x): Request queue is full
I removed the username and IP, obviously.
Any idea how to stop this?
I have about 5 Thousand users using horde that login ever 1-5 minutes
to refresh their page. I assume it is a setting, but I am confused as
to why it doesn't happen almost right away. It seems to take some
time to build up.
Please help! This is taking my webmail system down hourly.
On Oct 8, 2008, at 8:01 AM, David Cunningham wrote:
After a few hours of running, I get tons of the following errors in
my logs:dovecot: Oct 08 07:41:50 Error: auth(default):
ldap(user@domain,x.x.x.x): Request queue is fullI removed the username and IP, obviously.
Any idea how to stop this?
I have about 5 Thousand users using horde that login ever 1-5
minutes to refresh their page. I assume it is a setting, but I am
confused as to why it doesn't happen almost right away. It seems to
take some time to build up.Please help! This is taking my webmail system down hourly.
dovecot -n?
Hunch is login_max_processes_count is too low. http://wiki.dovecot.org/LoginProcess
hth, JL
Here is my dovecot -n:
# 1.1.3: /etc/dovecot.conf log_path: /var/log/dovecot info_log_path: /var/log/dovecot-info login_dir: /var/run/dovecot/login login_executable(default): /usr/libexec/dovecot/imap-login login_executable(imap): /usr/libexec/dovecot/imap-login login_executable(pop3): /usr/libexec/dovecot/pop3-login login_greeting: MECnet Mail System, Authorized Use Only, Please Log In. login_process_per_connection: no login_process_size: 1024 login_max_processes_count: 1024 login_max_connections: 1024 max_mail_processes: 50000 verbose_proctitle: yes first_valid_uid: 50 mail_uid: 93 mail_gid: 12 mail_location: maildir:/var/spool/maildirs/%d/%n/Maildir mail_executable(default): /usr/libexec/dovecot/imap mail_executable(imap): /usr/libexec/dovecot/imap mail_executable(pop3): /usr/libexec/dovecot/pop3 mail_process_size: 1024 mail_plugins(default): quota imap_quota mail_plugins(imap): quota imap_quota mail_plugins(pop3): mail_plugin_dir(default): /usr/lib64/dovecot/imap mail_plugin_dir(imap): /usr/lib64/dovecot/imap mail_plugin_dir(pop3): /usr/lib64/dovecot/pop3 namespace: type: private prefix: INBOX. inbox: yes list: yes subscriptions: yes auth default: verbose: yes passdb: driver: ldap args: /etc/dovecot-ldap.conf plugin: quota: maildir
My buddy and I found
#define DB_LDAP_MAX_QUEUE_SIZE 1024
in the db-ldap.h file in the source.
We believe we hitting this threshold for some reason. So, we are
looking to increase this to 8192. However, when trying to build the
RPM source from atrpms, I get this:
# rpmbuild -ba dovecot.spec error: line 1: Unknown tag: %bcond_without inotify
Any help?
Dave
Quoting Jurvis LaSalle <lasalle@idi.harvard.edu>:
On Oct 8, 2008, at 8:01 AM, David Cunningham wrote:
After a few hours of running, I get tons of the following errors in
my logs:dovecot: Oct 08 07:41:50 Error: auth(default):
ldap(user@domain,x.x.x.x): Request queue is fullI removed the username and IP, obviously.
Any idea how to stop this?
I have about 5 Thousand users using horde that login ever 1-5
minutes to refresh their page. I assume it is a setting, but I am
confused as to why it doesn't happen almost right away. It seems
to take some time to build up.Please help! This is taking my webmail system down hourly.
dovecot -n?
Hunch is login_max_processes_count is too low. http://wiki.dovecot.org/LoginProcess
hth, JL
Unfortantely, it just happened again! <sigh>
I am going to implement my increased queue change and see what happens.
Dave
Quoting Jurvis LaSalle <lasalle@idi.harvard.edu>:
On Oct 8, 2008, at 8:01 AM, David Cunningham wrote:
After a few hours of running, I get tons of the following errors in
my logs:dovecot: Oct 08 07:41:50 Error: auth(default):
ldap(user@domain,x.x.x.x): Request queue is fullI removed the username and IP, obviously.
Any idea how to stop this?
I have about 5 Thousand users using horde that login ever 1-5
minutes to refresh their page. I assume it is a setting, but I am
confused as to why it doesn't happen almost right away. It seems
to take some time to build up.Please help! This is taking my webmail system down hourly.
dovecot -n?
Hunch is login_max_processes_count is too low. http://wiki.dovecot.org/LoginProcess
hth, JL
David Cunningham wrote:
I have about 5 Thousand users using horde that login ever 1-5 minutes to refresh their page. I assume it is a setting, but I am confused as to
This may not related to the real reason of your problem, but I recommend up-imapproxy (http://www.imapproxy.org/) for such setups, which caches connections (and therefore logins). You could try to enable dovecot's auth_cache instead.
I agree. In fact, I may have found a DNS issue that may have been
causing login sessions to hang and thus reach max too quickly. The
last few hours have been stable. So, I am keeping my fingers crossed.
I have also recompiled dovecot and changed the setting in db-ldap.h
that reads:
#define DB_LDAP_MAX_QUEUE_SIZE 1024
to
#define DB_LDAP_MAX_QUEUE_SIZE 8192
I have not implemented this change yet as the system has been stable
for a few hours. If it happens again, I will implement this change
which will hopefully allow my queued logins to reach 8192, rather than
1024, before crashing.
I have considered IMAPproxy. This will likely make things better, but
will also not fix the underlying issue.
Plus, I am a bit worried about leaving about 5000 imap sessions open
on my server at one time. It's a bit scary. Any opinions on this?
Dave
Quoting Jakob Hirsch <jh@plonk.de>:
David Cunningham wrote:
I have about 5 Thousand users using horde that login ever 1-5
minutes to refresh their page. I assume it is a setting, but I am
confused as toThis may not related to the real reason of your problem, but I recommend up-imapproxy (http://www.imapproxy.org/) for such setups, which caches connections (and therefore logins). You could try to enable dovecot's auth_cache instead.
David Cunningham schrieb:
I have about 5 Thousand users using horde that login ever 1-5 minutes to refresh their page.
Imapproxy can be configured to cache connections only for a defined amount of time, eg. close connections not used for more than a few minutes. Setting is cache_expiration_time in imapproxy.conf.
Remember: Horde creates a new imap connection for every page click, imapproxy heavily reduces this login/logout noise.
If your 5k users are all using webmail *simultaneously* I don't know how the proxy will perform.
On Wed, 2008-10-08 at 10:48 -0400, David Cunningham wrote:
I agree. In fact, I may have found a DNS issue that may have been
causing login sessions to hang and thus reach max too quickly. The
last few hours have been stable. So, I am keeping my fingers crossed.I have also recompiled dovecot and changed the setting in db-ldap.h
that reads:#define DB_LDAP_MAX_QUEUE_SIZE 1024
to
#define DB_LDAP_MAX_QUEUE_SIZE 8192
If you're getting more than 1024 requests queued, something's wrong or you have hundreds or logins per second. Which one is it? (5000 users logging in once per minute is still only 83/sec)
The queue keeps increasing if the LDAP server isn't replying to old requests. So have you looked at the LDAP server side if it's running too slow?
Anyway two things you could do here:
Enable auth cache with large enough size so Dovecot doesn't consult LDAP server nearly as much.
Increase the number of auth processes (auth { .. count=5 }), so that you'll use more connections and hopefully the LDAP server likes that better than one connection sending lots of requests. Or maybe not.
Thank you, I will try the caching.
Dave
Quoting Timo Sirainen <tss@iki.fi>:
On Wed, 2008-10-08 at 10:48 -0400, David Cunningham wrote:
I agree. In fact, I may have found a DNS issue that may have been causing login sessions to hang and thus reach max too quickly. The last few hours have been stable. So, I am keeping my fingers crossed.
I have also recompiled dovecot and changed the setting in db-ldap.h that reads:
#define DB_LDAP_MAX_QUEUE_SIZE 1024
to
#define DB_LDAP_MAX_QUEUE_SIZE 8192
If you're getting more than 1024 requests queued, something's wrong or you have hundreds or logins per second. Which one is it? (5000 users logging in once per minute is still only 83/sec)
The queue keeps increasing if the LDAP server isn't replying to old requests. So have you looked at the LDAP server side if it's running too slow?
Anyway two things you could do here:
Enable auth cache with large enough size so Dovecot doesn't consult LDAP server nearly as much.
Increase the number of auth processes (auth { .. count=5 }), so that you'll use more connections and hopefully the LDAP server likes that better than one connection sending lots of requests. Or maybe not.
Simply changing auth_cache_size to a non-zero number enables caching, correct?
How big is too big?
Where does it cache it?
Here is what I set:
auth_cache_size = 1048576
I was hoping for 1GB worth of cache.
I have 16GB of memory on the system, so memory is not an issue if it
stores it in memory as opposed to disk.
Dave
Quoting David Cunningham <davec@mecnet.net>:
Thank you, I will try the caching.
Dave
Quoting Timo Sirainen <tss@iki.fi>:
On Wed, 2008-10-08 at 10:48 -0400, David Cunningham wrote:
I agree. In fact, I may have found a DNS issue that may have been causing login sessions to hang and thus reach max too quickly. The last few hours have been stable. So, I am keeping my fingers crossed.
I have also recompiled dovecot and changed the setting in db-ldap.h that reads:
#define DB_LDAP_MAX_QUEUE_SIZE 1024
to
#define DB_LDAP_MAX_QUEUE_SIZE 8192
If you're getting more than 1024 requests queued, something's wrong or you have hundreds or logins per second. Which one is it? (5000 users logging in once per minute is still only 83/sec)
The queue keeps increasing if the LDAP server isn't replying to old requests. So have you looked at the LDAP server side if it's running too slow?
Anyway two things you could do here:
Enable auth cache with large enough size so Dovecot doesn't consult LDAP server nearly as much.
Increase the number of auth processes (auth { .. count=5 }), so that you'll use more connections and hopefully the LDAP server likes that better than one connection sending lots of requests. Or maybe not.
On Wed, 2008-10-08 at 16:13 -0400, David Cunningham wrote:
Simply changing auth_cache_size to a non-zero number enables caching, correct?
How big is too big?
If it uses up too much of your memory. Although you'll probably also need to change auth_process_size then, because by default it kills the process if it grows larger than 256 MB. You could also disable it by setting it to 0.
Where does it cache it?
Here is what I set:
auth_cache_size = 1048576
I was hoping for 1GB worth of cache.
I think you could do with a lot less :) I think a single user takes maybe 50-100 bytes normally. Even with 200 bytes your 5000 users would fit into 1 MB. Of course negative hits are also cached, so the rest of the memory would go to them. I'd decrease it to 10 MB or so :)
Also you might want to increase auth_cache_ttl from the default hour.
On Wed, 2008-10-08 at 08:01 -0400, David Cunningham wrote:
After a few hours of running, I get tons of the following errors in my logs:
dovecot: Oct 08 07:41:50 Error: auth(default):
ldap(user@domain,x.x.x.x): Request queue is full
BTW. I improved this error message slightly to also tell how many seconds old data is in the queue. http://hg.dovecot.org/dovecot-1.1/rev/0329dc4df5ed
I guess you're using auth binds? If you weren't, I think it wouldn't be possible to fill the queue.
Well, most of my issues are gone with adding auth cache. However, I
am having an issue. Sometimes, even though cache incorrect passwords
is disabled, new passwords do not work. It would seem that once a
user logs in with one password successfully the cache does not
automatically retry if the user tries a different passwords. I would
think that the auth cache should check to see if the password changed
on the ldap server if something other than the cached password is
entered.
Is this something wrong with my configuraiton, or the auth code itself?
Thanks,
Dave
Quoting Timo Sirainen <tss@iki.fi>:
On Wed, 2008-10-08 at 08:01 -0400, David Cunningham wrote:
After a few hours of running, I get tons of the following errors in my logs:
dovecot: Oct 08 07:41:50 Error: auth(default): ldap(user@domain,x.x.x.x): Request queue is full
BTW. I improved this error message slightly to also tell how many seconds old data is in the queue. http://hg.dovecot.org/dovecot-1.1/rev/0329dc4df5ed
I guess you're using auth binds? If you weren't, I think it wouldn't be possible to fill the queue.
On 11/19/2008 10:17 PM, David Cunningham wrote:
Well, most of my issues are gone with adding auth cache. However, I am having an issue. Sometimes, even though cache incorrect passwords is disabled, new passwords do not work. It would seem that once a user logs in with one password successfully the cache does not automatically retry if the user tries a different passwords. I would think that the auth cache should check to see if the password changed on the ldap server if something other than the cached password is entered.
Is this something wrong with my configuraiton, or the auth code itself?
Maybe it is the mail client doing the caching... have you tested this on the command line?
--
Best regards,
Charles
Yes, i telnet to port 143 and enter everything manually.
Dave
Quoting Charles Marcus <CMarcus@Media-Brokers.com>:
On 11/19/2008 10:17 PM, David Cunningham wrote:
Well, most of my issues are gone with adding auth cache. However, I am having an issue. Sometimes, even though cache incorrect passwords is disabled, new passwords do not work. It would seem that once a user logs in with one password successfully the cache does not automatically retry if the user tries a different passwords. I would think that the auth cache should check to see if the password changed on the ldap server if something other than the cached password is entered.
Is this something wrong with my configuraiton, or the auth code itself?
Maybe it is the mail client doing the caching... have you tested this on the command line?
--
Best regards,
Charles
No one else with opinions on this?
Dave
Quoting David Cunningham <davec@mecnet.net>:
Yes, i telnet to port 143 and enter everything manually.
Dave
Quoting Charles Marcus <CMarcus@Media-Brokers.com>:
On 11/19/2008 10:17 PM, David Cunningham wrote:
Well, most of my issues are gone with adding auth cache. However, I am having an issue. Sometimes, even though cache incorrect passwords is disabled, new passwords do not work. It would seem that once a user logs in with one password successfully the cache does not automatically retry if the user tries a different passwords. I would think that the auth cache should check to see if the password changed on the ldap server if something other than the cached password is entered.
Is this something wrong with my configuraiton, or the auth code itself?
Maybe it is the mail client doing the caching... have you tested this on the command line?
--
Best regards,
Charles
On Wed, 2008-11-19 at 22:17 -0500, David Cunningham wrote:
Well, most of my issues are gone with adding auth cache. However, I
am having an issue. Sometimes, even though cache incorrect passwords
is disabled,
Do you mean auth_cache_negative_ttl=0 by this? It only affects "user not found" caching.
new passwords do not work. It would seem that once a
user logs in with one password successfully the cache does not
automatically retry if the user tries a different passwords. I would
think that the auth cache should check to see if the password changed
on the ldap server if something other than the cached password is
entered.Is this something wrong with my configuraiton, or the auth code itself?
The way it should work is that:
- User logs in with password X which succeeds.
- Password is changed to Y.
- User logs in with password Y. Dovecot sees that X != Y, but it sees that the previous auth succeeded, so it'll do an auth lookup, sees that the password was changed and caches it.
But this can also happen:
- User logs in with password X which succeeds.
- Password is changed to Y.
- User logs in with password X, which succeeds.
Or:
- User logs in with password X which succeeds.
- User logs in with password Y. Dovecot sees that X != Y, but it sees that the previous auth succeeded, so it'll do an auth lookup and sees that the password wasn't changed.
- Password is changed to Y.
- User logs in with password Y. Dovecot sees that X != Y, but it sees that the previous auth failed, so it doesn't bother doing another lookup.
Can you consistently make Dovecot behave differently as described above?
On Fri, 2008-11-21 at 21:38 +0200, Timo Sirainen wrote:
On Wed, 2008-11-19 at 22:17 -0500, David Cunningham wrote:
Well, most of my issues are gone with adding auth cache. However, I
am having an issue. Sometimes, even though cache incorrect passwords
is disabled,Do you mean auth_cache_negative_ttl=0 by this? It only affects "user not found" caching.
new passwords do not work. It would seem that once a
user logs in with one password successfully the cache does not
automatically retry if the user tries a different passwords. I would
think that the auth cache should check to see if the password changed
on the ldap server if something other than the cached password is
entered.Is this something wrong with my configuraiton, or the auth code itself?
The way it should work is that:
I also added these to Wiki with slightly better wording: http://wiki.dovecot.org/Authentication/Caching
I think the last thing you say is exactly what is happening to me. I
think the user is updating the password, but a slight delay in my LDAP
replication is causing them to try the new password before it is
actually the new password.
Yes, I was refering to auth_cache_negative_ttl=0. I didn't realize
that was user not found only. Is there any way to force the cache to
check the password for anything that was not previously cached as
being the correct password?
Dave
Quoting Timo Sirainen <tss@iki.fi>:
On Wed, 2008-11-19 at 22:17 -0500, David Cunningham wrote:
Well, most of my issues are gone with adding auth cache. However, I am having an issue. Sometimes, even though cache incorrect passwords is disabled,
Do you mean auth_cache_negative_ttl=0 by this? It only affects "user not found" caching.
new passwords do not work. It would seem that once a user logs in with one password successfully the cache does not automatically retry if the user tries a different passwords. I would think that the auth cache should check to see if the password changed on the ldap server if something other than the cached password is entered.
Is this something wrong with my configuraiton, or the auth code itself?
The way it should work is that:
- User logs in with password X which succeeds.
- Password is changed to Y.
- User logs in with password Y. Dovecot sees that X != Y, but it sees that the previous auth succeeded, so it'll do an auth lookup, sees that the password was changed and caches it.
But this can also happen:
- User logs in with password X which succeeds.
- Password is changed to Y.
- User logs in with password X, which succeeds.
Or:
- User logs in with password X which succeeds.
- User logs in with password Y. Dovecot sees that X != Y, but it sees that the previous auth succeeded, so it'll do an auth lookup and sees that the password wasn't changed.
- Password is changed to Y.
- User logs in with password Y. Dovecot sees that X != Y, but it sees that the previous auth failed, so it doesn't bother doing another lookup.
Can you consistently make Dovecot behave differently as described above?
On Fri, 2008-11-21 at 14:50 -0500, David Cunningham wrote:
Is there any way to force the cache to
check the password for anything that was not previously cached as
being the correct password?
Nope. Hmm. Perhaps there should be a different TTL for that. I don't really like adding new settings though.
For now you can at least do it by modifying sources: src/auth/passdb-cache.c:
if (ret == 0 && node->last_success) {Change it to:
if (ret == 0) {Wowa, that's easy enough. I will do that the next time that I upgrade
in a few weeks.
Dave
Quoting Timo Sirainen <tss@iki.fi>:
On Fri, 2008-11-21 at 14:50 -0500, David Cunningham wrote:
Is there any way to force the cache to check the password for anything that was not previously cached as being the correct password?
Nope. Hmm. Perhaps there should be a different TTL for that. I don't really like adding new settings though.
For now you can at least do it by modifying sources: src/auth/passdb-cache.c:
if (ret == 0 && node->last_success) {Change it to:
if (ret == 0) {
participants (6)
-
Charles Marcus
-
David Cunningham
-
Jakob Hirsch
-
Jurvis LaSalle
-
Oli Schacher
-
Timo Sirainen