[Dovecot] Dovecot (IMAP) Hangs sometimes
Hi there!
I'm using Dovecot 1.0.7 in RHEL 5.4 64bits as a IMAP/POP3 server.
Dovecot authenticates users against an AD server via kerberos (pam configuration) and clients connect to it via IMAPs and POP3s with SSL. Webmail is in localhost and connects locally also with SSL.
All is fine and people use the email server normally and with no problems.
But, I have noticed that dovecot hangs sometimes. Usually twice a day somewhere between 09:40 and 10:00 a.m and in the afternoon at about 3:00 p.m.
The problem starts when webmail takes long to authenticate and show folders and stuff. After a while it times out and doesn't connect. Then the clients (e.g. outlook, thunderbird) start taking long to connect and then they timeout. Important: Only IMAP connections timeout! POP3 continues ok. This latency keeps on going until dovecot finally hangs (after about 10 to 15 minutes). The problem is solved when I restart/reload dovecot.
I'm not sure as to what's going on! I check the log, but it has been hard to pinpoint an exact event or to detect when the problem begins.
Any ideas?
BTW: what's the real difference between reload and restart?
Thanks a lot! Mario
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Thu, 7 Jan 2010, Mario Gomide wrote:
I'm using Dovecot 1.0.7 in RHEL 5.4 64bits as a IMAP/POP3 server.
and stuff. After a while it times out and doesn't connect. Then the clients (e.g. outlook, thunderbird) start taking long to connect and then they timeout. Important: Only IMAP connections timeout! POP3 continues ok. This latency keeps on going until dovecot finally hangs (after about 10 to 15 minutes). The problem is solved when I restart/reload dovecot.
I had similiar problems once. The problem went away after I increased the maximum number of open file descriptors (ulimit -n).
Maybe, you should upgrade, v1.0 is pretty old.
Regards,
Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux)
iQEVAwUBS0XjFb+Vh58GPL/cAQJyVQgApvvpmRWvk+tTepmMrKhMjUxlqBOKZFDF rBmzJ7EvwTCnQziSMlH/ckvg9Z/ccNofBgaosTLXGik9P6jZFzTAcDu6ztNvPYPt 2ldnvDJhUoQCUL+8i594pOy5vs1edm5Tg36d/XHuexL+WiG3a3NuOBu2AgkuVfoa HLk4FNKfKQagquHcpG1z3+R9QJC4rYarg+lU7rNAebF0hEkCyNsZn6DsvRITP4BQ qlauMRS6jQ35o/AYMXuRfJrnektFUFUbT1oyRKwLRF9O3FcSt5fZZ0p6Mv4JLtXZ u8g40iYSF3nfo7J0SFlJjEVbqRjZMtlpNrhPLroa26ztk0GR9YNwtQ== =cyU4 -----END PGP SIGNATURE-----
Interesting.... But I used to have dovecot work with the same number of users (up to 9,000) and with the default number of fole descriptors with no problems... I can give it a try, though.
I can't upgrade dovecot until Red Hat releases a new version.
On Thu, Jan 7, 2010 at 11:35 AM, Steffen Kaiser < skdovecot@smail.inf.fh-brs.de> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Thu, 7 Jan 2010, Mario Gomide wrote:
I'm using Dovecot 1.0.7 in RHEL 5.4 64bits as a IMAP/POP3 server.
and stuff. After a while it times out and doesn't connect. Then the
clients (e.g. outlook, thunderbird) start taking long to connect and then they timeout. Important: Only IMAP connections timeout! POP3 continues ok. This latency keeps on going until dovecot finally hangs (after about 10 to 15 minutes). The problem is solved when I restart/reload dovecot.
I had similiar problems once. The problem went away after I increased the maximum number of open file descriptors (ulimit -n).
Maybe, you should upgrade, v1.0 is pretty old.
Regards,
- -- Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux)
iQEVAwUBS0XjFb+Vh58GPL/cAQJyVQgApvvpmRWvk+tTepmMrKhMjUxlqBOKZFDF rBmzJ7EvwTCnQziSMlH/ckvg9Z/ccNofBgaosTLXGik9P6jZFzTAcDu6ztNvPYPt 2ldnvDJhUoQCUL+8i594pOy5vs1edm5Tg36d/XHuexL+WiG3a3NuOBu2AgkuVfoa HLk4FNKfKQagquHcpG1z3+R9QJC4rYarg+lU7rNAebF0hEkCyNsZn6DsvRITP4BQ qlauMRS6jQ35o/AYMXuRfJrnektFUFUbT1oyRKwLRF9O3FcSt5fZZ0p6Mv4JLtXZ u8g40iYSF3nfo7J0SFlJjEVbqRjZMtlpNrhPLroa26ztk0GR9YNwtQ== =cyU4 -----END PGP SIGNATURE-----
On Thu, 7 Jan 2010 11:08:54 -0200, Mario Gomide <mariogomide@gmail.com> wrote:
Webmail is in localhost and connects locally also with SSL.
The problem starts when webmail takes long to authenticate and show folders and stuff. After a while it times out and doesn't connect. Any ideas?
It may be unrelated to your current problem, but, why in the world are you having a local process connect to another local process via SSL? That's akin to encrypting the traffic between the left and right hemispheres of your brain for fear someone might be listening in to the unencrypted traffic occurring within you skull. Silly isn't it?
One thing your configuration will certainly do is suck CPU horsepower, eat up memory unnecessarily, and make things slower than they should/could be.
-- Stan
Hello,
is there a way to enable some plugin (in particular I'm thinking of lazy_expunge) on a per-user basis?
I use Dovecot 1.2.9 with Maildir and virtual users in a MySQL database.
Thanks for your support, Igor.
On 8.1.2010, at 9.58, mailing@securitylabs.it wrote:
is there a way to enable some plugin (in particular I'm thinking of lazy_expunge) on a per-user basis?
I use Dovecot 1.2.9 with Maildir and virtual users in a MySQL database.
Yes. Return mail_plugins setting from your user_query. For example:
user_query = select .., mail_plugins from users where ..
BTW. Don't post to list by replying to an existing mail. It preserves In-Reply-To: header and messes up threading.
On 08/01/2010 09:15, Timo Sirainen wrote:
On 8.1.2010, at 9.58, mailing@securitylabs.it wrote:
is there a way to enable some plugin (in particular I'm thinking of lazy_expunge) on a per-user basis?
I use Dovecot 1.2.9 with Maildir and virtual users in a MySQL database.
Yes. Return mail_plugins setting from your user_query. For example:
user_query = select .., mail_plugins from users where ..
Thanks. One clarification. If I have:
protocol imap { mail_plugins = quota imap_quota mail_log
in my conf file, I have to add only lazy_expunge in the MySQL field and it "adds" to the plugins specified in the conf? Or I have to add every plugin I'd like to use in the MySQL field?
Igor
On 8.1.2010, at 12.14, mailing@securitylabs.it wrote:
user_query = select .., mail_plugins from users where ..
Thanks. One clarification. If I have:
protocol imap { mail_plugins = quota imap_quota mail_log
in my conf file, I have to add only lazy_expunge in the MySQL field and it "adds" to the plugins specified in the conf? Or I have to add every plugin I'd like to use in the MySQL field?
Returning mail_plugins from userdb overrides the setting, so it needs to return all of them. But of course you can do something like:
select concat('quota imap_quota mail_log ', extra_plugins) as mail_plugins
or whatever.
Exactly! I agree with you! I used to have webmail connect locally via unencrypted IMAP on port 143, but the problem occurred even more frequently. So I thought that it could be dovecot not being able to work out both protocols... I had it like this: protocol imap { listen = 127.0.0.1:143 ssl_listen = *:[another port] } Then I commented the listen line, and things got better, but not good enough yet. Strange, uh?
I know I should not be using webmail locally via SSL. But I got less errors like that. Altough, I know once I get the problem solved, I can go back to the correct solution.
Just for the record: My mail server is running super fast! The problem happens only at specific moments, twice a day, only.
Otherwise, what's the other configuration that could be eating up resources?
BTW, I got plenty of CPU (2xQuadCore 3.0) and memory (32GB).
On Fri, Jan 8, 2010 at 5:07 AM, Stan Hoeppner <stan@hardwarefreak.com>wrote:
On Thu, 7 Jan 2010 11:08:54 -0200, Mario Gomide <mariogomide@gmail.com> wrote:
Webmail is in localhost and connects locally also with SSL.
The problem starts when webmail takes long to authenticate and show folders and stuff. After a while it times out and doesn't connect. Any ideas?
It may be unrelated to your current problem, but, why in the world are you having a local process connect to another local process via SSL? That's akin to encrypting the traffic between the left and right hemispheres of your brain for fear someone might be listening in to the unencrypted traffic occurring within you skull. Silly isn't it?
One thing your configuration will certainly do is suck CPU horsepower, eat up memory unnecessarily, and make things slower than they should/could be.
-- Stan
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Fri, 8 Jan 2010, Mario Gomide wrote:
Then I commented the listen line, and things got better, but not good enough yet. Strange, uh?
I never tried that.
Just for the record: My mail server is running super fast! The problem happens only at specific moments, twice a day, only.
specific moments? Which and what are the spesifics?
If I remember correctly, I made logs of the number of file descriptors of the "dovecot" process; each time the number _almost_ reached the max (say one or two are not allocated yet), I saw the stalls _and_ when I manually logged off users (kill -15 imap-process) the number did not decreased. Well, not much anyway. Sort of, it looked like a fd leak.
Since I have ulimit -n in my init.d, no such problem came up again. The previous ulimit was lower than the max Dovecot could handle.
Regards,
Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux)
iQEVAwUBS0c0yb+Vh58GPL/cAQKYTQf/RRDR9hgWhjjO2hivS/Lncm1fO5cNVnqZ 6dTT4Z70M+kgC/yPgGbWEMjEPthDORGeIFMBQof2XdBYAlc/B9TiPH4gmhvkhesU sv5V8Rq9f02LPWmYAAwiCa4BZ+zQDBPQDfBajJhrhUsYfvrSwEW7Jgb9QUXl/00G h4s1S2ptrg/3NzKFctyUgkxEyDRLEa+eqCNXNUBwakRfb+4dvkjcaKPCclA9bw51 CxxtffJNR8oDxQoFCe+tXZDc5FWOxUAffkEDft1Qg9rabqMupfKw45Lecg8rtsgN vU7D0Fxb18OF7p2JOelYA8sfeCCOLMXj91utH87/VK7Jy7IO2keSUw== =n6J7 -----END PGP SIGNATURE-----
On 8.1.2010, at 15.36, Steffen Kaiser wrote:
If I remember correctly, I made logs of the number of file descriptors of the "dovecot" process; each time the number _almost_ reached the max (say one or two are not allocated yet), I saw the stalls _and_ when I manually logged off users (kill -15 imap-process) the number did not decreased. Well, not much anyway. Sort of, it looked like a fd leak.
If Dovecot runs out of file descriptors, it logs an error. So the most important part: Check if there are any errors in logs. http://wiki.dovecot.org/Logging
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Fri, 8 Jan 2010, Timo Sirainen wrote:
If I remember correctly, I made logs of the number of file descriptors of the "dovecot" process; each time the number _almost_ reached the max (say one or two are not allocated yet), I saw the stalls _and_ when I manually logged off users (kill -15 imap-process) the number did not decreased. Well, not much anyway.
Sort of, it looked like a fd leak.
If Dovecot runs out of file descriptors, it logs an error. So the most important part: Check if there are any errors in logs. http://wiki.dovecot.org/Logging
Well, I know it should, but it does not. Maybe it is something different.
I changed my init.d in July 2008, but unfortunately I do not find anything I posted in the mailinglist.
There is no sign in the logs, regardless of the auth_debug or mail_debug setting. A telnet to port 143 just hangs without any "connected".
However, this is Dovecot v1.0, so I see no reason to dig into it.
Regards,
Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux)
iQEVAwUBS0dHub+Vh58GPL/cAQJ2Bgf/VBx1tvRqc4mJIC/Bjc3ZhebZEWmm0NSj UaSU3uXum1QCaPZ8BOtZlM4lTFNhnIVvTQj9lzzKeDlu8UB2/OlvghWkqJfk7GpN UJJmlhadEkTqL8iUvRFyz2caClsEN0g5BvCjE/VlBREdeAWHJ/UQvCa7u97m+UZw CG8EMN1eZ7TgPLoLeEmkTcZ3eNL3e9ia2lYFjLYzG0pvsg+VeTZMv+nihN0PmGhY /K5o051kX3CG9fU9TVLhURlW/1HLVV4Cm6yhdb9nCy+KrJZSb+fVCZk5X0PxUUrL LdNZE2DMm7M9t81lUcmMZ2o9AW+dM21q1kg9SGpU3AQumAZ5HY0fLg== =CV2a -----END PGP SIGNATURE-----
The specific moments are at about 9:40 am and 2:30 pm. Exactly when I have most of my users connecting simultaneously (when they arrive in the morning and back from lunch break). At these exact moments, dovecot hangs.
Anyways, I took a look at http://wiki.dovecot.org/LoginProcess. I increased the default login_max_processes_count from 128 to 256 and restarted dovecot.
Right before doing this (2:24pm, at this time, imap was getting slow already and dovecot was about to hang) I checked my processes every second: ps -ef | grep imap-login | grep dovecot | wc -l and the results varied from 149 to 152.
After setting login_max_processes_count to 256, I hope the problem is fixed. I didn't do anything related to open file descriptors.
Mario
On Fri, Jan 8, 2010 at 11:36 AM, Steffen Kaiser < skdovecot@smail.inf.fh-brs.de> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Fri, 8 Jan 2010, Mario Gomide wrote:
Then I commented the listen line, and things got better, but not good
enough yet. Strange, uh?
I never tried that.
Just for the record: My mail server is running super fast! The problem
happens only at specific moments, twice a day, only.
specific moments? Which and what are the spesifics?
If I remember correctly, I made logs of the number of file descriptors of the "dovecot" process; each time the number _almost_ reached the max (say one or two are not allocated yet), I saw the stalls _and_ when I manually logged off users (kill -15 imap-process) the number did not decreased. Well, not much anyway. Sort of, it looked like a fd leak.
Since I have ulimit -n in my init.d, no such problem came up again. The previous ulimit was lower than the max Dovecot could handle.
Regards,
- -- Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux)
iQEVAwUBS0c0yb+Vh58GPL/cAQKYTQf/RRDR9hgWhjjO2hivS/Lncm1fO5cNVnqZ 6dTT4Z70M+kgC/yPgGbWEMjEPthDORGeIFMBQof2XdBYAlc/B9TiPH4gmhvkhesU sv5V8Rq9f02LPWmYAAwiCa4BZ+zQDBPQDfBajJhrhUsYfvrSwEW7Jgb9QUXl/00G h4s1S2ptrg/3NzKFctyUgkxEyDRLEa+eqCNXNUBwakRfb+4dvkjcaKPCclA9bw51 CxxtffJNR8oDxQoFCe+tXZDc5FWOxUAffkEDft1Qg9rabqMupfKw45Lecg8rtsgN vU7D0Fxb18OF7p2JOelYA8sfeCCOLMXj91utH87/VK7Jy7IO2keSUw== =n6J7 -----END PGP SIGNATURE-----
Yeap! It worked just fine!
No problems at all now! Thanks for the help! Mario
On Fri, Jan 8, 2010 at 2:58 PM, Mario Gomide <mariogomide@gmail.com> wrote:
The specific moments are at about 9:40 am and 2:30 pm. Exactly when I have most of my users connecting simultaneously (when they arrive in the morning and back from lunch break). At these exact moments, dovecot hangs.
Anyways, I took a look at http://wiki.dovecot.org/LoginProcess. I increased the default login_max_processes_count from 128 to 256 and restarted dovecot.
Right before doing this (2:24pm, at this time, imap was getting slow already and dovecot was about to hang) I checked my processes every second: ps -ef | grep imap-login | grep dovecot | wc -l and the results varied from 149 to 152.
After setting login_max_processes_count to 256, I hope the problem is fixed. I didn't do anything related to open file descriptors.
Mario
On Fri, Jan 8, 2010 at 11:36 AM, Steffen Kaiser < skdovecot@smail.inf.fh-brs.de> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Fri, 8 Jan 2010, Mario Gomide wrote:
Then I commented the listen line, and things got better, but not good
enough yet. Strange, uh?
I never tried that.
Just for the record: My mail server is running super fast! The problem
happens only at specific moments, twice a day, only.
specific moments? Which and what are the spesifics?
If I remember correctly, I made logs of the number of file descriptors of the "dovecot" process; each time the number _almost_ reached the max (say one or two are not allocated yet), I saw the stalls _and_ when I manually logged off users (kill -15 imap-process) the number did not decreased. Well, not much anyway. Sort of, it looked like a fd leak.
Since I have ulimit -n in my init.d, no such problem came up again. The previous ulimit was lower than the max Dovecot could handle.
Regards,
- -- Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux)
iQEVAwUBS0c0yb+Vh58GPL/cAQKYTQf/RRDR9hgWhjjO2hivS/Lncm1fO5cNVnqZ 6dTT4Z70M+kgC/yPgGbWEMjEPthDORGeIFMBQof2XdBYAlc/B9TiPH4gmhvkhesU sv5V8Rq9f02LPWmYAAwiCa4BZ+zQDBPQDfBajJhrhUsYfvrSwEW7Jgb9QUXl/00G h4s1S2ptrg/3NzKFctyUgkxEyDRLEa+eqCNXNUBwakRfb+4dvkjcaKPCclA9bw51 CxxtffJNR8oDxQoFCe+tXZDc5FWOxUAffkEDft1Qg9rabqMupfKw45Lecg8rtsgN vU7D0Fxb18OF7p2JOelYA8sfeCCOLMXj91utH87/VK7Jy7IO2keSUw== =n6J7 -----END PGP SIGNATURE-----
participants (5)
-
mailing＠securitylabs.it
-
Mario Gomide
-
Stan Hoeppner
-
Steffen Kaiser
-
Timo Sirainen