Re: [Dovecot] High level of pop3 popping causing server to become unresponsive
Missed CCing on last reply. See below..
Also, would having Nagios checking the number of messages in a mailbox cause issues with the popping of messages? And/Or would having a user accessing the mailbox from two different applications cause issues? Ie. from Outlook and mobile device? I am under the impression that a lock file is generated which should deal with the issue of contention, deleting mail, etc.
Thanks for any information.
Kevin
Sorry for the delay in responding, long weekend in Canada...
When trying to SSH into the server, the server prompts for user name then password, then just hangs. Same if there is already a console connection over, when trying to SU to root, it just hangs after entering the password.
Our passwords are in the shadow file, and because this is a server dedicated for this task, there is only the default linux users, and maybe 8 other user accounts in the shadow file.
There shouldn't be high IO as all this box does is postfix, pop3ad and dovecot. This box is seeing less then 3000 emails a day, and only has 5 mailboxes on it.
Thanks for the continued suggestions...
Kevin
On Sat, May 19, 2012 at 3:33 PM, Timo Sirainen tss@iki.fi wrote:
On Fri, 2012-05-18 at 09:21 -0400, Root Kev wrote:
During the last time that the load went up, it became unable to login / su to root for the entire period that dovecot was running, we had to kill dovecot and go back to Popa3d until the mailq was cleared up. We are running CentOS 5.6 server. Based on TOP running at the time the CPU usage was running under 10%. Once Dovecot was killed, we were then able to log in /su again.
Like Kelsey said, a very high disk IO might explain this, although normally the login should still eventually succeed. Another thing I'm wondering is if some process limit reached. How does the login/su fail, does it just hang or immediately fail with some error?
We were under the impression that checking to shadow directly should be the fastest and least amount of overhead, is any of the other ways to connect have less load on authentication to PAM?
Your passwords really are in /etc/shadow file, not LDAP/something else? I don't think the problem is with authentication. Reading /etc/shadow is pretty fast (unless maybe if it's a huge file) and it anyway can't block login/su from working.
Am 23.05.2012 16:01, schrieb Root Kev:
Missed CCing on last reply. See below..
Also, would having Nagios checking the number of messages in a mailbox cause issues with the popping of messages? And/Or would having a user accessing the mailbox from two different applications cause issues? Ie. from Outlook and mobile device? I am under the impression that a lock file is generated which should deal with the issue of contention, deleting mail, etc.
Thanks for any information.
Kevin
i have a different setup as yours ( virtual , sql etc ) with dove 2.0.20 and extrem high pop3 traffic no problem so far
same with low pop3 traffic setup dove 2.1.6 no problem reported
anyway use imap
where are your confs and logs ?
Sorry for the delay in responding, long weekend in Canada...
When trying to SSH into the server, the server prompts for user name then password, then just hangs. Same if there is already a console connection over, when trying to SU to root, it just hangs after entering the password.
Our passwords are in the shadow file, and because this is a server dedicated for this task, there is only the default linux users, and maybe 8 other user accounts in the shadow file.
There shouldn't be high IO as all this box does is postfix, pop3ad and dovecot. This box is seeing less then 3000 emails a day, and only has 5 mailboxes on it.
Thanks for the continued suggestions...
Kevin
On Sat, May 19, 2012 at 3:33 PM, Timo Sirainen tss@iki.fi wrote:
On Fri, 2012-05-18 at 09:21 -0400, Root Kev wrote:
During the last time that the load went up, it became unable to login / su to root for the entire period that dovecot was running, we had to kill dovecot and go back to Popa3d until the mailq was cleared up. We are running CentOS 5.6 server. Based on TOP running at the time the CPU usage was running under 10%. Once Dovecot was killed, we were then able to log in /su again.
Like Kelsey said, a very high disk IO might explain this, although normally the login should still eventually succeed. Another thing I'm wondering is if some process limit reached. How does the login/su fail, does it just hang or immediately fail with some error?
We were under the impression that checking to shadow directly should be the fastest and least amount of overhead, is any of the other ways to connect have less load on authentication to PAM?
Your passwords really are in /etc/shadow file, not LDAP/something else? I don't think the problem is with authentication. Reading /etc/shadow is pretty fast (unless maybe if it's a huge file) and it anyway can't block login/su from working.
-- Best Regards
MfG Robert Schetterer
Germany/Munich/Bavaria
Currently cannot use IMAP as our application to access the mailbox currently is only setup to access pop3 mailboxes. We are currently using Dovecot 2.1.4. Below is the majority of our configs, it is mostly basic: protocols = pop3 listen = ***Address here*** base_dir = /var/run/dovecot1/ instance_name = Popper login_greeting = Popper mail_location = mbox:/var/empty:INBOX=/opt/mailstore/spool/mail/%u:INDEX=MEMORY !include conf.d/*.conf !include_try local.conf disable_plaintext_auth = no auth_mechanisms = plain log_path = syslog syslog_facility = mail auth_verbose = yes log_timestamp = "%b %d %H:%M:%S " namespace inbox { inbox = yes } mail_privileged_group = mail lock_method = fcntl mbox_read_locks = fcntl mbox_write_locks = dotlock fcntl pop3_uidl_format = %08Xu%08Xv passdb { driver = checkpassword args = /usr/bin/checkpassword } userdb { driver = prefetch }
On 24 May 2012, at 16:54, Root Kev
Currently cannot use IMAP as our application to access the mailbox currently is only setup to access pop3 mailboxes. We are currently using Dovecot 2.1.4. Below is the majority of our configs, it is mostly basic:
protocols = pop3 listen = ***Address here*** base_dir = /var/run/dovecot1/ instance_name = Popper login_greeting = Popper mail_location = mbox:/var/empty:INBOX=/opt/mailstore/spool/mail/%u:INDEX=MEMORY
!include conf.d/*.conf !include_try local.conf
disable_plaintext_auth = no auth_mechanisms = plain log_path = syslog syslog_facility = mail auth_verbose = yes log_timestamp = "%b %d %H:%M:%S " namespace inbox { inbox = yes } mail_privileged_group = mail lock_method = fcntl mbox_read_locks = fcntl mbox_write_locks = dotlock fcntl pop3_uidl_format = %08Xu%08Xv passdb { driver = checkpassword args = /usr/bin/checkpassword } userdb { driver = prefetch }
Hi, Having a system with a third of our users on POP3 (2300000 of them), no trouble with dovecot (v2.1.5, on CentOS 5). But one thing surprises me in your config, the INDEX=MEMORY in the location parameter. That means that for each POP3 connection, dovecot has to read each and every mails to create the index in memory. That might be why the machine becomes unresponsive. Unless you have a specific reason to use memory index (and I would be curious to know about it), I would suggest to remove this and keep standard file indexes, and your performances should improve a lot. Regards, Thierry
So the best way to would be to remove it that part completely, or should it be stored somewhere on disk?
Thanks again,
Kevin
Hi,
Having a system with a third of our users on POP3 (2300000 of them), no trouble with dovecot (v2.1.5, on CentOS 5). But one thing surprises me in your config, the INDEX=MEMORY in the location parameter. That means that for each POP3 connection, dovecot has to read each and every mails to create the index in memory. That might be why the machine becomes unresponsive. Unless you have a specific reason to use memory index (and I would be curious to know about it), I would suggest to remove this and keep standard file indexes, and your performances should improve a lot.
Regards,
Thierry
On 25 May 2012, at 20:00, Root Kev root.kev@gmail.com wrote:
So the best way to would be to remove it that part completely, or should it be stored somewhere on disk?
Thanks again,
Kevin yes, best is to remove ":INDEX=MEMORY" part, and it will store indexes in your INBOX path, which is fine.
If you want to put it in a different path, you can have something like INDEX=/path/to/indexes/%u
Regards,
ThierryHi,
Having a system with a third of our users on POP3 (2300000 of them), no trouble with dovecot (v2.1.5, on CentOS 5). But one thing surprises me in your config, the INDEX=MEMORY in the location parameter. That means that for each POP3 connection, dovecot has to read each and every mails to create the index in memory. That might be why the machine becomes unresponsive. Unless you have a specific reason to use memory index (and I would be curious to know about it), I would suggest to remove this and keep standard file indexes, and your performances should improve a lot.
Regards,
Thierry
Thierry,
Had a chance to test this change this morning, and in my test environment, this does drastically improve the ability to ssh and su during heavy pop3 load (in test environment, change of 10-15sec to 1-2sec login). While this is better, the popping is still slowing down authentication on the box during load. Is there anything else that might be changed to increase speed?
Just a recap, I basically have between 2-8 mailboxes on each server, and an application that creates two connections to each mailbox, processes each message that comes in, and then deleted each message from the pop3 mailbox. During high load times (500+ messages) with the settings that I had, I was unable to reconnect to the server, nor su to root. Ended up having to stop the dovecot process and revert to popa3d (luckily had a ssh session still open to the box).
This is the config from my test environment: root@devsmtp:~# dovecot -c /usr/local/etc/dovecot/dovecot-test.conf -n # 2.1.4: /usr/local/etc/dovecot/dovecot-test.conf # OS: Linux 2.6.32-33-generic-pae i686 Ubuntu 10.04.4 LTS ext4 auth_verbose = yes base_dir = /var/run/dovecot-test/ disable_plaintext_auth = no instance_name = dovecot-test listen = <removed> login_greeting = Dovecot-Test for removing index mail_location = mbox:/var/empty:INBOX=/var/mail/%u mail_privileged_group = mail namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = shadow } protocols = pop3 service pop3-login { inet_listener pop3 { port = 130 } service_count = 0 } ssl = no userdb { driver = passwd } protocol pop3 { pop3_uidl_format = %08Xu%08Xv }
And from one of my production servers that has the issue: # 2.1.4: /usr/local/etc/dovecot/dovecot.conf-ml # OS: Linux 2.6.18-194.17.1.el5 x86_64 Red Hat Enterprise Linux Server release 5.6 (Tikanga) ext3 auth_verbose = yes base_dir = /var/run/dovecotml/ disable_plaintext_auth = no instance_name = Popper listen = <removed> login_greeting = Popper mail_location = mbox:/var/empty:INBOX=/opt/mailstore/spool/mail/%u:INDEX=MEMORY mail_privileged_group = mail namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = shadow } protocols = pop3 ssl = no userdb { args = blocking=yes driver = passwd } protocol pop3 { pop3_uidl_format = %08Xu%08Xv }
On Fri, May 25, 2012 at 2:24 PM, Thierry de Montaudry thierry@odry.netwrote:
On 25 May 2012, at 20:00, Root Kev root.kev@gmail.com wrote:
So the best way to would be to remove it that part completely, or should it be stored somewhere on disk?
Thanks again,
Kevin
yes, best is to remove ":INDEX=MEMORY" part, and it will store indexes in your INBOX path, which is fine.
If you want to put it in a different path, you can have something like INDEX=/path/to/indexes/%u
Regards,
Thierry
Hi,
Having a system with a third of our users on POP3 (2300000 of them), no trouble with dovecot (v2.1.5, on CentOS 5). But one thing surprises me in your config, the INDEX=MEMORY in the location parameter. That means that for each POP3 connection, dovecot has to read each and every mails to create the index in memory. That might be why the machine becomes unresponsive. Unless you have a specific reason to use memory index (and I would be curious to know about it), I would suggest to remove this and keep standard file indexes, and your performances should improve a lot.
Regards,
Thierry
On 30.5.2012, at 20.25, Root Kev wrote:
mail_location = mbox:/var/empty:INBOX=/var/mail/%u
Note that it's not a good idea for different users to share a single directory for indexes, which is what this does. Preferably /var/empty wouldn't be even writable to the users so this wouldn't happen accidentally. You could instead use something like:
mail_location = mbox:/var/empty:INBOX=/var/mail/%u:INDEX=/var/index/%u
But if the clients always just download + delete everything, the index files shouldn't make any (positive) difference.
Had a chance to test this change this morning, and in my test environment, this does drastically improve the ability to ssh and su during heavy pop3 load (in test environment, change of 10-15sec to 1-2sec login). While this
If you think the problem is authentication, try with passdb + userdb static (http://wiki2.dovecot.org/PasswordDatabase/Static) or passwd-file (http://wiki2.dovecot.org/AuthDatabase/PasswdFile) and see if you can reproduce the slowdowns with them.
On 2012-05-24 10:54 AM, Root Kev root.kev@gmail.com wrote:
Below is the majority of our configs, it is mostly basic:
protocols = pop3
Pleease don't post copy/pastes from your config files...
Always provide output of doveconf -n
This has multiple benefits - doveconf -n output is easier to read, will identify typos, and will make it evident if you are editing the wrong config file(s)...
--
Best regards,
Charles
participants (5)
-
Charles Marcus
-
Robert Schetterer
-
Root Kev
-
Thierry de Montaudry
-
Timo Sirainen