[Dovecot] Gah, pop3 falling over
Why would I be seeing a million of these?
dovecot: 2008-10-27 09:12:33 Warning: chdir(/mail/mail/domain/u/username/) blocked for 18 secs
Two servers were working fine and then all of a sudden this morning bam 500 load and theres a million dovecots and tons of these lines in the logs.
It is like everyone gets up at exactly 8:30 and hits the pop3 server at once.
Anyway, dovecot 1.1.5, nfs mount, debian lenny/sid amd64, 2.6.26-1-amd64 i had the performance mode options for login configured, i just set back to the single process per login though, I think that hasnt fallen over that way but I really dont remember anymore. I included the dovecot -n using the suggest performance login settings? Do I have something totally out of whack?
# 1.1.5: /etc/dovecot/dovecot.conf base_dir: /var/run/dovecot/ log_path: /var/log/dovecot/dovecot.log info_log_path: /var/log/dovecot/info.log log_timestamp: %Y-%m-%d %H:%M:%S protocols: imap pop3 listen(default): 208.200.248.2 208.200.248.7 listen(imap): 208.200.248.2 208.200.248.7 listen(pop3): 208.200.248.2 disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable(default): /usr/local/libexec/dovecot/imap-login login_executable(imap): /usr/local/libexec/dovecot/imap-login login_executable(pop3): /usr/local/libexec/dovecot/pop3-login login_greeting: Hello there. login_log_format_elements: <%u> <%r> login_log_format: %s login_process_per_connection: no login_process_size: 128 login_processes_count: 4 login_max_processes_count: 8 login_max_connections: 128 valid_chroot_dirs: /mail/mail mail_max_userip_connections(default): 15 mail_max_userip_connections(imap): 15 mail_max_userip_connections(pop3): 4 verbose_proctitle: yes first_valid_uid: 8 last_valid_uid: 8 first_valid_gid: 8 last_valid_gid: 8 mail_access_groups: mail mail_privileged_group: mail mail_uid: 8 mail_gid: 8 mail_location: maildir:~/:INDEX=/dovecot-index mailbox_idle_check_interval: 60 mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes mailbox_list_index_disable: no mail_executable(default): /usr/local/libexec/dovecot/imap mail_executable(imap): /usr/local/libexec/dovecot/imap mail_executable(pop3): /usr/local/libexec/dovecot/pop3 mail_plugins(default): quota imap_quota mail_plugins(imap): quota imap_quota mail_plugins(pop3): mail_plugin_dir(default): /usr/local/lib/dovecot/imap mail_plugin_dir(imap): /usr/local/lib/dovecot/imap mail_plugin_dir(pop3): /usr/local/lib/dovecot/pop3 mail_log_prefix: %Ls <%u>: imap_client_workarounds(default): outlook-idle delay-newmail imap_client_workarounds(imap): outlook-idle delay-newmail imap_client_workarounds(pop3): pop3_no_flag_updates(default): no pop3_no_flag_updates(imap): no pop3_no_flag_updates(pop3): yes pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh pop3_logout_format(default): top=%t/%p, retr=%r/%b, del=%d/%m, size=%s pop3_logout_format(imap): top=%t/%p, retr=%r/%b, del=%d/%m, size=%s pop3_logout_format(pop3): retr=%r/%b, del=%d/%m, size=%s auth default: mechanisms: plain login default_realm: olypen.com cache_size: 8192 cache_ttl: 300 worker_max_count: 60 passdb: driver: sql args: /etc/dovecot/dovecot-sql.conf userdb: driver: prefetch userdb: driver: sql args: /etc/dovecot/dovecot-sql.conf socket: type: listen client: path: /var/run/dovecot/auth-client mode: 438 user: mail group: mail master: path: /var/run/dovecot/auth-master mode: 438 user: mail group: mail plugin: quota: maildir quota_rule: Trash:ignore quota_rule2: Sent:ignore quota_rule3: Spam:ignore quota_rule4: Archive:ignore quota_rule5: Sent Items:ignore quota_rule6: Deleted Items:ignore auth_socket_path: /var/run/dovecot/auth-master
On Mon, Oct 27, 2008 at 9:20 AM, Brandon Lamb brandonlamb@gmail.com wrote:
Why would I be seeing a million of these?
dovecot: 2008-10-27 09:12:33 Warning: chdir(/mail/mail/domain/u/username/) blocked for 18 secs
Two servers were working fine and then all of a sudden this morning bam 500 load and theres a million dovecots and tons of these lines in the logs.
It is like everyone gets up at exactly 8:30 and hits the pop3 server at once.
Anyway, dovecot 1.1.5, nfs mount, debian lenny/sid amd64, 2.6.26-1-amd64 i had the performance mode options for login configured, i just set back to the single process per login though, I think that hasnt fallen over that way but I really dont remember anymore. I included the dovecot -n using the suggest performance login settings? Do I have something totally out of whack?
# 1.1.5: /etc/dovecot/dovecot.conf base_dir: /var/run/dovecot/ log_path: /var/log/dovecot/dovecot.log info_log_path: /var/log/dovecot/info.log log_timestamp: %Y-%m-%d %H:%M:%S protocols: imap pop3 listen(default): 208.200.248.2 208.200.248.7 listen(imap): 208.200.248.2 208.200.248.7 listen(pop3): 208.200.248.2 disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable(default): /usr/local/libexec/dovecot/imap-login login_executable(imap): /usr/local/libexec/dovecot/imap-login login_executable(pop3): /usr/local/libexec/dovecot/pop3-login login_greeting: Hello there. login_log_format_elements: <%u> <%r> login_log_format: %s login_process_per_connection: no login_process_size: 128 login_processes_count: 4 login_max_processes_count: 8 login_max_connections: 128 valid_chroot_dirs: /mail/mail mail_max_userip_connections(default): 15 mail_max_userip_connections(imap): 15 mail_max_userip_connections(pop3): 4 verbose_proctitle: yes first_valid_uid: 8 last_valid_uid: 8 first_valid_gid: 8 last_valid_gid: 8 mail_access_groups: mail mail_privileged_group: mail mail_uid: 8 mail_gid: 8 mail_location: maildir:~/:INDEX=/dovecot-index mailbox_idle_check_interval: 60 mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes mailbox_list_index_disable: no mail_executable(default): /usr/local/libexec/dovecot/imap mail_executable(imap): /usr/local/libexec/dovecot/imap mail_executable(pop3): /usr/local/libexec/dovecot/pop3 mail_plugins(default): quota imap_quota mail_plugins(imap): quota imap_quota mail_plugins(pop3): mail_plugin_dir(default): /usr/local/lib/dovecot/imap mail_plugin_dir(imap): /usr/local/lib/dovecot/imap mail_plugin_dir(pop3): /usr/local/lib/dovecot/pop3 mail_log_prefix: %Ls <%u>: imap_client_workarounds(default): outlook-idle delay-newmail imap_client_workarounds(imap): outlook-idle delay-newmail imap_client_workarounds(pop3): pop3_no_flag_updates(default): no pop3_no_flag_updates(imap): no pop3_no_flag_updates(pop3): yes pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh pop3_logout_format(default): top=%t/%p, retr=%r/%b, del=%d/%m, size=%s pop3_logout_format(imap): top=%t/%p, retr=%r/%b, del=%d/%m, size=%s pop3_logout_format(pop3): retr=%r/%b, del=%d/%m, size=%s auth default: mechanisms: plain login default_realm: olypen.com cache_size: 8192 cache_ttl: 300 worker_max_count: 60 passdb: driver: sql args: /etc/dovecot/dovecot-sql.conf userdb: driver: prefetch userdb: driver: sql args: /etc/dovecot/dovecot-sql.conf socket: type: listen client: path: /var/run/dovecot/auth-client mode: 438 user: mail group: mail master: path: /var/run/dovecot/auth-master mode: 438 user: mail group: mail plugin: quota: maildir quota_rule: Trash:ignore quota_rule2: Sent:ignore quota_rule3: Spam:ignore quota_rule4: Archive:ignore quota_rule5: Sent Items:ignore quota_rule6: Deleted Items:ignore auth_socket_path: /var/run/dovecot/auth-master
Well, I just rebooted all 4 servers and now the load is a cool 0.50 and under.... What would make them all be around 10 (well 40 to 100++) but then back down to 5 to 10 after restarting dovecot. But then a reboot and they come back and stay at under 1 for a day.
On Mon, Oct 27, 2008 at 9:20 AM, Brandon Lamb brandonlamb@gmail.com wrote:
Why would I be seeing a million of these?
dovecot: 2008-10-27 09:12:33 Warning: chdir(/mail/mail/domain/u/username/) blocked for 18 secs
Two servers were working fine and then all of a sudden this morning bam 500 load and theres a million dovecots and tons of these lines in the logs.
It is like everyone gets up at exactly 8:30 and hits the pop3 server at once.
Anyway, dovecot 1.1.5, nfs mount, debian lenny/sid amd64, 2.6.26-1-amd64 i had the performance mode options for login configured, i just set back to the single process per login though, I think that hasnt fallen over that way but I really dont remember anymore. I included the dovecot -n using the suggest performance login settings? Do I have something totally out of whack?
# 1.1.5: /etc/dovecot/dovecot.conf base_dir: /var/run/dovecot/ log_path: /var/log/dovecot/dovecot.log info_log_path: /var/log/dovecot/info.log log_timestamp: %Y-%m-%d %H:%M:%S protocols: imap pop3 listen(default): 208.200.248.2 208.200.248.7 listen(imap): 208.200.248.2 208.200.248.7 listen(pop3): 208.200.248.2 disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable(default): /usr/local/libexec/dovecot/imap-login login_executable(imap): /usr/local/libexec/dovecot/imap-login login_executable(pop3): /usr/local/libexec/dovecot/pop3-login login_greeting: Hello there. login_log_format_elements: <%u> <%r> login_log_format: %s login_process_per_connection: no login_process_size: 128 login_processes_count: 4 login_max_processes_count: 8 login_max_connections: 128 valid_chroot_dirs: /mail/mail mail_max_userip_connections(default): 15 mail_max_userip_connections(imap): 15 mail_max_userip_connections(pop3): 4 verbose_proctitle: yes first_valid_uid: 8 last_valid_uid: 8 first_valid_gid: 8 last_valid_gid: 8 mail_access_groups: mail mail_privileged_group: mail mail_uid: 8 mail_gid: 8 mail_location: maildir:~/:INDEX=/dovecot-index mailbox_idle_check_interval: 60 mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes mailbox_list_index_disable: no mail_executable(default): /usr/local/libexec/dovecot/imap mail_executable(imap): /usr/local/libexec/dovecot/imap mail_executable(pop3): /usr/local/libexec/dovecot/pop3 mail_plugins(default): quota imap_quota mail_plugins(imap): quota imap_quota mail_plugins(pop3): mail_plugin_dir(default): /usr/local/lib/dovecot/imap mail_plugin_dir(imap): /usr/local/lib/dovecot/imap mail_plugin_dir(pop3): /usr/local/lib/dovecot/pop3 mail_log_prefix: %Ls <%u>: imap_client_workarounds(default): outlook-idle delay-newmail imap_client_workarounds(imap): outlook-idle delay-newmail imap_client_workarounds(pop3): pop3_no_flag_updates(default): no pop3_no_flag_updates(imap): no pop3_no_flag_updates(pop3): yes pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh pop3_logout_format(default): top=%t/%p, retr=%r/%b, del=%d/%m, size=%s pop3_logout_format(imap): top=%t/%p, retr=%r/%b, del=%d/%m, size=%s pop3_logout_format(pop3): retr=%r/%b, del=%d/%m, size=%s auth default: mechanisms: plain login default_realm: olypen.com cache_size: 8192 cache_ttl: 300 worker_max_count: 60 passdb: driver: sql args: /etc/dovecot/dovecot-sql.conf userdb: driver: prefetch userdb: driver: sql args: /etc/dovecot/dovecot-sql.conf socket: type: listen client: path: /var/run/dovecot/auth-client mode: 438 user: mail group: mail master: path: /var/run/dovecot/auth-master mode: 438 user: mail group: mail plugin: quota: maildir quota_rule: Trash:ignore quota_rule2: Sent:ignore quota_rule3: Spam:ignore quota_rule4: Archive:ignore quota_rule5: Sent Items:ignore quota_rule6: Deleted Items:ignore auth_socket_path: /var/run/dovecot/auth-master
dotlock_use_excl = yes fsync_disable = no lock_method = fcntl
Could I be running into some nfs locking issue? Wouldnt a reboot clear any nfs um, handles or whatnot that the mail server(s) might have? Do these look like the right values for using NFS? Should I try using lock_method = dotlock instead?
On Mon, 2008-10-27 at 09:51 -0700, Brandon Lamb wrote:
Two servers were working fine and then all of a sudden this morning bam 500 load and theres a million dovecots and tons of these lines in the logs.
I guess the load is about I/O load, not CPU load?
It is like everyone gets up at exactly 8:30 and hits the pop3 server at once.
Anyway, dovecot 1.1.5, nfs mount, debian lenny/sid amd64, 2.6.26-1-amd64 i had the performance mode options for login configured, i just set back to the single process per login though, I think that hasnt fallen over that way but I really dont remember anymore. I included the dovecot -n using the suggest performance login settings? Do I have something totally out of whack?
If the virtual -> physical size change didn't help, I guess that wasn't the problem then. Another possibility could be Maildir++ quota working non-optimally. You could try temporarily disabling it for POP3 to see if that helps.
dotlock_use_excl = yes fsync_disable = no lock_method = fcntl
Could I be running into some nfs locking issue? Wouldnt a reboot clear any nfs um, handles or whatnot that the mail server(s) might have? Do these look like the right values for using NFS? Should I try using lock_method = dotlock instead?
I guess you could try lock_method=dotlock too.
participants (2)
-
Brandon Lamb
-
Timo Sirainen