Hi,
if you try the following command during the server has a high load:
# ps -ostat,pid,time,wchan='WCHAN-xxxxxxxxxxxxxxxxxxxx',cmd ax |grep D
Do you get back something like this?
STAT PID TIME WCHAN-xxxxxxxxxxxxxxxxxxxx CMD D 18713 00:00:00 synchronize_srcu dovecot/imap D 18736 00:00:00 synchronize_srcu dovecot/imap D 18775 00:00:05 synchronize_srcu dovecot/imap D 20330 00:00:00 synchronize_srcu dovecot/imap D 20357 00:00:00 synchronize_srcu dovecot/imap D 20422 00:00:00 synchronize_srcu dovecot/imap D 20687 00:00:00 synchronize_srcu dovecot/imap S+ 20913 00:00:00 pipe_wait grep D
If yes, it could be a problem with Inotify in your kernel. You can try to disable inotify in the kernel with:
echo 0 > /proc/sys/fs/inotify/max_user_watches echo 0 > /proc/sys/fs/inotify/max_user_instances
Full article: http://thread.gmane.org/gmane.linux.kernel/1315430
For me this resolved the problem. Load goes down to < 1.00
Regards Urban
Am 21.08.2013 12:37, schrieb Kavish Karkera:
Hi,
We have a serious issue running on our POP/IMAP servers these days. The load average of a servers spikes up to 400-500 as a uptime command result, for a particular time period , to be specific mostly in noon time and evening, but it last for few minutes only.
We have 2 servers running dovecot 1.1.20 , in loadbanlancer, We have used KEEPLIVE (1.1.13) for loadbalacing.
Server specification. Operating System : CentOS 5.5 64bit CPU cores : 16 RAM : 8GB
Mail and Indexes are mounted on NFS (NetApp).
Below is the dovecot -n ... (top results during high spike)
#############################################################################################
# 1.1.20: /usr/local/etc/dovecot.conf # OS: Linux 2.6.28 x86_64 CentOS release 5.5 (Final) log_path: /var/log/dovecot-info.log info_log_path: /var/log/dovecot-info.log syslog_facility: local1 protocols: imap imaps pop3 pop3s listen(default): *:143 listen(imap): *:143 listen(pop3): *:110 ssl_listen(default): *:993 ssl_listen(imap): *:993 ssl_listen(pop3): *:995 ssl_cert_file: /usr/local/etc/ssl/certs/dovecot.pem ssl_key_file: /usr/local/etc/ssl/private/dovecot.pem disable_plaintext_auth: no login_dir: /usr/local/var/run/dovecot/login login_executable(default): /usr/local/libexec/dovecot/imap-login login_executable(imap): /usr/local/libexec/dovecot/imap-login login_executable(pop3): /usr/local/libexec/dovecot/pop3-login login_greeting: Welcome to Popserver. login_process_per_connection: no max_mail_processes: 1024 mail_max_userip_connections(default): 100 mail_max_userip_connections(imap): 100 mail_max_userip_connections(pop3): 50 verbose_proctitle: yes first_valid_uid: 99 first_valid_gid: 99 mail_location: maildir:~/Maildir:INDEX=/indexes/%h:CONTROL=/indexes/%h mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes lock_method: dotlock mail_executable(default): /usr/local/libexec/dovecot/imap mail_executable(imap): /usr/local/libexec/dovecot/imap mail_executable(pop3): /usr/local/libexec/dovecot/pop3 mail_plugins(default): quota imap_quota mail_plugins(imap): quota imap_quota mail_plugins(pop3): quota mail_plugin_dir(default): /usr/local/lib/dovecot/imap mail_plugin_dir(imap): /usr/local/lib/dovecot/imap mail_plugin_dir(pop3): /usr/local/lib/dovecot/pop3 pop3_no_flag_updates(default): no pop3_no_flag_updates(imap): no pop3_no_flag_updates(pop3): yes pop3_lock_session(default): no pop3_lock_session(imap): no pop3_lock_session(pop3): yes pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls lda: postmaster_address: admin@research.com mail_plugins: cmusieve quota mail_log mail_plugin_dir: /usr/local/lib/dovecot/lda auth_socket_path: /var/run/dovecot/auth-master auth default: worker_max_count: 15 passdb: driver: sql args: /usr/local/etc/dovecot-mysql.conf userdb: driver: sql args: /usr/local/etc/dovecot-mysql.conf userdb: driver: prefetch socket: type: listen client: path: /var/run/dovecot/auth-client mode: 432 user: nobody group: nobody master: path: /var/run/dovecot/auth-master mode: 384 user: nobody group: nobody plugin: quota_warning: storage=95%% /usr/local/bin/quota-warning.sh 95 %u quota_warning2: storage=80%% /usr/local/bin/quota-warning.sh 80 %u quota: maildir:storage=64 ##############################################################################################
##############################################################################################
top - 12:08:31 up 206 days, 10:45, 3 users, load average: 189.88, 82.07, 55.97 Tasks: 771 total, 1 running, 767 sleeping, 1 stopped, 2 zombie Cpu(s): 8.3%us, 7.6%sy, 0.0%ni, 8.3%id, 75.0%wa, 0.0%hi, 0.8%si, 0.0%st Mem: 16279824k total, 11913788k used, 4366036k free, 334308k buffers Swap: 4192956k total, 0k used, 4192956k free, 10359492k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 408 mysql 18 0 384m 38m 4412 S 52.8 0.2 42221:44 mysqld 29326 nobody 15 0 22688 10m 1112 D 3.9 0.1 0:00.05 imap 29313 nobody 16 0 14892 4892 1000 S 3.1 0.0 0:00.07 imap 28129 nobody 16 0 23688 4148 1140 S 2.3 0.0 0:00.10 imap 28936 nobody 16 0 25048 13m 1108 S 2.3 0.1 0:00.06 imap 29302 nobody 16 0 15168 3924 1000 S 2.3 0.0 0:00.05 imap 29307 nobody 16 0 12864 2892 1000 S 2.3 0.0 0:00.03 imap 21956 root 15 0 13272 1652 800 R 1.6 0.0 0:06.46 top 29119 nobody 16 0 26000 9324 1096 D 1.6 0.1 0:00.11 imap 29158 nobody 16 0 12684 2728 1028 S 1.6 0.0 0:00.03 imap 29204 nobody 16 0 14400 4232 1000 S 1.6 0.0 0:00.08 imap 29262 nobody 16 0 25436 2776 1120 S 1.6 0.0 0:00.02 imap 3397 root 10 -5 0 0 0 S 0.8 0.0 28:02.90 rpciod/2 3401 root 10 -5 0 0 0 S 0.8 0.0 337:30.77 rpciod/6 4218 root 16 0 8948 1548 588 S 0.8 0.0 8:05.53 dovecot 10214 root 15 0 50868 1968 1468 S 0.8 0.0 0:30.08 dovecot-auth 27454 nobody 16 0 19220 5048 1256 D 0.8 0.0 0:00.17 imap 27820 nobody 16 0 18916 4704 1252 D 0.8 0.0 0:00.08 imap 27925 nobody 16 0 11180 1524 1000 S 0.8 0.0 0:00.07 imap 28094 nobody 16 0 11536 2068 1148 S 0.8 0.0 0:00.13 imap 28150 nobody 16 0 12528 2396 1104 S 0.8 0.0 0:00.08 imap 28196 nobody 16 0 13976 3792 1148 S 0.8 0.0 0:00.21 imap 28292 nobody 16 0 11984 1808 1004 S 0.8 0.0 0:00.04 imap 28343 nobody 16 0 11584 2052 1152 D 0.8 0.0 0:00.02 imap 28352 nobody 15 0 11108 1492 1004 S 0.8 0.0 0:00.04 imap 28413 nobody 16 0 11220 1568 996 S 0.8 0.0 0:00.02 imap 28434 nobody 16 0 12492 2444 1004 S 0.8 0.0 0:00.05 imap 28467 nobody 16 0 11852 2044 1004 D 0.8 0.0 0:00.06 imap 28558 nobody 16 0 11904 2316 1168 S 0.8 0.0 0:00.02 imap 28643 nobody 16 0 11312 1688 1000 S 0.8 0.0 0:00.04 imap 28711 nobody 15 0 12048 1868 1004 S 0.8 0.0 0:00.04 imap 28779 nobody 16 0 23392 10m 1000 D 0.8 0.1 0:00.05 imap
############################################################################################
Regards, Kavish Karkera