Hi,
I have a DELL PE R610 (32GB RAM 2x Six Core CPU and about 1,4 TB RAID 10) running with 20.000 Mailaccounts behind 2 Dovecot IMAP/POP3 Proxies on a Debian Lenny.
The Server was running about 1 year without any problems. 15Min Load was between 0,5 and max 8. No high IOWAIT. CPU Idletime about 98%.
But since yesterday morning the Systemload on the Server has been increased over 500. I Think this is very high. The strange thing: there was no IOWAIT and the CPU Idle time was allways the same on about 98%.
The total amount of IMAP Sessions is about 300 - 600.
Current vmstat and iostat:
# vmstat 1 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 1 0 0 27040576 635460 3495600 0 0 14 15 2 31 0 0 99 0 0 0 0 27040320 635468 3496064 0 0 804 455 1383 1281 0 0 98 1 0 0 0 27047016 634964 3489312 0 0 216 156 1841 1292 1 0 98 1 0 0 0 27047140 635028 3489012 0 0 240 619 1629 1658 0 0 96 3 0 0 0 27047264 635120 3489172 0 0 92 0 1069 881 0 0 100 0 0 0 0 27047388 635120 3489256 0 0 0 46 1404 1265 0 0 100 0 0 0 0 27047512 635136 3489312 0 0 128 471 1539 1354 0 0 99 1 0 0 0 27047388 635156 3489384 0 0 12 360 1108 952 0 0 99 0 0 0 0 27047516 635160 3489408 0 0 104 12 893 677 0 0 99 0 ^C
# iostat -k Linux 3.0.28-vs2.3.2.3-rol-em64t (mailstore4) 16.05.2012 _x86_64_ (24 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle 0,08 0,00 0,09 0,34 0,00 99,49
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn sda 67,35 636,28 1080,48 31337690 53215361 dm-0 73,53 591,55 893,93 29134837 44027496 dm-1 15,06 42,40 181,79 2088493 8953661 drbd1 51,21 316,06 277,94 15566325 13689004 drbd0 9,22 23,50 80,38 1157641 3958796
Current Load: 08:55:44 up 14:04, 3 users, load average: 19,64, 14,47, 10,49 Load is increasing.
The only strange thing I can see is this: # ps -ostat,pid,time,wchan='WCHAN-xxxxxxxxxxxxxxxxxxxx',cmd ax |grep D STAT PID TIME WCHAN-xxxxxxxxxxxxxxxxxxxx CMD D 18713 00:00:00 synchronize_srcu dovecot/imap D 18736 00:00:00 synchronize_srcu dovecot/imap D 18775 00:00:05 synchronize_srcu dovecot/imap D 20330 00:00:00 synchronize_srcu dovecot/imap D 20357 00:00:00 synchronize_srcu dovecot/imap D 20422 00:00:00 synchronize_srcu dovecot/imap D 20687 00:00:00 synchronize_srcu dovecot/imap S+ 20913 00:00:00 pipe_wait grep D
There are many imap processes in D State. Amount is increasing. I think they are delayed and are wating for some event. I have no idea on which event they are waiting.
And many of this in "mail.err" Log: May 16 08:52:11 dcot-rolmail-1.rolmail.net dovecot: master: Error: service(imap): Process 17468 is ignoring idle SIGINT May 16 08:52:24 dcot-rolmail-1.rolmail.net dovecot: master: Error: service(imap): Process 20307 is ignoring idle SIGINT May 16 08:52:25 dcot-rolmail-1.rolmail.net dovecot: master: Error: service(imap): Process 20318 is ignoring idle SIGINT May 16 08:52:26 dcot-rolmail-1.rolmail.net dovecot: master: Error: service(imap): Process 18964 is ignoring idle SIGINT May 16 08:52:28 dcot-rolmail-1.rolmail.net dovecot: master: Error: service(imap): Process 19244 is ignoring idle SIGINT May 16 08:54:22 dcot-rolmail-1.rolmail.net dovecot: master: Error: service(imap): Process 21177 is ignoring idle SIGINT May 16 08:54:41 dcot-rolmail-1.rolmail.net dovecot: master: Error: service(imap): Process 20647 is ignoring idle SIGINT May 16 08:55:10 dcot-rolmail-1.rolmail.net dovecot: master: Error: service(imap): Process 18836 is ignoring idle SIGINT May 16 08:55:17 dcot-rolmail-1.rolmail.net dovecot: master: Error: service(imap): Process 18857 is ignoring idle SIGINT May 16 08:55:19 dcot-rolmail-1.rolmail.net dovecot: master: Error: service(imap): Process 21176 is ignoring idle SIGINT May 16 08:55:24 dcot-rolmail-1.rolmail.net dovecot: master: Error: service(imap): Process 20688 is ignoring idle SIGINT May 16 08:55:25 dcot-rolmail-1.rolmail.net dovecot: master: Error: service(imap): Process 20973 is ignoring idle SIGINT May 16 08:56:44 dcot-rolmail-1.rolmail.net dovecot: master: Error: service(imap): Process 20326 is ignoring idle SIGINT
Have you any Idea ho I can troubleshoot this problem?
Some more technical information about the system: # uname -a Linux mailstore4 3.0.28-vs2.3.2.3-rol-em64t #1 SMP Thu May 3 09:31:08 CEST 2012 x86_64 GNU/Linux
Dovecot Packages on Backend: installed: # dpkg -l |grep dovecot ii debian-dovecot-auto-keyring 2010.01.30 GnuPG archive keys of the Automatic Dovecot Debian repository ii dovecot-common 2:2.0.13-0~auto+54 secure mail server that supports mbox, maildir, dbox and mdbox mailboxes ii dovecot-imapd 2:2.0.13-0~auto+54 secure IMAP server that supports mbox, maildir, dbox and mdbox mailboxes ii dovecot-lmtpd 2:2.0.13-0~auto+54 secure LMTP server for Dovecot ii dovecot-managesieved 2:2.0.13-0~auto+54 secure ManageSieve server for Dovecot ii dovecot-mysql 2:2.0.13-0~auto+54 MySQL support for Dovecot ii dovecot-pop3d 2:2.0.13-0~auto+54 secure POP3 server that supports mbox, maildir, dbox and mdbox mailboxes ii dovecot-sieve 2:2.0.13-0~auto+54 sieve filters support for Dovecot
Dovecot Packages on both Proxies installed: ii debian-dovecot-auto-keyring 2010.01.30 GnuPG archive keys of the Automatic Dovecot ii dovecot-common 2:2.0.17-0~auto+4 Transitional package for dovecot ii dovecot-core 2:2.0.17-0~auto+4 secure mail server that supports mbox, maild ii dovecot-gssapi 2:2.0.17-0~auto+4 GSSAPI authentication support for Dovecot ii dovecot-imapd 2:2.0.17-0~auto+4 secure IMAP server that supports mbox, maild ii dovecot-managesieved 2:2.0.17-0~auto+4 secure ManageSieve server for Dovecot ii dovecot-mysql 2:2.0.17-0~auto+4 MySQL support for Dovecot ii dovecot-pgsql 2:2.0.17-0~auto+4 PostgreSQL support for Dovecot ii dovecot-pop3d 2:2.0.17-0~auto+4 secure POP3 server that supports mbox, maild ii dovecot-sieve 2:2.0.17-0~auto+4 sieve filters support for Dovecot ii dovecot-sqlite 2:2.0.17-0~auto+4 SQLite support for Dovecot
# doveconf -n # 2.0.13 (02d97fb66047): /etc/dovecot/dovecot.conf # OS: Linux 3.0.28-vs2.3.2.3-rol-em64t x86_64 Debian 6.0.2 ext4 auth_cache_negative_ttl = 0 auth_cache_size = 40 M auth_cache_ttl = 12 hours auth_mechanisms = plain login auth_username_format = %Lu auth_verbose = yes deliver_log_format = msgid=%m: %$ %p %w disable_plaintext_auth = no login_trusted_networks = 195.254.252.207 195.254.252.208 2a02:2e8:0:2:0:143:0:1 2a02:2e8:0:2:0:143:0:2 mail_gid = mailstore mail_location = mdbox:/home/vmail/%d/%n:INDEX=/home/dovecotindex/%d/%n mail_plugins = " quota zlib" mail_uid = mailstore managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave imapflags notify mdbox_rotate_size = 5 M passdb { args = /etc/dovecot/dovecot-sql-account.conf driver = sql } plugin { mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid box msgid size from mail_log_group_events = no quota = dict:Storage used::file:%h/dovecot-quota sieve = ~/.dovecot.sieve sieve_dir = ~/sieve sieve_extensions = +notify +imapflags sieve_max_redirects = 10 zlib_save = gz zlib_save_level = 5 } protocols = imap pop3 lmtp sieve service auth-worker { process_min_avail = 12 service_count = 0 vsz_limit = 512 M } service imap-login { inet_listener imap { port = 143 } process_min_avail = 12 service_count = 0 vsz_limit = 256 M } service imap { process_min_avail = 12 service_count = 0 vsz_limit = 512 M } service lmtp { inet_listener lmtp { address = * port = 24 } unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0666 user = postfix } vsz_limit = 512 M } service pop3-login { inet_listener pop3 { port = 110 } process_min_avail = 12 service_count = 0 vsz_limit = 256 M } service pop3 { service_count = 0 vsz_limit = 256 M } ssl = no ssl_cert =
Thanks and regards Urban Loesch