[Dovecot] Dovecot performance issues with many writes
We are currently experiencing performance issue with our Dovecot system which we believe is caused by excessive writes to the dovecot files. The confusing thing is that we are seeing more writes than reads on our Dovecot volume when you would assume that most of the IO should be reads from customers checking their mail.
We're seeing reads vs. writes similar to the following:
# iostat -d 5 -x Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util dm-4 0.00 0.00 485.80 865.80 3886.40 6926.40 8.00 28.69 19.63 0.70 94.00 drbd0 0.00 0.00 485.80 865.80 3886.40 6926.40 8.00 111.42 94.44 0.74 99.60
We can't figure out why we would be seeing twice as many writes as we are reads. Could this have something to do with the index files?
Appreciate any assistance on this issue.
# dovecot -n # 1.2.9: //etc/dovecot.conf # OS: Linux 2.6.18-164.9.1.el5 x86_64 CentOS release 5.4 (Final) base_dir: /var/run/dovecot/ log_path: /var/log/dovecot.log protocols: imap pop3 ssl_cert_file: /etc/postfix/ssl/smtp1.domain.com.cert ssl_key_file: /etc/postfix/ssl/smtp1.domain.com.key disable_plaintext_auth: no login_dir: /var/run/dovecot//login login_executable(default): //libexec/dovecot/imap-login login_executable(imap): //libexec/dovecot/imap-login login_executable(pop3): //libexec/dovecot/pop3-login login_user: postfix login_greeting: Welcome login_process_per_connection: no login_processes_count: 2 mail_max_userip_connections: 50 mail_uid: 500 mail_gid: 500 mail_location: maildir:~/Maildir mail_debug: yes mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes mail_executable(default): /etc/dovecot/last-login-imap.sh mail_executable(imap): /etc/dovecot/last-login-imap.sh mail_executable(pop3): /etc/dovecot/last-login-pop3.sh mail_plugins: quota mail_plugin_dir(default): //lib/dovecot/imap mail_plugin_dir(imap): //lib/dovecot/imap mail_plugin_dir(pop3): //lib/dovecot/pop3 pop3_uidl_format: %f lda: postmaster_address: postmaster@domain.com auth_socket_path: /var/run/dovecot/auth-master log_path: /var/log/dovecot-deliver.log info_log_path: /var/log/dovecot-deliver.log fsync_disable: no auth default: mechanisms: PLAIN LOGIN CRAM-MD5 DIGEST-MD5 APOP user: vmail username_translation: %@ verbose: yes passdb: driver: ldap args: /etc/dovecot/dovecot-ldap.conf userdb: driver: ldap args: /etc/dovecot/dovecot-ldap.conf socket: type: listen client: path: /var/spool/postfix/private/auth mode: 432 user: postfix group: postfix master: path: /var/run/dovecot/auth-master mode: 384 user: vmail group: vmail plugin: quota: maildir:User quota quota_rule: *:storage=0
On 18.11.2011, at 0.20, Andy Robbins wrote:
# iostat -d 5 -x Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util dm-4 0.00 0.00 485.80 865.80 3886.40 6926.40 8.00 28.69 19.63 0.70 94.00 drbd0 0.00 0.00 485.80 865.80 3886.40 6926.40 8.00 111.42 94.44 0.74 99.60
These devices don't look like NFS, but you have:
mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes
So what filesystem is that? Are they storing only mails, or other stuff too (like logs)? The mail_nfs_*=yes are probably not needed and are slowing things up at least somewhat..
lda: fsync_disable: no
This is explicitly set for LDA, but it's the default already elsewhere. Maybe you wanted to try fsync_disable=yes elsewhere except in LDA?
Well, the iostat command was run from the NFS server and dovecot was run from the mail server where it is mounted, hence the discrepancy there.
On Thu, Nov 17, 2011 at 4:34 PM, Timo Sirainen tss@iki.fi wrote:
On 18.11.2011, at 0.20, Andy Robbins wrote:
# iostat -d 5 -x Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util dm-4 0.00 0.00 485.80 865.80 3886.40 6926.40 8.00 28.69 19.63 0.70 94.00 drbd0 0.00 0.00 485.80 865.80 3886.40 6926.40 8.00 111.42 94.44 0.74 99.60
These devices don't look like NFS, but you have:
mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes
So what filesystem is that? Are they storing only mails, or other stuff too (like logs)? The mail_nfs_*=yes are probably not needed and are slowing things up at least somewhat..
lda: fsync_disable: no
This is explicitly set for LDA, but it's the default already elsewhere. Maybe you wanted to try fsync_disable=yes elsewhere except in LDA?
Do you have more than one Dovecot server? If only one, you can still disable the mail_nfs_* settings. Also it would be interesting to see nfsstat numbers from the Dovecot server, compared to those iostat numbers..
On 18.11.2011, at 0.38, Andy Robbins wrote:
Well, the iostat command was run from the NFS server and dovecot was run from the mail server where it is mounted, hence the discrepancy there.
On Thu, Nov 17, 2011 at 4:34 PM, Timo Sirainen tss@iki.fi wrote: On 18.11.2011, at 0.20, Andy Robbins wrote:
# iostat -d 5 -x Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util dm-4 0.00 0.00 485.80 865.80 3886.40 6926.40 8.00 28.69 19.63 0.70 94.00 drbd0 0.00 0.00 485.80 865.80 3886.40 6926.40 8.00 111.42 94.44 0.74 99.60
These devices don't look like NFS, but you have:
mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes
So what filesystem is that? Are they storing only mails, or other stuff too (like logs)? The mail_nfs_*=yes are probably not needed and are slowing things up at least somewhat..
lda: fsync_disable: no
This is explicitly set for LDA, but it's the default already elsewhere. Maybe you wanted to try fsync_disable=yes elsewhere except in LDA?
Actually, some of the information I provided previously was incorrect. We have four mail servers in our cluster which each have the NFS share mounted. These four servers handle all SMTP and sending of mail. However, it was thought that we could boost performance by simply running POP3 and IMAP directly from the NFS server, since nothing besides Dovecot is currently running there. So, here is the dovecot -n output from our IMAP/POP3 server where the file system is local storage and is where we believe the performance issues are originating.
# dovecot -n # 1.2.9: /etc/dovecot/dovecot.conf Warning: fd limit 1024 is lower than what Dovecot can use under full load (more than 2304). Either grow the limit or change login_max_processes_count and max_mail_processes settings # OS: Linux 2.6.32-24-server x86_64 Ubuntu 10.04.1 LTS protocols: imap pop3 ssl_cert_file: /etc/dovecot/smtp1.domain.com.cert ssl_key_file: /etc/dovecot/smtp1.domain.com.key disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable(default): /usr/lib/dovecot/imap-login login_executable(imap): /usr/lib/dovecot/imap-login login_executable(pop3): /usr/lib/dovecot/pop3-login login_greeting: Welcome login_process_per_connection: no login_processes_count: 2 max_mail_processes: 2048 mail_max_userip_connections: 50 mail_uid: 500 mail_gid: 500 mail_location: maildir:~/Maildir fsync_disable: yes maildir_very_dirty_syncs: yes mbox_write_locks: fcntl dotlock mail_executable(default): /etc/dovecot/last-login-imap.sh mail_executable(imap): /etc/dovecot/last-login-imap.sh mail_executable(pop3): /etc/dovecot/last-login-pop3.sh mail_plugins: quota mail_plugin_dir(default): /usr/lib/dovecot/modules/imap mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3 pop3_uidl_format: %f auth default: mechanisms: PLAIN LOGIN CRAM-MD5 DIGEST-MD5 APOP username_translation: %@ verbose: yes passdb: driver: ldap args: /etc/dovecot/dovecot-ldap.conf userdb: driver: ldap args: /etc/dovecot/dovecot-ldap.conf plugin: quota: maildir:User quota quota_rule: *:storage=0
So you can see that we have already tried a number of performance modifications such as setting fsync_disable: yes with little or no effect. If I run nfsstat on the email servers where the volumes are mounted and only running SMTP I get:
# nfsstat Client rpc stats: calls retrans authrefrsh 51762532 72668 0
Client nfs v3: null getattr setattr lookup access readlink 0 0% 9639844 18% 3846547 7% 10495207 20% 11930633 23% 8 0% read write create mkdir symlink mknod 7011062 13% 5372278 10% 1593131 3% 95 0% 0 0% 0 0% remove rmdir rename link readdir readdirplus 830662 1% 0 0% 865484 1% 2576 0% 110845 0% 24963 0% fsstat fsinfo pathconf commit 2074 0% 8 0% 0 0% 37113 0%
My apologies for the somewhat garbled output, there, but I it seems to me that we're not witnessing the same higher IO writes as compared to reads.
Thanks for your help so far. Hopefully, this sheds more light on your previous questions.
On Thu, Nov 17, 2011 at 4:42 PM, Timo Sirainen tss@iki.fi wrote:
Do you have more than one Dovecot server? If only one, you can still disable the mail_nfs_* settings. Also it would be interesting to see nfsstat numbers from the Dovecot server, compared to those iostat numbers..
On 18.11.2011, at 0.38, Andy Robbins wrote:
Well, the iostat command was run from the NFS server and dovecot was run from the mail server where it is mounted, hence the discrepancy there.
On Thu, Nov 17, 2011 at 4:34 PM, Timo Sirainen tss@iki.fi wrote: On 18.11.2011, at 0.20, Andy Robbins wrote:
# iostat -d 5 -x Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util dm-4 0.00 0.00 485.80 865.80 3886.40 6926.40 8.00 28.69 19.63 0.70 94.00 drbd0 0.00 0.00 485.80 865.80 3886.40 6926.40 8.00 111.42 94.44 0.74 99.60
These devices don't look like NFS, but you have:
mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes
So what filesystem is that? Are they storing only mails, or other stuff too (like logs)? The mail_nfs_*=yes are probably not needed and are slowing things up at least somewhat..
lda: fsync_disable: no
This is explicitly set for LDA, but it's the default already elsewhere. Maybe you wanted to try fsync_disable=yes elsewhere except in LDA?
On 11/21/2011 12:34 PM, Andy Robbins wrote:
Warning: fd limit 1024 is lower than what Dovecot can use under full load (more than 2304). Either grow the limit or change login_max_processes_count and max_mail_processes settings
Why haven't you increased the file descriptor limit?
-- Stan
I don't see anything obviously bad in your setup. And since you said you're running Dovecot in the IMAP server, perhaps it's not even Dovecot at all that is causing the writes?.. Here are anyway some thoughts:
Log files? Make sure syslog isn't fsyncing each Dovecot log line to disk (or just set log_path = /var/log/dovecot.log and it definitely won't fsync)
Try if disabling index files lowers writes (I don't see why it would make a big difference): mail_location = maildir:~/Maildir:INDEX=MEMORY
Try if temporarily disabling Maildir++ quota makes any difference.
There aren't really any other potential writers in Dovecot..
On Mon, 2011-11-21 at 12:34 -0600, Andy Robbins wrote:
Actually, some of the information I provided previously was incorrect. We have four mail servers in our cluster which each have the NFS share mounted. These four servers handle all SMTP and sending of mail. However, it was thought that we could boost performance by simply running POP3 and IMAP directly from the NFS server, since nothing besides Dovecot is currently running there. So, here is the dovecot -n output from our IMAP/POP3 server where the file system is local storage and is where we believe the performance issues are originating.
# dovecot -n # 1.2.9: /etc/dovecot/dovecot.conf Warning: fd limit 1024 is lower than what Dovecot can use under full load (more than 2304). Either grow the limit or change login_max_processes_count and max_mail_processes settings # OS: Linux 2.6.32-24-server x86_64 Ubuntu 10.04.1 LTS protocols: imap pop3 ssl_cert_file: /etc/dovecot/smtp1.domain.com.cert ssl_key_file: /etc/dovecot/smtp1.domain.com.key disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable(default): /usr/lib/dovecot/imap-login login_executable(imap): /usr/lib/dovecot/imap-login login_executable(pop3): /usr/lib/dovecot/pop3-login login_greeting: Welcome login_process_per_connection: no login_processes_count: 2 max_mail_processes: 2048 mail_max_userip_connections: 50 mail_uid: 500 mail_gid: 500 mail_location: maildir:~/Maildir fsync_disable: yes maildir_very_dirty_syncs: yes mbox_write_locks: fcntl dotlock mail_executable(default): /etc/dovecot/last-login-imap.sh mail_executable(imap): /etc/dovecot/last-login-imap.sh mail_executable(pop3): /etc/dovecot/last-login-pop3.sh mail_plugins: quota mail_plugin_dir(default): /usr/lib/dovecot/modules/imap mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3 pop3_uidl_format: %f auth default: mechanisms: PLAIN LOGIN CRAM-MD5 DIGEST-MD5 APOP username_translation: %@ verbose: yes passdb: driver: ldap args: /etc/dovecot/dovecot-ldap.conf userdb: driver: ldap args: /etc/dovecot/dovecot-ldap.conf plugin: quota: maildir:User quota quota_rule: *:storage=0
So you can see that we have already tried a number of performance modifications such as setting fsync_disable: yes with little or no effect. If I run nfsstat on the email servers where the volumes are mounted and only running SMTP I get:
# nfsstat Client rpc stats: calls retrans authrefrsh 51762532 72668 0
Client nfs v3: null getattr setattr lookup access readlink 0 0% 9639844 18% 3846547 7% 10495207 20% 11930633 23% 8 0% read write create mkdir symlink mknod 7011062 13% 5372278 10% 1593131 3% 95 0% 0 0% 0 0% remove rmdir rename link readdir readdirplus 830662 1% 0 0% 865484 1% 2576 0% 110845 0% 24963 0% fsstat fsinfo pathconf commit 2074 0% 8 0% 0 0% 37113 0%
My apologies for the somewhat garbled output, there, but I it seems to me that we're not witnessing the same higher IO writes as compared to reads.
Thanks for your help so far. Hopefully, this sheds more light on your previous questions.
On Thu, Nov 17, 2011 at 4:42 PM, Timo Sirainen tss@iki.fi wrote:
Do you have more than one Dovecot server? If only one, you can still disable the mail_nfs_* settings. Also it would be interesting to see nfsstat numbers from the Dovecot server, compared to those iostat numbers..
On 18.11.2011, at 0.38, Andy Robbins wrote:
Well, the iostat command was run from the NFS server and dovecot was run from the mail server where it is mounted, hence the discrepancy there.
On Thu, Nov 17, 2011 at 4:34 PM, Timo Sirainen tss@iki.fi wrote: On 18.11.2011, at 0.20, Andy Robbins wrote:
# iostat -d 5 -x Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util dm-4 0.00 0.00 485.80 865.80 3886.40 6926.40 8.00 28.69 19.63 0.70 94.00 drbd0 0.00 0.00 485.80 865.80 3886.40 6926.40 8.00 111.42 94.44 0.74 99.60
These devices don't look like NFS, but you have:
mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes
So what filesystem is that? Are they storing only mails, or other stuff too (like logs)? The mail_nfs_*=yes are probably not needed and are slowing things up at least somewhat..
lda: fsync_disable: no
This is explicitly set for LDA, but it's the default already elsewhere. Maybe you wanted to try fsync_disable=yes elsewhere except in LDA?
On Thu, Nov 17, 2011 at 03:30:07PM -0600, Andy Robbins wrote:
We are currently experiencing performance issue with our Dovecot system which we believe is caused by excessive writes to the dovecot files. The confusing thing is that we are seeing more writes than reads on our Dovecot volume when you would assume that most of the IO should be reads from customers checking their mail.
Have you mounted the fs without noatime options?
-jf
Am Donnerstag, 17. November 2011, 22:30:07 schrieb Andy Robbins:
We are currently experiencing performance issue with our Dovecot system which we believe is caused by excessive writes to the dovecot files. The confusing thing is that we are seeing more writes than reads on our Dovecot volume when you would assume that most of the IO should be reads from customers checking their mail.
Yes, but your read-requests are answered from your local kernel-buffer / cache and aren't really read from your hard-disk.
(At least if you have enough free memory for a cache, that's large enought.)
Incoming mails ARE written on the disk, but read later on just from your cache.
Peer
--
Heinlein Professional Linux Support GmbH Linux: Akademie - Support - Hosting
http://www.heinlein-support.de Tel: 030 / 40 50 51 - 0 Fax: 030 / 40 50 51 - 19
Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin
participants (5)
-
Andy Robbins
-
Jan-Frode Myklebust
-
Peer Heinlein
-
Stan Hoeppner
-
Timo Sirainen