Actually, some of the information I provided previously was incorrect. We have four mail servers in our cluster which each have the NFS share mounted. These four servers handle all SMTP and sending of mail. However, it was thought that we could boost performance by simply running POP3 and IMAP directly from the NFS server, since nothing besides Dovecot is currently running there. So, here is the dovecot -n output from our IMAP/POP3 server where the file system is local storage and is where we believe the performance issues are originating.
# dovecot -n # 1.2.9: /etc/dovecot/dovecot.conf Warning: fd limit 1024 is lower than what Dovecot can use under full load (more than 2304). Either grow the limit or change login_max_processes_count and max_mail_processes settings # OS: Linux 2.6.32-24-server x86_64 Ubuntu 10.04.1 LTS protocols: imap pop3 ssl_cert_file: /etc/dovecot/smtp1.domain.com.cert ssl_key_file: /etc/dovecot/smtp1.domain.com.key disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable(default): /usr/lib/dovecot/imap-login login_executable(imap): /usr/lib/dovecot/imap-login login_executable(pop3): /usr/lib/dovecot/pop3-login login_greeting: Welcome login_process_per_connection: no login_processes_count: 2 max_mail_processes: 2048 mail_max_userip_connections: 50 mail_uid: 500 mail_gid: 500 mail_location: maildir:~/Maildir fsync_disable: yes maildir_very_dirty_syncs: yes mbox_write_locks: fcntl dotlock mail_executable(default): /etc/dovecot/last-login-imap.sh mail_executable(imap): /etc/dovecot/last-login-imap.sh mail_executable(pop3): /etc/dovecot/last-login-pop3.sh mail_plugins: quota mail_plugin_dir(default): /usr/lib/dovecot/modules/imap mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3 pop3_uidl_format: %f auth default: mechanisms: PLAIN LOGIN CRAM-MD5 DIGEST-MD5 APOP username_translation: %@ verbose: yes passdb: driver: ldap args: /etc/dovecot/dovecot-ldap.conf userdb: driver: ldap args: /etc/dovecot/dovecot-ldap.conf plugin: quota: maildir:User quota quota_rule: *:storage=0
So you can see that we have already tried a number of performance modifications such as setting fsync_disable: yes with little or no effect. If I run nfsstat on the email servers where the volumes are mounted and only running SMTP I get:
# nfsstat Client rpc stats: calls retrans authrefrsh 51762532 72668 0
Client nfs v3: null getattr setattr lookup access readlink 0 0% 9639844 18% 3846547 7% 10495207 20% 11930633 23% 8 0% read write create mkdir symlink mknod 7011062 13% 5372278 10% 1593131 3% 95 0% 0 0% 0 0% remove rmdir rename link readdir readdirplus 830662 1% 0 0% 865484 1% 2576 0% 110845 0% 24963 0% fsstat fsinfo pathconf commit 2074 0% 8 0% 0 0% 37113 0%
My apologies for the somewhat garbled output, there, but I it seems to me that we're not witnessing the same higher IO writes as compared to reads.
Thanks for your help so far. Hopefully, this sheds more light on your previous questions.
On Thu, Nov 17, 2011 at 4:42 PM, Timo Sirainen tss@iki.fi wrote:
Do you have more than one Dovecot server? If only one, you can still disable the mail_nfs_* settings. Also it would be interesting to see nfsstat numbers from the Dovecot server, compared to those iostat numbers..
On 18.11.2011, at 0.38, Andy Robbins wrote:
Well, the iostat command was run from the NFS server and dovecot was run from the mail server where it is mounted, hence the discrepancy there.
On Thu, Nov 17, 2011 at 4:34 PM, Timo Sirainen tss@iki.fi wrote: On 18.11.2011, at 0.20, Andy Robbins wrote:
# iostat -d 5 -x Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util dm-4 0.00 0.00 485.80 865.80 3886.40 6926.40 8.00 28.69 19.63 0.70 94.00 drbd0 0.00 0.00 485.80 865.80 3886.40 6926.40 8.00 111.42 94.44 0.74 99.60
These devices don't look like NFS, but you have:
mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes
So what filesystem is that? Are they storing only mails, or other stuff too (like logs)? The mail_nfs_*=yes are probably not needed and are slowing things up at least somewhat..
lda: fsync_disable: no
This is explicitly set for LDA, but it's the default already elsewhere. Maybe you wanted to try fsync_disable=yes elsewhere except in LDA?