[Dovecot] Dovecot performance issues with many writes

[Andy Robbins]

Actually, some of the information I provided previously was incorrect. We
have four mail servers in our cluster which each have the NFS share
mounted. These four servers handle all SMTP and sending of mail. However,
it was thought that we could boost performance by simply running POP3 and
IMAP directly from the NFS server, since nothing besides Dovecot is
currently running there. So, here is the dovecot -n output from our
IMAP/POP3 server where the file system is local storage and is where we
believe the performance issues are originating.

# dovecot -n
# 1.2.9: /etc/dovecot/dovecot.conf
Warning: fd limit 1024 is lower than what Dovecot can use under full load
(more than 2304). Either grow the limit or change login_max_processes_count
and max_mail_processes settings
# OS: Linux 2.6.32-24-server x86_64 Ubuntu 10.04.1 LTS
protocols: imap pop3
ssl_cert_file: /etc/dovecot/smtp1.domain.com.cert
ssl_key_file: /etc/dovecot/smtp1.domain.com.key
disable_plaintext_auth: no
login_dir: /var/run/dovecot/login
login_executable(default): /usr/lib/dovecot/imap-login
login_executable(imap): /usr/lib/dovecot/imap-login
login_executable(pop3): /usr/lib/dovecot/pop3-login
login_greeting: Welcome
login_process_per_connection: no
login_processes_count: 2
max_mail_processes: 2048
mail_max_userip_connections: 50
mail_uid: 500
mail_gid: 500
mail_location: maildir:~/Maildir
fsync_disable: yes
maildir_very_dirty_syncs: yes
mbox_write_locks: fcntl dotlock
mail_executable(default): /etc/dovecot/last-login-imap.sh
mail_executable(imap): /etc/dovecot/last-login-imap.sh
mail_executable(pop3): /etc/dovecot/last-login-pop3.sh
mail_plugins: quota
mail_plugin_dir(default): /usr/lib/dovecot/modules/imap
mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap
mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3
pop3_uidl_format: %f
auth default:
  mechanisms: PLAIN LOGIN CRAM-MD5 DIGEST-MD5 APOP
  username_translation: %@
  verbose: yes
  passdb:
    driver: ldap
    args: /etc/dovecot/dovecot-ldap.conf
  userdb:
    driver: ldap
    args: /etc/dovecot/dovecot-ldap.conf
plugin:
  quota: maildir:User quota
  quota_rule: *:storage=0

So you can see that we have already tried a number of performance
modifications such as setting fsync_disable: yes with little or no effect.
If I run nfsstat on the email servers where the volumes are mounted and
only running SMTP I get:

# nfsstat
Client rpc stats:
calls      retrans    authrefrsh
51762532   72668      0

Client nfs v3:
null         getattr      setattr      lookup       access
readlink
0         0% 9639844  18% 3846547   7% 10495207 20% 11930633 23% 8
0%
read         write        create       mkdir        symlink
mknod
7011062  13% 5372278  10% 1593131   3% 95        0% 0         0% 0
0%
remove       rmdir        rename       link         readdir
readdirplus
830662    1% 0         0% 865484    1% 2576      0% 110845    0% 24963
0%
fsstat       fsinfo       pathconf     commit
2074      0% 8         0% 0         0% 37113     0%

My apologies for the somewhat garbled output, there, but I it seems to me
that we're not witnessing the same higher IO writes as compared to reads.

Thanks for your help so far. Hopefully, this sheds more light on your
previous questions.



On Thu, Nov 17, 2011 at 4:42 PM, Timo Sirainen <tss at iki.fi> wrote:

> Do you have more than one Dovecot server? If only one, you can still
> disable the mail_nfs_* settings. Also it would be interesting to see
> nfsstat numbers from the Dovecot server, compared to those iostat numbers..
>
> On 18.11.2011, at 0.38, Andy Robbins wrote:
>
> > Well, the iostat command was run from the NFS server and dovecot was run
> from the mail server where it is mounted, hence the discrepancy there.
> >
> >
> > On Thu, Nov 17, 2011 at 4:34 PM, Timo Sirainen <tss at iki.fi> wrote:
> > On 18.11.2011, at 0.20, Andy Robbins wrote:
> >
> > > # iostat -d 5 -x
> > > Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s
> avgrq-sz
> > > avgqu-sz   await  svctm  %util
> > > dm-4              0.00     0.00  485.80  865.80  3886.40  6926.40
> > > 8.00    28.69   19.63   0.70  94.00
> > > drbd0             0.00     0.00  485.80  865.80  3886.40  6926.40
> > > 8.00   111.42   94.44   0.74  99.60
> >
> > These devices don't look like NFS, but you have:
> >
> > > mmap_disable: yes
> > > mail_nfs_storage: yes
> > > mail_nfs_index: yes
> >
> > So what filesystem is that? Are they storing only mails, or other stuff
> too (like logs)? The mail_nfs_*=yes are probably not needed and are slowing
> things up at least somewhat..
> >
> > > lda:
> > >  fsync_disable: no
> >
> > This is explicitly set for LDA, but it's the default already elsewhere.
> Maybe you wanted to try fsync_disable=yes elsewhere except in LDA?
> >
> >
> >
>
>