[Dovecot] Dovecot performance issues with many writes

Mon Nov 21 23:41:57 EET 2011

I don't see anything obviously bad in your setup. And since you said
you're running Dovecot in the IMAP server, perhaps it's not even Dovecot
at all that is causing the writes?.. Here are anyway some thoughts:

 * Log files? Make sure syslog isn't fsyncing each Dovecot log line to
disk (or just set log_path = /var/log/dovecot.log and it definitely
won't fsync)

 * Try if disabling index files lowers writes (I don't see why it would
make a big difference): mail_location = maildir:~/Maildir:INDEX=MEMORY

 * Try if temporarily disabling Maildir++ quota makes any difference.

There aren't really any other potential writers in Dovecot..

On Mon, 2011-11-21 at 12:34 -0600, Andy Robbins wrote:
> Actually, some of the information I provided previously was incorrect. We
> have four mail servers in our cluster which each have the NFS share
> mounted. These four servers handle all SMTP and sending of mail. However,
> it was thought that we could boost performance by simply running POP3 and
> IMAP directly from the NFS server, since nothing besides Dovecot is
> currently running there. So, here is the dovecot -n output from our
> IMAP/POP3 server where the file system is local storage and is where we
> believe the performance issues are originating.
> 
> # dovecot -n
> # 1.2.9: /etc/dovecot/dovecot.conf
> Warning: fd limit 1024 is lower than what Dovecot can use under full load
> (more than 2304). Either grow the limit or change login_max_processes_count
> and max_mail_processes settings
> # OS: Linux 2.6.32-24-server x86_64 Ubuntu 10.04.1 LTS
> protocols: imap pop3
> ssl_cert_file: /etc/dovecot/smtp1.domain.com.cert
> ssl_key_file: /etc/dovecot/smtp1.domain.com.key
> disable_plaintext_auth: no
> login_dir: /var/run/dovecot/login
> login_executable(default): /usr/lib/dovecot/imap-login
> login_executable(imap): /usr/lib/dovecot/imap-login
> login_executable(pop3): /usr/lib/dovecot/pop3-login
> login_greeting: Welcome
> login_process_per_connection: no
> login_processes_count: 2
> max_mail_processes: 2048
> mail_max_userip_connections: 50
> mail_uid: 500
> mail_gid: 500
> mail_location: maildir:~/Maildir
> fsync_disable: yes
> maildir_very_dirty_syncs: yes
> mbox_write_locks: fcntl dotlock
> mail_executable(default): /etc/dovecot/last-login-imap.sh
> mail_executable(imap): /etc/dovecot/last-login-imap.sh
> mail_executable(pop3): /etc/dovecot/last-login-pop3.sh
> mail_plugins: quota
> mail_plugin_dir(default): /usr/lib/dovecot/modules/imap
> mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap
> mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3
> pop3_uidl_format: %f
> auth default:
>   mechanisms: PLAIN LOGIN CRAM-MD5 DIGEST-MD5 APOP
>   username_translation: %@
>   verbose: yes
>   passdb:
>     driver: ldap
>     args: /etc/dovecot/dovecot-ldap.conf
>   userdb:
>     driver: ldap
>     args: /etc/dovecot/dovecot-ldap.conf
> plugin:
>   quota: maildir:User quota
>   quota_rule: *:storage=0
> 
> So you can see that we have already tried a number of performance
> modifications such as setting fsync_disable: yes with little or no effect.
> If I run nfsstat on the email servers where the volumes are mounted and
> only running SMTP I get:
> 
> # nfsstat
> Client rpc stats:
> calls      retrans    authrefrsh
> 51762532   72668      0
> 
> Client nfs v3:
> null         getattr      setattr      lookup       access
> readlink
> 0         0% 9639844  18% 3846547   7% 10495207 20% 11930633 23% 8
> 0%
> read         write        create       mkdir        symlink
> mknod
> 7011062  13% 5372278  10% 1593131   3% 95        0% 0         0% 0
> 0%
> remove       rmdir        rename       link         readdir
> readdirplus
> 830662    1% 0         0% 865484    1% 2576      0% 110845    0% 24963
> 0%
> fsstat       fsinfo       pathconf     commit
> 2074      0% 8         0% 0         0% 37113     0%
> 
> My apologies for the somewhat garbled output, there, but I it seems to me
> that we're not witnessing the same higher IO writes as compared to reads.
> 
> Thanks for your help so far. Hopefully, this sheds more light on your
> previous questions.
> 
> 
> 
> On Thu, Nov 17, 2011 at 4:42 PM, Timo Sirainen <tss at iki.fi> wrote:
> 
> > Do you have more than one Dovecot server? If only one, you can still
> > disable the mail_nfs_* settings. Also it would be interesting to see
> > nfsstat numbers from the Dovecot server, compared to those iostat numbers..
> >
> > On 18.11.2011, at 0.38, Andy Robbins wrote:
> >
> > > Well, the iostat command was run from the NFS server and dovecot was run
> > from the mail server where it is mounted, hence the discrepancy there.
> > >
> > >
> > > On Thu, Nov 17, 2011 at 4:34 PM, Timo Sirainen <tss at iki.fi> wrote:
> > > On 18.11.2011, at 0.20, Andy Robbins wrote:
> > >
> > > > # iostat -d 5 -x
> > > > Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s
> > avgrq-sz
> > > > avgqu-sz   await  svctm  %util
> > > > dm-4              0.00     0.00  485.80  865.80  3886.40  6926.40
> > > > 8.00    28.69   19.63   0.70  94.00
> > > > drbd0             0.00     0.00  485.80  865.80  3886.40  6926.40
> > > > 8.00   111.42   94.44   0.74  99.60
> > >
> > > These devices don't look like NFS, but you have:
> > >
> > > > mmap_disable: yes
> > > > mail_nfs_storage: yes
> > > > mail_nfs_index: yes
> > >
> > > So what filesystem is that? Are they storing only mails, or other stuff
> > too (like logs)? The mail_nfs_*=yes are probably not needed and are slowing
> > things up at least somewhat..
> > >
> > > > lda:
> > > >  fsync_disable: no
> > >
> > > This is explicitly set for LDA, but it's the default already elsewhere.
> > Maybe you wanted to try fsync_disable=yes elsewhere except in LDA?
> > >
> > >
> > >
> >
> >