[Dovecot] Dotlock dovecot-uidlist errors / NFS / High Load

[list at airstreamcomm.net]

As of late our four node dovecot 1.2.13 cluster has been experiencing a
massive number of these dotlock errors:

Created dotlock file's timestamp is different than current time
(1295480202 vs 1295479784): /mail/user/Maildir/dovecot-uidlist

These dotlock errors correspond with very high load averages, and
eventually we have to turn off all but one server to stop them from
occurring.  We first assumed this trend was related to the NFS storage, but
we could not find a networking issue or NFS related problem to speak of. 
We run the mail storage on NFS which is hosted on a Centos 5.5 host, and
mounted with the following options:

udp,nodev,noexec,nosuid.  

Secondly we thought the issues were due to NTP as the time stamps vary so
widely, so we rebuilt our NTP servers and found closer stratum 1 source
clocks to synchronize to hoping it would alleviate the problem but the
dotlock errors returned after about 12 hours.  We have fcntl locking set in
our configuration file, but it is our understanding from look at the source
code that this file is locked with dotlock.  

Any help troubleshooting is appreciated.

Thanks,

Michael


# 1.2.13: /etc/dovecot.conf
# OS: Linux 2.6.18-194.8.1.el5 x86_64 CentOS release 5.5 (Final) 
protocols: imap pop3
listen(default): *:143
listen(imap): *:143
listen(pop3): *:110
shutdown_clients: no
login_dir: /var/run/dovecot/login
login_executable(default): /usr/libexec/dovecot/imap-login
login_executable(imap): /usr/libexec/dovecot/imap-login
login_executable(pop3): /usr/libexec/dovecot/pop3-login
login_process_per_connection: no
login_process_size: 128
login_processes_count: 4
login_max_processes_count: 256
login_max_connections: 386
first_valid_uid: 300
mail_location: maildir:~/Maildir
mmap_disable: yes
dotlock_use_excl: no
mail_nfs_storage: yes
mail_nfs_index: yes
mail_executable(default): /usr/libexec/dovecot/imap
mail_executable(imap): /usr/libexec/dovecot/imap
mail_executable(pop3): /usr/libexec/dovecot/pop3
mail_plugin_dir(default): /usr/lib64/dovecot/imap
mail_plugin_dir(imap): /usr/lib64/dovecot/imap
mail_plugin_dir(pop3): /usr/lib64/dovecot/pop3
auth default:
  username_format: %Ln
  worker_max_count: 50
  passdb:
    driver: pam
  userdb:
    driver: passwd