[Dovecot] NFS performance and 1.1.3 - what can you (unofficially) get away with?
Hi,
Does anyone have experience with bending the NFS recommendations to get better performance?
The question is has anyone, with Maildir and the INDEX= on NFS (i.e. dovecot.index and dovecot.index.cache, set mail_nfs_index to no. If so, was it better to turn maildir_copy_preserve_filename to on or did it help to turn if off. The other question is did you play with turning off atime updates or changing the acmin* values?
I know that the recommendation is do not do this. Fortunately Timo's screams won't reach my ears for another ten hours. I'm just interested if anyone has done this. You are most welcome to reply offline.
I figure that the worst that can happen is that the dovecot.index.cache file will become corrupt, and dovecot will then rebuild it. As long as this doesn't happen all the time, it should not be a big deal. I'm guessing that this is less likely maildir_copy_preserve_filename turned on. My old (courier) servers is that I was able to get away with disabling atime updates and bumping up acmin* but then there was no central index to check.
I admit that this is a nutty way to go since part of the whole point of 1.1.X is its NFS settings. Yet this is a place with some pretty nutty users. (i.e. 2+GB inboxes, 100K messages, one/minute connections, ...)
---Jack
P.S. As a total aside people migrating from 1.0.X to 1.1.3 may wish to have to use a different INDEX directory in 1.1.3 than the one you used in 1.0.7. Doing this got rid of my index.cache and transaction log error messages. This may be why testing isn't showing anything.
P.P.S. I've tried imapproxy for my webmail but it doesn't seem to have made much of a difference on the INDEX I/O.
On Wed, 2008-09-10 at 10:58 -0700, Jack Stewart wrote:
The question is has anyone, with Maildir and the INDEX= on NFS (i.e. dovecot.index and dovecot.index.cache, set mail_nfs_index to no.
How much worse is the mail_nfs_index=yes? Last I heard it made hardly a difference.
If so, was it better to turn maildir_copy_preserve_filename to on or did it help to turn if off.
That shouldn't make a noticeable performance difference. If it's "yes" it does one more uidlist lookup, but pretty much always the uidlist is already in memory in any case so it doesn't matter.
The other question is did you play with turning off atime updates or changing the acmin* values?
Dovecot doesn't care about atimes, so feel free to disable them.
I figure that the worst that can happen is that the dovecot.index.cache file will become corrupt, and dovecot will then rebuild it.
It's not the worst that can happen, but index file errors are probably more likely than other errors..
Timo Sirainen wrote:
I figure that the worst that can happen is that the dovecot.index.cache file will become corrupt, and dovecot will then rebuild it.
It's not the worst that can happen, but index file errors are probably more likely than other errors..
I concur! I currently run one backend NFS server with Dovecot installed on it. In the process of testing adding another dovecot front-end server, mounting homedirs via NFS, I managed to delete my entire Inbox!
How? Dovecot on box1 was using local access, and box2 was using NFS.
I connected to the same account via both dovecots (imap) at the same
time. On box 2 I read my mail and deleted a couple just fine. Then went
back to box 1, deleted another and the expunge wiped everything out.
Lucky for me I do zfs snapshots nightly and this was right away in the morning.
I now have a local NFS mount on box1 to allow for box2 (It's FreeBSD, so it still doesn't work right, but at least it errors out 'gracefully' now).
Rick
The performance hit was bad. When I tried "mail_nfs_index = yes" the load went from 0.5 to 120+ (on each of three servers).
My RPM of 1.1.3 includes the Redhat patches from their source RPM for 1.0.7. I'm checking the patches now and most seem benign but there is some mbox locking changes. I'm using maildir so it shouldn't make a difference yet I shouldn't need the patches either (using the source RPM was a management recommendation).
"mail_nfs_storage" is set to yes. I forgot to put in dovecot -n so it is below.
Right now I'm running with local index.cache files for each servers (using sticky connections) and that seems to work fine.
---Jack
Timo Sirainen wrote:
On Wed, 2008-09-10 at 10:58 -0700, Jack Stewart wrote:
The question is has anyone, with Maildir and the INDEX= on NFS (i.e. dovecot.index and dovecot.index.cache, set mail_nfs_index to no.
How much worse is the mail_nfs_index=yes? Last I heard it made hardly a difference.
If so, was it better to turn maildir_copy_preserve_filename to on or did it help to turn if off.
That shouldn't make a noticeable performance difference. If it's "yes" it does one more uidlist lookup, but pretty much always the uidlist is already in memory in any case so it doesn't matter.
The other question is did you play with turning off atime updates or changing the acmin* values?
Dovecot doesn't care about atimes, so feel free to disable them.
I figure that the worst that can happen is that the dovecot.index.cache file will become corrupt, and dovecot will then rebuild it.
It's not the worst that can happen, but index file errors are probably more likely than other errors..
/opt/dovecot/sbin/dovecot -n -c /etc/dovecot.conf # 1.1.3: /etc/dovecot.conf Warning: fd limit 1024 is lower than what Dovecot can use under full load (more than 8192). Either grow the limit or change login_max_processes_count and max_mail_processes settings base_dir: /var/run/dovecot/default/ syslog_facility: local4 listen(default): *:143 listen(imap): *:143 listen(pop3): *:110 ssl_listen(default): *:993 ssl_listen(imap): *:993 ssl_listen(pop3): *:995 ssl_ca_file: /etc/pki/dovecot/certs/caltech-ca.pem ssl_cert_file: /etc/pki/dovecot/certs/imap.caltech.edu.pem ssl_key_file: /etc/pki/dovecot/private/imap.caltech.edu.key login_dir: /var/run/dovecot/default//login login_executable(default): /opt/dovecot/libexec/dovecot/imap-login login_executable(imap): /opt/dovecot/libexec/dovecot/imap-login login_executable(pop3): /opt/dovecot/libexec/dovecot/pop3-login login_greeting_capability: yes login_processes_count: 16 login_max_processes_count: 2048 max_mail_processes: 4096 mail_max_userip_connections: 2000 verbose_proctitle: yes mail_location: maildir:/var/spool/maildir/%1Ln/%Ln:INDEX=/var/spool/indexes/%1Ln/%Ln:CONTROL=/var/spool/dovecot/uidl/imap/%1Ln/%Ln mail_debug: yes mmap_disable: yes dotlock_use_excl: no mail_nfs_storage: yes mail_executable(default): /opt/dovecot/libexec/dovecot/imap mail_executable(imap): /opt/dovecot/libexec/dovecot/imap mail_executable(pop3): /opt/dovecot/libexec/dovecot/pop3 mail_plugins(default): fts fts_squat mail_plugins(imap): fts fts_squat mail_plugins(pop3): mail_plugin_dir(default): /opt/dovecot/lib/dovecot/imap mail_plugin_dir(imap): /opt/dovecot/lib/dovecot/imap mail_plugin_dir(pop3): /opt/dovecot/lib/dovecot/pop3 imap_client_workarounds(default): delay-newmail outlook-idle netscape-eoh imap_client_workarounds(imap): delay-newmail outlook-idle netscape-eoh imap_client_workarounds(pop3): pop3_uidl_format(default): %v-%u pop3_uidl_format(imap): %v-%u pop3_uidl_format(pop3): %08Xu%08Xv pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh namespace: type: private separator: . prefix: Mail. inbox: yes list: yes subscriptions: yes auth default: mechanisms: plain login passdb: driver: ldap args: /etc/dovecot.conf-ldap userdb: driver: static args: uid=vmail gid=mail home=/var/spool/maildir/%1Ln/%Ln socket: type: listen master: path: /var/run/dovecot/default/auth-master mode: 384 user: vmail group: mail plugin: fts: squat fts_squat: partial=4 full=8
On Wed, 2008-09-10 at 12:20 -0700, Jack Stewart wrote:
The performance hit was bad. When I tried "mail_nfs_index = yes" the load went from 0.5 to 120+ (on each of three servers).
It shouldn't have been anything that bad. I could send you some patches that reduce what mail_nfs_index=yes does. It would be interesting to know if the problem is just one or few specific calls or all of them cumulatively.
dotlock_use_excl: no
"yes" should be safe (and faster) here.
Thanks. I would like to give the patches a try.
I've removed all of the redhat patches except for the one that tells dovecot where to find the certs (why does redhat use pki? why?) but I haven't installed this version yet. Are there any build/configure twiddles that might help?
Just as an update, I tried 1.1.3 with the nfs settings below. At first, it was horrid. Then I deleted the old index cache files. Now it seems to be working well (with webmail - major service is not yet moved). The load is under 1.
I've managed to descend into the twilight zone yet again. Just rolling one server and just rolling webmail for now - we'll see where it goes.
---Jack
Timo Sirainen wrote:
On Wed, 2008-09-10 at 12:20 -0700, Jack Stewart wrote:
The performance hit was bad. When I tried "mail_nfs_index = yes" the load went from 0.5 to 120+ (on each of three servers).
It shouldn't have been anything that bad. I could send you some patches that reduce what mail_nfs_index=yes does. It would be interesting to know if the problem is just one or few specific calls or all of them cumulatively.
dotlock_use_excl: no
"yes" should be safe (and faster) here.
-- Jack Stewart jstewart@caltech.edu / 626-395-4690 http://www.imss.caltech.edu
participants (3)
-
Jack Stewart
-
Rick Romero
-
Timo Sirainen