[Dovecot] NFS cache flush tester
http://dovecot.org/tools/nfstest.c
I wrote a test utility that tests different ways to flush NFS attribute cache and data cache. Please test in your NFS setup and show me the results, so I can make Dovecot v1.1 works well with NFS.
Run the test several times (3 should be enough) to make sure the output stays the same, because the attribute cache flushes might incorrectly say "OK". And of course attribute cache must not be disabled for this test to work.
If you have other ideas how to flush either cache please tell me and I'll update the test utility. I'd like to have a way to flush the cache without having the NFS client send anything to server. Unfortunately all of the below ways do that.
Results for Linux v2.6.15 and v2.6.18:
Info: Testing attribute cache.. Info: Attr cache flush fchown(-1, -1): failed Info: Attr cache flush fchown(uid, gid): OK Info: Attr cache flush fchmod(mode): OK Info: Attr cache flush chown(-1, -1): failed Info: Attr cache flush chown(uid, gid): OK Info: Attr cache flush chmod(mode): OK Info: Testing data cache.. Info: data cache: Appends weren't noticed (ret = 0) Info: - Attribute cache flush helped Info: data cache (no caching): failed Info: data cache (attr cache): failed Info: data cache (lockf()): OK Info: data cache (flock(shared)): OK Info: data cache (flock(exclusive)): OK
On Wed, 11 Jul 2007, Timo Sirainen wrote:
Results for FreeBSD 6.2-RELEASE-p5 clients using a NetApp NFS server:
Info: Connected: client Info: Testing attribute cache.. Info: Attr cache flush fchown(-1, -1): OK Info: Attr cache flush fchown(uid, -1): OK Info: Attr cache flush fchmod(mode): OK Info: Attr cache flush chown(-1, -1): OK Info: Attr cache flush chown(uid, -1): OK Info: Attr cache flush chmod(mode): OK Info: Testing data cache.. Info: data cache: Appends weren't noticed (ret = 0) Info: - Attribute cache flush helped Info: data cache (no caching): failed Info: data cache (attr cache): OK Info: data cache (lockf()): failed Info: data cache (flock(shared)): failed Info: data cache (flock(exclusive)): failed Info: data cache (O_DIRECT): failed
-Doug
On Wed, 2007-07-11 at 22:21 +0300, Timo Sirainen wrote:
I've done several updates for this. Updated results for Linux 2.6:
Info: O_EXCL works Info: Testing attribute cache.. Info: Attr cache flush fchown(-1, -1): failed Info: Attr cache flush fchown(uid, -1): OK Info: Attr cache flush fchmod(mode): OK Info: Attr cache flush chown(-1, -1): failed Info: Attr cache flush chown(uid, -1): OK Info: Attr cache flush chmod(mode): OK Info: Testing write flushing.. Info: Write flush no caching: failed Info: Write flush fcntl(shared): OK Info: Write flush fcntl(exclusive): OK Info: Write flush flock(shared): OK Info: Write flush flock(exclusive): OK Info: Write flush reopen: OK Info: Testing data cache.. Info: data cache: Reading EOF requires attribute cache flush Info: data cache (no caching): failed Info: data cache (attr cache): OK Info: data cache (fcntl(shared)): OK Info: data cache (fcntl(exclusive)): OK Info: data cache (flock(shared)): OK Info: data cache (flock(exclusive)): OK Info: data cache (dotlock): failed Info: data cache (O_DIRECT): OK
On Thu, 12 Jul 2007, Timo Sirainen wrote:
Different result this time:
Info: Connected: client Info: O_EXCL works Info: Testing attribute cache.. Fatal: open(/mnt/nfs/blah) failed: Stale NFS file handle
This is on FreeBSD 6.2-RELEASE-p5 and a NetApp NFS server. I'm wondering if it is the same issue with the NFS client cache that I had to workaround in the past with dotlocking (http://www.dovecot.org/list/dovecot/2007-May/022883.html).
Before I implement the same workaround, I wanted to check if it would invalidate the test results assuming the workaround worked?
-Doug
On Thu, 12 Jul 2007, Timo Sirainen wrote:
Hmm. I updated the nfstest.c to now just retry if this happens. Does it help?
Yes, that worked.
Here are the results for FreeBSD 6.2-RELEASE-p5 and a NetApp NFS server. FYI, the fcntl errors also appeared on the server instance.
Info: Connected: client Info: O_EXCL works Info: Testing attribute cache.. Info: Attr cache flush fchown(-1, -1): OK Info: Attr cache flush fchown(uid, -1): OK Info: Attr cache flush fchmod(mode): OK Info: Attr cache flush chown(-1, -1): OK Info: Attr cache flush chown(uid, -1): OK Info: Attr cache flush chmod(mode): OK Info: Testing write flushing.. Info: Write flush no caching: failed Info: Write flush fcntl(shared): failed Info: Write flush fcntl(exclusive): failed Info: Write flush flock(shared): failed Info: Write flush flock(exclusive): failed Info: Write flush reopen: OK Info: Testing data cache.. Info: data cache: Reading EOF requires attribute cache flush Info: Data cache flush no caching: failed Info: Data cache flush attr cache: OK Error: fcntl(setlk, read) failed: Operation not supported Info: Data cache flush fcntl(shared): failed Error: fcntl(setlk, write) failed: Operation not supported Info: Data cache flush fcntl(exclusive): failed Info: Data cache flush flock(shared): failed Info: Data cache flush flock(exclusive): failed Info: Data cache flush dotlock: failed Info: Data cache flush O_DIRECT: failed
-Doug
On Thu, Jul 12, 2007 at 03:39:21AM +0300, Timo Sirainen wrote:
On Wed, 2007-07-11 at 15:43 -0700, Doug Council wrote:
Would you be able to enable lockd and see what these show then?
Running rpc.lockd and rpc.statd on FreeBSD 6.2-STABLE #17: Sun Jun 24 22:11:00 EDT 2007 with a NetApp filer as the server, ran 3 times and compared output (same):
Info: Connected: client Info: O_EXCL works Info: Testing attribute cache.. Info: Attr cache flush fchown(-1, -1): OK Info: Attr cache flush fchown(uid, -1): OK Info: Attr cache flush fchmod(mode): OK Info: Attr cache flush chown(-1, -1): OK Info: Attr cache flush chown(uid, -1): OK Info: Attr cache flush chmod(mode): OK Info: Testing write flushing.. Info: Write flush no caching: failed Info: Write flush fcntl(shared): failed Info: Write flush fcntl(exclusive): failed Info: Write flush flock(shared): failed Info: Write flush flock(exclusive): failed Info: Write flush reopen: OK Info: Testing data cache.. Info: data cache: Reading EOF requires attribute cache flush Info: Data cache flush no caching: failed Info: Data cache flush attr cache: OK Info: Data cache flush fcntl(shared): failed Info: Data cache flush fcntl(exclusive): failed Info: Data cache flush flock(shared): failed Info: Data cache flush flock(exclusive): failed Info: Data cache flush dotlock: failed Info: Data cache flush O_DIRECT: failed
On Thu, 12 Jul 2007, Timo Sirainen wrote:
The NetApp filer is in production, so I am not able to.
But, Adam has posted his results using rpc.lockd/statd on FreeBSD 6.2 with a NetApp filer in http://www.dovecot.org/list/dovecot/2007-July/024070.html.
-Doug
Timo Sirainen wrote:
What's a good command to compile it? I have a Linux 2.6.20 NFS client connected to a Linux 2.4.34 NFS client.
$ gcc ~/tmp/nfstest.c -I lib In file included from lib/lib.h:24, from /home/users/andrewz/tmp/nfstest.c:17: lib/compat.h:31:3: error: invalid preprocessing directive ## In file included from /usr/include/sys/socket.h:35, from lib/compat.h:80, from lib/lib.h:24, from /home/users/andrewz/tmp/nfstest.c:17: /usr/include/bits/socket.h:36: error: conflicting types for ‘socklen_t’ lib/compat.h:51: error: previous declaration of ‘socklen_t’ was here In file included from /usr/include/netinet/in.h:23, from lib/compat.h:81, from lib/lib.h:24, from /home/users/andrewz/tmp/nfstest.c:17: /usr/include/stdint.h:111: error: conflicting types for ‘uint_fast32_t’ lib/compat.h:46: error: previous declaration of ‘uint_fast32_t’ was here /usr/include/stdint.h:141: error: conflicting types for ‘uintmax_t’ lib/compat.h:38: error: previous declaration of ‘uintmax_t’ was here
Andrew
Timo Sirainen wrote:
** Test 1 Machine 1: Fedora Core 6, Linux 2.6.20-1.2952.fc6 #1 SMP Wed May 16 17:59:13 EDT 2007 i686 athlon i386 GNU/Linux Machine 2: Fedora Core 7, Linux 2.6.21-1.3228.fc7 #1 SMP Tue Jun 12 15:37:31 EDT 2007 i686 i686 i386 GNU/Linux NFS Server: Trustix Secure Linux 2.2, Linux 2.4.34-1tr #1 Thu Jan 4 11:36:05 GMT 2007 i686 i686 i386 GNU/Linux
Because the server is Linux 2.4, that implies the protocol is NFS version 3.
Info: Connected: client Info: O_EXCL works Info: Testing attribute cache.. Info: Attr cache flush fchown(-1, -1): failed Info: Attr cache flush fchown(uid, -1): OK Info: Attr cache flush fchmod(mode): OK Info: Attr cache flush chown(-1, -1): failed Info: Attr cache flush chown(uid, -1): OK Info: Attr cache flush chmod(mode): OK Info: Testing write flushing.. Info: Write flush no caching: failed Info: Write flush fcntl(shared): OK Info: Write flush fcntl(exclusive): OK Info: Write flush flock(shared): OK Info: Write flush flock(exclusive): OK Info: Write flush reopen: OK Info: Testing data cache.. Info: data cache: Reading EOF requires attribute cache flush Info: Data cache flush no caching: failed Info: Data cache flush attr cache: failed Info: Data cache flush fcntl(shared): OK Info: Data cache flush fcntl(exclusive): OK Info: Data cache flush flock(shared): OK Info: Data cache flush flock(exclusive): OK Info: Data cache flush dotlock: failed Info: Data cache flush O_DIRECT: OK
** Test 2 Machines 1 and 2 switched roles.
Info: Connected: client Info: O_EXCL works Info: Testing attribute cache.. Info: Attr cache flush fchown(-1, -1): failed Info: Attr cache flush fchown(uid, -1): OK Info: Attr cache flush fchmod(mode): OK Info: Attr cache flush chown(-1, -1): failed Info: Attr cache flush chown(uid, -1): OK Info: Attr cache flush chmod(mode): OK Info: Testing write flushing.. Info: Write flush no caching: failed Info: Write flush fcntl(shared): OK Info: Write flush fcntl(exclusive): OK Info: Write flush flock(shared): OK Info: Write flush flock(exclusive): OK Info: Write flush reopen: OK Info: Testing data cache.. Info: data cache: Reading EOF requires attribute cache flush Info: Data cache flush no caching: failed Info: Data cache flush attr cache: failed Info: Data cache flush fcntl(shared): OK Info: Data cache flush fcntl(exclusive): OK Info: Data cache flush flock(shared): OK Info: Data cache flush flock(exclusive): OK Info: Data cache flush dotlock: failed Info: Data cache flush O_DIRECT: OK
Andrew
On Wed, 2007-07-11 at 22:12 -0600, Andrew Ziem wrote:
Info: Data cache flush O_DIRECT: OK
I suppose this means that your kernel was compiled with CONFIG_NFS_DIRECTIO=y? Interesting that it is enabled by default in Fedora and Debian. The kernel itself recommends not to use it by default: "This kernel build option defaults OFF to avoid exposing system administrators unwittingly to a potentially hazardous feature."
Timo Sirainen wrote:
Yes, apparently so. I think this is the correct file:
Andrew
Timo Sirainen wrote:
Results for Solaris 8 talking to a NetApp filer (OS: ONTAP 7.2.2p3)
Info: Connected: client Info: O_EXCL works Info: Testing attribute cache.. Info: Attr cache flush fchown(-1, -1): OK Info: Attr cache flush fchown(uid, -1): OK Info: Attr cache flush fchmod(mode): OK Info: Attr cache flush chown(-1, -1): OK Info: Attr cache flush chown(uid, -1): OK Info: Attr cache flush chmod(mode): OK Info: Testing write flushing.. Info: Write flush no caching: failed Info: Write flush fcntl(shared): OK Info: Write flush fcntl(exclusive): OK Info: Write flush flock(shared): failed Info: Write flush flock(exclusive): failed Info: Write flush reopen: OK Info: Testing data cache.. Info: data cache: Reading EOF requires attribute cache flush Info: Data cache flush no caching: failed Info: Data cache flush attr cache: OK Info: Data cache flush fcntl(shared): OK Info: Data cache flush fcntl(exclusive): OK Info: Data cache flush flock(shared): failed Info: Data cache flush flock(exclusive): failed Info: Data cache flush dotlock: failed Info: Data cache flush O_DIRECT: failed
-- --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- Christopher Wakelin, c.d.wakelin@reading.ac.uk IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439 Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094
On Thu, 2007-07-12 at 12:45 +0100, Chris Wakelin wrote:
Results for Solaris 8 talking to a NetApp filer (OS: ONTAP 7.2.2p3)
Thanks, could you try once more with an updated nfstest.c version? I added "dup+close" which works for data and write cache flushing with Linux. I really hope it works with Solaris+FreeBSD too.
Timo Sirainen wrote:
OK, here it is!
Info: Connected: client Info: O_EXCL works Info: Testing attribute cache.. Info: Attr cache flush fchown(-1, -1): OK Info: Attr cache flush fchown(uid, -1): OK Info: Attr cache flush fchmod(mode): OK Info: Attr cache flush chown(-1, -1): OK Info: Attr cache flush chown(uid, -1): OK Info: Attr cache flush chmod(mode): OK Info: Attr cache flush dup+close: failed Info: Testing write flushing.. Info: Write flush no caching: failed Info: Write flush fcntl(shared): OK Info: Write flush fcntl(exclusive): OK Info: Write flush flock(shared): failed Info: Write flush flock(exclusive): failed Info: Write flush reopen: OK Info: Write flush dup+close: failed Info: Write flush attr cache: OK Info: Testing data cache.. Info: data cache: Reading EOF requires attribute cache flush Info: Data cache flush no caching: failed Info: Data cache flush attr cache: OK Info: Data cache flush fcntl(shared): OK Info: Data cache flush fcntl(exclusive): OK Info: Data cache flush flock(shared): failed Info: Data cache flush flock(exclusive): failed Info: Data cache flush dotlock: failed Info: Data cache flush O_DIRECT: failed Info: Data cache flush dup+close: OK
Best Wishes, Chris
-- --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- Christopher Wakelin, c.d.wakelin@reading.ac.uk IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439 Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094
On Thu, 12 Jul 2007, Timo Sirainen wrote:
FreeBSD 6.2-RELEASE-p5 and a NetApp NFS server:
Info: Connected: client Info: O_EXCL works Info: Testing attribute cache.. Info: Attr cache flush fchown(-1, -1): OK Info: Attr cache flush fchown(uid, -1): OK Info: Attr cache flush fchmod(mode): OK Info: Attr cache flush chown(-1, -1): OK Info: Attr cache flush chown(uid, -1): OK Info: Attr cache flush chmod(mode): OK Info: Attr cache flush dup+close: failed Info: Testing write flushing.. Info: Write flush no caching: failed Info: Write flush fcntl(shared): failed Info: Write flush fcntl(exclusive): failed Info: Write flush flock(shared): failed Info: Write flush flock(exclusive): failed Info: Write flush reopen: OK Info: Write flush dup+close: failed Info: Write flush attr cache: failed Info: Testing data cache.. Info: data cache: Reading EOF requires attribute cache flush Info: Data cache flush no caching: failed Info: Data cache flush attr cache: OK Error: fcntl(setlk, read) failed: Operation not supported Info: Data cache flush fcntl(shared): failed Error: fcntl(setlk, write) failed: Operation not supported Info: Data cache flush fcntl(exclusive): failed Info: Data cache flush flock(shared): failed Info: Data cache flush flock(exclusive): failed Info: Data cache flush dotlock: failed Info: Data cache flush O_DIRECT: failed Info: Data cache flush dup+close: failed
-Doug
Timo Sirainen wrote:
Solaris 8 diff (compilation needed -lrt as well as the usual -lnsl and -lsocket):
Info: Write flush fdatasync: OK Info: Data cache flush fdatasync: failed
Solaris 10 was the same as Solaris 8 for the previous nfstest.c. I'll let you know if it differs for fdatasync (seems unlikely).
Best Wishes, Chris
-- --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- Christopher Wakelin, c.d.wakelin@reading.ac.uk IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439 Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094
On Thu, 12 Jul 2007, Timo Sirainen wrote:
So still nothing usable. Updated nfstest.c once again to include fdatasync() test. It has to work.
fdatasync() isn't implemented in FreeBSD (see http://www.freebsd.org/cgi/query-pr.cgi?pr=64875). It defaults to mapping fdatasync() to fsync().
Results for FreeBSD 6.2-RELEASE-p5 and NetApp NFS server:
Info: Connected: client Info: O_EXCL works Info: Testing attribute cache.. Info: Attr cache flush fchown(-1, -1): OK Info: Attr cache flush fchown(uid, -1): OK Info: Attr cache flush fchmod(mode): OK Info: Attr cache flush chown(-1, -1): OK Info: Attr cache flush chown(uid, -1): OK Info: Attr cache flush chmod(mode): OK Info: Attr cache flush dup+close: failed Info: Testing write flushing.. Info: Write flush no caching: failed Info: Write flush fcntl(shared): failed Info: Write flush fcntl(exclusive): failed Info: Write flush flock(shared): failed Info: Write flush flock(exclusive): failed Info: Write flush reopen: OK Info: Write flush dup+close: failed Info: Write flush attr cache: failed Info: Write flush fdatasync: OK Info: Testing data cache.. Info: data cache: Reading EOF requires attribute cache flush Info: Data cache flush no caching: failed Info: Data cache flush attr cache: OK Error: fcntl(setlk, read) failed: Operation not supported Info: Data cache flush fcntl(shared): failed Error: fcntl(setlk, write) failed: Operation not supported Info: Data cache flush fcntl(exclusive): failed Info: Data cache flush flock(shared): failed Info: Data cache flush flock(exclusive): failed Info: Data cache flush dotlock: failed Info: Data cache flush dup+close: failed Info: Data cache flush fdatasync: failed
And with O_DIRECT enabled via vfs.nfs.nfs_directio_enable:
Info: Connected: client Info: O_EXCL works Info: Testing attribute cache.. Info: Attr cache flush fchown(-1, -1): OK Info: Attr cache flush fchown(uid, -1): OK Info: Attr cache flush fchmod(mode): OK Info: Attr cache flush chown(-1, -1): OK Info: Attr cache flush chown(uid, -1): OK Info: Attr cache flush chmod(mode): OK Info: Attr cache flush dup+close: failed Info: Testing write flushing.. Info: Write flush no caching: failed Info: Write flush fcntl(shared): failed Info: Write flush fcntl(exclusive): failed Info: Write flush flock(shared): failed Info: Write flush flock(exclusive): failed Info: Write flush reopen: OK Info: Write flush dup+close: failed Info: Write flush attr cache: failed Info: Write flush fdatasync: OK Info: Testing data cache.. Info: data cache: Reading EOF requires attribute cache flush Info: Data cache flush no caching: failed Info: Data cache flush attr cache: OK Error: fcntl(setlk, read) failed: Operation not supported Info: Data cache flush fcntl(shared): failed Error: fcntl(setlk, write) failed: Operation not supported Info: Data cache flush fcntl(exclusive): failed Info: Data cache flush flock(shared): failed Info: Data cache flush flock(exclusive): failed Info: Data cache flush dotlock: failed Info: Data cache flush dup+close: failed Info: Data cache flush fdatasync: failed Info: Data cache flush O_DIRECT: OK
-Doug
Timo Sirainen wrote:
Latest version used (2007-07-12 ~11am PST) for tests.
Machine 1: CentOS 4 (RHEL 4 clone) Linux 2.6.9-42.0.8.ELsmp #1 SMP Tue Jan 30 12:33:47 EST 2007 i686 i686 i386 GNU/Linux
Machine 2: CentOS 4 (RHEL 4 clone) Linux 2.6.9-42.0.10.ELsmp #1 SMP Tue Feb 27 10:11:19 EST 2007 i686 i686 i386 GNU/Linux
NFS Server: SnapServer 4500 (linux software RAID based appliance) Linux 2.4.19-gos-up #1 Wed Jul 12 23:00:50 PDT 2006 i686 i686 i386 GNU/Linux
machine 1: server machine 2: client
Info: Connected: client Info: O_EXCL works Info: Testing attribute cache.. Info: Attr cache flush fchown(-1, -1): failed Info: Attr cache flush fchown(uid, -1): OK Info: Attr cache flush fchmod(mode): OK Info: Attr cache flush chown(-1, -1): failed Info: Attr cache flush chown(uid, -1): OK Info: Attr cache flush chmod(mode): OK Info: Attr cache flush dup+close: OK Info: Testing write flushing.. Info: Write flush no caching: failed Info: Write flush fcntl(shared): OK Info: Write flush fcntl(exclusive): OK Info: Write flush flock(shared): failed Info: Write flush flock(exclusive): failed Info: Write flush reopen: OK Info: Write flush dup+close: OK Info: Write flush attr cache: failed Info: Testing data cache.. Info: data cache: Reading EOF requires attribute cache flush Info: Data cache flush no caching: failed Info: Data cache flush attr cache: failed Info: Data cache flush fcntl(shared): OK Info: Data cache flush fcntl(exclusive): OK Info: Data cache flush flock(shared): failed Info: Data cache flush flock(exclusive): failed Info: Data cache flush dotlock: failed Info: Data cache flush O_DIRECT: OK Info: Data cache flush dup+close: OK
machine 1: client machine 2: server
Info: Connected: client Info: O_EXCL works Info: Testing attribute cache.. Info: Attr cache flush fchown(-1, -1): failed Info: Attr cache flush fchown(uid, -1): OK Info: Attr cache flush fchmod(mode): OK Info: Attr cache flush chown(-1, -1): failed Info: Attr cache flush chown(uid, -1): OK Info: Attr cache flush chmod(mode): OK Info: Attr cache flush dup+close: OK Info: Testing write flushing.. Info: Write flush no caching: failed Info: Write flush fcntl(shared): OK Info: Write flush fcntl(exclusive): OK Info: Write flush flock(shared): failed Info: Write flush flock(exclusive): failed Info: Write flush reopen: OK Info: Write flush dup+close: OK Info: Write flush attr cache: failed Info: Testing data cache.. Info: data cache: Reading EOF requires attribute cache flush Info: Data cache flush no caching: failed Info: Data cache flush attr cache: failed Info: Data cache flush fcntl(shared): OK Info: Data cache flush fcntl(exclusive): OK Info: Data cache flush flock(shared): failed Info: Data cache flush flock(exclusive): failed Info: Data cache flush dotlock: failed Info: Data cache flush O_DIRECT: OK Info: Data cache flush dup+close: OK
-- Troy Engel | Systems Engineer Fluid, Inc | http://www.fluid.com
On Wed, 2007-07-11 at 22:21 +0300, Timo Sirainen wrote:
v1.1 code does now NFS cache flushing with mail_nfs_storage=yes and mail_nfs_index=yes. I haven't tried yet how well it works, but at least it finally should work without disabling attribute caches or anything.
For read caching I ended up using:
#ifdef __linux__
- fcntl() lock + unlock #else
- fchown(uid, -1)
Would have been nice to avoid the fcntl lock, but I couldn't find any other way (I even checked Linux sources). It also means that either you must use lockd with Linux or you must mount the filesystem with nolock option.
participants (7)
-
Adam McDougall
-
Andrew Ziem
-
Chris Wakelin
-
Cor Bosman
-
Doug Council
-
Timo Sirainen
-
Troy Engel