Re: [Dovecot] I stream read - stale NFS file handle (reboot of server)
NFS Security 101 for NFSv2 and v3 (NOT NFSv4 a long time ago I was part of the discussion group for NFSv4 spec the short comings of v2 and v3 have been fixed)
SRV: Server Exports File System /abc/123 access only to host=xyz.domain.com XYZ: Client Mount mount's SRV:/abc/123 SRV: "mountd" gets a request from SRV check access list, and if the client is allowed access, returns File Handle for the top of the mount point (I will call this the FileSystemHandle, this changes after every reboot) XYZ: Client talks to the SRV NFS Server "nfsd" using FileSystemHandle (which represents /abc/123), To find a FileHandle for a file the client sends the NFS server the FileSystemHandle, then finds the next File Handle (/mydir) and the next FileHandle (myfile) and the client has found the FileHandle for /abc/123/mydir/myfile
NFSD (v2/v3) is stateless other than the information provided by mountd
(mount requests) and lockd (file locking). When you share/export a file
system, as part of this a FileSystemHandle is generated and stored
somewhere for /abc/123 most likely store in the kernel.
SRV: Reboots, a new FileSystemHandle is allocated XYZ: Reports Stale File Handle for everything (if not most things) which was mounted from SRV or the mount point on the client does not respond.
A long time ago FileSystemHandle would stay the same between reboots and you wouldn't get this problem other than on an individual file.
To handle deleted files which are in use by NFS clients some servers rename them to .nfs* because if one client deleted and other clients where accessing the file then they would get Stale NFS handle. Once in a while a NFS server will do find $dir -type f -name .nfs\* -mtime +7 -mount -exec rm -f {} \; to clean up.
NFSv4 does not use mountd or lockd. My advice to anyone on old versions of NFS upgrade to NFSv4 as soon as possible and implement KerbV5 for improved authentication and disallow NFSv2/3 access.
If you do not get the Stale File Handle error when the server reboots, it most likely means the FileSystemHandle is not changing between reboots, but then you may have more security issues. Software which Clusters NFS Servers ensures that the node which takes over, uses the same FileSystemHandle (which may only change if the whole Cluster is shutdown)
Cheers Damon.
I was part of the discussion group for NFSv4 spec the short comings of v2 and v3 have been fixed I'm a bit surprised by this. Which "discussion group"?
NFSD (v2/v3) is stateless other than the information provided by mountd (mount requests) and lockd (file locking). NFS is stateless save the state information remembered by statd. After all, that's the whole point of statd (and the NLM grace period).
A long time ago FileSystemHandle would stay the same between reboots and you wouldn't get this problem other than on an individual file. My XNFS says: 7.1.3 Stateless Servers and Idempotency [...] With a few exceptions, rebooting the server must not invalidate distributed state information. (with the exceptions being unstable writes, teporary files etc.)
To handle deleted files which are in use by NFS clients some servers rename them to .nfs* because if one client deleted and other clients where accessing the file then they would get Stale NFS handle. This rename takes place on the client on a REMOVE, not on the server. Cf. either McKusick et. al. or RFC 1813, 3.3.12, IMPLEMENTATION.
Once in a while a NFS server will do find $dir -type f -name .nfs\* -mtime +7 -mount -exec rm -f {} \; to clean up. Do you have any reference for that?
If you do not get the Stale File Handle error when the server reboots, it most likely means the FileSystemHandle is not changing between reboots I.e., it conforms to the specs.
but then you may have more security issues. Could you please elaborate on this "secutity issues"?
-------- Original Message by Edgar --------
I was part of the discussion group for NFSv4 spec the short comings of v2 and v3 have been fixed
I'm a bit surprised by this. Which "discussion group"?
The RFC, one for NFSv4.0
NFSD (v2/v3) is stateless other than the information provided by mountd (mount requests) and lockd (file locking).
NFS is stateless save the state information remembered by statd. After all, that's the whole point of statd (and the NLM grace period).
NFS v2/v3 has no concept of locking so a external protocol was added, still keeping NFS v2/v3 stateless.
A long time ago FileSystemHandle would stay the same between reboots and you wouldn't get this problem other than on an individual file.
My XNFS says:
7.1.3 Stateless Servers and Idempotency
[...]
With a few exceptions, rebooting the server must not invalidate distributed state information.
(with the exceptions being unstable writes, teporary files etc.)
Yes the spec says this, and at first everyone implemented it this way, and then everyone changed it due to Security issues, maybe no-one went back and updated the spec. The FileSystemHandle (ie File Handle at the top of exported file system) is now changed on every-reboot, unless your in a Cluster configuration.
To handle deleted files which are in use by NFS clients some servers rename them to .nfs* because if one client deleted and other clients where accessing the file then they would get Stale NFS handle.
This rename takes place on the client on a REMOVE, not on the server. Cf. either McKusick et. al. or RFC 1813, 3.3.12, IMPLEMENTATION.
Bad sentence the client deletes the file, and if it is still in use at the time the client goes to delete it, it is renamed instead .nfsNNNN
Once in a while a NFS server will do find $dir -type f -name .nfs\* -mtime +7 -mount -exec rm -f {} \; to clean up.
Do you have any reference for that?
http://www.google.com.au/search?hl=en&q=%2Bfind+%22%2B-name+.nfs*%22+%2Brm+%2Bmtime http://www.google.com.au/search?hl=en&q=nfsfind #can be found in shell scripts e.g. nfsfind Look at your crontabs on your nfs server. The above find and remove .nfs files on a server is a cleanup, e.g. client crashes before a in-use file is deleted
If you do not get the Stale File Handle error when the server reboots, it most likely means the FileSystemHandle is not changing between reboots
I.e., it conforms to the specs.
The specs had problems
but then you may have more security issues.
Could you please elaborate on this "secutity issues"?
NFSD (v2/3) is stateless and trusts the security information (UDP, RPC requests using AUTH_UNIX) from the NFS client . The client is effectively responsible for the security when talking to a NFSv2/3 client. (NFSv4 does too (the default is AUTH_SYS,AUTH_UNIX) unless you setup the security to use RPCSEC_GSS or AUTH_DH). The first step taken by a NFSv2/3 client is to get the FileHandle for an exported path from mountd which checks the access lists for the NFS share, this FileHandle is used to talk to NFSD for the top of the export/shared path.
Cheers.
I must admit getting somewhat tired of this discussion, but I simply don't want people investigating the original problem being distracted.
EF> I'm a bit surprised by this. Which "discussion group"? DA> The RFC, one for NFSv4.0 Oh, you mean you posted to nfs4@ietf.org? Oh yes, http://www.nfsv4.org/nfsv4-wg-archive-dec-96-jan-03/author.html confirms this.
DA> Yes the spec says this, and at first everyone implemented it this DA> way, and then everyone changed it due to Security issues, maybe DA> no-one went back and updated the spec. The FileSystemHandle (ie File DA> Handle at the top of exported file system) is now changed on DA> every-reboot, unless your in a Cluster configuration. OK, you are suggesting that the designers of the NFS protocol went into great lengths making NFS immune to server crashes by making the operations idempotent and the protocol stateless; then the designers of the NLM protocol went into great lenghts making locking immune to server crashes by idempotent operations, introduction of the stat deamon and the locking deamon grace period -- and then later, they break all this for what you call "security reasons" and forget updating the protocol specification? Really?
DA> The specs had problems Ah yes.
DA> but then you may have more security issues. EF> Could you please elaborate on this "secutity issues"? DA> NFSD (v2/3) is stateless and trusts the security information (UDP, DA> RPC requests using AUTH_UNIX) from the NFS client. [...] I'm quite aware of all this. What I was asking for were those "security issues" that you claim to be solved by randomizing the inode-to-filehandle-relationship on every server reboot.
I think you are confusing inode generation number randomization with NFS file handles. Randomizing generation numbers makes file handles much harder to guess, addressing the security issues you mention. But generation numbers are part of the on-disk inode and so don't change on server reboots. They don't need in order to address the security issues. But changing the inode-to-fiilehandle relation on reboots would break NFS's immunity to server crashes. And it would break it for no reason whatsoever.
I'm quite aware of all this. What I was asking for were those "security issues" that you claim to be solved>by randomizing the inode-to-filehandle-relationship on every server reboot.
I have not really wanted to spell it out.
The following is mainly about NFSv2/v3.
Client requests a File Handle for a export file system from mountd, mountd checks its access list, and if the client has access returns a File Handle (for an exported file system). This File Handle can then be used to get directory listing from the top of the export file system.
NFSD gets a File Handle and UID GID information and returns information or data e.g. read a file, get a directory listing based on the File Handle.
Security Issue File Handles were/are fixed. e.g. something like Major/Minor device number of the disk the file-system was on and inode ie a sort of formula, which has a stable file handle for every file/directory/etc.
The problem (I hope it is still not) is that once you have the file handle for a file or directory you can access any file on a system, exported or not. Because of the formula way of creating file handles makes it possible to guess the file handle for say /, which could be accessed even if it was not exported. Why, because NFSD (and the protocol) assume everyone is nice, and that if you have a file handle then mountd must have given it to you, and hence NFSD just accepts it, so you could read /etc/password, even if /etc was not exported.
NFSD did not re-check access lists for the same things as mountd, if it did check access lists it would slow NFSD down, so some mountd/nfsd implementations made a portion of the file handle random at the time of export, this fixes the security issue of "skipping mountd" or being able to easily "guess" file-handles.
Now (I hope) CPU performance has increased, and some NFSD talk to mountd to get a list of IP address which are able to present File Handles. i.e. NFDS does some basic IP address checking, in the past it did not check anything that mountd did. These implementations which check the IP for example, may have gone back to a stable file handle between reboots.
I can not give you a list Unix flavours which do random file handles at export (if they did, or if they still do), all I can say is that it's a possible cause of "Stale File Handle" after a reboot, when it is not expected to happen.
OpenSolaris ZFS nfs share presents the same NFS file-handle but NFSD checks with mountd about the client IP address.
Some light reading http://utcc.utoronto.ca/~cks/space/blog/unix/NFSServerSecurity http://utcc.utoronto.ca/~cks/space/blog/solaris/SolarisMountdInnards http://fixunix.com/sgi/110779-stale-nfs-file-handle.html # second post
This is a lot more detail http://www.antibozo.net/ogata/webtools/nfs/README.txt http://www.fsl.cs.sunysb.edu/docs/nfscrack-tr/index.html
If you would like to check the file handle for an export file system before and after a reboot you can use the following tool
"nfs is an NFS protocol client package that allows you to test security of NFS implementations. It includes a mountd client that enumerates exported filesystems and their root filehandles, and an NFS client that allows you to perform file transfers, directory listings, object renaming, and most other procedures supported by NFS. The NFS client resembles an FTP client, and includes command line completion and a history mechanism. It supports versions 2 and 3 of the NFS protocol, and can use TCP or UDP transport. Both programs compile successfully on Linux, FreeBSD, IRIX, and Solaris." http://www.antibozo.net/ogata/webtools/index.html
e.g. the file handle for /export/home ./mtq localhost Connected to mountd server at udp localhost:45743. /export/home: fe9566320838bf750a00030000000000b70100000a00030000000000b7010000
Cheers
participants (2)
-
Damon Atkins
-
Edgar Fuß