"Ihor" == Ihor Rusyn via dovecot <dovecot@dovecot.org> writes:
Quick update -- we can reproduce this reliably, specifically on NFS. We've also tested the same optimize path on ext4 (local) and CephFS, and neither shows the problem. Here's why we think that lines up:
The core issue is unlink()/rmdir() semantics for files/directories that are still open by the calling process. On a local POSIX filesystem (ext4) and on CephFS, unlinking a file that's still open by the same process is fully supported: the directory entry is removed immediately, and the file's data blocks stay allocated until the last file descriptor closes. So rmdir() on the parent directory succeeds right away, because as far as the filesystem is concerned the directory is already empty -- the still-open file just has no name anymore.
NFS can't do that. There's no way for an NFS client to tell the server "unlink this, but I still have it open." When a client unlinks a file it still holds open, the NFS client silently works around it with what's usually called "silly rename": instead of an actual unlink, it renames the file to a hidden .nfsXXXX name, so the directory entry survives until the file is finally closed and the client can clean it up. In fts_flatcurve_xapian_optimize_box_do(), the shard's write handles are still open when the old current.* directory is deleted -- fine on ext4/CephFS, but on NFS it leaves .nfsXXXX entries behind, so the rmdir() on that directory fails with ENOTEMPTY. That failure doesn't appear to be surfaced anywhere -- unlink_directory() reports success regardless -- so the directory is left behind, and once the open handles are eventually closed, what remains is an empty current.* directory that is still treated as the active shard.
To me this screams that when the rmdir() fails, instead of just bailing, the code should move the directory to another name and then create a new directory.
But I'm not sure if those .nfs### files every get cleaned up, so it would have to be part of the scan, purge, clean process to go through and remove those files by hand.
So I personally love using NFS filesystems for storage, but I can see how the semantics of the way the tool is written don't quite match how NFS works in the real world.
So the options could be:
Document clearly that this is not supported on NFS volumes
patch the code to handle it more gracefully (like I suggested above)
Change how the code works to not depend on this rmdir() action at all.
Just an interested bystander... :-)
John