Hi Jens,
Quick update — we can reproduce this reliably, specifically on NFS. We've also tested the same optimize path on ext4 (local) and CephFS, and neither shows the problem. Here's why we think that lines up:
The core issue is unlink()/rmdir() semantics for files/directories that are still open by the calling process. On a local POSIX filesystem (ext4) and on CephFS, unlinking a file that's still open by the same process is fully supported: the directory entry is removed immediately, and the file's data blocks stay allocated until the last file descriptor closes. So rmdir() on the parent directory succeeds right away, because as far as the filesystem is concerned the directory is already empty — the still-open file just has no name anymore.
NFS can't do that. There's no way for an NFS client to tell the server "unlink this, but I still have it open." When a client unlinks a file it still holds open, the NFS client silently works around it with what's usually called "silly rename": instead of an actual unlink, it renames the file to a hidden .nfsXXXX name, so the directory entry survives until the file is finally closed and the client can clean it up. In fts_flatcurve_xapian_optimize_box_do(), the shard's write handles are still open when the old current.* directory is deleted — fine on ext4/CephFS, but on NFS it leaves .nfsXXXX entries behind, so the rmdir() on that directory fails with ENOTEMPTY. That failure doesn't appear to be surfaced anywhere — unlink_directory() reports success regardless — so the directory is left behind, and once the open handles are eventually closed, what remains is an empty current.* directory that is still treated as the active shard.
We also tested whether mail_nfs_storage/mail_nfs_index change anything here — enabled both on our stand and reproduced the exact same failure chain (optimize succeeds silently, current.* ends up empty, next delivery fails to index with "Failed to open glass revision file," subsequent search fails with an internal error). That tracks with the code: as far as we can tell, fts-flatcurve doesn't consult either setting anywhere — they only affect the mail_index (MAIL_INDEX_OPEN_FLAG_NFS_FLUSH) and dotlock behavior in lib-storage, not the Xapian glass files flatcurve manages itself. So this doesn't look like something those settings, or NFS locking configuration in general, can fix.
Let us know if it'd help to see the exact optimize-path trace, or anything else from our reproduction.
On Wed, Aug 12, 2026 at 12:12 PM Jens Urban <Jens.Urban@freenet.ag> wrote:
Hello Ihor, hello Michael,
We tested this on our setup (2.4.4-5+debian12 (8b687aa65c)) as well and could not reproduce the issue. We are using a proprietary storage implementation (not NFS), with the following Dovecot settings:
mail_nfs_storage = yes mail_nfs_index = yes
After running doveadm fts optimize -u <user>, the index files were consolidated into a single index.* directory, and the existing current.* directory was removed. No current.* directory remained after the optimization. We then manually triggered an FTS search (we do not use fts_autoindex), after which a new current.* directory was created correctly with a complete Xapian database.
Given that the reported setup uses NFSv3 with local_lock=none, I also wonder whether NFS locking and cache semantics may be involved here. In particular, I would verify that the required NFS locking infrastructure is working correctly before assuming that this is a generic flatcurve optimization issue.
———————————————————————————————————————— In addition, based on our testing of FTS Flatcurve, I would recommend reviewing the following settings:
- fts_flatcurve_min_term_size We use: *fts_flatcurve_min_term_size = 1*
instead of the default value of 2. During testing we found that indexing respects fts_flatcurve_min_term_size and therefore drops one-character terms, while query tokenization may still generate such terms. This can result in false negatives, for example when searching for URLs or identifiers containing single-character tokens.
A simple example is: doveadm fts tokenize -u <USER> ' http://www.youtube.com/watch?v=kgu_fz1zytE'
which generates a "v" token even with fts:flatcurve_min_term_size=2. In our measurements, changing min_term_size from 2 to 1 resulted in only a negligible increase in index size.
- fts_flatcurve_substring_search We deliberately use: *fts_flatcurve_substring_search = no*
In our tests, enabling substring search caused massive index growth due to the additional index-only tokens, while the query tokens remained identical. Email addresses are already split into searchable tokens by the email-address tokenizer, so we found little practical benefit from enabling substring search in our environment.
———————————————————————————————————————— There are many other FTS settings worth tuning for the specific environment. In particular, we found that fts_autoindex can cause significant performance impact during indexing for users with mailboxes containing tens of thousands of messages. In such environments, it may be preferable to disable automatic indexing and perform the indexing asynchronously using a background service instead.
Regards Jens Urban
*Von: *Michael Slusarz via dovecot <dovecot@dovecot.org> *Datum: *Mittwoch, 12. August 2026 um 02:24 *An: *Ihor Rusyn <0k.a.b.a0@gmail.com>; Ihor Rusyn via dovecot < dovecot@dovecot.org> *Betreff: *Re: fts-flatcurve 2.4.4: after "doveadm fts optimize" a mailbox stops indexing new mail (empty current.<id> shard, missing iamglass)
On 08/07/2026 10:22 AM MDT Ihor Rusyn via dovecot <dovecot@dovecot.org> wrote:
Hello,
"doveadm fts optimize" leaves the mailbox in a state where no new mail can be indexed. The existing index survives and searches over already-indexed mail keep working, so the damage is invisible until the next message arrives.
Version: 2.4.4 (8b687aa65c), packaged in a container image FTS: fts_flatcurve, substring_search = yes, min_term_size default (2) fts_autoindex = yes, fts_search_add_missing = yes Index store: NFSv3 (rsize/wsize 32768, local_lock=none) Mail store: NFSv3, maildir
[snip]
I cannot reproduce this, probably due to some combination of NFS latency and semantics. (Xapian itself notes that NFS operation can be problematic. https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fxapian.org...) <https://xapian.org/docs/admin_notes.html>
It's likely fixed by adding fts_flatcurve_xapian_refresh() in fts_flatcurve_xapian_optimize_box_do(), but I am unable to verify.
michael
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
-- Best regards, Ihor Rusyn
Hi Jens,
Quick update -- we can reproduce this reliably, specifically on NFS. We've also tested the same optimize path on ext4 (local) and CephFS, and neither shows the problem. Here's why we think that lines up:
The core issue is unlink()/rmdir() semantics for files/directories that are still open by the calling process. On a local POSIX filesystem (ext4) and on CephFS, unlinking a file that's still open by the same process is fully supported: the directory entry is removed immediately, and the file's data blocks stay allocated until the last file descriptor closes. So rmdir() on the parent directory succeeds right away, because as far as the filesystem is concerned the directory is already empty -- the still-open file just has no name anymore.
NFS can't do that. There's no way for an NFS client to tell the server "unlink this, but I still have it open." When a client unlinks a file it still holds open, the NFS client silently works around it with what's usually called "silly rename": instead of an actual unlink, it renames the file to a hidden .nfsXXXX name, so the directory entry survives until the file is finally closed and the client can clean it up. In fts_flatcurve_xapian_optimize_box_do(), the shard's write handles are still open when the old current.* directory is deleted -- fine on ext4/CephFS, but on NFS it leaves .nfsXXXX entries behind, so the rmdir() on that directory fails with ENOTEMPTY. That failure doesn't appear to be surfaced anywhere -- unlink_directory() reports success regardless -- so the directory is left behind, and once the open handles are eventually closed, what remains is an empty current.* directory that is still treated as the active shard.
We also tested whether mail_nfs_storage/mail_nfs_index change anything here -- enabled both on our stand and reproduced the exact same failure chain (optimize succeeds silently, current.* ends up empty, next delivery fails to index with "Failed to open glass revision file," subsequent search fails with an internal error). That tracks with the code: as far as we can tell, fts-flatcurve doesn't consult either setting anywhere -- they only affect the mail_index (MAIL_INDEX_OPEN_FLAG_NFS_FLUSH) and dotlock behavior in lib-storage, not the Xapian glass files flatcurve manages itself. So this doesn't look like something those settings, or NFS locking configuration in general, can fix.
Let us know if it'd help to see the exact optimize-path trace, or anything else from our reproduction.
On Wed, Aug 12, 2026 at 12:12PM Jens Urban <[1]Jens.Urban@freenet.ag> wrote:
Hello Ihor, hello Michael,
We tested this on our setup (2.4.4-5+debian12 (8b687aa65c)) as well and
could not reproduce the issue. We are using a proprietary storage
implementation (not NFS), with the following Dovecot settings:
mail_nfs_storage = yes
mail_nfs_index = yes
After running doveadm fts optimize -u <user>, the index files were
consolidated into a single index.* directory, and the existing current.*
directory was removed. No current.* directory remained after the
optimization. We then manually triggered an FTS search (we do not use
fts_autoindex), after which a new current.* directory was created
correctly with a complete Xapian database.
Given that the reported setup uses NFSv3 with local_lock=none, I also
wonder whether NFS locking and cache semantics may be involved here. In
particular, I would verify that the required NFS locking infrastructure
is working correctly before assuming that this is a generic flatcurve
optimization issue.
--------------------------------------------------------------------------------
In addition, based on our testing of FTS Flatcurve, I would recommend
reviewing the following settings:
1. fts_flatcurve_min_term_size
We use: fts_flatcurve_min_term_size = 1
instead of the default value of 2. During testing we found that indexing
respects fts_flatcurve_min_term_size and therefore drops one-character
terms, while query tokenization may still generate such terms. This can
result in false negatives, for example when searching for URLs or
identifiers containing single-character tokens.
A simple example is: doveadm fts tokenize -u <USER>
'[2]http://www.youtube.com/watch?v=kgu_fz1zytE'
which generates a "v" token even with fts:flatcurve_min_term_size=2.
In our measurements, changing min_term_size from 2 to 1 resulted in only
a negligible increase in index size.
2. fts_flatcurve_substring_search
We deliberately use: fts_flatcurve_substring_search = no
In our tests, enabling substring search caused massive index growth due
to the additional index-only tokens, while the query tokens remained
identical. Email addresses are already split into searchable tokens by
the email-address tokenizer, so we found little practical benefit from
enabling substring search in our environment.
--------------------------------------------------------------------------------
There are many other FTS settings worth tuning for the specific
environment. In particular, we found that fts_autoindex can cause
significant performance impact during indexing for users with mailboxes
containing tens of thousands of messages. In such environments, it may
be preferable to disable automatic indexing and perform the indexing
asynchronously using a background service instead.
Regards
Jens Urban
Von: Michael Slusarz via dovecot <[3]dovecot@dovecot.org>
Datum: Mittwoch, 12. August 2026 um 02:24
An: Ihor Rusyn <[4]0k.a.b.a0@gmail.com>; Ihor Rusyn via dovecot
<[5]dovecot@dovecot.org>
Betreff: Re: fts-flatcurve 2.4.4: after "doveadm fts optimize" a mailbox
stops indexing new mail (empty current.<id> shard, missing iamglass)
> On 08/07/2026 10:22 AM MDT Ihor Rusyn via dovecot
<[6]dovecot@dovecot.org> wrote:
>
> Hello,
>
> "doveadm fts optimize" leaves the mailbox in a state where no new mail
can
> be
> indexed. The existing index survives and searches over already-indexed
mail
> keep
> working, so the damage is invisible until the next message arrives.
>
> Version: 2.4.4 (8b687aa65c), packaged in a container image
> FTS: fts_flatcurve, substring_search = yes, min_term_size
default
> (2)
> fts_autoindex = yes, fts_search_add_missing = yes
> Index store: NFSv3 (rsize/wsize 32768, local_lock=none)
> Mail store: NFSv3, maildir
[snip]
I cannot reproduce this, probably due to some combination of NFS latency
and semantics. (Xapian itself notes that NFS operation can be
problematic.
[7]https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fxapian.org%2Fdocs%2Fadmin_notes.html&data=05%7C02%7Cjens.urban%40freenet.ag%7C23f8ee82ed544f08372d08def8080ff9%7C7d95deb30bca4c9da61e1fd6c47ed60c%7C0%7C0%7C639220910671970524%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=eqoHjH9Pbt8h0AGo%2FDPRw%2F15YYeuXSTRMr3r8isznpE%3D&reserved=0)
It's likely fixed by adding fts_flatcurve_xapian_refresh() in
fts_flatcurve_xapian_optimize_box_do(), but I am unable to verify.
michael
_______________________________________________
dovecot mailing list -- [8]dovecot@dovecot.org
To unsubscribe send an email to [9]dovecot-leave@dovecot.org-- Best regards, Ihor Rusyn
References
Visible links
- mailto:Jens.Urban@freenet.ag
- http://www.youtube.com/watch?v=kgu_fz1zytE
- mailto:dovecot@dovecot.org
- mailto:0k.a.b.a0@gmail.com
- mailto:dovecot@dovecot.org
- mailto:dovecot@dovecot.org
- https://xapian.org/docs/admin_notes.html
- mailto:dovecot@dovecot.org
- mailto:dovecot-leave@dovecot.org