fts-flatcurve 2.4.4: after "doveadm fts optimize" a mailbox stops indexing new mail (empty current.<id> shard, missing iamglass)
Hello,
"doveadm fts optimize" leaves the mailbox in a state where no new mail can be indexed. The existing index survives and searches over already-indexed mail keep working, so the damage is invisible until the next message arrives.
Version: 2.4.4 (8b687aa65c), packaged in a container image FTS: fts_flatcurve, substring_search = yes, min_term_size default (2) fts_autoindex = yes, fts_search_add_missing = yes Index store: NFSv3 (rsize/wsize 32768, local_lock=none) Mail store: NFSv3, maildir
Reproduction
Starting point: a healthy index.
doveadm fts flatcurve check -u user@example.com INBOX
INBOX guid=88885837bb33136365390000d09efc50 errors=0 shards=1
ls /indexes/.../.INBOX/fts-flatcurve/*/
current.1786117302443530: flintlock iamglass postlist.glass termlist.glass
doveadm search -u user@example.com mailbox INBOX text a | tail -1
88885837bb33136365390000d09efc50 58
Optimize. Exits 0, prints nothing.
doveadm fts optimize -u user@example.com
echo $?
0
State afterwards: the compacted data is in a new index.<n> directory, and the pre-existing current.<id> directory is still present but EMPTY.
ls /indexes/.../.INBOX/fts-flatcurve/*/
current.1786117302443530: index.1464: flintlock iamglass postlist.glass termlist.glass
ls -ld .../fts-flatcurve/current.1786117302443530
drwx------ 2 dovecot dovecot 4096 Aug 7 16:08 .../current.1786117302443530
doveadm fts flatcurve check -u user@example.com INBOX
Error: fts-flatcurve(INBOX): Cannot open DB (RO; current.1786117302443530); DatabaseNotFoundError: Couldn't detect type of database Error: fts-flatcurve(INBOX): Check failed; DatabaseOpeningError: Directory does not contain a Xapian database
Searching already-indexed mail still works at this point:
doveadm search -u user@example.com mailbox INBOX text a | tail -1
88885837bb33136365390000d09efc50 58
Deliver one message and index it. This is where it breaks:
printf 'From: a@b.c\nSubject: reproprobe\n\nbody\n' \
| doveadm save -u user@example.com -m INBOXInfo: Mailbox INBOX: save: box=INBOX, uid=59, subject=reproprobe
doveadm index -u user@example.com INBOX
Error: fts-flatcurve(INBOX): Cannot open DB (RW, current.1786117302443530): .../current.1786117302443530/iamglass: Failed to open glass revision file for reading (No such file or directory) Error: cmd index: Mailbox INBOX: Precache for UID=59 failed: Internal error
From now on SEARCH fails too, because fts_search_add_missing makes it index first:
doveadm search -u user@example.com mailbox INBOX text reproprobe
Error: Mailbox INBOX: indexer failed to index mailbox Error: cmd search: Searching mailbox INBOX failed: Internal error occurred.
Over IMAP the client sees: NO [SERVERBUG] Internal error occurred.
It does not recover by itself
"doveadm index" and autoindexing both take the same write path and fail the same way. "doveadm fts rescan" does not help either -- it also cannot open the empty shard. The only repair we found is to delete the folder's fts-flatcurve directory and index the folder again.
Impact
A quiet mailbox looks fine for as long as nothing is delivered to it, so this can sit unnoticed. We found it because a monitoring mailbox that receives one message per minute started returning SERVERBUG on SEARCH. In our case a maintenance run that called "doveadm fts optimize" per mailbox left 114 mailboxes in this state, which is all of the ones that had any mail.
Possible cause (not verified)
In fts_flatcurve_xapian_optimize_box_do() every shard is first opened for writing ("We need to lock all of the mailboxes so nothing changes while we are optimizing"), and the old current.*/index.* directories are then deleted from disk while those write handles are still open. A handle that is closed afterwards appears to recreate its directory without a glass revision file, which matches what is left behind.
Also, in the same function the result of
fts_flatcurve_xapian_rename_db(backend, dbpath, NULL, error_r)is discarded: rename_db() does not update the path it is given and returns the new path only through newpath_r, which is NULL at this call site.
I have not tried to confirm either of these beyond reading the code.
Not the same as the known root-owned iamglass report
There is an older report where "doveadm fts optimize -A" run from cron as root left iamglass owned by root, producing the same error string. That is not this: here the directories are owned by the mail user (dovecot:dovecot, 0700) and the revision file does not exist at all.
-- Best regards, Ihor Rusyn
Hello, "doveadm fts optimize" leaves the mailbox in a state where no new mail can be indexed. The existing index survives and searches over already-indexed mail keep working, so the damage is invisible until the next message arrives. Version: 2.4.4 (8b687aa65c), packaged in a container image FTS: fts_flatcurve, substring_search = yes, min_term_size default (2) fts_autoindex = yes, fts_search_add_missing = yes Index store: NFSv3 (rsize/wsize 32768, local_lock=none) Mail store: NFSv3, maildir Reproduction
Starting point: a healthy index.
doveadm fts flatcurve check -u [1]user@example.com INBOX
INBOX guid=88885837bb33136365390000d09efc50 errors=0 shards=1
ls /indexes/.../.INBOX/fts-flatcurve/*/
current.1786117302443530: flintlock iamglass postlist.glass termlist.glass
doveadm search -u [2]user@example.com mailbox INBOX text a | tail -1
88885837bb33136365390000d09efc50 58
Optimize. Exits 0, prints nothing.
doveadm fts optimize -u [3]user@example.com
echo $?
0
State afterwards: the compacted data is in a new index.<n> directory, and the pre-existing current.<id> directory is still present but EMPTY.
ls /indexes/.../.INBOX/fts-flatcurve/*/
current.1786117302443530: index.1464: flintlock iamglass postlist.glass termlist.glass
ls -ld .../fts-flatcurve/current.1786117302443530
drwx------ 2 dovecot dovecot 4096 Aug 7 16:08 .../current.1786117302443530
doveadm fts flatcurve check -u [4]user@example.com INBOX
Error: fts-flatcurve(INBOX): Cannot open DB (RO; current.1786117302443530); DatabaseNotFoundError: Couldn't detect type of database Error: fts-flatcurve(INBOX): Check failed; DatabaseOpeningError: Directory does not contain a Xapian database Searching already-indexed mail still works at this point:
doveadm search -u [5]user@example.com mailbox INBOX text a | tail -1
88885837bb33136365390000d09efc50 58
Deliver one message and index it. This is where it breaks:
printf 'From: a@b.c\nSubject: reproprobe\n\nbody\n' \
| doveadm save -u [6]user@example.com -m INBOXInfo: Mailbox INBOX: save: box=INBOX, uid=59, subject=reproprobe
doveadm index -u [7]user@example.com INBOX
Error: fts-flatcurve(INBOX): Cannot open DB (RW, current.1786117302443530): .../current.1786117302443530/iamglass: Failed to open glass revision file for reading (No such file or directory) Error: cmd index: Mailbox INBOX: Precache for UID=59 failed: Internal error
From now on SEARCH fails too, because fts_search_add_missing makes it index first:
doveadm search -u [8]user@example.com mailbox INBOX text reproprobe
Error: Mailbox INBOX: indexer failed to index mailbox Error: cmd search: Searching mailbox INBOX failed: Internal error occurred. Over IMAP the client sees: NO [SERVERBUG] Internal error occurred. It does not recover by itself
"doveadm index" and autoindexing both take the same write path and fail the same way. "doveadm fts rescan" does not help either -- it also cannot open the empty shard. The only repair we found is to delete the folder's fts-flatcurve directory and index the folder again. Impact
A quiet mailbox looks fine for as long as nothing is delivered to it, so this can sit unnoticed. We found it because a monitoring mailbox that receives one message per minute started returning SERVERBUG on SEARCH. In our case a maintenance run that called "doveadm fts optimize" per mailbox left 114 mailboxes in this state, which is all of the ones that had any mail. Possible cause (not verified)
In fts_flatcurve_xapian_optimize_box_do() every shard is first opened for writing ("We need to lock all of the mailboxes so nothing changes while we are optimizing"), and the old current.*/index.* directories are then deleted from disk while those write handles are still open. A handle that is closed afterwards appears to recreate its directory without a glass revision file, which matches what is left behind. Also, in the same function the result of fts_flatcurve_xapian_rename_db(backend, dbpath, NULL, error_r) is discarded: rename_db() does not update the path it is given and returns the new path only through newpath_r, which is NULL at this call site. I have not tried to confirm either of these beyond reading the code. Not the same as the known root-owned iamglass report
There is an older report where "doveadm fts optimize -A" run from cron as root left iamglass owned by root, producing the same error string. That is not this: here the directories are owned by the mail user (dovecot:dovecot, 0700) and the revision file does not exist at all.
-- Best regards, Ihor Rusyn
References
Visible links
- mailto:user@example.com
- mailto:user@example.com
- mailto:user@example.com
- mailto:user@example.com
- mailto:user@example.com
- mailto:user@example.com
- mailto:user@example.com
- mailto:user@example.com
On 08/07/2026 10:22 AM MDT Ihor Rusyn via dovecot <dovecot@dovecot.org> wrote:
Hello,
"doveadm fts optimize" leaves the mailbox in a state where no new mail can be indexed. The existing index survives and searches over already-indexed mail keep working, so the damage is invisible until the next message arrives.
Version: 2.4.4 (8b687aa65c), packaged in a container image FTS: fts_flatcurve, substring_search = yes, min_term_size default (2) fts_autoindex = yes, fts_search_add_missing = yes Index store: NFSv3 (rsize/wsize 32768, local_lock=none) Mail store: NFSv3, maildir
[snip]
I cannot reproduce this, probably due to some combination of NFS latency and semantics. (Xapian itself notes that NFS operation can be problematic. https://xapian.org/docs/admin_notes.html)
It's likely fixed by adding fts_flatcurve_xapian_refresh() in fts_flatcurve_xapian_optimize_box_do(), but I am unable to verify.
michael
Hi Michael,
Thanks for looking into this. I don't think fts_flatcurve_xapian_refresh() alone would close the gap here — as far as I can tell it re-opens/refreshes an already-open db handle (to pick up externally-committed changes or drop a stale NFS attribute cache), but it doesn't release the write handle the process is holding open against the *old* shard directory during optimize. On NFS, deleting a directory whose files are still open by the client doesn't unlink them — the files get silently renamed to .nfsXXXX, rmdir() on the directory then fails with ENOTEMPTY, and (as far as I can tell from the code) that failure isn't surfaced — the directory is left behind, still pointing at what should have been the removed shard, and empty of anything usable. A refresh call on the handle wouldn't change that, since the handle to the old path is what needs to be dropped before the delete, not refreshed afterward.
Happy to be wrong here since I haven't run this against your tree — is there something in the optimize path that already closes out the old shard's handles before the delete step that I'm missing?
If it'd help you reproduce it locally, I'm glad to send over the exact NFS client mount options and the relevant doveconf -n fts_flatcurve/mail_location section from our setup. NFS server behavior around silly-rename can vary a fair bit between implementations, so it might just be that your test NFS server doesn't hit the same path ours does.
On Tue, Aug 11, 2026 at 11:59 PM Michael Slusarz < michael.slusarz@dovecotpro.com> wrote:
On 08/07/2026 10:22 AM MDT Ihor Rusyn via dovecot <dovecot@dovecot.org> wrote:
Hello,
"doveadm fts optimize" leaves the mailbox in a state where no new mail can be indexed. The existing index survives and searches over already-indexed mail keep working, so the damage is invisible until the next message arrives.
Version: 2.4.4 (8b687aa65c), packaged in a container image FTS: fts_flatcurve, substring_search = yes, min_term_size default (2) fts_autoindex = yes, fts_search_add_missing = yes Index store: NFSv3 (rsize/wsize 32768, local_lock=none) Mail store: NFSv3, maildir
[snip]
I cannot reproduce this, probably due to some combination of NFS latency and semantics. (Xapian itself notes that NFS operation can be problematic. https://xapian.org/docs/admin_notes.html)
It's likely fixed by adding fts_flatcurve_xapian_refresh() in fts_flatcurve_xapian_optimize_box_do(), but I am unable to verify.
michael
-- Best regards, Ihor Rusyn
Hi Michael,
Thanks for looking into this. I don't think fts_flatcurve_xapian_refresh() alone would close the gap here -- as far as I can tell it re-opens/refreshes an already-open db handle (to pick up externally-committed changes or drop a stale NFS attribute cache), but it doesn't release the write handle the process is holding open against the old shard directory during optimize. On NFS, deleting a directory whose files are still open by the client doesn't unlink them -- the files get silently renamed to .nfsXXXX, rmdir() on the directory then fails with ENOTEMPTY, and (as far as I can tell from the code) that failure isn't surfaced -- the directory is left behind, still pointing at what should have been the removed shard, and empty of anything usable. A refresh call on the handle wouldn't change that, since the handle to the old path is what needs to be dropped before the delete, not refreshed afterward.
Happy to be wrong here since I haven't run this against your tree -- is there something in the optimize path that already closes out the old shard's handles before the delete step that I'm missing?
If it'd help you reproduce it locally, I'm glad to send over the exact NFS client mount options and the relevant doveconf -n fts_flatcurve/mail_location section from our setup. NFS server behavior around silly-rename can vary a fair bit between implementations, so it might just be that your test NFS server doesn't hit the same path ours does.
On Tue, Aug 11, 2026 at 11:59PM Michael Slusarz <[1]michael.slusarz@dovecotpro.com> wrote:
> On 08/07/2026 10:22 AM MDT Ihor Rusyn via dovecot
<[2]dovecot@dovecot.org> wrote:
>
> Hello,
>
> "doveadm fts optimize" leaves the mailbox in a state where no new mail
can
> be
> indexed. The existing index survives and searches over already-indexed
mail
> keep
> working, so the damage is invisible until the next message arrives.
>
> Version: 2.4.4 (8b687aa65c), packaged in a container image
> FTS: fts_flatcurve, substring_search = yes, min_term_size
default
> (2)
> fts_autoindex = yes, fts_search_add_missing = yes
> Index store: NFSv3 (rsize/wsize 32768, local_lock=none)
> Mail store: NFSv3, maildir
[snip]
I cannot reproduce this, probably due to some combination of NFS latency
and semantics. (Xapian itself notes that NFS operation can be
problematic. [3]https://xapian.org/docs/admin_notes.html)
It's likely fixed by adding fts_flatcurve_xapian_refresh() in
fts_flatcurve_xapian_optimize_box_do(), but I am unable to verify.
michael-- Best regards, Ihor Rusyn
References
Visible links
- mailto:michael.slusarz@dovecotpro.com
- mailto:dovecot@dovecot.org
- https://xapian.org/docs/admin_notes.html
On 08/12/2026 4:11 AM MDT Ihor Rusyn via dovecot <dovecot@dovecot.org> wrote:
Thanks for looking into this. I don't think fts_flatcurve_xapian_refresh() alone would close the gap here — as far as I can tell it re-opens/refreshes an already-open db handle (to pick up externally-committed changes or drop a stale NFS attribute cache), but it doesn't release the write handle the process is holding open against the *old* shard directory during optimize.
fts_flatcurve_xapian_refresh() closes all active write DB handles against the mailbox, and destroys their underlying C++ Xapian write database objects. (read handles are irrelevant because they don't lock).
michael
Hello Ihor, hello Michael,
We tested this on our setup (2.4.4-5+debian12 (8b687aa65c)) as well and could not reproduce the issue. We are using a proprietary storage implementation (not NFS), with the following Dovecot settings:
mail_nfs_storage = yes mail_nfs_index = yes
After running doveadm fts optimize -u <user>, the index files were consolidated into a single index.* directory, and the existing current.* directory was removed. No current.* directory remained after the optimization. We then manually triggered an FTS search (we do not use fts_autoindex), after which a new current.* directory was created correctly with a complete Xapian database.
Given that the reported setup uses NFSv3 with local_lock=none, I also wonder whether NFS locking and cache semantics may be involved here. In particular, I would verify that the required NFS locking infrastructure is working correctly before assuming that this is a generic flatcurve optimization issue.
———————————————————————————————————————— In addition, based on our testing of FTS Flatcurve, I would recommend reviewing the following settings:
- fts_flatcurve_min_term_size We use: fts_flatcurve_min_term_size = 1
instead of the default value of 2. During testing we found that indexing respects fts_flatcurve_min_term_size and therefore drops one-character terms, while query tokenization may still generate such terms. This can result in false negatives, for example when searching for URLs or identifiers containing single-character tokens.
A simple example is: doveadm fts tokenize -u <USER> 'http://www.youtube.com/watch?v=kgu_fz1zytE'
which generates a "v" token even with fts:flatcurve_min_term_size=2. In our measurements, changing min_term_size from 2 to 1 resulted in only a negligible increase in index size.
- fts_flatcurve_substring_search We deliberately use: fts_flatcurve_substring_search = no
In our tests, enabling substring search caused massive index growth due to the additional index-only tokens, while the query tokens remained identical. Email addresses are already split into searchable tokens by the email-address tokenizer, so we found little practical benefit from enabling substring search in our environment.
———————————————————————————————————————— There are many other FTS settings worth tuning for the specific environment. In particular, we found that fts_autoindex can cause significant performance impact during indexing for users with mailboxes containing tens of thousands of messages. In such environments, it may be preferable to disable automatic indexing and perform the indexing asynchronously using a background service instead.
Regards Jens Urban
Von: Michael Slusarz via dovecot <dovecot@dovecot.org> Datum: Mittwoch, 12. August 2026 um 02:24 An: Ihor Rusyn <0k.a.b.a0@gmail.com>; Ihor Rusyn via dovecot <dovecot@dovecot.org> Betreff: Re: fts-flatcurve 2.4.4: after "doveadm fts optimize" a mailbox stops indexing new mail (empty current.<id> shard, missing iamglass)
On 08/07/2026 10:22 AM MDT Ihor Rusyn via dovecot <dovecot@dovecot.org> wrote:
Hello,
"doveadm fts optimize" leaves the mailbox in a state where no new mail can be indexed. The existing index survives and searches over already-indexed mail keep working, so the damage is invisible until the next message arrives.
Version: 2.4.4 (8b687aa65c), packaged in a container image FTS: fts_flatcurve, substring_search = yes, min_term_size default (2) fts_autoindex = yes, fts_search_add_missing = yes Index store: NFSv3 (rsize/wsize 32768, local_lock=none) Mail store: NFSv3, maildir
[snip]
I cannot reproduce this, probably due to some combination of NFS latency and semantics. (Xapian itself notes that NFS operation can be problematic. https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fxapian.org%2Fdocs%2Fadmin_notes.html&data=05%7C02%7Cjens.urban%40freenet.ag%7C23f8ee82ed544f08372d08def8080ff9%7C7d95deb30bca4c9da61e1fd6c47ed60c%7C0%7C0%7C639220910671970524%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=eqoHjH9Pbt8h0AGo%2FDPRw%2F15YYeuXSTRMr3r8isznpE%3D&reserved=0)<https://xapian.org/docs/admin_notes.html>
It's likely fixed by adding fts_flatcurve_xapian_refresh() in fts_flatcurve_xapian_optimize_box_do(), but I am unable to verify.
michael
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
Hello Ihor, hello Michael, We tested this on our setup (2.4.4-5+debian12 (8b687aa65c)) as well and could not reproduce the issue. We are using a proprietary storage implementation (not NFS), with the following Dovecot settings: mail_nfs_storage = yes mail_nfs_index = yes After running doveadm fts optimize -u <user>, the index files were consolidated into a single index.* directory, and the existing current.* directory was removed. No current.* directory remained after the optimization. We then manually triggered an FTS search (we do not use fts_autoindex), after which a new current.* directory was created correctly with a complete Xapian database. Given that the reported setup uses NFSv3 with local_lock=none, I also wonder whether NFS locking and cache semantics may be involved here. In particular, I would verify that the required NFS locking infrastructure is working correctly before assuming that this is a generic flatcurve optimization issue. aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR" In addition, based on our testing of FTS Flatcurve, I would recommend reviewing the following settings:
- fts_flatcurve_min_term_size We use: fts_flatcurve_min_term_size = 1 instead of the default value of 2. During testing we found that indexing respects fts_flatcurve_min_term_size and therefore drops one-character terms, while query tokenization may still generate such terms. This can result in false negatives, for example when searching for URLs or identifiers containing single-character tokens. A simple example is: doveadm fts tokenize -u <USER> 'http://www.youtube.com/watch?v=kgu_fz1zytE' which generates a "v" token even with fts:flatcurve_min_term_size=2. In our measurements, changing min_term_size from 2 to 1 resulted in only a negligible increase in index size.
- fts_flatcurve_substring_search We deliberately use: fts_flatcurve_substring_search = no In our tests, enabling substring search caused massive index growth due to the additional index-only tokens, while the query tokens remained identical. Email addresses are already split into searchable tokens by the email-address tokenizer, so we found little practical benefit from enabling substring search in our environment. aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR"aEUR" There are many other FTS settings worth tuning for the specific environment. In particular, we found that fts_autoindex can cause significant performance impact during indexing for users with mailboxes containing tens of thousands of messages. In such environments, it may be preferable to disable automatic indexing and perform the indexing asynchronously using a background service instead. Regards Jens Urban Von: Michael Slusarz via dovecot <dovecot@dovecot.org> Datum: Mittwoch, 12. August 2026 um 02:24 An: Ihor Rusyn <0k.a.b.a0@gmail.com>; Ihor Rusyn via dovecot <dovecot@dovecot.org> Betreff: Re: fts-flatcurve 2.4.4: after "doveadm fts optimize" a mailbox stops indexing new mail (empty current.<id> shard, missing iamglass)
On 08/07/2026 10:22 AM MDT Ihor Rusyn via dovecot <dovecot@dovecot.org> wrote:
Hello,
"doveadm fts optimize" leaves the mailbox in a state where no new mail can be indexed. The existing index survives and searches over already-indexed mail keep working, so the damage is invisible until the next message arrives.
Version: 2.4.4 (8b687aa65c), packaged in a container image FTS: fts_flatcurve, substring_search = yes, min_term_size default (2) fts_autoindex = yes, fts_search_add_missing = yes Index store: NFSv3 (rsize/wsize 32768, local_lock=none) Mail store: NFSv3, maildir
[snip]
I cannot reproduce this, probably due to some combination of NFS latency and semantics. (Xapian itself notes that NFS operation can be problematic. [1]https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fxapian.org...)
It's likely fixed by adding fts_flatcurve_xapian_refresh() in fts_flatcurve_xapian_optimize_box_do(), but I am unable to verify.
michael
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
References
Visible links
Hi Jens,
Quick update — we can reproduce this reliably, specifically on NFS. We've also tested the same optimize path on ext4 (local) and CephFS, and neither shows the problem. Here's why we think that lines up:
The core issue is unlink()/rmdir() semantics for files/directories that are still open by the calling process. On a local POSIX filesystem (ext4) and on CephFS, unlinking a file that's still open by the same process is fully supported: the directory entry is removed immediately, and the file's data blocks stay allocated until the last file descriptor closes. So rmdir() on the parent directory succeeds right away, because as far as the filesystem is concerned the directory is already empty — the still-open file just has no name anymore.
NFS can't do that. There's no way for an NFS client to tell the server "unlink this, but I still have it open." When a client unlinks a file it still holds open, the NFS client silently works around it with what's usually called "silly rename": instead of an actual unlink, it renames the file to a hidden .nfsXXXX name, so the directory entry survives until the file is finally closed and the client can clean it up. In fts_flatcurve_xapian_optimize_box_do(), the shard's write handles are still open when the old current.* directory is deleted — fine on ext4/CephFS, but on NFS it leaves .nfsXXXX entries behind, so the rmdir() on that directory fails with ENOTEMPTY. That failure doesn't appear to be surfaced anywhere — unlink_directory() reports success regardless — so the directory is left behind, and once the open handles are eventually closed, what remains is an empty current.* directory that is still treated as the active shard.
We also tested whether mail_nfs_storage/mail_nfs_index change anything here — enabled both on our stand and reproduced the exact same failure chain (optimize succeeds silently, current.* ends up empty, next delivery fails to index with "Failed to open glass revision file," subsequent search fails with an internal error). That tracks with the code: as far as we can tell, fts-flatcurve doesn't consult either setting anywhere — they only affect the mail_index (MAIL_INDEX_OPEN_FLAG_NFS_FLUSH) and dotlock behavior in lib-storage, not the Xapian glass files flatcurve manages itself. So this doesn't look like something those settings, or NFS locking configuration in general, can fix.
Let us know if it'd help to see the exact optimize-path trace, or anything else from our reproduction.
On Wed, Aug 12, 2026 at 12:12 PM Jens Urban <Jens.Urban@freenet.ag> wrote:
Hello Ihor, hello Michael,
We tested this on our setup (2.4.4-5+debian12 (8b687aa65c)) as well and could not reproduce the issue. We are using a proprietary storage implementation (not NFS), with the following Dovecot settings:
mail_nfs_storage = yes mail_nfs_index = yes
After running doveadm fts optimize -u <user>, the index files were consolidated into a single index.* directory, and the existing current.* directory was removed. No current.* directory remained after the optimization. We then manually triggered an FTS search (we do not use fts_autoindex), after which a new current.* directory was created correctly with a complete Xapian database.
Given that the reported setup uses NFSv3 with local_lock=none, I also wonder whether NFS locking and cache semantics may be involved here. In particular, I would verify that the required NFS locking infrastructure is working correctly before assuming that this is a generic flatcurve optimization issue.
———————————————————————————————————————— In addition, based on our testing of FTS Flatcurve, I would recommend reviewing the following settings:
- fts_flatcurve_min_term_size We use: *fts_flatcurve_min_term_size = 1*
instead of the default value of 2. During testing we found that indexing respects fts_flatcurve_min_term_size and therefore drops one-character terms, while query tokenization may still generate such terms. This can result in false negatives, for example when searching for URLs or identifiers containing single-character tokens.
A simple example is: doveadm fts tokenize -u <USER> ' http://www.youtube.com/watch?v=kgu_fz1zytE'
which generates a "v" token even with fts:flatcurve_min_term_size=2. In our measurements, changing min_term_size from 2 to 1 resulted in only a negligible increase in index size.
- fts_flatcurve_substring_search We deliberately use: *fts_flatcurve_substring_search = no*
In our tests, enabling substring search caused massive index growth due to the additional index-only tokens, while the query tokens remained identical. Email addresses are already split into searchable tokens by the email-address tokenizer, so we found little practical benefit from enabling substring search in our environment.
———————————————————————————————————————— There are many other FTS settings worth tuning for the specific environment. In particular, we found that fts_autoindex can cause significant performance impact during indexing for users with mailboxes containing tens of thousands of messages. In such environments, it may be preferable to disable automatic indexing and perform the indexing asynchronously using a background service instead.
Regards Jens Urban
*Von: *Michael Slusarz via dovecot <dovecot@dovecot.org> *Datum: *Mittwoch, 12. August 2026 um 02:24 *An: *Ihor Rusyn <0k.a.b.a0@gmail.com>; Ihor Rusyn via dovecot < dovecot@dovecot.org> *Betreff: *Re: fts-flatcurve 2.4.4: after "doveadm fts optimize" a mailbox stops indexing new mail (empty current.<id> shard, missing iamglass)
On 08/07/2026 10:22 AM MDT Ihor Rusyn via dovecot <dovecot@dovecot.org> wrote:
Hello,
"doveadm fts optimize" leaves the mailbox in a state where no new mail can be indexed. The existing index survives and searches over already-indexed mail keep working, so the damage is invisible until the next message arrives.
Version: 2.4.4 (8b687aa65c), packaged in a container image FTS: fts_flatcurve, substring_search = yes, min_term_size default (2) fts_autoindex = yes, fts_search_add_missing = yes Index store: NFSv3 (rsize/wsize 32768, local_lock=none) Mail store: NFSv3, maildir
[snip]
I cannot reproduce this, probably due to some combination of NFS latency and semantics. (Xapian itself notes that NFS operation can be problematic. https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fxapian.org...) <https://xapian.org/docs/admin_notes.html>
It's likely fixed by adding fts_flatcurve_xapian_refresh() in fts_flatcurve_xapian_optimize_box_do(), but I am unable to verify.
michael
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
-- Best regards, Ihor Rusyn
Hi Jens,
Quick update -- we can reproduce this reliably, specifically on NFS. We've also tested the same optimize path on ext4 (local) and CephFS, and neither shows the problem. Here's why we think that lines up:
The core issue is unlink()/rmdir() semantics for files/directories that are still open by the calling process. On a local POSIX filesystem (ext4) and on CephFS, unlinking a file that's still open by the same process is fully supported: the directory entry is removed immediately, and the file's data blocks stay allocated until the last file descriptor closes. So rmdir() on the parent directory succeeds right away, because as far as the filesystem is concerned the directory is already empty -- the still-open file just has no name anymore.
NFS can't do that. There's no way for an NFS client to tell the server "unlink this, but I still have it open." When a client unlinks a file it still holds open, the NFS client silently works around it with what's usually called "silly rename": instead of an actual unlink, it renames the file to a hidden .nfsXXXX name, so the directory entry survives until the file is finally closed and the client can clean it up. In fts_flatcurve_xapian_optimize_box_do(), the shard's write handles are still open when the old current.* directory is deleted -- fine on ext4/CephFS, but on NFS it leaves .nfsXXXX entries behind, so the rmdir() on that directory fails with ENOTEMPTY. That failure doesn't appear to be surfaced anywhere -- unlink_directory() reports success regardless -- so the directory is left behind, and once the open handles are eventually closed, what remains is an empty current.* directory that is still treated as the active shard.
We also tested whether mail_nfs_storage/mail_nfs_index change anything here -- enabled both on our stand and reproduced the exact same failure chain (optimize succeeds silently, current.* ends up empty, next delivery fails to index with "Failed to open glass revision file," subsequent search fails with an internal error). That tracks with the code: as far as we can tell, fts-flatcurve doesn't consult either setting anywhere -- they only affect the mail_index (MAIL_INDEX_OPEN_FLAG_NFS_FLUSH) and dotlock behavior in lib-storage, not the Xapian glass files flatcurve manages itself. So this doesn't look like something those settings, or NFS locking configuration in general, can fix.
Let us know if it'd help to see the exact optimize-path trace, or anything else from our reproduction.
On Wed, Aug 12, 2026 at 12:12PM Jens Urban <[1]Jens.Urban@freenet.ag> wrote:
Hello Ihor, hello Michael,
We tested this on our setup (2.4.4-5+debian12 (8b687aa65c)) as well and
could not reproduce the issue. We are using a proprietary storage
implementation (not NFS), with the following Dovecot settings:
mail_nfs_storage = yes
mail_nfs_index = yes
After running doveadm fts optimize -u <user>, the index files were
consolidated into a single index.* directory, and the existing current.*
directory was removed. No current.* directory remained after the
optimization. We then manually triggered an FTS search (we do not use
fts_autoindex), after which a new current.* directory was created
correctly with a complete Xapian database.
Given that the reported setup uses NFSv3 with local_lock=none, I also
wonder whether NFS locking and cache semantics may be involved here. In
particular, I would verify that the required NFS locking infrastructure
is working correctly before assuming that this is a generic flatcurve
optimization issue.
--------------------------------------------------------------------------------
In addition, based on our testing of FTS Flatcurve, I would recommend
reviewing the following settings:
1. fts_flatcurve_min_term_size
We use: fts_flatcurve_min_term_size = 1
instead of the default value of 2. During testing we found that indexing
respects fts_flatcurve_min_term_size and therefore drops one-character
terms, while query tokenization may still generate such terms. This can
result in false negatives, for example when searching for URLs or
identifiers containing single-character tokens.
A simple example is: doveadm fts tokenize -u <USER>
'[2]http://www.youtube.com/watch?v=kgu_fz1zytE'
which generates a "v" token even with fts:flatcurve_min_term_size=2.
In our measurements, changing min_term_size from 2 to 1 resulted in only
a negligible increase in index size.
2. fts_flatcurve_substring_search
We deliberately use: fts_flatcurve_substring_search = no
In our tests, enabling substring search caused massive index growth due
to the additional index-only tokens, while the query tokens remained
identical. Email addresses are already split into searchable tokens by
the email-address tokenizer, so we found little practical benefit from
enabling substring search in our environment.
--------------------------------------------------------------------------------
There are many other FTS settings worth tuning for the specific
environment. In particular, we found that fts_autoindex can cause
significant performance impact during indexing for users with mailboxes
containing tens of thousands of messages. In such environments, it may
be preferable to disable automatic indexing and perform the indexing
asynchronously using a background service instead.
Regards
Jens Urban
Von: Michael Slusarz via dovecot <[3]dovecot@dovecot.org>
Datum: Mittwoch, 12. August 2026 um 02:24
An: Ihor Rusyn <[4]0k.a.b.a0@gmail.com>; Ihor Rusyn via dovecot
<[5]dovecot@dovecot.org>
Betreff: Re: fts-flatcurve 2.4.4: after "doveadm fts optimize" a mailbox
stops indexing new mail (empty current.<id> shard, missing iamglass)
> On 08/07/2026 10:22 AM MDT Ihor Rusyn via dovecot
<[6]dovecot@dovecot.org> wrote:
>
> Hello,
>
> "doveadm fts optimize" leaves the mailbox in a state where no new mail
can
> be
> indexed. The existing index survives and searches over already-indexed
mail
> keep
> working, so the damage is invisible until the next message arrives.
>
> Version: 2.4.4 (8b687aa65c), packaged in a container image
> FTS: fts_flatcurve, substring_search = yes, min_term_size
default
> (2)
> fts_autoindex = yes, fts_search_add_missing = yes
> Index store: NFSv3 (rsize/wsize 32768, local_lock=none)
> Mail store: NFSv3, maildir
[snip]
I cannot reproduce this, probably due to some combination of NFS latency
and semantics. (Xapian itself notes that NFS operation can be
problematic.
[7]https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fxapian.org%2Fdocs%2Fadmin_notes.html&data=05%7C02%7Cjens.urban%40freenet.ag%7C23f8ee82ed544f08372d08def8080ff9%7C7d95deb30bca4c9da61e1fd6c47ed60c%7C0%7C0%7C639220910671970524%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=eqoHjH9Pbt8h0AGo%2FDPRw%2F15YYeuXSTRMr3r8isznpE%3D&reserved=0)
It's likely fixed by adding fts_flatcurve_xapian_refresh() in
fts_flatcurve_xapian_optimize_box_do(), but I am unable to verify.
michael
_______________________________________________
dovecot mailing list -- [8]dovecot@dovecot.org
To unsubscribe send an email to [9]dovecot-leave@dovecot.org-- Best regards, Ihor Rusyn
References
Visible links
- mailto:Jens.Urban@freenet.ag
- http://www.youtube.com/watch?v=kgu_fz1zytE
- mailto:dovecot@dovecot.org
- mailto:0k.a.b.a0@gmail.com
- mailto:dovecot@dovecot.org
- mailto:dovecot@dovecot.org
- https://xapian.org/docs/admin_notes.html
- mailto:dovecot@dovecot.org
- mailto:dovecot-leave@dovecot.org
"Ihor" == Ihor Rusyn via dovecot <dovecot@dovecot.org> writes:
Quick update -- we can reproduce this reliably, specifically on NFS. We've also tested the same optimize path on ext4 (local) and CephFS, and neither shows the problem. Here's why we think that lines up:
The core issue is unlink()/rmdir() semantics for files/directories that are still open by the calling process. On a local POSIX filesystem (ext4) and on CephFS, unlinking a file that's still open by the same process is fully supported: the directory entry is removed immediately, and the file's data blocks stay allocated until the last file descriptor closes. So rmdir() on the parent directory succeeds right away, because as far as the filesystem is concerned the directory is already empty -- the still-open file just has no name anymore.
NFS can't do that. There's no way for an NFS client to tell the server "unlink this, but I still have it open." When a client unlinks a file it still holds open, the NFS client silently works around it with what's usually called "silly rename": instead of an actual unlink, it renames the file to a hidden .nfsXXXX name, so the directory entry survives until the file is finally closed and the client can clean it up. In fts_flatcurve_xapian_optimize_box_do(), the shard's write handles are still open when the old current.* directory is deleted -- fine on ext4/CephFS, but on NFS it leaves .nfsXXXX entries behind, so the rmdir() on that directory fails with ENOTEMPTY. That failure doesn't appear to be surfaced anywhere -- unlink_directory() reports success regardless -- so the directory is left behind, and once the open handles are eventually closed, what remains is an empty current.* directory that is still treated as the active shard.
To me this screams that when the rmdir() fails, instead of just bailing, the code should move the directory to another name and then create a new directory.
But I'm not sure if those .nfs### files every get cleaned up, so it would have to be part of the scan, purge, clean process to go through and remove those files by hand.
So I personally love using NFS filesystems for storage, but I can see how the semantics of the way the tool is written don't quite match how NFS works in the real world.
So the options could be:
Document clearly that this is not supported on NFS volumes
patch the code to handle it more gracefully (like I suggested above)
Change how the code works to not depend on this rmdir() action at all.
Just an interested bystander... :-)
John
participants (4)
-
Ihor Rusyn
-
Jens Urban
-
John Stoffel
-
Michael Slusarz