Hello,

Summary

fts_flatcurve_xapian_optimize_rebuild() aborts on the first document whose termlist is missing, which makes shard compaction impossible for the affected folder from that point on. The failure is silent — the operation reports success — so shards accumulate without bound. We have seen folders reach several thousand shards, with an index orders of magnitude larger than the messages it covers.

Present in 2.4.4, 2.4.5 and current main (checked at 732917749a).

Symptom

doveadm fts optimize -u <user> Error: fts-flatcurve: Optimize failed: DocNotFoundError: No termlist for document 257091

exit code 0, shard count unchanged

In production this is logged by indexer-worker and repeats every 20-30 minutes, once per delivery, each run leaving another shard behind. Affected folders also keep an optimize directory containing an empty Xapian database.

Mechanism

1. Long-lived, high-churn folders end up with shards holding disjoint UID ranges. db->compact() with DBCOMPACT_NO_RENUMBER throws InvalidOperationError, which the code expects and handles.
2. It falls back to fts_flatcurve_xapian_optimize_rebuild(), which walks the MSet and copies each document into a fresh database.
3. iter.get_document() sits **outside** the try block. When a document in the MSet has no termlist it throws DocNotFoundError, which the local handler never sees.
4. The exception propagates to fts_flatcurve_xapian_optimize_box_do(), is caught by catch (Xapian::Error &e), sets failed = TRUE — and that path does return 0.

The caller therefore sees success. Old shards are never deleted, the merge never happens, and every subsequent message adds another shard. The folder can never recover on its own.

Reproduction

Reproducible from scratch in about five minutes on a scratch mailbox, with generated messages only. Three conditions have to coincide; raising uidnext alone does not reproduce it.

U=test@example.com

1. push uidnext past 2^31

doveadm mailbox update -u $U --min-next-uid 2277107224 INBOX

2. + 3. churn: grow, index into many shards, expunge half.

the expunged messages leave their documents behind in older shards.

for round in $(seq 1 10); do n=$(doveadm -f flow mailbox status -u $U messages INBOX | tail -1 | sed 's/.*messages=//') [ "$n" -gt 400 ] && n=400 doveadm copy -u $U INBOX mailbox INBOX 1:$n doveadm -o fts_flatcurve_rotate_count=25 index -u $U INBOX doveadm expunge -u $U mailbox INBOX 1:$((n/2)) done

doveadm fts optimize -u $U

Rounds 1-9 compact cleanly to 2 shards. The tenth breaks: 1,838 live messages, 84 shards, uidnext 2277110749 — and optimize then fails as above and never merges again.

Suggested fix

Move get_document() inside the try, and skip an unreadable document instead of aborting the rebuild. A document whose termlist is gone belongs to a message that no longer exists, so there is nothing to carry into the optimized database; the rebuild exists precisely to salvage databases the native compact refused.

Patch attached (fts-flatcurve-optimize-rebuild.diff).

Other Xapian::Error types keep aborting the rebuild, so genuine corruption is still not silently swallowed.

Separately, the return 0 on failed in fts_flatcurve_xapian_optimize_box_do() looks questionable regardless of this fix: a failed optimize is reported to the caller as success, which is why this went unnoticed for so long.

Notes

Affected folders are long-lived and high-churn: most of the UIDs they have issued belong to messages that were deleted years ago. Folders that compact normally stay at 9-11 shards, so the ones in this state stand out immediately by shard count alone — which is also the cheapest way to find them:

find <index root> -type d -path '*/fts-flatcurve/*' -prune -printf '%h\n'
| sort | uniq -c | sort -rn

-- Best regards, Ihor Rusyn

Hello,

Summary

fts_flatcurve_xapian_optimize_rebuild() aborts on the first document whose termlist is missing, which makes shard compaction impossible for the affected folder from that point on. The failure is silent -- the operation reports success -- so shards accumulate without bound. We have seen folders reach several thousand shards, with an index orders of magnitude larger than the messages it covers. Present in 2.4.4, 2.4.5 and current main (checked at 732917749a).

Symptom

   doveadm fts optimize -u <user>
   Error: fts-flatcurve: Optimize failed: DocNotFoundError: No termlist for document 257091
   # exit code 0, shard count unchanged

In production this is logged by indexer-worker and repeats every 20-30 minutes, once per delivery, each run leaving another shard behind. Affected folders also keep an optimize directory containing an empty Xapian database.

Mechanism

1. Long-lived, high-churn folders end up with shards holding disjoint UID ranges. db->compact() with DBCOMPACT_NO_RENUMBER throws InvalidOperationError, which the code expects and handles.
2. It falls back to fts_flatcurve_xapian_optimize_rebuild(), which walks the MSet and copies each document into a fresh database.
3. iter.get_document() sits **outside** the try block. When a document in the MSet has no termlist it throws DocNotFoundError, which the local handler never sees.
4. The exception propagates to fts_flatcurve_xapian_optimize_box_do(), is caught by catch (Xapian::Error &e), sets failed = TRUE -- and that path does return 0. The caller therefore sees success. Old shards are never deleted, the merge never happens, and every subsequent message adds another shard. The folder can never recover on its own.

Reproduction

Reproducible from scratch in about five minutes on a scratch mailbox, with generated messages only. Three conditions have to coincide; raising uidnext alone does not reproduce it. U=[1]test@example.com # 1. push uidnext past 2^31 doveadm mailbox update -u $U --min-next-uid 2277107224 INBOX # 2. + 3. churn: grow, index into many shards, expunge half. # the expunged messages leave their documents behind in older shards. for round in $(seq 1 10); do n=$(doveadm -f flow mailbox status -u $U messages INBOX | tail -1 | sed 's/.*messages=//') [ "$n" -gt 400 ] && n=400 doveadm copy -u $U INBOX mailbox INBOX 1:$n doveadm -o fts_flatcurve_rotate_count=25 index -u $U INBOX doveadm expunge -u $U mailbox INBOX 1:$((n/2)) done doveadm fts optimize -u $U Rounds 1-9 compact cleanly to 2 shards. The tenth breaks: 1,838 live messages, 84 shards, uidnext 2277110749 -- and optimize then fails as above and never merges again.

Suggested fix

Move get_document() inside the try, and skip an unreadable document instead of aborting the rebuild. A document whose termlist is gone belongs to a message that no longer exists, so there is nothing to carry into the optimized database; the rebuild exists precisely to salvage databases the native compact refused. Patch attached (fts-flatcurve-optimize-rebuild.diff). Other Xapian::Error types keep aborting the rebuild, so genuine corruption is still not silently swallowed. Separately, the return 0 on failed in fts_flatcurve_xapian_optimize_box_do() looks questionable regardless of this fix: a failed optimize is reported to the caller as success, which is why this went unnoticed for so long.

Notes

Affected folders are long-lived and high-churn: most of the UIDs they have issued belong to messages that were deleted years ago. Folders that compact normally stay at 9-11 shards, so the ones in this state stand out immediately by shard count alone -- which is also the cheapest way to find them: find <index root> -type d -path '*/fts-flatcurve/*' -prune -printf '%h\n'
| sort | uniq -c | sort -rn

Best regards, Ihor Rusyn

References

Visible links

1. mailto:test@example.com