Hello,
"doveadm fts optimize" leaves the mailbox in a state where no new mail can be indexed. The existing index survives and searches over already-indexed mail keep working, so the damage is invisible until the next message arrives.
Version: 2.4.4 (8b687aa65c), packaged in a container image FTS: fts_flatcurve, substring_search = yes, min_term_size default (2) fts_autoindex = yes, fts_search_add_missing = yes Index store: NFSv3 (rsize/wsize 32768, local_lock=none) Mail store: NFSv3, maildir
Reproduction
Starting point: a healthy index.
doveadm fts flatcurve check -u user@example.com INBOX
INBOX guid=88885837bb33136365390000d09efc50 errors=0 shards=1
ls /indexes/.../.INBOX/fts-flatcurve/*/
current.1786117302443530: flintlock iamglass postlist.glass termlist.glass
doveadm search -u user@example.com mailbox INBOX text a | tail -1
88885837bb33136365390000d09efc50 58
Optimize. Exits 0, prints nothing.
doveadm fts optimize -u user@example.com
echo $?
0
State afterwards: the compacted data is in a new index.<n> directory, and the pre-existing current.<id> directory is still present but EMPTY.
ls /indexes/.../.INBOX/fts-flatcurve/*/
current.1786117302443530: index.1464: flintlock iamglass postlist.glass termlist.glass
ls -ld .../fts-flatcurve/current.1786117302443530
drwx------ 2 dovecot dovecot 4096 Aug 7 16:08 .../current.1786117302443530
doveadm fts flatcurve check -u user@example.com INBOX
Error: fts-flatcurve(INBOX): Cannot open DB (RO; current.1786117302443530); DatabaseNotFoundError: Couldn't detect type of database Error: fts-flatcurve(INBOX): Check failed; DatabaseOpeningError: Directory does not contain a Xapian database
Searching already-indexed mail still works at this point:
doveadm search -u user@example.com mailbox INBOX text a | tail -1
88885837bb33136365390000d09efc50 58
Deliver one message and index it. This is where it breaks:
printf 'From: a@b.c\nSubject: reproprobe\n\nbody\n' \
| doveadm save -u user@example.com -m INBOXInfo: Mailbox INBOX: save: box=INBOX, uid=59, subject=reproprobe
doveadm index -u user@example.com INBOX
Error: fts-flatcurve(INBOX): Cannot open DB (RW, current.1786117302443530): .../current.1786117302443530/iamglass: Failed to open glass revision file for reading (No such file or directory) Error: cmd index: Mailbox INBOX: Precache for UID=59 failed: Internal error
From now on SEARCH fails too, because fts_search_add_missing makes it index first:
doveadm search -u user@example.com mailbox INBOX text reproprobe
Error: Mailbox INBOX: indexer failed to index mailbox Error: cmd search: Searching mailbox INBOX failed: Internal error occurred.
Over IMAP the client sees: NO [SERVERBUG] Internal error occurred.
It does not recover by itself
"doveadm index" and autoindexing both take the same write path and fail the same way. "doveadm fts rescan" does not help either -- it also cannot open the empty shard. The only repair we found is to delete the folder's fts-flatcurve directory and index the folder again.
Impact
A quiet mailbox looks fine for as long as nothing is delivered to it, so this can sit unnoticed. We found it because a monitoring mailbox that receives one message per minute started returning SERVERBUG on SEARCH. In our case a maintenance run that called "doveadm fts optimize" per mailbox left 114 mailboxes in this state, which is all of the ones that had any mail.
Possible cause (not verified)
In fts_flatcurve_xapian_optimize_box_do() every shard is first opened for writing ("We need to lock all of the mailboxes so nothing changes while we are optimizing"), and the old current.*/index.* directories are then deleted from disk while those write handles are still open. A handle that is closed afterwards appears to recreate its directory without a glass revision file, which matches what is left behind.
Also, in the same function the result of
fts_flatcurve_xapian_rename_db(backend, dbpath, NULL, error_r)is discarded: rename_db() does not update the path it is given and returns the new path only through newpath_r, which is NULL at this call site.
I have not tried to confirm either of these beyond reading the code.
Not the same as the known root-owned iamglass report
There is an older report where "doveadm fts optimize -A" run from cron as root left iamglass owned by root, producing the same error string. That is not this: here the directories are owned by the mail user (dovecot:dovecot, 0700) and the revision file does not exist at all.
-- Best regards, Ihor Rusyn
Hello, "doveadm fts optimize" leaves the mailbox in a state where no new mail can be indexed. The existing index survives and searches over already-indexed mail keep working, so the damage is invisible until the next message arrives. Version: 2.4.4 (8b687aa65c), packaged in a container image FTS: fts_flatcurve, substring_search = yes, min_term_size default (2) fts_autoindex = yes, fts_search_add_missing = yes Index store: NFSv3 (rsize/wsize 32768, local_lock=none) Mail store: NFSv3, maildir Reproduction
Starting point: a healthy index.
doveadm fts flatcurve check -u [1]user@example.com INBOX
INBOX guid=88885837bb33136365390000d09efc50 errors=0 shards=1
ls /indexes/.../.INBOX/fts-flatcurve/*/
current.1786117302443530: flintlock iamglass postlist.glass termlist.glass
doveadm search -u [2]user@example.com mailbox INBOX text a | tail -1
88885837bb33136365390000d09efc50 58
Optimize. Exits 0, prints nothing.
doveadm fts optimize -u [3]user@example.com
echo $?
0
State afterwards: the compacted data is in a new index.<n> directory, and the pre-existing current.<id> directory is still present but EMPTY.
ls /indexes/.../.INBOX/fts-flatcurve/*/
current.1786117302443530: index.1464: flintlock iamglass postlist.glass termlist.glass
ls -ld .../fts-flatcurve/current.1786117302443530
drwx------ 2 dovecot dovecot 4096 Aug 7 16:08 .../current.1786117302443530
doveadm fts flatcurve check -u [4]user@example.com INBOX
Error: fts-flatcurve(INBOX): Cannot open DB (RO; current.1786117302443530); DatabaseNotFoundError: Couldn't detect type of database Error: fts-flatcurve(INBOX): Check failed; DatabaseOpeningError: Directory does not contain a Xapian database Searching already-indexed mail still works at this point:
doveadm search -u [5]user@example.com mailbox INBOX text a | tail -1
88885837bb33136365390000d09efc50 58
Deliver one message and index it. This is where it breaks:
printf 'From: a@b.c\nSubject: reproprobe\n\nbody\n' \
| doveadm save -u [6]user@example.com -m INBOXInfo: Mailbox INBOX: save: box=INBOX, uid=59, subject=reproprobe
doveadm index -u [7]user@example.com INBOX
Error: fts-flatcurve(INBOX): Cannot open DB (RW, current.1786117302443530): .../current.1786117302443530/iamglass: Failed to open glass revision file for reading (No such file or directory) Error: cmd index: Mailbox INBOX: Precache for UID=59 failed: Internal error
From now on SEARCH fails too, because fts_search_add_missing makes it index first:
doveadm search -u [8]user@example.com mailbox INBOX text reproprobe
Error: Mailbox INBOX: indexer failed to index mailbox Error: cmd search: Searching mailbox INBOX failed: Internal error occurred. Over IMAP the client sees: NO [SERVERBUG] Internal error occurred. It does not recover by itself
"doveadm index" and autoindexing both take the same write path and fail the same way. "doveadm fts rescan" does not help either -- it also cannot open the empty shard. The only repair we found is to delete the folder's fts-flatcurve directory and index the folder again. Impact
A quiet mailbox looks fine for as long as nothing is delivered to it, so this can sit unnoticed. We found it because a monitoring mailbox that receives one message per minute started returning SERVERBUG on SEARCH. In our case a maintenance run that called "doveadm fts optimize" per mailbox left 114 mailboxes in this state, which is all of the ones that had any mail. Possible cause (not verified)
In fts_flatcurve_xapian_optimize_box_do() every shard is first opened for writing ("We need to lock all of the mailboxes so nothing changes while we are optimizing"), and the old current.*/index.* directories are then deleted from disk while those write handles are still open. A handle that is closed afterwards appears to recreate its directory without a glass revision file, which matches what is left behind. Also, in the same function the result of fts_flatcurve_xapian_rename_db(backend, dbpath, NULL, error_r) is discarded: rename_db() does not update the path it is given and returns the new path only through newpath_r, which is NULL at this call site. I have not tried to confirm either of these beyond reading the code. Not the same as the known root-owned iamglass report
There is an older report where "doveadm fts optimize -A" run from cron as root left iamglass owned by root, producing the same error string. That is not this: here the directories are owned by the mail user (dovecot:dovecot, 0700) and the revision file does not exist at all.
-- Best regards, Ihor Rusyn
References
Visible links
- mailto:user@example.com
- mailto:user@example.com
- mailto:user@example.com
- mailto:user@example.com
- mailto:user@example.com
- mailto:user@example.com
- mailto:user@example.com
- mailto:user@example.com