Hello,

"doveadm fts optimize" leaves the mailbox in a state where no new mail can be indexed. The existing index survives and searches over already-indexed mail keep working, so the damage is invisible until the next message arrives.

Version: 2.4.4 (8b687aa65c), packaged in a container image FTS: fts_flatcurve, substring_search = yes, min_term_size default (2) fts_autoindex = yes, fts_search_add_missing = yes Index store: NFSv3 (rsize/wsize 32768, local_lock=none) Mail store: NFSv3, maildir

Reproduction

1. Starting point: a healthy index.

   doveadm fts flatcurve check -u user@example.com INBOX

   INBOX guid=88885837bb33136365390000d09efc50 errors=0 shards=1

   ls /indexes/.../.INBOX/fts-flatcurve/*/

   current.1786117302443530: flintlock iamglass postlist.glass termlist.glass

   doveadm search -u user@example.com mailbox INBOX text a | tail -1

   88885837bb33136365390000d09efc50 58

2. Optimize. Exits 0, prints nothing.

   doveadm fts optimize -u user@example.com

   echo $?

   0

3. State afterwards: the compacted data is in a new index.<n> directory, and the pre-existing current.<id> directory is still present but EMPTY.

   ls /indexes/.../.INBOX/fts-flatcurve/*/

   current.1786117302443530: index.1464: flintlock iamglass postlist.glass termlist.glass

   ls -ld .../fts-flatcurve/current.1786117302443530

   drwx------ 2 dovecot dovecot 4096 Aug 7 16:08 .../current.1786117302443530

   doveadm fts flatcurve check -u user@example.com INBOX

   Error: fts-flatcurve(INBOX): Cannot open DB (RO; current.1786117302443530); DatabaseNotFoundError: Couldn't detect type of database Error: fts-flatcurve(INBOX): Check failed; DatabaseOpeningError: Directory does not contain a Xapian database

   Searching already-indexed mail still works at this point:

   doveadm search -u user@example.com mailbox INBOX text a | tail -1

   88885837bb33136365390000d09efc50 58

4. Deliver one message and index it. This is where it breaks:

   printf 'From: a@b.c\nSubject: reproprobe\n\nbody\n' \

   | doveadm save -u user@example.com -m INBOX

   Info: Mailbox INBOX: save: box=INBOX, uid=59, subject=reproprobe

   doveadm index -u user@example.com INBOX

   Error: fts-flatcurve(INBOX): Cannot open DB (RW, current.1786117302443530): .../current.1786117302443530/iamglass: Failed to open glass revision file for reading (No such file or directory) Error: cmd index: Mailbox INBOX: Precache for UID=59 failed: Internal error

5. From now on SEARCH fails too, because fts_search_add_missing makes it index first:

   doveadm search -u user@example.com mailbox INBOX text reproprobe

   Error: Mailbox INBOX: indexer failed to index mailbox Error: cmd search: Searching mailbox INBOX failed: Internal error occurred.

   Over IMAP the client sees: NO [SERVERBUG] Internal error occurred.

It does not recover by itself

"doveadm index" and autoindexing both take the same write path and fail the same way. "doveadm fts rescan" does not help either -- it also cannot open the empty shard. The only repair we found is to delete the folder's fts-flatcurve directory and index the folder again.

Impact

A quiet mailbox looks fine for as long as nothing is delivered to it, so this can sit unnoticed. We found it because a monitoring mailbox that receives one message per minute started returning SERVERBUG on SEARCH. In our case a maintenance run that called "doveadm fts optimize" per mailbox left 114 mailboxes in this state, which is all of the ones that had any mail.

Possible cause (not verified)

In fts_flatcurve_xapian_optimize_box_do() every shard is first opened for writing ("We need to lock all of the mailboxes so nothing changes while we are optimizing"), and the old current.*/index.* directories are then deleted from disk while those write handles are still open. A handle that is closed afterwards appears to recreate its directory without a glass revision file, which matches what is left behind.

Also, in the same function the result of

fts_flatcurve_xapian_rename_db(backend, dbpath, NULL, error_r)

is discarded: rename_db() does not update the path it is given and returns the new path only through newpath_r, which is NULL at this call site.

I have not tried to confirm either of these beyond reading the code.

Not the same as the known root-owned iamglass report

There is an older report where "doveadm fts optimize -A" run from cron as root left iamglass owned by root, producing the same error string. That is not this: here the directories are owned by the mail user (dovecot:dovecot, 0700) and the revision file does not exist at all.

-- Best regards, Ihor Rusyn

Hello, "doveadm fts optimize" leaves the mailbox in a state where no new mail can be indexed. The existing index survives and searches over already-indexed mail keep working, so the damage is invisible until the next message arrives. Version: 2.4.4 (8b687aa65c), packaged in a container image FTS: fts_flatcurve, substring_search = yes, min_term_size default (2) fts_autoindex = yes, fts_search_add_missing = yes Index store: NFSv3 (rsize/wsize 32768, local_lock=none) Mail store: NFSv3, maildir Reproduction

1. Starting point: a healthy index.

   doveadm fts flatcurve check -u [1]user@example.com INBOX

   INBOX guid=88885837bb33136365390000d09efc50 errors=0 shards=1

   ls /indexes/.../.INBOX/fts-flatcurve/*/

   current.1786117302443530: flintlock iamglass postlist.glass termlist.glass

   doveadm search -u [2]user@example.com mailbox INBOX text a | tail -1

   88885837bb33136365390000d09efc50 58

2. Optimize. Exits 0, prints nothing.

   doveadm fts optimize -u [3]user@example.com

   echo $?

   0

3. State afterwards: the compacted data is in a new index.<n> directory, and the pre-existing current.<id> directory is still present but EMPTY.

   ls /indexes/.../.INBOX/fts-flatcurve/*/

   current.1786117302443530: index.1464: flintlock iamglass postlist.glass termlist.glass

   ls -ld .../fts-flatcurve/current.1786117302443530

   drwx------ 2 dovecot dovecot 4096 Aug 7 16:08 .../current.1786117302443530

   doveadm fts flatcurve check -u [4]user@example.com INBOX

   Error: fts-flatcurve(INBOX): Cannot open DB (RO; current.1786117302443530); DatabaseNotFoundError: Couldn't detect type of database Error: fts-flatcurve(INBOX): Check failed; DatabaseOpeningError: Directory does not contain a Xapian database Searching already-indexed mail still works at this point:

   doveadm search -u [5]user@example.com mailbox INBOX text a | tail -1

   88885837bb33136365390000d09efc50 58

4. Deliver one message and index it. This is where it breaks:

   printf 'From: a@b.c\nSubject: reproprobe\n\nbody\n' \

   | doveadm save -u [6]user@example.com -m INBOX

   Info: Mailbox INBOX: save: box=INBOX, uid=59, subject=reproprobe

   doveadm index -u [7]user@example.com INBOX

   Error: fts-flatcurve(INBOX): Cannot open DB (RW, current.1786117302443530): .../current.1786117302443530/iamglass: Failed to open glass revision file for reading (No such file or directory) Error: cmd index: Mailbox INBOX: Precache for UID=59 failed: Internal error

5. From now on SEARCH fails too, because fts_search_add_missing makes it index first:

   doveadm search -u [8]user@example.com mailbox INBOX text reproprobe

   Error: Mailbox INBOX: indexer failed to index mailbox Error: cmd search: Searching mailbox INBOX failed: Internal error occurred. Over IMAP the client sees: NO [SERVERBUG] Internal error occurred. It does not recover by itself

"doveadm index" and autoindexing both take the same write path and fail the same way. "doveadm fts rescan" does not help either -- it also cannot open the empty shard. The only repair we found is to delete the folder's fts-flatcurve directory and index the folder again. Impact

A quiet mailbox looks fine for as long as nothing is delivered to it, so this can sit unnoticed. We found it because a monitoring mailbox that receives one message per minute started returning SERVERBUG on SEARCH. In our case a maintenance run that called "doveadm fts optimize" per mailbox left 114 mailboxes in this state, which is all of the ones that had any mail. Possible cause (not verified)

In fts_flatcurve_xapian_optimize_box_do() every shard is first opened for writing ("We need to lock all of the mailboxes so nothing changes while we are optimizing"), and the old current.*/index.* directories are then deleted from disk while those write handles are still open. A handle that is closed afterwards appears to recreate its directory without a glass revision file, which matches what is left behind. Also, in the same function the result of fts_flatcurve_xapian_rename_db(backend, dbpath, NULL, error_r) is discarded: rename_db() does not update the path it is given and returns the new path only through newpath_r, which is NULL at this call site. I have not tried to confirm either of these beyond reading the code. Not the same as the known root-owned iamglass report

There is an older report where "doveadm fts optimize -A" run from cron as root left iamglass owned by root, producing the same error string. That is not this: here the directories are owned by the mail user (dovecot:dovecot, 0700) and the revision file does not exist at all.

-- Best regards, Ihor Rusyn

References

Visible links

1. mailto:user@example.com
2. mailto:user@example.com
3. mailto:user@example.com
4. mailto:user@example.com
5. mailto:user@example.com
6. mailto:user@example.com
7. mailto:user@example.com
8. mailto:user@example.com