Hello,
"doveadm fts optimize" leaves the mailbox in a state where no new mail can
be
indexed. The existing index survives and searches over already-indexed mail
keep
working, so the damage is invisible until the next message arrives.
Version: 2.4.4 (8b687aa65c), packaged in a container image
FTS: fts_flatcurve, substring_search = yes, min_term_size default
(2)
fts_autoindex = yes, fts_search_add_missing = yes
Index store: NFSv3 (rsize/wsize 32768, local_lock=none)
Mail store: NFSv3, maildir
Reproduction
------------
1) Starting point: a healthy index.
# doveadm fts flatcurve check -u user(a)example.com INBOX
INBOX guid=88885837bb33136365390000d09efc50 errors=0 shards=1
# ls /indexes/.../.INBOX/fts-flatcurve/*/
current.1786117302443530: flintlock iamglass postlist.glass
termlist.glass
# doveadm search -u user(a)example.com mailbox INBOX text a | tail -1
88885837bb33136365390000d09efc50 58
2) Optimize. Exits 0, prints nothing.
# doveadm fts optimize -u user(a)example.com
# echo $?
0
3) State afterwards: the compacted data is in a new index.<n> directory,
and the
pre-existing current.<id> directory is still present but EMPTY.
# ls /indexes/.../.INBOX/fts-flatcurve/*/
current.1786117302443530:
index.1464: flintlock iamglass postlist.glass termlist.glass
# ls -ld .../fts-flatcurve/current.1786117302443530
drwx------ 2 dovecot dovecot 4096 Aug 7 16:08
.../current.1786117302443530
# doveadm fts flatcurve check -u user(a)example.com INBOX
Error: fts-flatcurve(INBOX): Cannot open DB (RO;
current.1786117302443530);
DatabaseNotFoundError: Couldn't detect type of database
Error: fts-flatcurve(INBOX): Check failed; DatabaseOpeningError:
Directory does not contain a Xapian database
Searching already-indexed mail still works at this point:
# doveadm search -u user(a)example.com mailbox INBOX text a | tail -1
88885837bb33136365390000d09efc50 58
4) Deliver one message and index it. This is where it breaks:
# printf 'From: a(a)b.c\nSubject: reproprobe\n\nbody\n' \
| doveadm save -u user(a)example.com -m INBOX
Info: Mailbox INBOX: save: box=INBOX, uid=59, subject=reproprobe
# doveadm index -u user(a)example.com INBOX
Error: fts-flatcurve(INBOX): Cannot open DB (RW,
current.1786117302443530):
.../current.1786117302443530/iamglass: Failed to open glass
revision
file for reading (No such file or directory)
Error: cmd index: Mailbox INBOX: Precache for UID=59 failed: Internal
error
5) From now on SEARCH fails too, because fts_search_add_missing makes it
index
first:
# doveadm search -u user(a)example.com mailbox INBOX text reproprobe
Error: Mailbox INBOX: indexer failed to index mailbox
Error: cmd search: Searching mailbox INBOX failed: Internal error
occurred.
Over IMAP the client sees: NO [SERVERBUG] Internal error occurred.
It does not recover by itself
-----------------------------
"doveadm index" and autoindexing both take the same write path and fail the
same
way. "doveadm fts rescan" does not help either -- it also cannot open the
empty
shard. The only repair we found is to delete the folder's fts-flatcurve
directory and index the folder again.
Impact
------
A quiet mailbox looks fine for as long as nothing is delivered to it, so
this can
sit unnoticed. We found it because a monitoring mailbox that receives one
message
per minute started returning SERVERBUG on SEARCH. In our case a maintenance
run
that called "doveadm fts optimize" per mailbox left 114 mailboxes in this
state,
which is all of the ones that had any mail.
Possible cause (not verified)
-----------------------------
In fts_flatcurve_xapian_optimize_box_do() every shard is first opened for
writing
("We need to lock all of the mailboxes so nothing changes while we are
optimizing"), and the old current.*/index.* directories are then deleted
from
disk while those write handles are still open. A handle that is closed
afterwards
appears to recreate its directory without a glass revision file, which
matches
what is left behind.
Also, in the same function the result of
fts_flatcurve_xapian_rename_db(backend, dbpath, NULL, error_r)
is discarded: rename_db() does not update the path it is given and returns
the
new path only through newpath_r, which is NULL at this call site.
I have not tried to confirm either of these beyond reading the code.
Not the same as the known root-owned iamglass report
----------------------------------------------------
There is an older report where "doveadm fts optimize -A" run from cron as
root
left iamglass owned by root, producing the same error string. That is not
this:
here the directories are owned by the mail user (dovecot:dovecot, 0700) and
the
revision file does not exist at all.
--
Best regards,
Ihor Rusyn
Hello,
"doveadm fts optimize" leaves the mailbox in a state where no new mail can
be
indexed. The existing index survives and searches over already-indexed
mail keep
working, so the damage is invisible until the next message arrives.
Version: 2.4.4 (8b687aa65c), packaged in a container image
FTS: fts_flatcurve, substring_search = yes, min_term_size default
(2)
fts_autoindex = yes, fts_search_add_missing = yes
Index store: NFSv3 (rsize/wsize 32768, local_lock=none)
Mail store: NFSv3, maildir
Reproduction
------------
1) Starting point: a healthy index.
# doveadm fts flatcurve check -u [1]user(a)example.com INBOX
INBOX guid=88885837bb33136365390000d09efc50 errors=0 shards=1
# ls /indexes/.../.INBOX/fts-flatcurve/*/
current.1786117302443530: flintlock iamglass postlist.glass
termlist.glass
# doveadm search -u [2]user(a)example.com mailbox INBOX text a | tail -1
88885837bb33136365390000d09efc50 58
2) Optimize. Exits 0, prints nothing.
# doveadm fts optimize -u [3]user(a)example.com
# echo $?
0
3) State afterwards: the compacted data is in a new index.<n> directory,
and the
pre-existing current.<id> directory is still present but EMPTY.
# ls /indexes/.../.INBOX/fts-flatcurve/*/
current.1786117302443530:
index.1464: flintlock iamglass postlist.glass termlist.glass
# ls -ld .../fts-flatcurve/current.1786117302443530
drwx------ 2 dovecot dovecot 4096 Aug 7 16:08
.../current.1786117302443530
# doveadm fts flatcurve check -u [4]user(a)example.com INBOX
Error: fts-flatcurve(INBOX): Cannot open DB (RO;
current.1786117302443530);
DatabaseNotFoundError: Couldn't detect type of database
Error: fts-flatcurve(INBOX): Check failed; DatabaseOpeningError:
Directory does not contain a Xapian database
Searching already-indexed mail still works at this point:
# doveadm search -u [5]user(a)example.com mailbox INBOX text a | tail -1
88885837bb33136365390000d09efc50 58
4) Deliver one message and index it. This is where it breaks:
# printf 'From: a(a)b.c\nSubject: reproprobe\n\nbody\n' \
| doveadm save -u [6]user(a)example.com -m INBOX
Info: Mailbox INBOX: save: box=INBOX, uid=59, subject=reproprobe
# doveadm index -u [7]user(a)example.com INBOX
Error: fts-flatcurve(INBOX): Cannot open DB (RW,
current.1786117302443530):
.../current.1786117302443530/iamglass: Failed to open glass
revision
file for reading (No such file or directory)
Error: cmd index: Mailbox INBOX: Precache for UID=59 failed: Internal
error
5) From now on SEARCH fails too, because fts_search_add_missing makes it
index
first:
# doveadm search -u [8]user(a)example.com mailbox INBOX text reproprobe
Error: Mailbox INBOX: indexer failed to index mailbox
Error: cmd search: Searching mailbox INBOX failed: Internal error
occurred.
Over IMAP the client sees: NO [SERVERBUG] Internal error occurred.
It does not recover by itself
-----------------------------
"doveadm index" and autoindexing both take the same write path and fail
the same
way. "doveadm fts rescan" does not help either -- it also cannot open the
empty
shard. The only repair we found is to delete the folder's fts-flatcurve
directory and index the folder again.
Impact
------
A quiet mailbox looks fine for as long as nothing is delivered to it, so
this can
sit unnoticed. We found it because a monitoring mailbox that receives one
message
per minute started returning SERVERBUG on SEARCH. In our case a
maintenance run
that called "doveadm fts optimize" per mailbox left 114 mailboxes in this
state,
which is all of the ones that had any mail.
Possible cause (not verified)
-----------------------------
In fts_flatcurve_xapian_optimize_box_do() every shard is first opened for
writing
("We need to lock all of the mailboxes so nothing changes while we are
optimizing"), and the old current.*/index.* directories are then deleted
from
disk while those write handles are still open. A handle that is closed
afterwards
appears to recreate its directory without a glass revision file, which
matches
what is left behind.
Also, in the same function the result of
fts_flatcurve_xapian_rename_db(backend, dbpath, NULL, error_r)
is discarded: rename_db() does not update the path it is given and returns
the
new path only through newpath_r, which is NULL at this call site.
I have not tried to confirm either of these beyond reading the code.
Not the same as the known root-owned iamglass report
----------------------------------------------------
There is an older report where "doveadm fts optimize -A" run from cron as
root
left iamglass owned by root, producing the same error string. That is not
this:
here the directories are owned by the mail user (dovecot:dovecot, 0700)
and the
revision file does not exist at all.
--
Best regards,
Ihor Rusyn
References
Visible links
1. mailto:user@example.com
2. mailto:user@example.com
3. mailto:user@example.com
4. mailto:user@example.com
5. mailto:user@example.com
6. mailto:user@example.com
7. mailto:user@example.com
8. mailto:user@example.com