Panic with FTS_BACKEND_FLAG_TOKENIZED_INPUT and virtual mailboxes: file index-search-result.c: line 132

Originally posted and debugged here: https://github.com/slusarz/dovecot-fts-flatcurve/issues/66

I have to following panic with fts flatcurve enabled:

Panic: file index-search-result.c: line 132 (index_search_result_update_flags): assertion failed: (result->search_args->args == &search_arg)
Error: Raw backtrace:
/usr/lib/dovecot/libdovecot.so.0(backtrace_append+0x46) [0x74e9890a2846] ->
/usr/lib/dovecot/libdovecot.so.0(backtrace_get+0x22) [0x74e9890a2962] ->
/usr/lib/dovecot/libdovecot.so.0(+0x10d81b) [0x74e9890af81b] ->
/usr/lib/dovecot/libdovecot.so.0(+0x10d8b7) [0x74e9890af8b7] ->
/usr/lib/dovecot/libdovecot.so.0(+0x5e2e2) [0x74e9890002e2] ->
/usr/lib/dovecot/libdovecot-storage.so.0(+0x518a4) [0x74e9891d38a4] ->
/usr/lib/dovecot/modules/lib20_virtual_plugin.so(virtual_storage_sync_init+0x274e) [0x74e988bf8a9e] ->
/usr/lib/dovecot/libdovecot-storage.so.0(mailbox_sync_init+0x5c) [0x74e9891eff9c] ->
/usr/lib/dovecot/libdovecot-storage.so.0(mailbox_sync+0x39) [0x74e9891f0039] -> 
dovecot/imap(cmd_select_full+0x1df) [0x62354bd8326f] -> 
dovecot/imap(command_exec+0xa4) [0x62354bd8b444] -> 
dovecot/imap(+0x22392) [0x62354bd89392] -> 
dovecot/imap(+0x22444) [0x62354bd89444] -> 
dovecot/imap(client_handle_input+0x1bd) [0x62354bd8989d] -> 
dovecot/imap(client_input+0x74) [0x62354bd89e64] -> 
/usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x6d) [0x74e9890c603d] -> 
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x13a) [0x74e9890c77aa] -> 
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x54) [0x74e9890c60e4] -> 
/usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x40) [0x74e9890c62a0] -> 
/usr/lib/dovecot/libdovecot.so.0(master_service_run+0x17) [0x74e989036ad7] -> 
dovecot/imap(main+0x570) [0x62354bd7ad20] -> 
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x74e988c29d90] -> 
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x74e988c29e40] -> 
dovecot/imap(_start+0x25) [0x62354bd7ade5]

It happens when accessing virtual folder with filters like this:

something-*/excep-dev*
  header x-abc-exception "1"

The error happens on subsequent access of the folder:

• Removing dovecot.index{,.log} files allows to access the virtual folder again
• dovecot.index gets recreated on access and the folder can be accessed multiple times
• Marking an e-mail read triggers the error, though! Now each next access raises the error
• Removing dovecot.index{,.log} restores access again and the e-mails are still marked as read (so this information is not lost), until the next e-mail is marked as read: then the folder cannot be accessed again

Slusarz debugged the backtrace and wrote a detailed description at https://github.com/slusarz/dovecot-fts-flatcurve/issues/66#issuecomment-2294... which I'll quote here:

This is not a flatcurve bug - it is a bug within Dovecot's indexing/fts handling.

(Detailed debug below, so that core team can understand the problem.)

As discussed above, this bug is triggered with virtual mailboxes on subsequent accesses. The initial access will work fine; the assert crash only happens when the Dovecot lib-storage index code reads the indexes created for the virtual mailboxes.

index_search_result_update_appends() in lib-storage/index/index-search-result.c

* Inside this function,  a "temporary search parameter to limit the search only to the new messages" is added to the mail search args.  Later, it is checked that the mail_search args points to this same search parameter after the search is executed.  It is this assert check that is failing, because the mail args is changing during the course of the search.

* `i_assert(result->search_args->args == &search_arg);`

The search args are being altered in plugins/fts/fts-search.args.c, fts_search_args_expand(). Specifically, in that function, either fts_search_args_expand_language_top_level() or fts_search_args_expand_tree() is being called, which can replace the current args parameter with a new pointer.

fts_search_args_expand() is being called in plugins//fts/fts-search.c by fts_search_try_lookup() when the FTS backend has the FTS_BACKEND_FLAG_TOKENIZED_INPUT flag set. Flatcurve has this flag set.

This flag is defined in plugins/fts/fts-api-private.h. Comments say:

        /* Tokenize all the input. update_build_more() will be called a single
           directly indexable token at a time. Searching will modify the search
           args so that lookup() sees only tokens that can be directly
           searched. */

Thus, the comments explicitly say the search args will be modified if this flag is set. This modification is incompatible with the assert check in index_search_result_update_appends(), which requires that the search args not be modified.

The core team will have to determine whether this behavior is correct, or if the assert check is invalid.

You can find some more discussion at https://github.com/slusarz/dovecot-fts-flatcurve/issues/66, but I think the most important part (the very detailed analysis from Slusarz, thanks again!) might be enough to understand and maybe fix the problem.

Relevant information about the mail system:

# 2.3.21 (47349e2482): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.21 (f6cd4b8e)
# OS: Linux 6.5.0-45-generic x86_64 Ubuntu 22.04.4 LTS
[...]
namespace {
  location = virtual:~/virtual:LAYOUT=maildir++
  prefix = _/
  separator = /
}
[...]
plugin {
  acl = vfile
  acl_shared_dict = file:/var/lib/dovecot/db/shared
  fts = flatcurve
  fts_autoindex = yes
  fts_filters = normalizer-icu snowball stopwords
  fts_filters_en = lowercase snowball stopwords
  fts_languages = de en
  fts_tokenizer_email_address = maxlen=100
  fts_tokenizer_generic = algorithm=simple maxlen=30
  fts_tokenizers = generic email-address
  sieve = file:~/sieve;active=~/.dovecot.sieve
}
[...]

I can provide more information if you need. Thanks for looking into it!