September 2026 - dovecot

SEGV in fts_search_merge_scores_or in 2.4.5
by Patrick Cernko 05 Oct '26

05 Oct '26

Hi Devs, yesterday I had to SEGV from imap processes (2.4.5) which dumped core. The backtraces were similar: > #0 0x00007fdb234556b4 in fts_search_merge_scores_or (dest=0x7ffde96b1720, src=0x7ffde96b1680) at ./src/plugins/fts/fts-search.c:274 > 274 src_map[srci].uid < src2_map[src2i].uid) { > (gdb) bt > #0 0x00007fdb234556b4 in fts_search_merge_scores_or (dest=0x7ffde96b1720, src=0x7ffde96b1680) at ./src/plugins/fts/fts-search.c:274 > #1 fts_search_merge_scores_level (fctx=fctx@entry=0x55ce3f2347d0, args=0x55ce3f2fad10, idx=idx@entry=0x7ffde96b182c, and_args=false, scores=scores@entry=0x7ffde96b1720) > at ./src/plugins/fts/fts-search.c:329 > #2 0x00007fdb234555fa in fts_search_merge_scores_level (fctx=fctx@entry=0x55ce3f2347d0, args=0x55ce3f2fa878, idx=idx@entry=0x7ffde96b182c, and_args=true, scores=scores@entry=0x7ffde96b17c0) > at ./src/plugins/fts/fts-search.c:322 > #3 0x00007fdb234555fa in fts_search_merge_scores_level (fctx=fctx@entry=0x55ce3f2347d0, args=0x55ce3f22d458, idx=idx@entry=0x7ffde96b182c, and_args=and_args@entry=true, > scores=0x55ce3f2ffb68) at ./src/plugins/fts/fts-search.c:322 > #4 0x00007fdb2345b2ee in fts_search_merge_scores (fctx=0x55ce3f2347d0) at ./src/plugins/fts/fts-search.c:337 > #5 fts_search_try_lookup (fctx=0x55ce3f2347d0) at ./src/plugins/fts/fts-search.c:404 > #6 fts_search_lookup (fctx=fctx@entry=0x55ce3f2347d0) at ./src/plugins/fts/fts-search.c:414 > #7 0x00007fdb2345bc25 in fts_mailbox_search_init (t=<optimized out>, args=0x55ce3f22cd28, sort_program=<optimized out>, wanted_fields=0, wanted_headers=<optimized out>) > at ./src/plugins/fts/fts-storage.c:213 > #8 0x00007fdb266b7d99 in mailbox_search_init (t=0x55ce3f322fb0, args=args@entry=0x55ce3f22cd28, sort_program=sort_program@entry=0x0, wanted_fields=wanted_fields@entry=0, > wanted_headers=wanted_headers@entry=0x0) at ../lib-storage/mail-storage.c:2988 > #9 0x000055ce070d746c in imap_search_start (ctx=0x55ce3f1cb568, sargs=0x55ce3f22cd28, sort_program=0x0) at ./src/imap/imap-search.c:538 > #10 0x000055ce070c3702 in cmd_search (cmd=0x55ce3f1cb1a8) at ./src/imap/cmd-search.c:54 > #11 0x000055ce070c8394 in command_exec (cmd=cmd@entry=0x55ce3f1cb1a8) at ./src/imap/imap-commands.c:208 > #12 0x000055ce070cec20 in client_command_input (cmd=<optimized out>, cmd@entry=0x55ce3f1cb1a8) at ./src/imap/imap-client.c:1359 > #13 0x000055ce070cecb6 in client_command_input (cmd=<optimized out>, cmd@entry=0x55ce3f1cb1a8) at ./src/imap/imap-client.c:1429 > #14 0x000055ce070cef4d in client_command_input (cmd=<optimized out>) at ./src/imap/imap-client.c:1393 > #15 0x000055ce070cf135 in client_handle_next_command (client=0x55ce3f1c9788, remove_io_r=<synthetic pointer>) at ./src/imap/imap-client.c:1471 > #16 client_handle_input (client=client@entry=0x55ce3f1c9788) at ./src/imap/imap-client.c:1485 > #17 0x000055ce070cf527 in client_input (client=0x55ce3f1c9788) at ./src/imap/imap-client.c:1529 > #18 0x00007fdb26379cdb in io_loop_call_io (io=0x55ce3f1cb960) at ../lib/ioloop.c:749 > #19 0x00007fdb2637bb1b in io_loop_handler_run_internal (ioloop=ioloop@entry=0x55ce3f19b5c0) at ../lib/ioloop-epoll.c:222 > #20 0x00007fdb2637bbe4 in io_loop_handler_run (ioloop=ioloop@entry=0x55ce3f19b5c0) at ../lib/ioloop.c:803 > #21 0x00007fdb2637bde8 in io_loop_run (ioloop=0x55ce3f19b5c0) at ../lib/ioloop.c:774 > #22 0x00007fdb262c34be in master_service_run (service=0x55ce3f19ad10, callback=callback@entry=0x55ce070d3e30 <client_connected>) at ../lib-master/master-service.c:1280 > #23 0x000055ce070ba036 in main (argc=<optimized out>, argv=<optimized out>) at ./src/imap/main.c:588 in both cases the SEGV was caused by illegal access to `src_map[srci]`: > (gdb) print src_map[srci] > Cannot access memory at address 0x7fdb23091000 I can provide the coredumps to you (but not public) if you are interested. Best, -- Patrick Cernko <errror(a)errror.org>

2 1

Making AI Feedback Part of a Supervised Writing Process
by harri.evans90＠protonmail.com 30 Sep '26

30 Sep '26

Begin with the Student’s Reasoning When assessing AI assistance in academic writing, I start with the decisions a student must make. A polished essay offers limited evidence of learning if the writer cannot explain its argument, defend its sources, or justify its revisions. My approach therefore places the student’s reasoning at the center of the writing process and evaluates assistance according to whether it strengthens that reasoning. I consider academic writing software most useful when its function is defined within a supervised workflow. When evaluating a resource such as https://essaysbot.com/, I would first consider how its use fits the assignment’s learning objectives and the instructor’s requirements. Artificial intelligence can help a writer examine an outline, identify unclear explanations, and reconsider paragraph structure. Its educational value depends on how the student interprets the feedback and applies it to the assignment. A writing assistant should create opportunities for analysis while keeping intellectual decisions with the student. The following consultation framework uses an illustrative undergraduate sociology essay about remote work and workplace inequality. I use this example to show how an educator can connect digital assistance with planning, research, and independent revision without treating generated language as evidence of understanding. Establish the Task Before Requesting Output I would begin the consultation by reviewing the assignment instructions with the student. Before opening a digital tool, the writer should identify the research question, intended audience, assessment criteria, citation style, word count, and deadline. These details determine the scope of the essay and give automated feedback a relevant context. For the sociology assignment, a broad question about whether remote work is beneficial would need refinement. A more manageable inquiry could examine how access to remote employment affects workers with different caring responsibilities. The student would then develop a provisional thesis statement after reviewing the assigned readings and locating suitable evidence. At this stage, I would ask the student to prepare an outline independently. A language model could subsequently examine whether the proposed sections overlap or whether an important counterargument remains unaddressed. A focused prompt might request questions about the relationship between employment flexibility and promotion opportunities. This preserves an investigative research process because the student must resolve those questions through reading. Course expectations also belong in this initial discussion. I would check the instructor’s requirements for responsible use, disclosure, and permitted assistance before recommending a particular activity. Academic integrity becomes easier to manage when expectations are translated into specific actions within the student workflow. Move from Feedback to Deliberate Revision My preferred revision cycle addresses one substantive concern at a time. Asking a system to improve an entire essay can produce extensive suggestions with unclear priorities. Asking whether each topic sentence advances the central claim gives the student a defined problem and a basis for evaluating the response. Consider a paragraph describing reduced commuting costs. Its information may be relevant, yet its connection to workplace inequality may remain unexplained. Automated feedback could flag that gap. The student must decide whether the paragraph needs additional evidence, a clearer explanation, or removal. Each option requires a different judgment about the argument. I would review the overall reasoning before sentence clarity, editing, and proofreading. A reverse outline can help: the student summarizes the actual purpose of each paragraph in the existing draft, then compares that sequence with the intended argument. Repetition, unsupported claims, and missing transitions become easier to locate when attention shifts from individual sentences to the essay’s organization. To make the feedback loop educational, I would request a brief revision rationale for major changes. The student should explain why a claim was narrowed, a section moved, or a suggestion rejected. This guided practice encourages critical thinking and gives the educator a clearer view of developing writing skills. It also allows feedback to remain useful when the model’s recommendation is inappropriate. Keep Evidence and Attribution Verifiable I separate source quality from fluent expression. A generated draft may present an explanation confidently without establishing that its supporting material is accurate or relevant. I would therefore require the student to verify factual claims against sources they have retrieved and read, including the context and limitations of the original research. In the sociology example, evidence about professional employees in one industry may not support conclusions about all workers. The writer must examine the study population, methods, and publication context before applying its findings. A tool can suggest verification questions, but the student remains responsible for answering them. Citation awareness should begin during note-taking. I recommend distinguishing direct quotations, paraphrases, and independent analysis before drafting. This practice supports originality and plagiarism awareness while preventing attribution problems from accumulating near the deadline. A system may help explain reference formatting or flag passages that appear to require a citation. Nevertheless, I would check each reference against the original publication and the required style guidance. Bibliographic accuracy includes both identifying a real source and confirming that it supports the associated claim. Assess Learning Through the Final Review During the final review, I would compare output quality with the rubric and ask the student to explain the essay’s reasoning in their own words. The conclusion should answer the research question within the limits of the evidence. Word count management should remove duplication while preserving necessary analysis, rather than reduce every section mechanically. For educators and academic consultants, instructional design should make these checkpoints visible. An outline, source notes, selected feedback, and a short revision explanation can provide useful evidence of the learning process without requiring an exhaustive record of every interaction. I judge AI-supported learning by the quality of the student’s decisions and their ability to apply the same principles independently. Learning support has achieved its purpose when the writer can recognize a weak explanation, investigate its cause, and choose a defensible revision. That capacity gives writing confidence a sound foundation and keeps authorship grounded in informed judgment.

1 0

Dovecot 2.4.5 on macOS 27 ARM64: anvil panic with imap-login pid=-1
by list2＠captainnet.net 30 Sep '26

30 Sep '26

Subject: Dovecot 2.4.5 on macOS 27 ARM64: anvil panic with imap-login pid=-1 I am testing Dovecot 2.4.5 on macOS 27 (Darwin 27.0.0), ARM64/Apple Silicon, migrating a working Dovecot 2.3.21.1 installation. IMAP itself appears to be functioning correctly. Authentication succeeds using CRAM-MD5, the userdb resolves the expected uid/gid/home, Maildir is opened correctly, and Roundcube can see all folders and messages. LMTP delivery also works. However, under multiple IMAP connections I repeatedly see anvil receive an imap-login connection with pid=-1: anvil: Warning: conn unix:anvil (uid=0): Handshake with duplicate service=imap-login pid=-1 - replacing the old connection This can then lead to: anvil: Panic: file anvil-connection.c: line 837: unreached anvil: Fatal: master: service(anvil): child ... killed with signal 6 followed by: anvil: Error: connect limit: disconnection for unknown (pid=..., user=testuser(a)example.com, service=imap, ...) With mail_debug=yes, the IMAP process itself looks healthy immediately around the failure. For example: Debug: Effective uid=350, gid=350, home=/private/var/mail/example.com/testuser@example.com Debug: Namespace inbox: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes Debug: maildir++: root=/private/var/mail/example.com/testuser@example.com/Maildir, inbox=/private/var/mail/example.com/testuser@example.com/Maildir Apple Mail successfully establishes TLS and authenticates: imap-login: Info: Logged in: user=<testuser(a)example.com>, method=CRAM-MD5, rip=192.0.2.10, lip=192.0.2.20, TLS Dovecot's own mailbox tools confirm that the mailboxes and messages exist: # doveadm mailbox list -u testuser(a)example.com -s Sent Drafts Archive Trash Junk # doveadm mailbox status -u testuser(a)example.com \ messages uidvalidity uidnext INBOX Sent Sent messages=1 INBOX messages=7 I initially experimented with disabling the anvil-auth-penalty listener, but reverted that. The problem also occurs with the stock/default Dovecot 2.4.5 anvil configuration: service anvil { executable = anvil process_limit = 1 process_min_avail = 1 unix_listener anvil { group = $SET:default_internal_group mode = 0660 } unix_listener anvil-auth-penalty { group = $SET:default_internal_group mode = 0660 type = penalty } unix_listener anvil-connect-limit { group = $SET:default_internal_group mode = 0660 type = connect-limit } } The effective default is currently: mail_max_userip_connections = 10 Dovecot 2.3.21.1 on the same ARM64 Mac, using the same users/Maildirs, does not exhibit this problem. The particularly suspicious part seems to be the imap-login pid=-1 registration immediately preceding the anvil failure. Is this a known 2.4.x issue on macOS/ARM64, and is there a supported way in 2.4.5 to disable anvil connection tracking as a workaround while retaining IMAP service? I can provide the complete doveconf -n, full backtrace, and additional debug logging if useful.

2 1

dovecot submission Outlook issue
by Adrian M 30 Sep '26

30 Sep '26

Hello, I'm testing dovecot submission and I have some issues with Outlook clients. Thunderbird works without a hitch, only Outlook cannot send mails. If someone has seen this please let me know ! The log looks like this: Sep 28 09:36:12 mail dovecot: submission(user1)<1419><Ii0QSIVc2OvAqHBK>: Disconnected: Connection closed (unfinished QUIT command) (state=GREETING) in=44 out=81 Sep 28 09:37:23 mail dovecot: submission(user1)<4909><zOBITIVc6OfAqHBK>: Disconnected: Connection closed (unfinished QUIT command) (state=GREETING) in=44 out=81 My dovecot config has the following lines in local.conf: submission_relay_host = 127.0.0.1 submission_relay_port = 174 submission_relay_trusted = yes service submission-login { inet_listener submission { port = 587 } inet_listener submissions { port = 465 ssl = yes } } Hello, I'm testing dovecot submission and I have some issues with Outlook clients. Thunderbird works without a hitch, only Outlook cannot send mails. If someone has seen this please let me know ! The log looks like this: Sep 28 09:36:12 mail dovecot: submission(user1)<1419><Ii0QSIVc2OvAqHBK>: Disconnected: Connection closed (unfinished QUIT command) (state=GREETING) in=44 out=81 Sep 28 09:37:23 mail dovecot: submission(user1)<4909><zOBITIVc6OfAqHBK>: Disconnected: Connection closed (unfinished QUIT command) (state=GREETING) in=44 out=81 My dovecot config has the following lines in local.conf: submission_relay_host = 127.0.0.1 submission_relay_port = 174 submission_relay_trusted = yes service submission-login { inet_listener submission { port = 587 } inet_listener submissions { port = 465 ssl = yes } }

3 3

Re: Migration to 2.4: Howto get the quota dict writing into a second DB
by Felipe Carneiro 28 Sep '26

28 Sep '26

Hello, This is regarding your message at https://dovecot.org/mailman3/archives/list/dovecot@dovecot.org/message/XW53… (I tried to reply on the thread but I don't have permission.) Were you able to get it working? I'm having the same problem with Dovecot 2.4. I've tried configuring it exactly as the documentation says, but I get this error when enabling the quota_clone plugin: *Error: quota_clone: dict_init_auto() failed: dict { .. } named list filter is missing* Same problem reported by other users: https://github.com/postfixadmin/postfixadmin/issues/935 Thanks in advance ! -- Best regards, Felipe Carneiro Hello, This is regarding your message at [1]https://dovecot.org/mailman3/archives/list/dovecot@dovecot.org/message/XW… (I tried to reply on the thread but I don't have permission.) Were you able to get it working? I'm having the same problem with Dovecot 2.4. I've tried configuring it exactly as the documentation says, but I get this error when enabling the quota_clone plugin: Error: quota_clone: dict_init_auto() failed: dict { .. } named list filter is missing Same problem reported by other users: [2]https://github.com/postfixadmin/postfixadmin/issues/935 Thanks in advance ! -- Best regards, Felipe Carneiro References Visible links 1. https://dovecot.org/mailman3/archives/list/dovecot@dovecot.org/message/XW53… 2. https://github.com/postfixadmin/postfixadmin/issues/935

2 4

Dovecot 2.4.5 - Oath2 proxy
by Jerome Dalle 28 Sep '26

28 Sep '26

Hello I'm trying to set the pssdbb static (as describe in the documentation) to forward my oath2 authentication to backend Unfortunately, dovecot is unable to start with error: auth: Fatal: Cannot initialize oauth2: Missing oauth2_introspection_mode setting If anyone can help ? Thanks Hello I'm trying to set the pssdbb static (as describe in the documentation) to forward my oath2 authentication to backend Unfortunately, dovecot is unable to start with error: auth: Fatal: Cannot initialize oauth2: Missing oauth2_introspection_mode setting If anyone can help ? Thanks

2 1

Dovecot Mailbox Viewer (Windows)
by jay＠designstack.co.uk 25 Sep '26

25 Sep '26

Hi Guys, I’ve been working on 'Dovecot Mailbox Viewer', a free, open-source Windows application for browsing Dovecot mdbox backups without setting up a Dovecot server. It started as a way to help a client access emails from a JetBackup/cPanel backup. You can open a .tar.gz archive or extracted folder, browse mailbox folders, search messages, view HTML emails, save attachments and export emails as pdf or .eml. The app is read-only and leaves the original backup untouched. Everything is processed locally, and remote images are blocked unless you choose to download them. There’s a standalone Windows executable available, so no Python installation is needed. Code, screenshots and downloads: https://github.com/DesignStack/Dovecot-Mailbox-Viewer It’s still early days, so feedback from the Dovecot community would be very welcome! Thanks! Jay

1 0

Crash on FreeBSD 14.4 with dovecot 2.4.5
by Frank Volf 24 Sep '26

24 Sep '26

Hi, On a daily basis I see a anvil crash in my logs for dovecot 2.4.5 on FreeBSD. Not sure if this is a FreeBSD specific issue, a dovecot issue, or a configuration issue (this is the first system converted from 2.3 configuration syntax): Sep 9 03:20:00 mail dovecot[53730]: anvil: Panic: file anvil-connection.c: line 837: unreached Sep 8 16:40:00 mail dovecot[53730]: anvil: Panic: file anvil-connection.c: line 837: unreached Sep 7 20:47:28 mail dovecot[6215]: anvil: Panic: file anvil-connection.c: line 838 (anvil_connection_destroy): assertion failed: (hash_conn == conn) Sep 6 17:07:24 mail dovecot[96610]: anvil: Panic: file anvil-connection.c: line 837: unreached Sep 6 20:40:01 mail dovecot[96610]: anvil: Panic: file anvil-connection.c: line 837: unreached Sep 6 21:53:59 mail dovecot[96610]: anvil: Panic: file anvil-connection.c: line 837: unreached The "assertion failed" only happened once, but the "unreached" happen on a daily basis. About 20 minutes (more or less) after each crash I see also the following message: Sep 8 16:56:32 mail dovecot[53730]: anvil: Error: connect limit: disconnection for unknown (pid=59834, user=xxxx, service=imap, ip=x.x.x.x, conn_guid=7925eb292e15a06abae9000050df13ba) Can somebody please give me some pointers on how to solve or troubleshoot this? Kind regards, Frank Hi, On a daily basis I see a anvil crash in my logs for dovecot 2.4.5 on FreeBSD. Not sure if this is a FreeBSD specific issue, a dovecot issue, or a configuration issue (this is the first system converted from 2.3 configuration syntax): Sep 9 03:20:00 mail dovecot[53730]: anvil: Panic: file anvil-connection.c: line 837: unreached Sep 8 16:40:00 mail dovecot[53730]: anvil: Panic: file anvil-connection.c: line 837: unreached Sep 7 20:47:28 mail dovecot[6215]: anvil: Panic: file anvil-connection.c: line 838 (anvil_connection_destroy): assertion failed: (hash_conn == conn) Sep 6 17:07:24 mail dovecot[96610]: anvil: Panic: file anvil-connection.c: line 837: unreached Sep 6 20:40:01 mail dovecot[96610]: anvil: Panic: file anvil-connection.c: line 837: unreached Sep 6 21:53:59 mail dovecot[96610]: anvil: Panic: file anvil-connection.c: line 837: unreached The "assertion failed" only happened once, but the "unreached" happen on a daily basis. About 20 minutes (more or less) after each crash I see also the following message: Sep 8 16:56:32 mail dovecot[53730]: anvil: Error: connect limit: disconnection for unknown (pid=59834, user=xxxx, service=imap, ip=x.x.x.x, conn_guid=7925eb292e15a06abae9000050df13ba) Can somebody please give me some pointers on how to solve or troubleshoot this? Kind regards, Frank

4 5

dict: ldap deref
by Tomas Habarta 23 Sep '26

23 Sep '26

Hello, just been testing sieve extlists against ldap dict in 2.4.5 -- works ok but need to narrow a search base but this would require setting deref other than default none for the ldap client. Seems that something like ldap_deref = always in dict_map {} section does not work (is not even documented, but config is parsed ok on the other hand) and value of the global ldap_deref is not picked up either. Question is whether the deref parameter is supposed to work and if yes, where to set it up correctly... Many thanks, Tomas

1 1

fts-flatcurve: optimize can never complete once a shard holds an unreadable document
by Ihor Rusyn 23 Sep '26

23 Sep '26

Hello, ## Summary `fts_flatcurve_xapian_optimize_rebuild()` aborts on the first document whose termlist is missing, which makes shard compaction impossible for the affected folder from that point on. The failure is silent — the operation reports success — so shards accumulate without bound. We have seen folders reach several thousand shards, with an index orders of magnitude larger than the messages it covers. Present in 2.4.4, 2.4.5 and current main (checked at 732917749a). ## Symptom doveadm fts optimize -u <user> Error: fts-flatcurve: Optimize failed: DocNotFoundError: No termlist for document 257091 # exit code 0, shard count unchanged In production this is logged by indexer-worker and repeats every 20-30 minutes, once per delivery, each run leaving another shard behind. Affected folders also keep an `optimize` directory containing an empty Xapian database. ## Mechanism 1. Long-lived, high-churn folders end up with shards holding disjoint UID ranges. `db->compact()` with `DBCOMPACT_NO_RENUMBER` throws `InvalidOperationError`, which the code expects and handles. 2. It falls back to `fts_flatcurve_xapian_optimize_rebuild()`, which walks the MSet and copies each document into a fresh database. 3. `iter.get_document()` sits **outside** the try block. When a document in the MSet has no termlist it throws `DocNotFoundError`, which the local handler never sees. 4. The exception propagates to `fts_flatcurve_xapian_optimize_box_do()`, is caught by `catch (Xapian::Error &e)`, sets `failed = TRUE` — and that path does `return 0`. The caller therefore sees success. Old shards are never deleted, the merge never happens, and every subsequent message adds another shard. The folder can never recover on its own. ## Reproduction Reproducible from scratch in about five minutes on a scratch mailbox, with generated messages only. Three conditions have to coincide; raising uidnext alone does not reproduce it. U=test(a)example.com # 1. push uidnext past 2^31 doveadm mailbox update -u $U --min-next-uid 2277107224 INBOX # 2. + 3. churn: grow, index into many shards, expunge half. # the expunged messages leave their documents behind in older shards. for round in $(seq 1 10); do n=$(doveadm -f flow mailbox status -u $U messages INBOX | tail -1 | sed 's/.*messages=//') [ "$n" -gt 400 ] && n=400 doveadm copy -u $U INBOX mailbox INBOX 1:$n doveadm -o fts_flatcurve_rotate_count=25 index -u $U INBOX doveadm expunge -u $U mailbox INBOX 1:$((n/2)) done doveadm fts optimize -u $U Rounds 1-9 compact cleanly to 2 shards. The tenth breaks: 1,838 live messages, 84 shards, uidnext 2277110749 — and optimize then fails as above and never merges again. ## Suggested fix Move `get_document()` inside the try, and skip an unreadable document instead of aborting the rebuild. A document whose termlist is gone belongs to a message that no longer exists, so there is nothing to carry into the optimized database; the rebuild exists precisely to salvage databases the native compact refused. Patch attached (`fts-flatcurve-optimize-rebuild.diff`). Other `Xapian::Error` types keep aborting the rebuild, so genuine corruption is still not silently swallowed. Separately, the `return 0` on `failed` in `fts_flatcurve_xapian_optimize_box_do()` looks questionable regardless of this fix: a failed optimize is reported to the caller as success, which is why this went unnoticed for so long. ## Notes Affected folders are long-lived and high-churn: most of the UIDs they have issued belong to messages that were deleted years ago. Folders that compact normally stay at 9-11 shards, so the ones in this state stand out immediately by shard count alone — which is also the cheapest way to find them: find <index root> -type d -path '*/fts-flatcurve/*' -prune -printf '%h\n' \ | sort | uniq -c | sort -rn -- Best regards, Ihor Rusyn Hello, ## Summary `fts_flatcurve_xapian_optimize_rebuild()` aborts on the first document whose termlist is missing, which makes shard compaction impossible for the affected folder from that point on. The failure is silent -- the operation reports success -- so shards accumulate without bound. We have seen folders reach several thousand shards, with an index orders of magnitude larger than the messages it covers. Present in 2.4.4, 2.4.5 and current main (checked at 732917749a). ## Symptom doveadm fts optimize -u <user> Error: fts-flatcurve: Optimize failed: DocNotFoundError: No termlist for document 257091 # exit code 0, shard count unchanged In production this is logged by indexer-worker and repeats every 20-30 minutes, once per delivery, each run leaving another shard behind. Affected folders also keep an `optimize` directory containing an empty Xapian database. ## Mechanism 1. Long-lived, high-churn folders end up with shards holding disjoint UID ranges. `db->compact()` with `DBCOMPACT_NO_RENUMBER` throws `InvalidOperationError`, which the code expects and handles. 2. It falls back to `fts_flatcurve_xapian_optimize_rebuild()`, which walks the MSet and copies each document into a fresh database. 3. `iter.get_document()` sits **outside** the try block. When a document in the MSet has no termlist it throws `DocNotFoundError`, which the local handler never sees. 4. The exception propagates to `fts_flatcurve_xapian_optimize_box_do()`, is caught by `catch (Xapian::Error &e)`, sets `failed = TRUE` -- and that path does `return 0`. The caller therefore sees success. Old shards are never deleted, the merge never happens, and every subsequent message adds another shard. The folder can never recover on its own. ## Reproduction Reproducible from scratch in about five minutes on a scratch mailbox, with generated messages only. Three conditions have to coincide; raising uidnext alone does not reproduce it. U=[1]test(a)example.com # 1. push uidnext past 2^31 doveadm mailbox update -u $U --min-next-uid 2277107224 INBOX # 2. + 3. churn: grow, index into many shards, expunge half. # the expunged messages leave their documents behind in older shards. for round in $(seq 1 10); do n=$(doveadm -f flow mailbox status -u $U messages INBOX | tail -1 | sed 's/.*messages=//') [ "$n" -gt 400 ] && n=400 doveadm copy -u $U INBOX mailbox INBOX 1:$n doveadm -o fts_flatcurve_rotate_count=25 index -u $U INBOX doveadm expunge -u $U mailbox INBOX 1:$((n/2)) done doveadm fts optimize -u $U Rounds 1-9 compact cleanly to 2 shards. The tenth breaks: 1,838 live messages, 84 shards, uidnext 2277110749 -- and optimize then fails as above and never merges again. ## Suggested fix Move `get_document()` inside the try, and skip an unreadable document instead of aborting the rebuild. A document whose termlist is gone belongs to a message that no longer exists, so there is nothing to carry into the optimized database; the rebuild exists precisely to salvage databases the native compact refused. Patch attached (`fts-flatcurve-optimize-rebuild.diff`). Other `Xapian::Error` types keep aborting the rebuild, so genuine corruption is still not silently swallowed. Separately, the `return 0` on `failed` in `fts_flatcurve_xapian_optimize_box_do()` looks questionable regardless of this fix: a failed optimize is reported to the caller as success, which is why this went unnoticed for so long. ## Notes Affected folders are long-lived and high-churn: most of the UIDs they have issued belong to messages that were deleted years ago. Folders that compact normally stay at 9-11 shards, so the ones in this state stand out immediately by shard count alone -- which is also the cheapest way to find them: find <index root> -type d -path '*/fts-flatcurve/*' -prune -printf '%h\n' \ | sort | uniq -c | sort -rn -- Best regards, Ihor Rusyn References Visible links 1. mailto:test@example.com

1 2