- dovecot - dovecot.org

IMAP access marking new messages as "old" for mutt
by Marc Haber 19 Aug '26

19 Aug '26

Hi, this might be a stupid question asked by someone who does not know enough about IMAP, but I'm going to tell you my user story anyway: tl;dr: can I keep dovecot from moving all "NEW" message to· "OLD" state when the Mailbox was opened once and nothing actually read? long version: My mail is delivered to a maildir on a machine I also have a shell account on. That Maildir is huge, it holds half a million messages. I usually access it directly with mutt accessing the spool. In the Mailbox, new messages automatically get the N flag and once I have opened it in mutt, I can mark it as "O" (old). I use that flag for messages that still need further work, I usually have a couple of hundred messages in that state. There is also a third state (I would call it "read") that doesnt display any flags in mutt. Occasionally, I need access to an attachment directly on my work box. In the past, I have moved that message to a sub-maildir which I then access with Thunderbird or neomutt via IMAP and dovecot (configured via userdb). I now want to give neomutt direct IMAP access to the BIG mailbox. I am impressed how quickly the neomutt/dovecot acesses this mailbox. However, once I have once opened that mailbox, all messages that had the N flag in the mutt-on-spool lose that flag immediately, replaced by the O flag. That destroys my method of seeing "O" messages as "needs work". Since that happens with both Thunderbird and neomutt when testing, I suspect, this is either mandated by the protocol (which would be a showstopper for me), or a dovecot issue, maybe even a thing that I could configure in dovecot. Hence, my question, can I configure dovecot to move a "NEW" message to "OLD" only if the message was actually READ/body-displayed by an IMAP client? And, if yes, how do I do that. Greetings Marc -- ----------------------------------------------------------------------------- Marc Haber | "I don't trust Computers. They | Mailadresse im Header Leimen, Germany | lose things." Winona Ryder | Fon: *49 6224 1600402 Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421

2 1

RE: ai dovecot
by Marc 16 Aug '26

16 Aug '26

No I am curious if they use llms, because I am getting more and more impressed by it. I see suggestions referring to RFC compliant code, which I have been missing in quite a few developers working for me. I think if you are full time developer this can be really helpful. Not to mention those news items where software is ported and scanned for vulnerabilities. > > So you'd like the developers to subject their code to LLMs for audit? > What idiocy is that! > > On Sun, Aug 16, 2026 at 1:33 AM Marc <Marc(a)f1-outsourcing.eu > <mailto:Marc@f1-outsourcing.eu> > wrote: > > > > > > > > > I am just a bit curious, are llm's being used to check the > code or > > help in any other way? > > > > > > > > Why does it bother you? > > > > > > > ??? Can't you read? I am curious, not bothered. > > > > > -- > > Best regards, > Odhiambo WASHINGTON, > Nairobi,KE > +254 7 3200 0004/+254 7 2274 3223 > In an Internet failure case, the #1 suspect is a constant: DNS. > "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) > [How to ask smart questions: http://www.catb.org/~esr/faqs/smart- > questions.html]

1 0

mbox - maildir transition using dovecot lmtp local user auth struggles
by "Tóth Attila" 14 Aug '26

14 Aug '26

Approaching age 49 I passed the time long before when I was excited about a new version or a software upgrade. Why poke something that works? From time to time I'm forced to upgrade hardware and software against my will otherwise could serve many more years. I became the victim of mbox -> maildir transition. First I converted the good old Sendmail to Postfix. Initially it still used procmail delivering in mbox mailboxes, now I converted the mboxes to maildir and changed the way of delivery by dovecot lmtp, since that is the software I use for IMAP access. At least there seem to be no data-loss after a couple of doveadm backup and sync retries. There are just 4 local users, so dovecot was set for pam authentication and it worked (for 20+ years, although there were some bumps while upgrading from 2.3 to 2.4 syntax). Now dovecot lmtp started bouncing mails, because it was trying to authenticate a username including the domain name as well and the backend returned unknown user, since expected local user names. This wasn't a problem with mbox/procmail. I asked AI for help and first suggested to use auth_username_format = %Ln. It soon turned out, that this syntax is obsolete. So I dig into it myself and tried auth_username_format = %{user | lower}. It wasn't picked up by dovecot, symptoms persisted. I also tried auth_username_translation = %{user | split('@') | list_first}, which turned out to be also unsuccessful. I also tried to solve the issue on the postfix side by introducing regex handling, but either lmtp_generic_maps = regexp:/etc/postfix/lmtp_generic or recipient_canonical_maps = regexp:/etc/postfix/recipient_canonical had no influence on the symptoms (regex was: /^([^@]+)@atoth\.sote\.hu$/ ${1}). Also tried virtual conversion and lda. Nothing solved the fricking problem. I ended up using a workaround I'm ashamed to share. I kept passdb pam, but for userdb I created a userdb-file, where I included users both with the local name only and the full email address. This shameful concept is functional, but I have to admit it is disappointing: several ways back and forth for solving an issue were all unsuccessful. And it is exhausting. Any ideas appreciated. BR: Dw. -- dr Tóth Attila, Radiológus, 06-20-825-8057 Attila Toth MD, Radiologist, +36-20-825-8057

1 0

Debian 13 Dovecot 2.4.1 LMTP Error Absent ~/sieve
by Conrad T. Pino 14 Aug '26

14 Aug '26

Hello Dovecot List, Our upgrade from Debian 9 Dovecot 2.2.27 to Debian 13 Dovecot 2.4.1 went well. Only recently we added a new mailbox and LMTP delivery failed. Dovecot redacted error from Postfix log: 451 4.2.0 <user@domain> Temporarily unable to access necessary Sieve scripts (in reply to end of DATA command)) Dovecot redacted errors from Dovecot log: dovecot[736]: lmtp(user@domain): Error: sieve: file storage: Performing t_readlink() on active sieve symlink '(null)' failed: readlink() failed: Bad address dovecot[736]: lmtp(user@domain): Error: sieve: Failed to access active Sieve script in parsonal storage 'default': Internal error occurred. Refer to server log for more information. [2026-08-13] (temporary failure) LMTP delivery works after manually creating ~/sieve directory with correct permissions. I know a PostfixAdmin postcreation script can automate directory creation. Have I overlooked Dovecot setting to have Dovecot create ~/sieve directory during LMTP delivery? Best regards, Conrad T. Pino

1 0

ai dovecot
by Marc 14 Aug '26

14 Aug '26

I am just a bit curious, are llm's being used to check the code or help in any other way?

1 0

BUG 2.4 regression: a wrong password over PLAIN is reported as a temporary failure, and the invalid_grant branch is now dead code
by Ihor Rusyn 14 Aug '26

14 Aug '26

Summary With passdb oauth2 doing a password grant, a wrong password makes dovecot answer the IMAP client a NO [UNAVAILABLE] Temporary authentication failure. and log it at Error: level. 2.3 answered AUTHENTICATIONFAILED for the same exchange. PASSDB_RESULT_PASSWORD_MISMATCH is still in db-oauth2.c, but for the password grant it can no longer be reached. Environment Dovecot *2.4.4 (8b687aa65c)*, the official dovecot/dovecot:2.4.4-root image, unmodified. Compared against *2.3.21*. Reproduction Any RFC 6749 compliant authorization server. Point passdb oauth2 at its token endpoint with grant_type=password and attempt an IMAP LOGIN with a wrong password. The server answers, as §5.2 prescribes: HTTP 400 {"error":"invalid_grant","error_description":"Invalid credentials"} *2.3.21:* NO [AUTHENTICATIONFAILED], logged at info level. *2.4.4:* NO [UNAVAILABLE] Temporary authentication failure, logged as auth(user,ip,sasl:plain): Error: oauth2-grant: oauth2 failed: Password grant failed: Invalid credentials imap-login: Login aborted: Logged out (auth service reported temporary failure, 1 attempts in 3 secs) (temp_fail) Cause src/auth/db-oauth2.c is identical in both versions: passdb_result = PASSDB_RESULT_INTERNAL_FAILURE; if (result->error != NULL) error = result->error; else { error = db_oauth2_field_find(result->fields, "error"); if (error == NULL) error = "OAuth2 server returned failure without error field"; else if (strcmp("invalid_grant", error) == 0) passdb_result = PASSDB_RESULT_PASSWORD_MISMATCH; } What changed is what reaches result->error. In *2.3.21*, src/lib-oauth2/oauth2-request.c:113-129 set error only when the JSON *parser* failed. A well-formed error body parsed fine, so error was NULL, res.error stayed NULL, and db-oauth2.c took the else branch, found invalid_grant among the fields and returned PASSWORD_MISMATCH. In *2.4.4*, src/lib-oauth2/oauth2-request.c:136-149 added: /* check if fields contain error now */ const struct oauth2_field *error_field = array_lsearch(&req->fields, "error", request_field_cmp); if (error_field != NULL) { error = error_field->value; /* "invalid_grant" */ /* check for detailed error */ error_field = array_lsearch(&req->fields, "error_description", request_field_cmp); if (error_field != NULL) error = error_field->value; /* overwritten: "Invalid credentials" */ } req->json_parsed_cb(req, error); oauth2_request_continue() then does if (error != NULL) res.error = error;, so result->error is always set on failure and db-oauth2.c never reaches the branch that recognises invalid_grant. The machine-readable code is replaced by the human-readable description before anything can act on it. Note that error_description is *optional* in RFC 6749 §5.2 — so a fully compliant server triggers this simply by being helpful. Omitting it does not help either: error would then hold invalid_grant and still be assigned to res.error, taking the same branch. Why it matters - Mail clients treat [UNAVAILABLE] as a server problem and retry rather than prompting for the password, so users are not told their password is wrong. - Monitoring reads a user error as a service failure — on a farm this is the difference between a quiet log and a page. - RFC 5530 defines AUTHENTICATIONFAILED for a failed authentication and UNAVAILABLE for a temporarily unavailable server. Reporting the former as the latter tells the client something untrue about the server. There is no anti-enumeration benefit in the current behaviour: the same provider response is returned for a wrong password and for a non-existent account, so both are indistinguishable to the client under either mapping. Summary With passdb oauth2 doing a password grant, a wrong password makes dovecot answer the IMAP client a NO [UNAVAILABLE] Temporary authentication failure. and log it at Error: level. 2.3 answered AUTHENTICATIONFAILED for the same exchange. PASSDB_RESULT_PASSWORD_MISMATCH is still in db-oauth2.c, but for the password grant it can no longer be reached. Environment Dovecot 2.4.4 (8b687aa65c), the official dovecot/dovecot:2.4.4-root image, unmodified. Compared against 2.3.21. Reproduction Any RFC 6749 compliant authorization server. Point passdb oauth2 at its token endpoint with grant_type=password and attempt an IMAP LOGIN with a wrong password. The server answers, as S5.2 prescribes: HTTP 400 {"error":"invalid_grant","error_description":"Invalid credentials"} 2.3.21: NO [AUTHENTICATIONFAILED], logged at info level. 2.4.4: NO [UNAVAILABLE] Temporary authentication failure, logged as auth(user,ip,sasl:plain): Error: oauth2-grant: oauth2 failed: Password grant failed: Invalid credentials imap-login: Login aborted: Logged out (auth service reported temporary failure, 1 attempts in 3 secs) (temp_fail) Cause src/auth/db-oauth2.c is identical in both versions: passdb_result = PASSDB_RESULT_INTERNAL_FAILURE; if (result->error != NULL) error = result->error; else { error = db_oauth2_field_find(result->fields, "error"); if (error == NULL) error = "OAuth2 server returned failure without error field"; else if (strcmp("invalid_grant", error) == 0) passdb_result = PASSDB_RESULT_PASSWORD_MISMATCH; } What changed is what reaches result->error. In 2.3.21, src/lib-oauth2/oauth2-request.c:113-129 set error only when the JSON parser failed. A well-formed error body parsed fine, so error was NULL, res.error stayed NULL, and db-oauth2.c took the else branch, found invalid_grant among the fields and returned PASSWORD_MISMATCH. In 2.4.4, src/lib-oauth2/oauth2-request.c:136-149 added: /* check if fields contain error now */ const struct oauth2_field *error_field = array_lsearch(&req->fields, "error", request_field_cmp); if (error_field != NULL) { error = error_field->value; /* "invalid_grant" */ /* check for detailed error */ error_field = array_lsearch(&req->fields, "error_description", request_field_cmp); if (error_field != NULL) error = error_field->value; /* overwritten: "Invalid credentials" */ } req->json_parsed_cb(req, error); oauth2_request_continue() then does if (error != NULL) res.error = error;, so result->error is always set on failure and db-oauth2.c never reaches the branch that recognises invalid_grant. The machine-readable code is replaced by the human-readable description before anything can act on it. Note that error_description is optional in RFC 6749 S5.2 -- so a fully compliant server triggers this simply by being helpful. Omitting it does not help either: error would then hold invalid_grant and still be assigned to res.error, taking the same branch. Why it matters o Mail clients treat [UNAVAILABLE] as a server problem and retry rather than prompting for the password, so users are not told their password is wrong. o Monitoring reads a user error as a service failure -- on a farm this is the difference between a quiet log and a page. o RFC 5530 defines AUTHENTICATIONFAILED for a failed authentication and UNAVAILABLE for a temporarily unavailable server. Reporting the former as the latter tells the client something untrue about the server. There is no anti-enumeration benefit in the current behaviour: the same provider response is returned for a wrong password and for a non-existent account, so both are indistinguishable to the client under either mapping.

1 0

IMAP4rev2: extended LIST drops \Sent, \Drafts, \Trash and \Junk unless the client sends the RFC 6154 return option that IMAP4rev2 does not define
by dennis＠dennisameling.com 14 Aug '26

14 Aug '26

SYSTEM INFORMATION ================== Dovecot version: 2.4.4 (8b687aa65c), Pigeonhole 2.4.4 (3beb331a) OS: Linux 6.12.76-linuxkit aarch64, Debian 13.5 (official dovecot/dovecot:2.4.4 container image, digest sha256:723e3392fe16c6fad8ddc605ea767cc01b4bad 9cd9f13eb1dbac15e79c89b2d4) Host: Docker 29.6.2 (linux/arm64) on macOS 26, Apple silicon CPU architecture: aarch64 (arm64) Filesystem: Docker named volume (overlay2) inside the Docker Desktop VM. Not NFS. Feature: experimental IMAP4rev2 (imap4rev2_enable, mail_utf8_extensions) Client: our own IMAP client library (Rust), but reproduced below with openssl s_client so that no client code is involved The configuration is the image's own /etc/dovecot/dovecot.conf, unmodified: nothing of ours is in it. Rather than paste 200 lines of your own defaults, "doveconf -n" for it is docker run --rm --entrypoint doveconf dovecot/dovecot:2.4.4 -n and we will gladly send the output if you would prefer it inline. The three settings that bear on this report are imap4rev2_enable = yes, mail_utf8_extensions = yes and mailbox_list_utf8 = yes, all three of which the image sets itself. SUMMARY ======= In an IMAP4rev2 session, an extended LIST (one carrying any RETURN (...) clause) omits the special-use mailbox attributes unless the client also sends RETURN (SPECIAL-USE). IMAP4rev2 folded those attributes into the base protocol and defines no SPECIAL-USE return option, so a client that implements only RFC 9051 has no defined way to ask for them, and therefore cannot obtain them from any extended LIST at all. The practical effect for such a client is that every folder loses its role the moment the client asks for anything else -- in our case RETURN (STATUS (UNSEEN)), to fetch unread counts in the same round trip. It cannot find the \Sent folder to place a copy of a sent message in, cannot find \Trash to move a deleted message to, and renders the mailbox list without role icons or ordering. We are not asking for a workaround: we already send RETURN (SPECIAL-USE) on both dialects, since Dovecot advertises the SPECIAL-USE capability in a rev2 session too. This report is about what a strictly RFC 9051 client is able to do. That decision in our client is here, in case it is of interest -- the comment above it records what we found on your server: https://github.com/allodia-eu/email-calendar-sync-engine/blob/fa85b88121132… REPRODUCTION ============ No configuration is required. dovecot/dovecot:2.4.4 ships imap4rev2_enable = yes and mail_utf8_extensions = yes in its own /etc/dovecot/dovecot.conf, and its "@mailbox_defaults = english" creates Drafts/Junk/Sent/Trash with special-use attributes. docker run -d --name dovecot-stock -p 127.0.0.1:11999:31993 \ -e USER_PASSWORD=stock-test-pw \ dovecot/dovecot:2.4.4 { printf 'a1 LOGIN alice(a)example.com stock-test-pw\r\n'; sleep 2; printf 'a2 ENABLE IMAP4rev2\r\n'; sleep 1; printf 'a3 LIST "" "*"\r\n'; sleep 1; printf 'a4 LIST "" "*" RETURN (STATUS (UNSEEN))\r\n'; sleep 1; printf 'a5 LIST "" "*" RETURN (SPECIAL-USE STATUS (UNSEEN))\r\n'; sleep 1; printf 'a6 LOGOUT\r\n'; sleep 1; } \ | openssl s_client -connect 127.0.0.1:11999 -quiet TRANSCRIPT ========== Verbatim; the greeting and the CAPABILITY lines are elided. S: * ENABLED IMAP4REV2 S: a2 OK Enabled (0.001 + 0.000 secs). C: a3 LIST "" "*" S: * LIST (\HasNoChildren \Drafts) "/" Drafts S: * LIST (\HasNoChildren \Junk) "/" Junk S: * LIST (\HasNoChildren \Sent) "/" Sent S: * LIST (\HasNoChildren \Trash) "/" Trash S: * LIST (\HasNoChildren) "/" INBOX S: a3 OK List completed (0.001 + 0.000 secs). --> correct: the role attributes are present C: a4 LIST "" "*" RETURN (STATUS (UNSEEN)) S: * LIST () "/" Drafts S: * STATUS Drafts (UNSEEN 0) S: * LIST () "/" Junk S: * STATUS Junk (UNSEEN 0) S: * LIST () "/" Sent S: * STATUS Sent (UNSEEN 0) S: * LIST () "/" Trash S: * STATUS Trash (UNSEEN 0) S: * LIST () "/" INBOX S: * STATUS INBOX (UNSEEN 0) S: a4 OK List completed (0.034 + 0.000 + 0.033 secs). --> the problem: every role attribute is gone, and IMAP4rev2 gives the client no option to ask for them back C: a5 LIST "" "*" RETURN (SPECIAL-USE STATUS (UNSEEN)) S: * LIST (\Trash) "/" Trash S: * STATUS Trash (UNSEEN 0) S: * LIST (\Sent) "/" Sent S: * STATUS Sent (UNSEEN 0) S: * LIST (\Junk) "/" Junk S: * STATUS Junk (UNSEEN 0) S: * LIST (\Drafts) "/" Drafts S: * STATUS Drafts (UNSEEN 0) S: * LIST () "/" INBOX S: * STATUS INBOX (UNSEEN 0) S: a5 OK List completed (0.002 + 0.000 + 0.001 secs). --> present again, but only via the RFC 6154 return option, which RFC 9051 does not define WHY WE BELIEVE a4 IS NOT CORRECT FOR IMAP4rev2 ============================================== 1. RFC 9051 section 7.3.1 makes the special-use attributes ordinary LIST mailbox attributes: "In addition to the base mailbox name attributes defined above, an IMAP server MAY also include any or all of the following attributes that denote 'role' (or 'special-use') of a mailbox. These attributes are included along with base attributes defined above." Appendix E item 2 lists "the list of new mailbox attributes from SPECIAL-USE [RFC6154]" among what IMAP4rev2 folded in. 2. RFC 9051 section 6.3.9.2 defines the return options for IMAP4rev2 as exactly SUBSCRIBED, CHILDREN and STATUS. There is no SPECIAL-USE return option. Nor is SPECIAL-USE among the selection options in section 6.3.9.1, which are SUBSCRIBED, REMOTE and RECURSIVEMATCH; it survives in RFC 9051 only as an unexplained C: A002 LIST (SPECIAL-USE) "" "*" in the NAMESPACE examples in section 6.3.10, which may itself be worth an erratum. 3. Section 6.3.9 says "a client MUST NOT send an option for which the server has not advertised support", so a strictly rev2 client will not invent one. Taken together, an RFC 9051-only client can obtain the role attributes on a basic LIST but never on an extended one, which means it cannot combine roles with RETURN (STATUS ...) -- the single round trip that RFC 5819 exists to provide. Since RFC 9051 section 6.3.9 also states that return options "MUST NOT cause the server to report information about additional mailbox names", but says nothing about removing base attributes, we read a4 as a case where an attribute defined as base data is being withheld. Note the contrast with \HasNoChildren in the same transcript: it also disappears in a4, but CHILDREN is a return option in RFC 9051, so a rev2 client can ask for it back. For special-use there is no equivalent. WHAT WE BELIEVE CORRECT BEHAVIOUR WOULD BE ========================================== In a session where IMAP4rev2 has been ENABLEd, include the special-use attributes in extended LIST responses without requiring RETURN (SPECIAL-USE) -- that is, treat them as the base mailbox attributes that RFC 9051 section 7.3.1 defines them to be. The RFC 6154 return option would remain accepted for IMAP4rev1 sessions. NOTE ON IMAP4rev1: WE BELIEVE THAT BEHAVIOUR IS CORRECT ======================================================= The same server on an IMAP4rev1 session behaves identically, and there we read it as compliant. RFC 6154 section 2 says "The extended LIST command MAY return SPECIAL-USE attributes even if the client does not specify the return option", and MAY is not MUST. This report concerns only IMAP4rev2 sessions, where the way to ask was removed along with the need to. CONTACT ======= Happy to test a patch. The reproduction is a single "docker run", and we keep both an IMAP4rev1 and an IMAP4rev2 Dovecot fixture in our CI, so a fix can be regression-tested on both dialects.

2 1

Request for official Dovecot 2.4 RPMs for Amazon Linux 2023
by Yashar Fakhari 14 Aug '26

14 Aug '26

Hello Dovecot team, Would you consider providing official Dovecot 2.4 RPMs for Amazon Linux 2023 on both x86_64 and aarch64? I needed Dovecot 2.4 for FTS Flatcurve because Amazon Linux 2023 currently provides Dovecot 2.3 without a Flatcurve package, while the standalone Flatcurve project for 2.3 is EOL and development has moved into Dovecot core. I have developed a working experimental build process, and I hope the lessons from it could help make Amazon Linux 2023 easier to add to your release workflow: The entire build runs through GitHub Actions using Amazon Linux 2023 containers. GitHub repo: https://github.com/YasharF/dovecot.2.4.ALinux2023 Workflow files .github/workflow:- watch.yml monitors Dovecot's RPM repository for new releases.- build.yml, using build/build.sh, rebuilds the RPMs for both architectures with two documented spec adjustments.- verify.yml runs basic verification tests.- publish.yml publishes verified artifacts as a DNF-compatible repository. I am currently using the resulting RPMs in production. This project is intended only as a temporary bridge until AWS or Dovecot provides official packages. Official, continuously updated Dovecot packages would remain valuable even if AWS later adopts a newer release. As a distribution, AWS/Amazon Linux may freeze the packaged version and provide primarily security fixes rather than each subsequent Dovecot feature and bug-fix release. Thank you for considering this. Let me know if I can contribute in any other way for this. Best regards, Yashar Fakharihttps://github.com/YasharFhttps://linkedin.com/in/YasharF Hello Dovecot team, Would you consider providing official Dovecot 2.4 RPMs for Amazon Linux 2023 on both x86_64 and aarch64? I needed Dovecot 2.4 for FTS Flatcurve because Amazon Linux 2023 currently provides Dovecot 2.3 without a Flatcurve package, while the standalone Flatcurve project for 2.3 is EOL and development has moved into Dovecot core. I have developed a working experimental build process, and I hope the lessons from it could help make Amazon Linux 2023 easier to add to your release workflow: The entire build runs through GitHub Actions using Amazon Linux 2023 containers. GitHub repo: https://github.com/YasharF/dovecot.2.4.ALinux2023 Workflow files .github/workflow: - watch.yml monitors Dovecot's RPM repository for new releases. - build.yml, using build/build.sh, rebuilds the RPMs for both architectures with two documented spec adjustments. - verify.yml runs basic verification tests. - publish.yml publishes verified artifacts as a DNF-compatible repository. I am currently using the resulting RPMs in production. This project is intended only as a temporary bridge until AWS or Dovecot provides official packages. Official, continuously updated Dovecot packages would remain valuable even if AWS later adopts a newer release. As a distribution, AWS/Amazon Linux may freeze the packaged version and provide primarily security fixes rather than each subsequent Dovecot feature and bug-fix release. Thank you for considering this. Let me know if I can contribute in any other way for this. Best regards, Yashar Fakhari https://github.com/YasharF https://linkedin.com/in/YasharF

2 1

fts-flatcurve 2.4.4: after "doveadm fts optimize" a mailbox stops indexing new mail (empty current.<id> shard, missing iamglass)
by Ihor Rusyn 13 Aug '26

13 Aug '26

Hello, "doveadm fts optimize" leaves the mailbox in a state where no new mail can be indexed. The existing index survives and searches over already-indexed mail keep working, so the damage is invisible until the next message arrives. Version: 2.4.4 (8b687aa65c), packaged in a container image FTS: fts_flatcurve, substring_search = yes, min_term_size default (2) fts_autoindex = yes, fts_search_add_missing = yes Index store: NFSv3 (rsize/wsize 32768, local_lock=none) Mail store: NFSv3, maildir Reproduction ------------ 1) Starting point: a healthy index. # doveadm fts flatcurve check -u user(a)example.com INBOX INBOX guid=88885837bb33136365390000d09efc50 errors=0 shards=1 # ls /indexes/.../.INBOX/fts-flatcurve/*/ current.1786117302443530: flintlock iamglass postlist.glass termlist.glass # doveadm search -u user(a)example.com mailbox INBOX text a | tail -1 88885837bb33136365390000d09efc50 58 2) Optimize. Exits 0, prints nothing. # doveadm fts optimize -u user(a)example.com # echo $? 0 3) State afterwards: the compacted data is in a new index.<n> directory, and the pre-existing current.<id> directory is still present but EMPTY. # ls /indexes/.../.INBOX/fts-flatcurve/*/ current.1786117302443530: index.1464: flintlock iamglass postlist.glass termlist.glass # ls -ld .../fts-flatcurve/current.1786117302443530 drwx------ 2 dovecot dovecot 4096 Aug 7 16:08 .../current.1786117302443530 # doveadm fts flatcurve check -u user(a)example.com INBOX Error: fts-flatcurve(INBOX): Cannot open DB (RO; current.1786117302443530); DatabaseNotFoundError: Couldn't detect type of database Error: fts-flatcurve(INBOX): Check failed; DatabaseOpeningError: Directory does not contain a Xapian database Searching already-indexed mail still works at this point: # doveadm search -u user(a)example.com mailbox INBOX text a | tail -1 88885837bb33136365390000d09efc50 58 4) Deliver one message and index it. This is where it breaks: # printf 'From: a(a)b.c\nSubject: reproprobe\n\nbody\n' \ | doveadm save -u user(a)example.com -m INBOX Info: Mailbox INBOX: save: box=INBOX, uid=59, subject=reproprobe # doveadm index -u user(a)example.com INBOX Error: fts-flatcurve(INBOX): Cannot open DB (RW, current.1786117302443530): .../current.1786117302443530/iamglass: Failed to open glass revision file for reading (No such file or directory) Error: cmd index: Mailbox INBOX: Precache for UID=59 failed: Internal error 5) From now on SEARCH fails too, because fts_search_add_missing makes it index first: # doveadm search -u user(a)example.com mailbox INBOX text reproprobe Error: Mailbox INBOX: indexer failed to index mailbox Error: cmd search: Searching mailbox INBOX failed: Internal error occurred. Over IMAP the client sees: NO [SERVERBUG] Internal error occurred. It does not recover by itself ----------------------------- "doveadm index" and autoindexing both take the same write path and fail the same way. "doveadm fts rescan" does not help either -- it also cannot open the empty shard. The only repair we found is to delete the folder's fts-flatcurve directory and index the folder again. Impact ------ A quiet mailbox looks fine for as long as nothing is delivered to it, so this can sit unnoticed. We found it because a monitoring mailbox that receives one message per minute started returning SERVERBUG on SEARCH. In our case a maintenance run that called "doveadm fts optimize" per mailbox left 114 mailboxes in this state, which is all of the ones that had any mail. Possible cause (not verified) ----------------------------- In fts_flatcurve_xapian_optimize_box_do() every shard is first opened for writing ("We need to lock all of the mailboxes so nothing changes while we are optimizing"), and the old current.*/index.* directories are then deleted from disk while those write handles are still open. A handle that is closed afterwards appears to recreate its directory without a glass revision file, which matches what is left behind. Also, in the same function the result of fts_flatcurve_xapian_rename_db(backend, dbpath, NULL, error_r) is discarded: rename_db() does not update the path it is given and returns the new path only through newpath_r, which is NULL at this call site. I have not tried to confirm either of these beyond reading the code. Not the same as the known root-owned iamglass report ---------------------------------------------------- There is an older report where "doveadm fts optimize -A" run from cron as root left iamglass owned by root, producing the same error string. That is not this: here the directories are owned by the mail user (dovecot:dovecot, 0700) and the revision file does not exist at all. -- Best regards, Ihor Rusyn Hello, "doveadm fts optimize" leaves the mailbox in a state where no new mail can be indexed. The existing index survives and searches over already-indexed mail keep working, so the damage is invisible until the next message arrives. Version: 2.4.4 (8b687aa65c), packaged in a container image FTS: fts_flatcurve, substring_search = yes, min_term_size default (2) fts_autoindex = yes, fts_search_add_missing = yes Index store: NFSv3 (rsize/wsize 32768, local_lock=none) Mail store: NFSv3, maildir Reproduction ------------ 1) Starting point: a healthy index. # doveadm fts flatcurve check -u [1]user(a)example.com INBOX INBOX guid=88885837bb33136365390000d09efc50 errors=0 shards=1 # ls /indexes/.../.INBOX/fts-flatcurve/*/ current.1786117302443530: flintlock iamglass postlist.glass termlist.glass # doveadm search -u [2]user(a)example.com mailbox INBOX text a | tail -1 88885837bb33136365390000d09efc50 58 2) Optimize. Exits 0, prints nothing. # doveadm fts optimize -u [3]user(a)example.com # echo $? 0 3) State afterwards: the compacted data is in a new index.<n> directory, and the pre-existing current.<id> directory is still present but EMPTY. # ls /indexes/.../.INBOX/fts-flatcurve/*/ current.1786117302443530: index.1464: flintlock iamglass postlist.glass termlist.glass # ls -ld .../fts-flatcurve/current.1786117302443530 drwx------ 2 dovecot dovecot 4096 Aug 7 16:08 .../current.1786117302443530 # doveadm fts flatcurve check -u [4]user(a)example.com INBOX Error: fts-flatcurve(INBOX): Cannot open DB (RO; current.1786117302443530); DatabaseNotFoundError: Couldn't detect type of database Error: fts-flatcurve(INBOX): Check failed; DatabaseOpeningError: Directory does not contain a Xapian database Searching already-indexed mail still works at this point: # doveadm search -u [5]user(a)example.com mailbox INBOX text a | tail -1 88885837bb33136365390000d09efc50 58 4) Deliver one message and index it. This is where it breaks: # printf 'From: a(a)b.c\nSubject: reproprobe\n\nbody\n' \ | doveadm save -u [6]user(a)example.com -m INBOX Info: Mailbox INBOX: save: box=INBOX, uid=59, subject=reproprobe # doveadm index -u [7]user(a)example.com INBOX Error: fts-flatcurve(INBOX): Cannot open DB (RW, current.1786117302443530): .../current.1786117302443530/iamglass: Failed to open glass revision file for reading (No such file or directory) Error: cmd index: Mailbox INBOX: Precache for UID=59 failed: Internal error 5) From now on SEARCH fails too, because fts_search_add_missing makes it index first: # doveadm search -u [8]user(a)example.com mailbox INBOX text reproprobe Error: Mailbox INBOX: indexer failed to index mailbox Error: cmd search: Searching mailbox INBOX failed: Internal error occurred. Over IMAP the client sees: NO [SERVERBUG] Internal error occurred. It does not recover by itself ----------------------------- "doveadm index" and autoindexing both take the same write path and fail the same way. "doveadm fts rescan" does not help either -- it also cannot open the empty shard. The only repair we found is to delete the folder's fts-flatcurve directory and index the folder again. Impact ------ A quiet mailbox looks fine for as long as nothing is delivered to it, so this can sit unnoticed. We found it because a monitoring mailbox that receives one message per minute started returning SERVERBUG on SEARCH. In our case a maintenance run that called "doveadm fts optimize" per mailbox left 114 mailboxes in this state, which is all of the ones that had any mail. Possible cause (not verified) ----------------------------- In fts_flatcurve_xapian_optimize_box_do() every shard is first opened for writing ("We need to lock all of the mailboxes so nothing changes while we are optimizing"), and the old current.*/index.* directories are then deleted from disk while those write handles are still open. A handle that is closed afterwards appears to recreate its directory without a glass revision file, which matches what is left behind. Also, in the same function the result of fts_flatcurve_xapian_rename_db(backend, dbpath, NULL, error_r) is discarded: rename_db() does not update the path it is given and returns the new path only through newpath_r, which is NULL at this call site. I have not tried to confirm either of these beyond reading the code. Not the same as the known root-owned iamglass report ---------------------------------------------------- There is an older report where "doveadm fts optimize -A" run from cron as root left iamglass owned by root, producing the same error string. That is not this: here the directories are owned by the mail user (dovecot:dovecot, 0700) and the revision file does not exist at all. -- Best regards, Ihor Rusyn References Visible links 1. mailto:user@example.com 2. mailto:user@example.com 3. mailto:user@example.com 4. mailto:user@example.com 5. mailto:user@example.com 6. mailto:user@example.com 7. mailto:user@example.com 8. mailto:user@example.com

4 6

Reports to this list held twice for message size — please release them from moderation
by Ihor Rusyn 11 Aug '26

11 Aug '26

Hello, I have twice tried to send a bug report to this list, and both times the message was blocked for exceeding the size limit with error "The message is larger than the 40 KB maximum size" The report is long because it is meant to be actionable: environment details, exact reproduction steps, log excerpts and strace output. That is the material a maintainer asks for first — and it is exactly what the size limit rejects. If this list is where technical reports belong, a limit that blocks them works against its purpose. Could you please release both of my held messages from moderation and reply on the thread? If long reports are not wanted here, I would appreciate a pointer to where they should go instead. The subject of email was «2.4.4 fts-flatcurve: fts optimize silently destroys a folder's index when the index is on NFS», sent on today from this address. -- Best regards, Ihor Ru Hello, I have twice tried to send a bug report to this list, and both times the message was blocked for exceeding the size limit with error "The message is larger than the 40 KB maximum size" The report is long because it is meant to be actionable: environment details, exact reproduction steps, log excerpts and strace output. That is the material a maintainer asks for first -- and it is exactly what the size limit rejects. If this list is where technical reports belong, a limit that blocks them works against its purpose. Could you please release both of my held messages from moderation and reply on the thread? If long reports are not wanted here, I would appreciate a pointer to where they should go instead. The subject of email was <<2.4.4 fts-flatcurve: fts optimize silently destroys a folder's index when the index is on NFS>>, sent on today from this address. -- Best regards, Ihor Ru

3 2