- dovecot - dovecot.org

mbox - maildir transition using dovecot lmtp local user auth struggles
by "Tóth Attila" 14 Aug '26

14 Aug '26

Approaching age 49 I passed the time long before when I was excited about a new version or a software upgrade. Why poke something that works? From time to time I'm forced to upgrade hardware and software against my will otherwise could serve many more years. I became the victim of mbox -> maildir transition. First I converted the good old Sendmail to Postfix. Initially it still used procmail delivering in mbox mailboxes, now I converted the mboxes to maildir and changed the way of delivery by dovecot lmtp, since that is the software I use for IMAP access. At least there seem to be no data-loss after a couple of doveadm backup and sync retries. There are just 4 local users, so dovecot was set for pam authentication and it worked (for 20+ years, although there were some bumps while upgrading from 2.3 to 2.4 syntax). Now dovecot lmtp started bouncing mails, because it was trying to authenticate a username including the domain name as well and the backend returned unknown user, since expected local user names. This wasn't a problem with mbox/procmail. I asked AI for help and first suggested to use auth_username_format = %Ln. It soon turned out, that this syntax is obsolete. So I dig into it myself and tried auth_username_format = %{user | lower}. It wasn't picked up by dovecot, symptoms persisted. I also tried auth_username_translation = %{user | split('@') | list_first}, which turned out to be also unsuccessful. I also tried to solve the issue on the postfix side by introducing regex handling, but either lmtp_generic_maps = regexp:/etc/postfix/lmtp_generic or recipient_canonical_maps = regexp:/etc/postfix/recipient_canonical had no influence on the symptoms (regex was: /^([^@]+)@atoth\.sote\.hu$/ ${1}). Also tried virtual conversion and lda. Nothing solved the fricking problem. I ended up using a workaround I'm ashamed to share. I kept passdb pam, but for userdb I created a userdb-file, where I included users both with the local name only and the full email address. This shameful concept is functional, but I have to admit it is disappointing: several ways back and forth for solving an issue were all unsuccessful. And it is exhausting. Any ideas appreciated. BR: Dw. -- dr Tóth Attila, Radiológus, 06-20-825-8057 Attila Toth MD, Radiologist, +36-20-825-8057

1 0

Debian 13 Dovecot 2.4.1 LMTP Error Absent ~/sieve
by Conrad T. Pino 14 Aug '26

14 Aug '26

Hello Dovecot List, Our upgrade from Debian 9 Dovecot 2.2.27 to Debian 13 Dovecot 2.4.1 went well. Only recently we added a new mailbox and LMTP delivery failed. Dovecot redacted error from Postfix log: 451 4.2.0 <user@domain> Temporarily unable to access necessary Sieve scripts (in reply to end of DATA command)) Dovecot redacted errors from Dovecot log: dovecot[736]: lmtp(user@domain): Error: sieve: file storage: Performing t_readlink() on active sieve symlink '(null)' failed: readlink() failed: Bad address dovecot[736]: lmtp(user@domain): Error: sieve: Failed to access active Sieve script in parsonal storage 'default': Internal error occurred. Refer to server log for more information. [2026-08-13] (temporary failure) LMTP delivery works after manually creating ~/sieve directory with correct permissions. I know a PostfixAdmin postcreation script can automate directory creation. Have I overlooked Dovecot setting to have Dovecot create ~/sieve directory during LMTP delivery? Best regards, Conrad T. Pino

1 0

ai dovecot
by Marc 14 Aug '26

14 Aug '26

I am just a bit curious, are llm's being used to check the code or help in any other way?

1 0

BUG 2.4 regression: a wrong password over PLAIN is reported as a temporary failure, and the invalid_grant branch is now dead code
by Ihor Rusyn 14 Aug '26

14 Aug '26

Summary With passdb oauth2 doing a password grant, a wrong password makes dovecot answer the IMAP client a NO [UNAVAILABLE] Temporary authentication failure. and log it at Error: level. 2.3 answered AUTHENTICATIONFAILED for the same exchange. PASSDB_RESULT_PASSWORD_MISMATCH is still in db-oauth2.c, but for the password grant it can no longer be reached. Environment Dovecot *2.4.4 (8b687aa65c)*, the official dovecot/dovecot:2.4.4-root image, unmodified. Compared against *2.3.21*. Reproduction Any RFC 6749 compliant authorization server. Point passdb oauth2 at its token endpoint with grant_type=password and attempt an IMAP LOGIN with a wrong password. The server answers, as §5.2 prescribes: HTTP 400 {"error":"invalid_grant","error_description":"Invalid credentials"} *2.3.21:* NO [AUTHENTICATIONFAILED], logged at info level. *2.4.4:* NO [UNAVAILABLE] Temporary authentication failure, logged as auth(user,ip,sasl:plain): Error: oauth2-grant: oauth2 failed: Password grant failed: Invalid credentials imap-login: Login aborted: Logged out (auth service reported temporary failure, 1 attempts in 3 secs) (temp_fail) Cause src/auth/db-oauth2.c is identical in both versions: passdb_result = PASSDB_RESULT_INTERNAL_FAILURE; if (result->error != NULL) error = result->error; else { error = db_oauth2_field_find(result->fields, "error"); if (error == NULL) error = "OAuth2 server returned failure without error field"; else if (strcmp("invalid_grant", error) == 0) passdb_result = PASSDB_RESULT_PASSWORD_MISMATCH; } What changed is what reaches result->error. In *2.3.21*, src/lib-oauth2/oauth2-request.c:113-129 set error only when the JSON *parser* failed. A well-formed error body parsed fine, so error was NULL, res.error stayed NULL, and db-oauth2.c took the else branch, found invalid_grant among the fields and returned PASSWORD_MISMATCH. In *2.4.4*, src/lib-oauth2/oauth2-request.c:136-149 added: /* check if fields contain error now */ const struct oauth2_field *error_field = array_lsearch(&req->fields, "error", request_field_cmp); if (error_field != NULL) { error = error_field->value; /* "invalid_grant" */ /* check for detailed error */ error_field = array_lsearch(&req->fields, "error_description", request_field_cmp); if (error_field != NULL) error = error_field->value; /* overwritten: "Invalid credentials" */ } req->json_parsed_cb(req, error); oauth2_request_continue() then does if (error != NULL) res.error = error;, so result->error is always set on failure and db-oauth2.c never reaches the branch that recognises invalid_grant. The machine-readable code is replaced by the human-readable description before anything can act on it. Note that error_description is *optional* in RFC 6749 §5.2 — so a fully compliant server triggers this simply by being helpful. Omitting it does not help either: error would then hold invalid_grant and still be assigned to res.error, taking the same branch. Why it matters - Mail clients treat [UNAVAILABLE] as a server problem and retry rather than prompting for the password, so users are not told their password is wrong. - Monitoring reads a user error as a service failure — on a farm this is the difference between a quiet log and a page. - RFC 5530 defines AUTHENTICATIONFAILED for a failed authentication and UNAVAILABLE for a temporarily unavailable server. Reporting the former as the latter tells the client something untrue about the server. There is no anti-enumeration benefit in the current behaviour: the same provider response is returned for a wrong password and for a non-existent account, so both are indistinguishable to the client under either mapping. Summary With passdb oauth2 doing a password grant, a wrong password makes dovecot answer the IMAP client a NO [UNAVAILABLE] Temporary authentication failure. and log it at Error: level. 2.3 answered AUTHENTICATIONFAILED for the same exchange. PASSDB_RESULT_PASSWORD_MISMATCH is still in db-oauth2.c, but for the password grant it can no longer be reached. Environment Dovecot 2.4.4 (8b687aa65c), the official dovecot/dovecot:2.4.4-root image, unmodified. Compared against 2.3.21. Reproduction Any RFC 6749 compliant authorization server. Point passdb oauth2 at its token endpoint with grant_type=password and attempt an IMAP LOGIN with a wrong password. The server answers, as S5.2 prescribes: HTTP 400 {"error":"invalid_grant","error_description":"Invalid credentials"} 2.3.21: NO [AUTHENTICATIONFAILED], logged at info level. 2.4.4: NO [UNAVAILABLE] Temporary authentication failure, logged as auth(user,ip,sasl:plain): Error: oauth2-grant: oauth2 failed: Password grant failed: Invalid credentials imap-login: Login aborted: Logged out (auth service reported temporary failure, 1 attempts in 3 secs) (temp_fail) Cause src/auth/db-oauth2.c is identical in both versions: passdb_result = PASSDB_RESULT_INTERNAL_FAILURE; if (result->error != NULL) error = result->error; else { error = db_oauth2_field_find(result->fields, "error"); if (error == NULL) error = "OAuth2 server returned failure without error field"; else if (strcmp("invalid_grant", error) == 0) passdb_result = PASSDB_RESULT_PASSWORD_MISMATCH; } What changed is what reaches result->error. In 2.3.21, src/lib-oauth2/oauth2-request.c:113-129 set error only when the JSON parser failed. A well-formed error body parsed fine, so error was NULL, res.error stayed NULL, and db-oauth2.c took the else branch, found invalid_grant among the fields and returned PASSWORD_MISMATCH. In 2.4.4, src/lib-oauth2/oauth2-request.c:136-149 added: /* check if fields contain error now */ const struct oauth2_field *error_field = array_lsearch(&req->fields, "error", request_field_cmp); if (error_field != NULL) { error = error_field->value; /* "invalid_grant" */ /* check for detailed error */ error_field = array_lsearch(&req->fields, "error_description", request_field_cmp); if (error_field != NULL) error = error_field->value; /* overwritten: "Invalid credentials" */ } req->json_parsed_cb(req, error); oauth2_request_continue() then does if (error != NULL) res.error = error;, so result->error is always set on failure and db-oauth2.c never reaches the branch that recognises invalid_grant. The machine-readable code is replaced by the human-readable description before anything can act on it. Note that error_description is optional in RFC 6749 S5.2 -- so a fully compliant server triggers this simply by being helpful. Omitting it does not help either: error would then hold invalid_grant and still be assigned to res.error, taking the same branch. Why it matters o Mail clients treat [UNAVAILABLE] as a server problem and retry rather than prompting for the password, so users are not told their password is wrong. o Monitoring reads a user error as a service failure -- on a farm this is the difference between a quiet log and a page. o RFC 5530 defines AUTHENTICATIONFAILED for a failed authentication and UNAVAILABLE for a temporarily unavailable server. Reporting the former as the latter tells the client something untrue about the server. There is no anti-enumeration benefit in the current behaviour: the same provider response is returned for a wrong password and for a non-existent account, so both are indistinguishable to the client under either mapping.

1 0

IMAP4rev2: extended LIST drops \Sent, \Drafts, \Trash and \Junk unless the client sends the RFC 6154 return option that IMAP4rev2 does not define
by dennis＠dennisameling.com 14 Aug '26

14 Aug '26

SYSTEM INFORMATION ================== Dovecot version: 2.4.4 (8b687aa65c), Pigeonhole 2.4.4 (3beb331a) OS: Linux 6.12.76-linuxkit aarch64, Debian 13.5 (official dovecot/dovecot:2.4.4 container image, digest sha256:723e3392fe16c6fad8ddc605ea767cc01b4bad 9cd9f13eb1dbac15e79c89b2d4) Host: Docker 29.6.2 (linux/arm64) on macOS 26, Apple silicon CPU architecture: aarch64 (arm64) Filesystem: Docker named volume (overlay2) inside the Docker Desktop VM. Not NFS. Feature: experimental IMAP4rev2 (imap4rev2_enable, mail_utf8_extensions) Client: our own IMAP client library (Rust), but reproduced below with openssl s_client so that no client code is involved The configuration is the image's own /etc/dovecot/dovecot.conf, unmodified: nothing of ours is in it. Rather than paste 200 lines of your own defaults, "doveconf -n" for it is docker run --rm --entrypoint doveconf dovecot/dovecot:2.4.4 -n and we will gladly send the output if you would prefer it inline. The three settings that bear on this report are imap4rev2_enable = yes, mail_utf8_extensions = yes and mailbox_list_utf8 = yes, all three of which the image sets itself. SUMMARY ======= In an IMAP4rev2 session, an extended LIST (one carrying any RETURN (...) clause) omits the special-use mailbox attributes unless the client also sends RETURN (SPECIAL-USE). IMAP4rev2 folded those attributes into the base protocol and defines no SPECIAL-USE return option, so a client that implements only RFC 9051 has no defined way to ask for them, and therefore cannot obtain them from any extended LIST at all. The practical effect for such a client is that every folder loses its role the moment the client asks for anything else -- in our case RETURN (STATUS (UNSEEN)), to fetch unread counts in the same round trip. It cannot find the \Sent folder to place a copy of a sent message in, cannot find \Trash to move a deleted message to, and renders the mailbox list without role icons or ordering. We are not asking for a workaround: we already send RETURN (SPECIAL-USE) on both dialects, since Dovecot advertises the SPECIAL-USE capability in a rev2 session too. This report is about what a strictly RFC 9051 client is able to do. That decision in our client is here, in case it is of interest -- the comment above it records what we found on your server: https://github.com/allodia-eu/email-calendar-sync-engine/blob/fa85b88121132… REPRODUCTION ============ No configuration is required. dovecot/dovecot:2.4.4 ships imap4rev2_enable = yes and mail_utf8_extensions = yes in its own /etc/dovecot/dovecot.conf, and its "@mailbox_defaults = english" creates Drafts/Junk/Sent/Trash with special-use attributes. docker run -d --name dovecot-stock -p 127.0.0.1:11999:31993 \ -e USER_PASSWORD=stock-test-pw \ dovecot/dovecot:2.4.4 { printf 'a1 LOGIN alice(a)example.com stock-test-pw\r\n'; sleep 2; printf 'a2 ENABLE IMAP4rev2\r\n'; sleep 1; printf 'a3 LIST "" "*"\r\n'; sleep 1; printf 'a4 LIST "" "*" RETURN (STATUS (UNSEEN))\r\n'; sleep 1; printf 'a5 LIST "" "*" RETURN (SPECIAL-USE STATUS (UNSEEN))\r\n'; sleep 1; printf 'a6 LOGOUT\r\n'; sleep 1; } \ | openssl s_client -connect 127.0.0.1:11999 -quiet TRANSCRIPT ========== Verbatim; the greeting and the CAPABILITY lines are elided. S: * ENABLED IMAP4REV2 S: a2 OK Enabled (0.001 + 0.000 secs). C: a3 LIST "" "*" S: * LIST (\HasNoChildren \Drafts) "/" Drafts S: * LIST (\HasNoChildren \Junk) "/" Junk S: * LIST (\HasNoChildren \Sent) "/" Sent S: * LIST (\HasNoChildren \Trash) "/" Trash S: * LIST (\HasNoChildren) "/" INBOX S: a3 OK List completed (0.001 + 0.000 secs). --> correct: the role attributes are present C: a4 LIST "" "*" RETURN (STATUS (UNSEEN)) S: * LIST () "/" Drafts S: * STATUS Drafts (UNSEEN 0) S: * LIST () "/" Junk S: * STATUS Junk (UNSEEN 0) S: * LIST () "/" Sent S: * STATUS Sent (UNSEEN 0) S: * LIST () "/" Trash S: * STATUS Trash (UNSEEN 0) S: * LIST () "/" INBOX S: * STATUS INBOX (UNSEEN 0) S: a4 OK List completed (0.034 + 0.000 + 0.033 secs). --> the problem: every role attribute is gone, and IMAP4rev2 gives the client no option to ask for them back C: a5 LIST "" "*" RETURN (SPECIAL-USE STATUS (UNSEEN)) S: * LIST (\Trash) "/" Trash S: * STATUS Trash (UNSEEN 0) S: * LIST (\Sent) "/" Sent S: * STATUS Sent (UNSEEN 0) S: * LIST (\Junk) "/" Junk S: * STATUS Junk (UNSEEN 0) S: * LIST (\Drafts) "/" Drafts S: * STATUS Drafts (UNSEEN 0) S: * LIST () "/" INBOX S: * STATUS INBOX (UNSEEN 0) S: a5 OK List completed (0.002 + 0.000 + 0.001 secs). --> present again, but only via the RFC 6154 return option, which RFC 9051 does not define WHY WE BELIEVE a4 IS NOT CORRECT FOR IMAP4rev2 ============================================== 1. RFC 9051 section 7.3.1 makes the special-use attributes ordinary LIST mailbox attributes: "In addition to the base mailbox name attributes defined above, an IMAP server MAY also include any or all of the following attributes that denote 'role' (or 'special-use') of a mailbox. These attributes are included along with base attributes defined above." Appendix E item 2 lists "the list of new mailbox attributes from SPECIAL-USE [RFC6154]" among what IMAP4rev2 folded in. 2. RFC 9051 section 6.3.9.2 defines the return options for IMAP4rev2 as exactly SUBSCRIBED, CHILDREN and STATUS. There is no SPECIAL-USE return option. Nor is SPECIAL-USE among the selection options in section 6.3.9.1, which are SUBSCRIBED, REMOTE and RECURSIVEMATCH; it survives in RFC 9051 only as an unexplained C: A002 LIST (SPECIAL-USE) "" "*" in the NAMESPACE examples in section 6.3.10, which may itself be worth an erratum. 3. Section 6.3.9 says "a client MUST NOT send an option for which the server has not advertised support", so a strictly rev2 client will not invent one. Taken together, an RFC 9051-only client can obtain the role attributes on a basic LIST but never on an extended one, which means it cannot combine roles with RETURN (STATUS ...) -- the single round trip that RFC 5819 exists to provide. Since RFC 9051 section 6.3.9 also states that return options "MUST NOT cause the server to report information about additional mailbox names", but says nothing about removing base attributes, we read a4 as a case where an attribute defined as base data is being withheld. Note the contrast with \HasNoChildren in the same transcript: it also disappears in a4, but CHILDREN is a return option in RFC 9051, so a rev2 client can ask for it back. For special-use there is no equivalent. WHAT WE BELIEVE CORRECT BEHAVIOUR WOULD BE ========================================== In a session where IMAP4rev2 has been ENABLEd, include the special-use attributes in extended LIST responses without requiring RETURN (SPECIAL-USE) -- that is, treat them as the base mailbox attributes that RFC 9051 section 7.3.1 defines them to be. The RFC 6154 return option would remain accepted for IMAP4rev1 sessions. NOTE ON IMAP4rev1: WE BELIEVE THAT BEHAVIOUR IS CORRECT ======================================================= The same server on an IMAP4rev1 session behaves identically, and there we read it as compliant. RFC 6154 section 2 says "The extended LIST command MAY return SPECIAL-USE attributes even if the client does not specify the return option", and MAY is not MUST. This report concerns only IMAP4rev2 sessions, where the way to ask was removed along with the need to. CONTACT ======= Happy to test a patch. The reproduction is a single "docker run", and we keep both an IMAP4rev1 and an IMAP4rev2 Dovecot fixture in our CI, so a fix can be regression-tested on both dialects.

2 1

Request for official Dovecot 2.4 RPMs for Amazon Linux 2023
by Yashar Fakhari 14 Aug '26

14 Aug '26

Hello Dovecot team, Would you consider providing official Dovecot 2.4 RPMs for Amazon Linux 2023 on both x86_64 and aarch64? I needed Dovecot 2.4 for FTS Flatcurve because Amazon Linux 2023 currently provides Dovecot 2.3 without a Flatcurve package, while the standalone Flatcurve project for 2.3 is EOL and development has moved into Dovecot core. I have developed a working experimental build process, and I hope the lessons from it could help make Amazon Linux 2023 easier to add to your release workflow: The entire build runs through GitHub Actions using Amazon Linux 2023 containers. GitHub repo: https://github.com/YasharF/dovecot.2.4.ALinux2023 Workflow files .github/workflow:- watch.yml monitors Dovecot's RPM repository for new releases.- build.yml, using build/build.sh, rebuilds the RPMs for both architectures with two documented spec adjustments.- verify.yml runs basic verification tests.- publish.yml publishes verified artifacts as a DNF-compatible repository. I am currently using the resulting RPMs in production. This project is intended only as a temporary bridge until AWS or Dovecot provides official packages. Official, continuously updated Dovecot packages would remain valuable even if AWS later adopts a newer release. As a distribution, AWS/Amazon Linux may freeze the packaged version and provide primarily security fixes rather than each subsequent Dovecot feature and bug-fix release. Thank you for considering this. Let me know if I can contribute in any other way for this. Best regards, Yashar Fakharihttps://github.com/YasharFhttps://linkedin.com/in/YasharF Hello Dovecot team, Would you consider providing official Dovecot 2.4 RPMs for Amazon Linux 2023 on both x86_64 and aarch64? I needed Dovecot 2.4 for FTS Flatcurve because Amazon Linux 2023 currently provides Dovecot 2.3 without a Flatcurve package, while the standalone Flatcurve project for 2.3 is EOL and development has moved into Dovecot core. I have developed a working experimental build process, and I hope the lessons from it could help make Amazon Linux 2023 easier to add to your release workflow: The entire build runs through GitHub Actions using Amazon Linux 2023 containers. GitHub repo: https://github.com/YasharF/dovecot.2.4.ALinux2023 Workflow files .github/workflow: - watch.yml monitors Dovecot's RPM repository for new releases. - build.yml, using build/build.sh, rebuilds the RPMs for both architectures with two documented spec adjustments. - verify.yml runs basic verification tests. - publish.yml publishes verified artifacts as a DNF-compatible repository. I am currently using the resulting RPMs in production. This project is intended only as a temporary bridge until AWS or Dovecot provides official packages. Official, continuously updated Dovecot packages would remain valuable even if AWS later adopts a newer release. As a distribution, AWS/Amazon Linux may freeze the packaged version and provide primarily security fixes rather than each subsequent Dovecot feature and bug-fix release. Thank you for considering this. Let me know if I can contribute in any other way for this. Best regards, Yashar Fakhari https://github.com/YasharF https://linkedin.com/in/YasharF

2 1

fts-flatcurve 2.4.4: after "doveadm fts optimize" a mailbox stops indexing new mail (empty current.<id> shard, missing iamglass)
by Ihor Rusyn 13 Aug '26

13 Aug '26

Hello, "doveadm fts optimize" leaves the mailbox in a state where no new mail can be indexed. The existing index survives and searches over already-indexed mail keep working, so the damage is invisible until the next message arrives. Version: 2.4.4 (8b687aa65c), packaged in a container image FTS: fts_flatcurve, substring_search = yes, min_term_size default (2) fts_autoindex = yes, fts_search_add_missing = yes Index store: NFSv3 (rsize/wsize 32768, local_lock=none) Mail store: NFSv3, maildir Reproduction ------------ 1) Starting point: a healthy index. # doveadm fts flatcurve check -u user(a)example.com INBOX INBOX guid=88885837bb33136365390000d09efc50 errors=0 shards=1 # ls /indexes/.../.INBOX/fts-flatcurve/*/ current.1786117302443530: flintlock iamglass postlist.glass termlist.glass # doveadm search -u user(a)example.com mailbox INBOX text a | tail -1 88885837bb33136365390000d09efc50 58 2) Optimize. Exits 0, prints nothing. # doveadm fts optimize -u user(a)example.com # echo $? 0 3) State afterwards: the compacted data is in a new index.<n> directory, and the pre-existing current.<id> directory is still present but EMPTY. # ls /indexes/.../.INBOX/fts-flatcurve/*/ current.1786117302443530: index.1464: flintlock iamglass postlist.glass termlist.glass # ls -ld .../fts-flatcurve/current.1786117302443530 drwx------ 2 dovecot dovecot 4096 Aug 7 16:08 .../current.1786117302443530 # doveadm fts flatcurve check -u user(a)example.com INBOX Error: fts-flatcurve(INBOX): Cannot open DB (RO; current.1786117302443530); DatabaseNotFoundError: Couldn't detect type of database Error: fts-flatcurve(INBOX): Check failed; DatabaseOpeningError: Directory does not contain a Xapian database Searching already-indexed mail still works at this point: # doveadm search -u user(a)example.com mailbox INBOX text a | tail -1 88885837bb33136365390000d09efc50 58 4) Deliver one message and index it. This is where it breaks: # printf 'From: a(a)b.c\nSubject: reproprobe\n\nbody\n' \ | doveadm save -u user(a)example.com -m INBOX Info: Mailbox INBOX: save: box=INBOX, uid=59, subject=reproprobe # doveadm index -u user(a)example.com INBOX Error: fts-flatcurve(INBOX): Cannot open DB (RW, current.1786117302443530): .../current.1786117302443530/iamglass: Failed to open glass revision file for reading (No such file or directory) Error: cmd index: Mailbox INBOX: Precache for UID=59 failed: Internal error 5) From now on SEARCH fails too, because fts_search_add_missing makes it index first: # doveadm search -u user(a)example.com mailbox INBOX text reproprobe Error: Mailbox INBOX: indexer failed to index mailbox Error: cmd search: Searching mailbox INBOX failed: Internal error occurred. Over IMAP the client sees: NO [SERVERBUG] Internal error occurred. It does not recover by itself ----------------------------- "doveadm index" and autoindexing both take the same write path and fail the same way. "doveadm fts rescan" does not help either -- it also cannot open the empty shard. The only repair we found is to delete the folder's fts-flatcurve directory and index the folder again. Impact ------ A quiet mailbox looks fine for as long as nothing is delivered to it, so this can sit unnoticed. We found it because a monitoring mailbox that receives one message per minute started returning SERVERBUG on SEARCH. In our case a maintenance run that called "doveadm fts optimize" per mailbox left 114 mailboxes in this state, which is all of the ones that had any mail. Possible cause (not verified) ----------------------------- In fts_flatcurve_xapian_optimize_box_do() every shard is first opened for writing ("We need to lock all of the mailboxes so nothing changes while we are optimizing"), and the old current.*/index.* directories are then deleted from disk while those write handles are still open. A handle that is closed afterwards appears to recreate its directory without a glass revision file, which matches what is left behind. Also, in the same function the result of fts_flatcurve_xapian_rename_db(backend, dbpath, NULL, error_r) is discarded: rename_db() does not update the path it is given and returns the new path only through newpath_r, which is NULL at this call site. I have not tried to confirm either of these beyond reading the code. Not the same as the known root-owned iamglass report ---------------------------------------------------- There is an older report where "doveadm fts optimize -A" run from cron as root left iamglass owned by root, producing the same error string. That is not this: here the directories are owned by the mail user (dovecot:dovecot, 0700) and the revision file does not exist at all. -- Best regards, Ihor Rusyn Hello, "doveadm fts optimize" leaves the mailbox in a state where no new mail can be indexed. The existing index survives and searches over already-indexed mail keep working, so the damage is invisible until the next message arrives. Version: 2.4.4 (8b687aa65c), packaged in a container image FTS: fts_flatcurve, substring_search = yes, min_term_size default (2) fts_autoindex = yes, fts_search_add_missing = yes Index store: NFSv3 (rsize/wsize 32768, local_lock=none) Mail store: NFSv3, maildir Reproduction ------------ 1) Starting point: a healthy index. # doveadm fts flatcurve check -u [1]user(a)example.com INBOX INBOX guid=88885837bb33136365390000d09efc50 errors=0 shards=1 # ls /indexes/.../.INBOX/fts-flatcurve/*/ current.1786117302443530: flintlock iamglass postlist.glass termlist.glass # doveadm search -u [2]user(a)example.com mailbox INBOX text a | tail -1 88885837bb33136365390000d09efc50 58 2) Optimize. Exits 0, prints nothing. # doveadm fts optimize -u [3]user(a)example.com # echo $? 0 3) State afterwards: the compacted data is in a new index.<n> directory, and the pre-existing current.<id> directory is still present but EMPTY. # ls /indexes/.../.INBOX/fts-flatcurve/*/ current.1786117302443530: index.1464: flintlock iamglass postlist.glass termlist.glass # ls -ld .../fts-flatcurve/current.1786117302443530 drwx------ 2 dovecot dovecot 4096 Aug 7 16:08 .../current.1786117302443530 # doveadm fts flatcurve check -u [4]user(a)example.com INBOX Error: fts-flatcurve(INBOX): Cannot open DB (RO; current.1786117302443530); DatabaseNotFoundError: Couldn't detect type of database Error: fts-flatcurve(INBOX): Check failed; DatabaseOpeningError: Directory does not contain a Xapian database Searching already-indexed mail still works at this point: # doveadm search -u [5]user(a)example.com mailbox INBOX text a | tail -1 88885837bb33136365390000d09efc50 58 4) Deliver one message and index it. This is where it breaks: # printf 'From: a(a)b.c\nSubject: reproprobe\n\nbody\n' \ | doveadm save -u [6]user(a)example.com -m INBOX Info: Mailbox INBOX: save: box=INBOX, uid=59, subject=reproprobe # doveadm index -u [7]user(a)example.com INBOX Error: fts-flatcurve(INBOX): Cannot open DB (RW, current.1786117302443530): .../current.1786117302443530/iamglass: Failed to open glass revision file for reading (No such file or directory) Error: cmd index: Mailbox INBOX: Precache for UID=59 failed: Internal error 5) From now on SEARCH fails too, because fts_search_add_missing makes it index first: # doveadm search -u [8]user(a)example.com mailbox INBOX text reproprobe Error: Mailbox INBOX: indexer failed to index mailbox Error: cmd search: Searching mailbox INBOX failed: Internal error occurred. Over IMAP the client sees: NO [SERVERBUG] Internal error occurred. It does not recover by itself ----------------------------- "doveadm index" and autoindexing both take the same write path and fail the same way. "doveadm fts rescan" does not help either -- it also cannot open the empty shard. The only repair we found is to delete the folder's fts-flatcurve directory and index the folder again. Impact ------ A quiet mailbox looks fine for as long as nothing is delivered to it, so this can sit unnoticed. We found it because a monitoring mailbox that receives one message per minute started returning SERVERBUG on SEARCH. In our case a maintenance run that called "doveadm fts optimize" per mailbox left 114 mailboxes in this state, which is all of the ones that had any mail. Possible cause (not verified) ----------------------------- In fts_flatcurve_xapian_optimize_box_do() every shard is first opened for writing ("We need to lock all of the mailboxes so nothing changes while we are optimizing"), and the old current.*/index.* directories are then deleted from disk while those write handles are still open. A handle that is closed afterwards appears to recreate its directory without a glass revision file, which matches what is left behind. Also, in the same function the result of fts_flatcurve_xapian_rename_db(backend, dbpath, NULL, error_r) is discarded: rename_db() does not update the path it is given and returns the new path only through newpath_r, which is NULL at this call site. I have not tried to confirm either of these beyond reading the code. Not the same as the known root-owned iamglass report ---------------------------------------------------- There is an older report where "doveadm fts optimize -A" run from cron as root left iamglass owned by root, producing the same error string. That is not this: here the directories are owned by the mail user (dovecot:dovecot, 0700) and the revision file does not exist at all. -- Best regards, Ihor Rusyn References Visible links 1. mailto:user@example.com 2. mailto:user@example.com 3. mailto:user@example.com 4. mailto:user@example.com 5. mailto:user@example.com 6. mailto:user@example.com 7. mailto:user@example.com 8. mailto:user@example.com

4 6

Global sieve variables don't work in 2.4.4
by inaugurator＠ya.ru 12 Aug '26

12 Aug '26

Hi everyone! I have created a minimal reproducible setup. First, let's prepare the environment and create the necessary directories: ```shell getent group vmail >/dev/null \ || groupadd -g 500 vmail; \ getent passwd vmail \ | awk -F: '$3 == "500" && $4 == "vmail"' >/dev/null \ || { id vmail >/dev/null 2>&1 && usermod -u 500 -g vmail vmail \ || useradd -u 500 -g vmail -M -s /usr/sbin/nologin vmail; } mkdir -p /mail && chown vmail:vmail -R /mail cat > /mail/global_bug.sieve <<EOF require ["variables", "include"]; global "global_var"; set "global_var" "hello"; EOF cat > /mail/before_bug.sieve <<EOF require ["variables", "include", "vnd.dovecot.debug"]; global "global_var"; include :global :optional "global_bug"; debug_log "\${global_var}"; EOF ``` Then, we configure dovecot.conf: ``` dovecot_config_version = 2.4.4 dovecot_storage_version = 2.4.4 auth_allow_cleartext = yes mail_driver = maildir mail_home = /mail/%{user} mail_inbox_path = .INBOX mail_uid = vmail mail_gid = vmail first_valid_uid = 500 protocols { imap = yes } mail_plugins { imap_sieve = yes } service imap-login { inet_listener imaps { port = 0 } } passdb anypass { driver = static fields { nopassword = yes } } sieve_global_extensions { vnd.dovecot.debug = yes } sieve_script global_bug { driver = file path = /mail/global_bug.sieve type = global } namespace inbox { inbox = yes separator = / mailbox INBOX { auto = subscribe sieve_script before_bug { driver = file path = /mail/before_bug.sieve cause = APPEND type = before } } } ``` If we run an APPEND command to the INBOX directory: ```shell curl -v -u "user1:hello" -k "imap://localhost:143/INBOX" --upload-file /tmp/test.eml ``` the log shows an empty message. ``` imap(user1)<3155434><YNISPNZYHtV/AAAB>: sieve: DEBUG: ``` However, if we change the configuration and the Sieve script by replacing "global" with "personal", I see my hello in debug log: ``` ... sieve_script global_bug { driver = file path = /mail/global_bug.sieve type = personal # !!! } ... require ["variables", "include", "vnd.dovecot.debug"]; global "global_var"; include :personal :optional "global_bug"; # !!! debug_log "\${global_var}"; ... imap(user1)<3155659><iiiIhNZYgpl/AAAB>: sieve: DEBUG: hello ``` Why am I unable to import a global variable into a before script if it is defined in a global script? Thank You in advance!

1 0

Reports to this list held twice for message size — please release them from moderation
by Ihor Rusyn 11 Aug '26

11 Aug '26

Hello, I have twice tried to send a bug report to this list, and both times the message was blocked for exceeding the size limit with error "The message is larger than the 40 KB maximum size" The report is long because it is meant to be actionable: environment details, exact reproduction steps, log excerpts and strace output. That is the material a maintainer asks for first — and it is exactly what the size limit rejects. If this list is where technical reports belong, a limit that blocks them works against its purpose. Could you please release both of my held messages from moderation and reply on the thread? If long reports are not wanted here, I would appreciate a pointer to where they should go instead. The subject of email was «2.4.4 fts-flatcurve: fts optimize silently destroys a folder's index when the index is on NFS», sent on today from this address. -- Best regards, Ihor Ru Hello, I have twice tried to send a bug report to this list, and both times the message was blocked for exceeding the size limit with error "The message is larger than the 40 KB maximum size" The report is long because it is meant to be actionable: environment details, exact reproduction steps, log excerpts and strace output. That is the material a maintainer asks for first -- and it is exactly what the size limit rejects. If this list is where technical reports belong, a limit that blocks them works against its purpose. Could you please release both of my held messages from moderation and reply on the thread? If long reports are not wanted here, I would appreciate a pointer to where they should go instead. The subject of email was <<2.4.4 fts-flatcurve: fts optimize silently destroys a folder's index when the index is on NFS>>, sent on today from this address. -- Best regards, Ihor Ru

3 2

2.4.4 fts-flatcurve: fts optimize silently destroys a folder's index when the index is on NFS
by Ihor Rusyn 11 Aug '26

11 Aug '26

Hello, Summary ======= On Dovecot 2.4.4 with fts_flatcurve and mail_index_path on NFS, "doveadm fts optimize" leaves behind an empty shard directory. That directory remains the shard every subsequent write opens, so from that point on the folder can no longer index new mail. The operation reports success and logs nothing at the time. The damage is permanent and, on a folder that receives no new mail, invisible. Two separate defects are involved, and both are reported below: (1) fts optimize deletes a shard directory whose files it still holds open. On a local filesystem the unlink succeeds invisibly; NFS cannot express "unlink but keep open" and silly-renames instead, so the directory survives. (2) unlink_directory() reports success when rmdir() failed, which is what makes (1) silent rather than loud. Environment =========== Dovecot 2.4.4 CE (8b687aa65c) container image sha256:d6b2f80db2e656a9250583712a82397e2752b35ad7e4e2e71c85fdc0d510137d base Debian 13 trixie fts_flatcurve (Xapian glass) libxapian30 1.4.29-3 (libxapian.so.30.14.1) substring_search = yes, min_term_size = 2 optimize_limit at its default of 10 dovecot_storage_version = 2.3.0 mail_index_path on NFSv3; mail storage on a separate NFS export NFS client is the host, not the container: Debian 12 bookworm, kernel 6.1.0-51-amd64 x86_64 nfs-common 1:2.6.2-4+deb12u1 mount options: rw,noatime,vers=3,rsize=32768,wsize=32768,hard, nordirplus,proto=tcp,timeo=600,retrans=2,sec=sys, local_lock=none Impact ====== An operator does not have to invoke optimize to be exposed: with the default optimize_limit = 10, flatcurve runs it itself once a folder accumulates ten shards. Every active folder therefore reaches this eventually. Already-indexed mail stays searchable, so the folder looks healthy. Only indexing of new mail fails. With fts_search_add_missing = yes, the first SEARCH after new mail arrives attempts to index it, fails, and the client receives NO [SERVERBUG]. I hit this while migrating a host from 2.3 to 2.4: 114 mailboxes were left in this state in a single run, and it surfaced only because a monitoring mailbox began answering SERVERBUG. Defect 1: fts optimize leaves an empty shard directory on NFS ============================================================= Steps to reproduce ------------------ 1. Configure fts_flatcurve with the index path on an NFSv3 mount. 2. Take a mailbox with mail in INBOX (a few hundred messages is enough) and build the index: doveadm index -u <user> INBOX Confirm one shard exists: ls <index>/.INBOX/fts-flatcurve/ -> current.<id> 3. Run: doveadm fts optimize -u <user> INBOX (Alternatively, let it fire on its own by driving the folder past optimize_limit shards.) 4. While it runs, list the shard directory repeatedly - .nfsXXXX entries appear inside current.<id>. 5. After it finishes, note the exit status is 0 and no error was logged. List the directory again: index.<n> was created as expected, and current.<id> is still there, now empty. 6. Deliver one new message, then: doveadm index -u <user> INBOX Expected vs actual ------------------ Expected: after optimize, only the compacted index.<n> remains, and subsequent indexing writes to a new shard. Actual: the stale empty current.<id> remains and is reopened for writing, so step 6 fails: Cannot open DB (RW, current.<id>): .../current.<id>/iamglass: Failed to open glass revision file for reading (No such file or directory) cmd index: Mailbox INBOX: Precache for UID=59 failed and IMAP SEARCH on that folder returns NO [SERVERBUG]. What I observed while diagnosing -------------------------------- strace of "doveadm fts optimize" on NFSv3 shows the old shard's files being unlinked while handles on them are still open, so the directory cannot be removed: unlink(...) = 0 (x7, every file in the shard) rmdir(...) = -1 ENOTEMPTY (Directory not empty) rename(...) = 0 (optimize -> index.N, proceeds regardless) Once the process exits, the NFS client reaps the .nfsXXXX files and the directory is simply empty. What does not help ------------------ NFSv4.1, mounting with nolock, "doveadm fts rescan", "doveadm index", and full re-indexing all take the same write path into the same empty shard and fail identically. Removing the empty directory (or the folder's whole fts-flatcurve tree) and re-indexing restores the folder; setting optimize_limit = 0 avoids the situation but leaves the index spread across many shards. Defect 2: unlink_directory() returns success when rmdir() failed ================================================================ The ENOTEMPTY above never reaches the caller. unlink_directory() in src/lib/unlink-directory.c returns 1 when the final rmdir() fails, the same value it returns on success: if (rmdir(dir) < 0 && errno != ENOENT) { *error_r = t_strdup_printf("rmdir(%s) failed: %m", dir); return errno == ENOENT ? 0 : 1; } The "errno != ENOENT" guard already establishes that errno is not ENOENT, so the ternary always yields 1. Callers cannot distinguish "directory removed" from "directory still there", and the error message it composed is discarded. Any condition that makes rmdir() fail with something other than ENOENT reproduces this, so the return value is wrong independently of NFS - every caller of unlink_directory() is affected. The code path is unchanged in current main. -- Best regards, Ihor Ru Hello, Summary ======= On Dovecot 2.4.4 with fts_flatcurve and mail_index_path on NFS, "doveadm fts optimize" leaves behind an empty shard directory. That directory remains the shard every subsequent write opens, so from that point on the folder can no longer index new mail. The operation reports success and logs nothing at the time. The damage is permanent and, on a folder that receives no new mail, invisible. Two separate defects are involved, and both are reported below: (1) fts optimize deletes a shard directory whose files it still holds open. On a local filesystem the unlink succeeds invisibly; NFS cannot express "unlink but keep open" and silly-renames instead, so the directory survives. (2) unlink_directory() reports success when rmdir() failed, which is what makes (1) silent rather than loud. Environment =========== Dovecot 2.4.4 CE (8b687aa65c) container image sha256:d6b2f80db2e656a9250583712a82397e2752b35ad7e4e2e71c85fdc0d510137d base Debian 13 trixie fts_flatcurve (Xapian glass) libxapian30 1.4.29-3 (libxapian.so.30.14.1) substring_search = yes, min_term_size = 2 optimize_limit at its default of 10 dovecot_storage_version = 2.3.0 mail_index_path on NFSv3; mail storage on a separate NFS export NFS client is the host, not the container: Debian 12 bookworm, kernel 6.1.0-51-amd64 x86_64 nfs-common 1:2.6.2-4+deb12u1 mount options: rw,noatime,vers=3,rsize=32768,wsize=32768,hard, nordirplus,proto=tcp,timeo=600,retrans=2,sec=sys, local_lock=none Impact ====== An operator does not have to invoke optimize to be exposed: with the default optimize_limit = 10, flatcurve runs it itself once a folder accumulates ten shards. Every active folder therefore reaches this eventually. Already-indexed mail stays searchable, so the folder looks healthy. Only indexing of new mail fails. With fts_search_add_missing = yes, the first SEARCH after new mail arrives attempts to index it, fails, and the client receives NO [SERVERBUG]. I hit this while migrating a host from 2.3 to 2.4: 114 mailboxes were left in this state in a single run, and it surfaced only because a monitoring mailbox began answering SERVERBUG. Defect 1: fts optimize leaves an empty shard directory on NFS ============================================================= Steps to reproduce ------------------ 1. Configure fts_flatcurve with the index path on an NFSv3 mount. 2. Take a mailbox with mail in INBOX (a few hundred messages is enough) and build the index: doveadm index -u <user> INBOX Confirm one shard exists: ls <index>/.INBOX/fts-flatcurve/ -> current.<id> 3. Run: doveadm fts optimize -u <user> INBOX (Alternatively, let it fire on its own by driving the folder past optimize_limit shards.) 4. While it runs, list the shard directory repeatedly - .nfsXXXX entries appear inside current.<id>. 5. After it finishes, note the exit status is 0 and no error was logged. List the directory again: index.<n> was created as expected, and current.<id> is still there, now empty. 6. Deliver one new message, then: doveadm index -u <user> INBOX Expected vs actual ------------------ Expected: after optimize, only the compacted index.<n> remains, and subsequent indexing writes to a new shard. Actual: the stale empty current.<id> remains and is reopened for writing, so step 6 fails: Cannot open DB (RW, current.<id>): .../current.<id>/iamglass: Failed to open glass revision file for reading (No such file or directory) cmd index: Mailbox INBOX: Precache for UID=59 failed and IMAP SEARCH on that folder returns NO [SERVERBUG]. What I observed while diagnosing -------------------------------- strace of "doveadm fts optimize" on NFSv3 shows the old shard's files being unlinked while handles on them are still open, so the directory cannot be removed: unlink(...) = 0 (x7, every file in the shard) rmdir(...) = -1 ENOTEMPTY (Directory not empty) rename(...) = 0 (optimize -> index.N, proceeds regardless) Once the process exits, the NFS client reaps the .nfsXXXX files and the directory is simply empty. What does not help ------------------ NFSv4.1, mounting with nolock, "doveadm fts rescan", "doveadm index", and full re-indexing all take the same write path into the same empty shard and fail identically. Removing the empty directory (or the folder's whole fts-flatcurve tree) and re-indexing restores the folder; setting optimize_limit = 0 avoids the situation but leaves the index spread across many shards. Defect 2: unlink_directory() returns success when rmdir() failed ================================================================ The ENOTEMPTY above never reaches the caller. unlink_directory() in src/lib/unlink-directory.c returns 1 when the final rmdir() fails, the same value it returns on success: if (rmdir(dir) < 0 && errno != ENOENT) { *error_r = t_strdup_printf("rmdir(%s) failed: %m", dir); return errno == ENOENT ? 0 : 1; } The "errno != ENOENT" guard already establishes that errno is not ENOENT, so the ternary always yields 1. Callers cannot distinguish "directory removed" from "directory still there", and the error message it composed is discarded. Any condition that makes rmdir() fail with something other than ENOENT reproduces this, so the return value is wrong independently of NFS - every caller of unlink_directory() is affected. The code path is unchanged in current main. -- Best regards, Ihor Ru

1 0