- dovecot - dovecot.org

fts-flatcurve 2.4.4: after "doveadm fts optimize" a mailbox stops indexing new mail (empty current.<id> shard, missing iamglass)
by Ihor Rusyn 12 Aug '26

12 Aug '26

Hello, "doveadm fts optimize" leaves the mailbox in a state where no new mail can be indexed. The existing index survives and searches over already-indexed mail keep working, so the damage is invisible until the next message arrives. Version: 2.4.4 (8b687aa65c), packaged in a container image FTS: fts_flatcurve, substring_search = yes, min_term_size default (2) fts_autoindex = yes, fts_search_add_missing = yes Index store: NFSv3 (rsize/wsize 32768, local_lock=none) Mail store: NFSv3, maildir Reproduction ------------ 1) Starting point: a healthy index. # doveadm fts flatcurve check -u user(a)example.com INBOX INBOX guid=88885837bb33136365390000d09efc50 errors=0 shards=1 # ls /indexes/.../.INBOX/fts-flatcurve/*/ current.1786117302443530: flintlock iamglass postlist.glass termlist.glass # doveadm search -u user(a)example.com mailbox INBOX text a | tail -1 88885837bb33136365390000d09efc50 58 2) Optimize. Exits 0, prints nothing. # doveadm fts optimize -u user(a)example.com # echo $? 0 3) State afterwards: the compacted data is in a new index.<n> directory, and the pre-existing current.<id> directory is still present but EMPTY. # ls /indexes/.../.INBOX/fts-flatcurve/*/ current.1786117302443530: index.1464: flintlock iamglass postlist.glass termlist.glass # ls -ld .../fts-flatcurve/current.1786117302443530 drwx------ 2 dovecot dovecot 4096 Aug 7 16:08 .../current.1786117302443530 # doveadm fts flatcurve check -u user(a)example.com INBOX Error: fts-flatcurve(INBOX): Cannot open DB (RO; current.1786117302443530); DatabaseNotFoundError: Couldn't detect type of database Error: fts-flatcurve(INBOX): Check failed; DatabaseOpeningError: Directory does not contain a Xapian database Searching already-indexed mail still works at this point: # doveadm search -u user(a)example.com mailbox INBOX text a | tail -1 88885837bb33136365390000d09efc50 58 4) Deliver one message and index it. This is where it breaks: # printf 'From: a(a)b.c\nSubject: reproprobe\n\nbody\n' \ | doveadm save -u user(a)example.com -m INBOX Info: Mailbox INBOX: save: box=INBOX, uid=59, subject=reproprobe # doveadm index -u user(a)example.com INBOX Error: fts-flatcurve(INBOX): Cannot open DB (RW, current.1786117302443530): .../current.1786117302443530/iamglass: Failed to open glass revision file for reading (No such file or directory) Error: cmd index: Mailbox INBOX: Precache for UID=59 failed: Internal error 5) From now on SEARCH fails too, because fts_search_add_missing makes it index first: # doveadm search -u user(a)example.com mailbox INBOX text reproprobe Error: Mailbox INBOX: indexer failed to index mailbox Error: cmd search: Searching mailbox INBOX failed: Internal error occurred. Over IMAP the client sees: NO [SERVERBUG] Internal error occurred. It does not recover by itself ----------------------------- "doveadm index" and autoindexing both take the same write path and fail the same way. "doveadm fts rescan" does not help either -- it also cannot open the empty shard. The only repair we found is to delete the folder's fts-flatcurve directory and index the folder again. Impact ------ A quiet mailbox looks fine for as long as nothing is delivered to it, so this can sit unnoticed. We found it because a monitoring mailbox that receives one message per minute started returning SERVERBUG on SEARCH. In our case a maintenance run that called "doveadm fts optimize" per mailbox left 114 mailboxes in this state, which is all of the ones that had any mail. Possible cause (not verified) ----------------------------- In fts_flatcurve_xapian_optimize_box_do() every shard is first opened for writing ("We need to lock all of the mailboxes so nothing changes while we are optimizing"), and the old current.*/index.* directories are then deleted from disk while those write handles are still open. A handle that is closed afterwards appears to recreate its directory without a glass revision file, which matches what is left behind. Also, in the same function the result of fts_flatcurve_xapian_rename_db(backend, dbpath, NULL, error_r) is discarded: rename_db() does not update the path it is given and returns the new path only through newpath_r, which is NULL at this call site. I have not tried to confirm either of these beyond reading the code. Not the same as the known root-owned iamglass report ---------------------------------------------------- There is an older report where "doveadm fts optimize -A" run from cron as root left iamglass owned by root, producing the same error string. That is not this: here the directories are owned by the mail user (dovecot:dovecot, 0700) and the revision file does not exist at all. -- Best regards, Ihor Rusyn Hello, "doveadm fts optimize" leaves the mailbox in a state where no new mail can be indexed. The existing index survives and searches over already-indexed mail keep working, so the damage is invisible until the next message arrives. Version: 2.4.4 (8b687aa65c), packaged in a container image FTS: fts_flatcurve, substring_search = yes, min_term_size default (2) fts_autoindex = yes, fts_search_add_missing = yes Index store: NFSv3 (rsize/wsize 32768, local_lock=none) Mail store: NFSv3, maildir Reproduction ------------ 1) Starting point: a healthy index. # doveadm fts flatcurve check -u [1]user(a)example.com INBOX INBOX guid=88885837bb33136365390000d09efc50 errors=0 shards=1 # ls /indexes/.../.INBOX/fts-flatcurve/*/ current.1786117302443530: flintlock iamglass postlist.glass termlist.glass # doveadm search -u [2]user(a)example.com mailbox INBOX text a | tail -1 88885837bb33136365390000d09efc50 58 2) Optimize. Exits 0, prints nothing. # doveadm fts optimize -u [3]user(a)example.com # echo $? 0 3) State afterwards: the compacted data is in a new index.<n> directory, and the pre-existing current.<id> directory is still present but EMPTY. # ls /indexes/.../.INBOX/fts-flatcurve/*/ current.1786117302443530: index.1464: flintlock iamglass postlist.glass termlist.glass # ls -ld .../fts-flatcurve/current.1786117302443530 drwx------ 2 dovecot dovecot 4096 Aug 7 16:08 .../current.1786117302443530 # doveadm fts flatcurve check -u [4]user(a)example.com INBOX Error: fts-flatcurve(INBOX): Cannot open DB (RO; current.1786117302443530); DatabaseNotFoundError: Couldn't detect type of database Error: fts-flatcurve(INBOX): Check failed; DatabaseOpeningError: Directory does not contain a Xapian database Searching already-indexed mail still works at this point: # doveadm search -u [5]user(a)example.com mailbox INBOX text a | tail -1 88885837bb33136365390000d09efc50 58 4) Deliver one message and index it. This is where it breaks: # printf 'From: a(a)b.c\nSubject: reproprobe\n\nbody\n' \ | doveadm save -u [6]user(a)example.com -m INBOX Info: Mailbox INBOX: save: box=INBOX, uid=59, subject=reproprobe # doveadm index -u [7]user(a)example.com INBOX Error: fts-flatcurve(INBOX): Cannot open DB (RW, current.1786117302443530): .../current.1786117302443530/iamglass: Failed to open glass revision file for reading (No such file or directory) Error: cmd index: Mailbox INBOX: Precache for UID=59 failed: Internal error 5) From now on SEARCH fails too, because fts_search_add_missing makes it index first: # doveadm search -u [8]user(a)example.com mailbox INBOX text reproprobe Error: Mailbox INBOX: indexer failed to index mailbox Error: cmd search: Searching mailbox INBOX failed: Internal error occurred. Over IMAP the client sees: NO [SERVERBUG] Internal error occurred. It does not recover by itself ----------------------------- "doveadm index" and autoindexing both take the same write path and fail the same way. "doveadm fts rescan" does not help either -- it also cannot open the empty shard. The only repair we found is to delete the folder's fts-flatcurve directory and index the folder again. Impact ------ A quiet mailbox looks fine for as long as nothing is delivered to it, so this can sit unnoticed. We found it because a monitoring mailbox that receives one message per minute started returning SERVERBUG on SEARCH. In our case a maintenance run that called "doveadm fts optimize" per mailbox left 114 mailboxes in this state, which is all of the ones that had any mail. Possible cause (not verified) ----------------------------- In fts_flatcurve_xapian_optimize_box_do() every shard is first opened for writing ("We need to lock all of the mailboxes so nothing changes while we are optimizing"), and the old current.*/index.* directories are then deleted from disk while those write handles are still open. A handle that is closed afterwards appears to recreate its directory without a glass revision file, which matches what is left behind. Also, in the same function the result of fts_flatcurve_xapian_rename_db(backend, dbpath, NULL, error_r) is discarded: rename_db() does not update the path it is given and returns the new path only through newpath_r, which is NULL at this call site. I have not tried to confirm either of these beyond reading the code. Not the same as the known root-owned iamglass report ---------------------------------------------------- There is an older report where "doveadm fts optimize -A" run from cron as root left iamglass owned by root, producing the same error string. That is not this: here the directories are owned by the mail user (dovecot:dovecot, 0700) and the revision file does not exist at all. -- Best regards, Ihor Rusyn References Visible links 1. mailto:user@example.com 2. mailto:user@example.com 3. mailto:user@example.com 4. mailto:user@example.com 5. mailto:user@example.com 6. mailto:user@example.com 7. mailto:user@example.com 8. mailto:user@example.com

3 5

Global sieve variables don't work in 2.4.4
by inaugurator＠ya.ru 12 Aug '26

12 Aug '26

Hi everyone! I have created a minimal reproducible setup. First, let's prepare the environment and create the necessary directories: ```shell getent group vmail >/dev/null \ || groupadd -g 500 vmail; \ getent passwd vmail \ | awk -F: '$3 == "500" && $4 == "vmail"' >/dev/null \ || { id vmail >/dev/null 2>&1 && usermod -u 500 -g vmail vmail \ || useradd -u 500 -g vmail -M -s /usr/sbin/nologin vmail; } mkdir -p /mail && chown vmail:vmail -R /mail cat > /mail/global_bug.sieve <<EOF require ["variables", "include"]; global "global_var"; set "global_var" "hello"; EOF cat > /mail/before_bug.sieve <<EOF require ["variables", "include", "vnd.dovecot.debug"]; global "global_var"; include :global :optional "global_bug"; debug_log "\${global_var}"; EOF ``` Then, we configure dovecot.conf: ``` dovecot_config_version = 2.4.4 dovecot_storage_version = 2.4.4 auth_allow_cleartext = yes mail_driver = maildir mail_home = /mail/%{user} mail_inbox_path = .INBOX mail_uid = vmail mail_gid = vmail first_valid_uid = 500 protocols { imap = yes } mail_plugins { imap_sieve = yes } service imap-login { inet_listener imaps { port = 0 } } passdb anypass { driver = static fields { nopassword = yes } } sieve_global_extensions { vnd.dovecot.debug = yes } sieve_script global_bug { driver = file path = /mail/global_bug.sieve type = global } namespace inbox { inbox = yes separator = / mailbox INBOX { auto = subscribe sieve_script before_bug { driver = file path = /mail/before_bug.sieve cause = APPEND type = before } } } ``` If we run an APPEND command to the INBOX directory: ```shell curl -v -u "user1:hello" -k "imap://localhost:143/INBOX" --upload-file /tmp/test.eml ``` the log shows an empty message. ``` imap(user1)<3155434><YNISPNZYHtV/AAAB>: sieve: DEBUG: ``` However, if we change the configuration and the Sieve script by replacing "global" with "personal", I see my hello in debug log: ``` ... sieve_script global_bug { driver = file path = /mail/global_bug.sieve type = personal # !!! } ... require ["variables", "include", "vnd.dovecot.debug"]; global "global_var"; include :personal :optional "global_bug"; # !!! debug_log "\${global_var}"; ... imap(user1)<3155659><iiiIhNZYgpl/AAAB>: sieve: DEBUG: hello ``` Why am I unable to import a global variable into a before script if it is defined in a global script? Thank You in advance!

1 0

Reports to this list held twice for message size — please release them from moderation
by Ihor Rusyn 11 Aug '26

11 Aug '26

Hello, I have twice tried to send a bug report to this list, and both times the message was blocked for exceeding the size limit with error "The message is larger than the 40 KB maximum size" The report is long because it is meant to be actionable: environment details, exact reproduction steps, log excerpts and strace output. That is the material a maintainer asks for first — and it is exactly what the size limit rejects. If this list is where technical reports belong, a limit that blocks them works against its purpose. Could you please release both of my held messages from moderation and reply on the thread? If long reports are not wanted here, I would appreciate a pointer to where they should go instead. The subject of email was «2.4.4 fts-flatcurve: fts optimize silently destroys a folder's index when the index is on NFS», sent on today from this address. -- Best regards, Ihor Ru Hello, I have twice tried to send a bug report to this list, and both times the message was blocked for exceeding the size limit with error "The message is larger than the 40 KB maximum size" The report is long because it is meant to be actionable: environment details, exact reproduction steps, log excerpts and strace output. That is the material a maintainer asks for first -- and it is exactly what the size limit rejects. If this list is where technical reports belong, a limit that blocks them works against its purpose. Could you please release both of my held messages from moderation and reply on the thread? If long reports are not wanted here, I would appreciate a pointer to where they should go instead. The subject of email was <<2.4.4 fts-flatcurve: fts optimize silently destroys a folder's index when the index is on NFS>>, sent on today from this address. -- Best regards, Ihor Ru

3 2

2.4.4 fts-flatcurve: fts optimize silently destroys a folder's index when the index is on NFS
by Ihor Rusyn 11 Aug '26

11 Aug '26

Hello, Summary ======= On Dovecot 2.4.4 with fts_flatcurve and mail_index_path on NFS, "doveadm fts optimize" leaves behind an empty shard directory. That directory remains the shard every subsequent write opens, so from that point on the folder can no longer index new mail. The operation reports success and logs nothing at the time. The damage is permanent and, on a folder that receives no new mail, invisible. Two separate defects are involved, and both are reported below: (1) fts optimize deletes a shard directory whose files it still holds open. On a local filesystem the unlink succeeds invisibly; NFS cannot express "unlink but keep open" and silly-renames instead, so the directory survives. (2) unlink_directory() reports success when rmdir() failed, which is what makes (1) silent rather than loud. Environment =========== Dovecot 2.4.4 CE (8b687aa65c) container image sha256:d6b2f80db2e656a9250583712a82397e2752b35ad7e4e2e71c85fdc0d510137d base Debian 13 trixie fts_flatcurve (Xapian glass) libxapian30 1.4.29-3 (libxapian.so.30.14.1) substring_search = yes, min_term_size = 2 optimize_limit at its default of 10 dovecot_storage_version = 2.3.0 mail_index_path on NFSv3; mail storage on a separate NFS export NFS client is the host, not the container: Debian 12 bookworm, kernel 6.1.0-51-amd64 x86_64 nfs-common 1:2.6.2-4+deb12u1 mount options: rw,noatime,vers=3,rsize=32768,wsize=32768,hard, nordirplus,proto=tcp,timeo=600,retrans=2,sec=sys, local_lock=none Impact ====== An operator does not have to invoke optimize to be exposed: with the default optimize_limit = 10, flatcurve runs it itself once a folder accumulates ten shards. Every active folder therefore reaches this eventually. Already-indexed mail stays searchable, so the folder looks healthy. Only indexing of new mail fails. With fts_search_add_missing = yes, the first SEARCH after new mail arrives attempts to index it, fails, and the client receives NO [SERVERBUG]. I hit this while migrating a host from 2.3 to 2.4: 114 mailboxes were left in this state in a single run, and it surfaced only because a monitoring mailbox began answering SERVERBUG. Defect 1: fts optimize leaves an empty shard directory on NFS ============================================================= Steps to reproduce ------------------ 1. Configure fts_flatcurve with the index path on an NFSv3 mount. 2. Take a mailbox with mail in INBOX (a few hundred messages is enough) and build the index: doveadm index -u <user> INBOX Confirm one shard exists: ls <index>/.INBOX/fts-flatcurve/ -> current.<id> 3. Run: doveadm fts optimize -u <user> INBOX (Alternatively, let it fire on its own by driving the folder past optimize_limit shards.) 4. While it runs, list the shard directory repeatedly - .nfsXXXX entries appear inside current.<id>. 5. After it finishes, note the exit status is 0 and no error was logged. List the directory again: index.<n> was created as expected, and current.<id> is still there, now empty. 6. Deliver one new message, then: doveadm index -u <user> INBOX Expected vs actual ------------------ Expected: after optimize, only the compacted index.<n> remains, and subsequent indexing writes to a new shard. Actual: the stale empty current.<id> remains and is reopened for writing, so step 6 fails: Cannot open DB (RW, current.<id>): .../current.<id>/iamglass: Failed to open glass revision file for reading (No such file or directory) cmd index: Mailbox INBOX: Precache for UID=59 failed and IMAP SEARCH on that folder returns NO [SERVERBUG]. What I observed while diagnosing -------------------------------- strace of "doveadm fts optimize" on NFSv3 shows the old shard's files being unlinked while handles on them are still open, so the directory cannot be removed: unlink(...) = 0 (x7, every file in the shard) rmdir(...) = -1 ENOTEMPTY (Directory not empty) rename(...) = 0 (optimize -> index.N, proceeds regardless) Once the process exits, the NFS client reaps the .nfsXXXX files and the directory is simply empty. What does not help ------------------ NFSv4.1, mounting with nolock, "doveadm fts rescan", "doveadm index", and full re-indexing all take the same write path into the same empty shard and fail identically. Removing the empty directory (or the folder's whole fts-flatcurve tree) and re-indexing restores the folder; setting optimize_limit = 0 avoids the situation but leaves the index spread across many shards. Defect 2: unlink_directory() returns success when rmdir() failed ================================================================ The ENOTEMPTY above never reaches the caller. unlink_directory() in src/lib/unlink-directory.c returns 1 when the final rmdir() fails, the same value it returns on success: if (rmdir(dir) < 0 && errno != ENOENT) { *error_r = t_strdup_printf("rmdir(%s) failed: %m", dir); return errno == ENOENT ? 0 : 1; } The "errno != ENOENT" guard already establishes that errno is not ENOENT, so the ternary always yields 1. Callers cannot distinguish "directory removed" from "directory still there", and the error message it composed is discarded. Any condition that makes rmdir() fail with something other than ENOENT reproduces this, so the return value is wrong independently of NFS - every caller of unlink_directory() is affected. The code path is unchanged in current main. -- Best regards, Ihor Ru Hello, Summary ======= On Dovecot 2.4.4 with fts_flatcurve and mail_index_path on NFS, "doveadm fts optimize" leaves behind an empty shard directory. That directory remains the shard every subsequent write opens, so from that point on the folder can no longer index new mail. The operation reports success and logs nothing at the time. The damage is permanent and, on a folder that receives no new mail, invisible. Two separate defects are involved, and both are reported below: (1) fts optimize deletes a shard directory whose files it still holds open. On a local filesystem the unlink succeeds invisibly; NFS cannot express "unlink but keep open" and silly-renames instead, so the directory survives. (2) unlink_directory() reports success when rmdir() failed, which is what makes (1) silent rather than loud. Environment =========== Dovecot 2.4.4 CE (8b687aa65c) container image sha256:d6b2f80db2e656a9250583712a82397e2752b35ad7e4e2e71c85fdc0d510137d base Debian 13 trixie fts_flatcurve (Xapian glass) libxapian30 1.4.29-3 (libxapian.so.30.14.1) substring_search = yes, min_term_size = 2 optimize_limit at its default of 10 dovecot_storage_version = 2.3.0 mail_index_path on NFSv3; mail storage on a separate NFS export NFS client is the host, not the container: Debian 12 bookworm, kernel 6.1.0-51-amd64 x86_64 nfs-common 1:2.6.2-4+deb12u1 mount options: rw,noatime,vers=3,rsize=32768,wsize=32768,hard, nordirplus,proto=tcp,timeo=600,retrans=2,sec=sys, local_lock=none Impact ====== An operator does not have to invoke optimize to be exposed: with the default optimize_limit = 10, flatcurve runs it itself once a folder accumulates ten shards. Every active folder therefore reaches this eventually. Already-indexed mail stays searchable, so the folder looks healthy. Only indexing of new mail fails. With fts_search_add_missing = yes, the first SEARCH after new mail arrives attempts to index it, fails, and the client receives NO [SERVERBUG]. I hit this while migrating a host from 2.3 to 2.4: 114 mailboxes were left in this state in a single run, and it surfaced only because a monitoring mailbox began answering SERVERBUG. Defect 1: fts optimize leaves an empty shard directory on NFS ============================================================= Steps to reproduce ------------------ 1. Configure fts_flatcurve with the index path on an NFSv3 mount. 2. Take a mailbox with mail in INBOX (a few hundred messages is enough) and build the index: doveadm index -u <user> INBOX Confirm one shard exists: ls <index>/.INBOX/fts-flatcurve/ -> current.<id> 3. Run: doveadm fts optimize -u <user> INBOX (Alternatively, let it fire on its own by driving the folder past optimize_limit shards.) 4. While it runs, list the shard directory repeatedly - .nfsXXXX entries appear inside current.<id>. 5. After it finishes, note the exit status is 0 and no error was logged. List the directory again: index.<n> was created as expected, and current.<id> is still there, now empty. 6. Deliver one new message, then: doveadm index -u <user> INBOX Expected vs actual ------------------ Expected: after optimize, only the compacted index.<n> remains, and subsequent indexing writes to a new shard. Actual: the stale empty current.<id> remains and is reopened for writing, so step 6 fails: Cannot open DB (RW, current.<id>): .../current.<id>/iamglass: Failed to open glass revision file for reading (No such file or directory) cmd index: Mailbox INBOX: Precache for UID=59 failed and IMAP SEARCH on that folder returns NO [SERVERBUG]. What I observed while diagnosing -------------------------------- strace of "doveadm fts optimize" on NFSv3 shows the old shard's files being unlinked while handles on them are still open, so the directory cannot be removed: unlink(...) = 0 (x7, every file in the shard) rmdir(...) = -1 ENOTEMPTY (Directory not empty) rename(...) = 0 (optimize -> index.N, proceeds regardless) Once the process exits, the NFS client reaps the .nfsXXXX files and the directory is simply empty. What does not help ------------------ NFSv4.1, mounting with nolock, "doveadm fts rescan", "doveadm index", and full re-indexing all take the same write path into the same empty shard and fail identically. Removing the empty directory (or the folder's whole fts-flatcurve tree) and re-indexing restores the folder; setting optimize_limit = 0 avoids the situation but leaves the index spread across many shards. Defect 2: unlink_directory() returns success when rmdir() failed ================================================================ The ENOTEMPTY above never reaches the caller. unlink_directory() in src/lib/unlink-directory.c returns 1 when the final rmdir() fails, the same value it returns on success: if (rmdir(dir) < 0 && errno != ENOENT) { *error_r = t_strdup_printf("rmdir(%s) failed: %m", dir); return errno == ENOENT ? 0 : 1; } The "errno != ENOENT" guard already establishes that errno is not ENOENT, so the ternary always yields 1. Callers cannot distinguish "directory removed" from "directory still there", and the error message it composed is discarded. Any condition that makes rmdir() fail with something other than ENOENT reproduces this, so the return value is wrong independently of NFS - every caller of unlink_directory() is affected. The code path is unchanged in current main. -- Best regards, Ihor Ru

1 0

Tribulations trying to pin-down viable sieve script constraints in connection with imap events.
by sjh_dovecot_org＠shic.co.uk 10 Aug '26

10 Aug '26

Hi Dovecot experts... I've recently started what I expected to be a tiny project to migrate from a legacy ad-hoc procmail by embracing Sieve scripting. I have two different frustrations - depending upon which approach I adopt to trigger spam-training actions. *Option 1*: Trigger actions when messages are moved to/from Spam folder. This mostly works... but has has a niggle I can't see how to overcome. I don't have only one 'Spam' folder - I have a hierarchy of folders - with names like "Spam.2026.July" and "Spam.Unsubscribed.2023". My 'learn ham' sieve script uses "imap.mailbox" from environment to determine if the email is being moved out of the Spam hierarchy. This works well. Unfortunately, when triggering 'learn spam' - the value of imap.mailbox is always the destination mailbox 'Spam' - not the source mailbox from which the spammy email has been moved. I can't see any way to trigger a spam reporting action, within Sieve scripts, if (and only if) a message is moved from a folder that does not have a name matching 'Spam.*' into the folder 'Spam'. This restriction is required so that adjustments to filed spam messages do not result in a single email being wrongly presented multiple times for learning as a spam example. This multiple reporting might otherwise happen if, for example, a message was moved between 'Spam' and 'Spam.2025.July" ; "Spam.2025.August" and "Spam.ForReview.2025" several times. Have I overlooked a reliable way to determine the source folder, in a Sieve script, when a message is moved (copied) to the Spam folder? *Option 2*: Trigger actions when the Thunderbird 'Junk' and 'Not Junk' buttons are pressed I accept it would be preferable to avoid depending upon client-specific flags... but I do expect Thunderbird to be the mail client... and marking messages as Junk/Spam in Thunderbird does promise a neat user interface experience. I've read RFC 6785 (also RFC 5228 and RFC 5232). I understand that the "imap.changedflags" environment value is set both when flags have been added and when removed... and that it is expected that hasflag should be used to determine if the imap.changedflag flag was added or removed. This makes perfect sense, in principle... but the actual behaviour I see is peculiar. With a sieve script /var/dovecot-sieve/scripts/flags_changed.sieve (compiled with sievec) like this: require ["environment", "imap4flags", "imapsieve", "variables", "vnd.dovecot.debug"]; if header :matches "subject" "*" { debug_log "FLAG-EMAIL Subject: ${1}"; } if environment :matches "imap.changedflags" "*" { debug_log "FLAG-EMAIL changedflags : ${1}"; } if hasflag :matches "*" { debug_log "FLAG-EMAIL hasflag : ${1}"; } and configuration in /etc/dovecot/conf.d/90-sieve.conf like this (N.B. user sieve script does not interact with flags... type=after and type=before behave similarly): sieve_script flags_changed { type = after cause = flag path = /var/dovecot-sieve/scripts/flags_changed.sieve } I experimented with the "Junk" and "Not Junk" buttons for an email with a distinctive subject. The following nested bullet-list summarises the values I found for various scenarios. My experiment followed this order of scenarios - there should have been no flag changes, for the email in question, between each of the scenarios in the test sequence. * Scenario 1 - Fresh, not-deterimed-to-be-spam email arrives in Inbox... - Sieve script invocation 1 * imap.changedflags => NonJunk * hasflag => NonJunk - Notes: As expected. * Scenario 2 - Same email is marked as read by Thunderbird after it has been displayed onscreen for a while... - Sieve script invocation 2 * imap.changedflags => \Seen * hasflag => \seen - Notes: Flags are case insensitive - but odd lower case distinction. What happened to NonJunk flag from Scenario 1? * Scenario 3 - Junk button pressed to manually mark this email as spam... - Sieve script invocation 3 * imap.changedflags => Junk * hasflag => \seen - Sieve script invocation 4 * imap.changedflags => NonJunk * hasflag => \seen - Notes: Why neither Junk nor NonJunk flags set for hasflag? * Scenario 4 - "Not Junk" button is pressed to manually mark this email as ham... - Sieve script invocation 5 * imap.changedflags => NonJunk * hasflag => \seen - Sieve script invocation 6 * imap.changedflags => Junk * hasflag => \seen - Sieve script invocation 7 * imap.changedflags => \Seen * hasflag => NonJunk Perhaps it is relevant... where I have captured the value from hasflag... that only one token is ever reported - even where I'd expect multiple flags present. I would have expected at least to see both 'Junk' and 'NotJunk' alongside '\seen' for hasflag at appropriate stages in the sequence of scenarios. Is there a problem with how I've used the :match idiom (with "*") intending to capture all the flags associated with the message? (If so, what approach should I have used instead?) Is there a bug which prevents using the hasflag feature to disambiguate between a flag reported for imap.changedflags having been added versus having been removed? If not, where have I gone wrong? Versions... Server-side: Debian Trixie: dovecot-core/stable,stable-security,now 1:2.4.1+dfsg1-6+deb13u6 amd64 [installed] dovecot-imapd/stable,stable-security,now 1:2.4.1+dfsg1-6+deb13u6 amd64 [installed] dovecot-sieve/stable,stable-security,now 1:2.4.1+dfsg1-6+deb13u6 amd64 [installed] Client-side: Windows 11 Thunderbird 140.12.1esr (64-bit) Steve NonJunk \seen 5 Junk \seen 6 \Seen NonJunk 7 Weird... Is NonJunk reported for hasflag because it is now the only flag associated with the message (after \Seen flag removed)?

3 5

[patch] proposed patch for managing timeouts on doveadm iteration with -A
by John Fawcett 10 Aug '26

10 Aug '26

Hi having further investigated the timeouts mentioned in the previous thread, I am submitting this patch as a potential solution. Basically with -A doveadm parameter, the 155 second timeout is applied to the whole end to end transaction. With -F parameter the 155 second timeout is applied to each single user operation. Changing the logic of -A to follow -F looks a bit invasive. So this patch does more or less the same thing, i.e. giving each user operation 155 seconds timeout, done in a different way. Whenever another username is returned, the timeout is reset. PS I got help from Claude on this. best regards John

3 5

Bug report: Pigeonhole Sieve delivery plugin fails to load in Dovecot 2.4 — undefined symbol mail_deliver_ctx_get_log_var_expand_table
by Developer Sub N Grill 08 Aug '26

08 Aug '26

Hello Dovecot Team, I am reporting what appears to be a packaging/ABI issue affecting the Pigeonhole Sieve delivery plugin in Dovecot 2.4. I have performed extensive diagnostics and would appreciate any guidance. Environment Operating System: Debian 13 (Trixie), amd64 Repository: Official Dovecot CE repository (repo.dovecot.org/ce-2.4-latest/debian/trixie) Installed package version:2:2.4.4-5+debian13 Installed packages: dovecot-core dovecot-sieve dovecot-lmtpd dovecot-imapd dovecot-pop3d dovecot-managesieved dovecot-pgsql All packages are installed at the exact same version. Problem Description When enabling the Sieve delivery plugin with: mail_plugins { sieve = yes } Dovecot fails to load the plugin with the following error: sievec: Fatal: Couldn't load required plugin /usr/lib/dovecot/modules/lib90_sieve_plugin.so: dlopen() failed: /usr/lib/dovecot/modules/lib90_sieve_plugin.so: undefined symbol: mail_deliver_ctx_get_log_var_expand_table As a result, message delivery fails entirely whenever the Sieve delivery plugin is enabled. Disabling the plugin restores normal mail delivery. Diagnostics Performed1. Package verification All installed packages are version 2:2.4.4-5+debian13 from the official Dovecot CE repository. There are no mixed package versions. 2. Symbol verification I searched for the symbol mail_deliver_ctx_get_log_var_expand_table throughout the Dovecot libraries. Results: nm -D on libdovecot-lda.so.0.0.0 → symbol not found readelf -sW on libdovecot-lda.so.0.0.0 → symbol not found Recursive search across all .so files under: /usr/lib/dovecot/ /usr/lib/x86_64-linux-gnu/dovecot/ → symbol not exported anywhere The symbol string exists inside both: lib90_sieve_plugin.so libdovecot-lda.so.0.0.0 but only as plain text (visible via grep), never as an exported ELF symbol. 3. Dependency verification ldd on lib90_sieve_plugin.so shows dependencies on: libdovecot-sieve.so.0 libdovecot-storage.so.0 libdovecot.so.0 It does not directly link against libdovecot-lda.so. 4. Standalone sievec Compiling a Sieve script works correctly: sievec /tmp/test.sieve Exit status is 0 as long as no configuration enables the Sieve delivery plugin. The failure only occurs once: mail_plugins { sieve = yes } is present in the configuration. Reproduced on a Clean Debian 13 Installation To rule out configuration or upgrade issues, I reproduced the exact same problem on a completely clean debian:13-slim container. I installed only the Debian stable packages: apt-get install dovecot-core dovecot-sieve dovecot-lmtpd (version 1:2.4.1+dfsg1-6+deb13u6) Using the same production configuration with: mail_plugins { sieve = yes } produces the exact same error: sievec: Fatal: Couldn't load required plugin ... undefined symbol: mail_deliver_ctx_get_log_var_expand_table This indicates the problem is not caused by: mixed package versions an upgrade issue local modifications custom configuration The issue reproduces consistently on both: Dovecot CE 2.4.4 Debian 13's packaged Dovecot 2.4.1 Questions Is mail_deliver_ctx_get_log_var_expand_table a new internal function that should be exported in Dovecot 2.4? Was there an ABI change affecting the Pigeonhole plugin? Is this a known issue in the Dovecot 2.4.x packages? If so, is a fix planned (for example, in 2.4.5)? Is there any recommended workaround for using Sieve during message delivery (e.g. moving spam to the Junk folder) while this issue is unresolved? I am trying to use the new Dovecot 2.4 configuration syntax (sieve_script_type, sieve_script_precedence, and sieve_script_path), since sieve_before and the plugin {} block were removed in 2.4. Thank you for your time. I would be happy to provide additional logs, the complete doveconf -n output, or a core dump if that would be helpful. Best regards, Miguel A Morales Server: Debian 13 (amd64) Dovecot: 2.4.4-5+debian13 Hello Dovecot Team, I am reporting what appears to be a packaging/ABI issue affecting the Pigeonhole Sieve delivery plugin in Dovecot 2.4. I have performed extensive diagnostics and would appreciate any guidance. Environment o Operating System: Debian 13 (Trixie), amd64 o Repository: Official Dovecot CE repository (repo.dovecot.org/ce-2.4-latest/debian/trixie) o Installed package version: 2:2.4.4-5+debian13 Installed packages: o dovecot-core o dovecot-sieve o dovecot-lmtpd o dovecot-imapd o dovecot-pop3d o dovecot-managesieved o dovecot-pgsql All packages are installed at the exact same version. Problem Description When enabling the Sieve delivery plugin with: mail_plugins { sieve = yes } Dovecot fails to load the plugin with the following error: sievec: Fatal: Couldn't load required plugin /usr/lib/dovecot/modules/lib90_sieve_plugin.so: dlopen() failed: /usr/lib/dovecot/modules/lib90_sieve_plugin.so: undefined symbol: mail_deliver_ctx_get_log_var_expand_table As a result, message delivery fails entirely whenever the Sieve delivery plugin is enabled. Disabling the plugin restores normal mail delivery. Diagnostics Performed 1. Package verification All installed packages are version 2:2.4.4-5+debian13 from the official Dovecot CE repository. There are no mixed package versions. 2. Symbol verification I searched for the symbol mail_deliver_ctx_get_log_var_expand_table throughout the Dovecot libraries. Results: o nm -D on libdovecot-lda.so.0.0.0 -> symbol not found o readelf -sW on libdovecot-lda.so.0.0.0 -> symbol not found o Recursive search across all .so files under: o /usr/lib/dovecot/ o /usr/lib/x86_64-linux-gnu/dovecot/ -> symbol not exported anywhere o The symbol string exists inside both: o lib90_sieve_plugin.so o libdovecot-lda.so.0.0.0 but only as plain text (visible via grep), never as an exported ELF symbol. 3. Dependency verification ldd on lib90_sieve_plugin.so shows dependencies on: o libdovecot-sieve.so.0 o libdovecot-storage.so.0 o libdovecot.so.0 It does not directly link against libdovecot-lda.so. 4. Standalone sievec Compiling a Sieve script works correctly: sievec /tmp/test.sieve Exit status is 0 as long as no configuration enables the Sieve delivery plugin. The failure only occurs once: mail_plugins { sieve = yes } is present in the configuration. Reproduced on a Clean Debian 13 Installation To rule out configuration or upgrade issues, I reproduced the exact same problem on a completely clean debian:13-slim container. I installed only the Debian stable packages: apt-get install dovecot-core dovecot-sieve dovecot-lmtpd (version 1:2.4.1+dfsg1-6+deb13u6) Using the same production configuration with: mail_plugins { sieve = yes } produces the exact same error: sievec: Fatal: Couldn't load required plugin ... undefined symbol: mail_deliver_ctx_get_log_var_expand_table This indicates the problem is not caused by: o mixed package versions o an upgrade issue o local modifications o custom configuration The issue reproduces consistently on both: o Dovecot CE 2.4.4 o Debian 13's packaged Dovecot 2.4.1 Questions 1. Is mail_deliver_ctx_get_log_var_expand_table a new internal function that should be exported in Dovecot 2.4? Was there an ABI change affecting the Pigeonhole plugin? 2. Is this a known issue in the Dovecot 2.4.x packages? If so, is a fix planned (for example, in 2.4.5)? 3. Is there any recommended workaround for using Sieve during message delivery (e.g. moving spam to the Junk folder) while this issue is unresolved? I am trying to use the new Dovecot 2.4 configuration syntax (sieve_script_type, sieve_script_precedence, and sieve_script_path), since sieve_before and the plugin {} block were removed in 2.4. Thank you for your time. I would be happy to provide additional logs, the complete doveconf -n output, or a core dump if that would be helpful. Best regards, Miguel A Morales Server: Debian 13 (amd64) Dovecot: 2.4.4-5+debian13

3 2

Can't connect to MySQL without SSL
by dovecot＠fagschlunger.at 08 Aug '26

08 Aug '26

Hi! I try to migrate from dovecot 2.3 to dovecot CE 2.4, but within a docker container. But I always get the following error from Dovecot CE: Error: auth: mysql(host.docker.internal): Connect failed to database (foobar): TLS/SSL error: self-signed certificate in certificate chain - waiting for 1 seconds before retry This is how I configured the mysql connection: mysql host.docker.internal { host = host.docker.internal port = 3306 user = ... password = ... dbname = foobar ssl = no } So MySQL runs on the host machine directly, not in a container - therefore I'm using "host.docker.internal" as an host. But I don't need SQL encryption, since both MySQL and Dovecot run on the same machine. So why the heck does it look like, Dovecot is trying to connect using SSL? Best regards, Billie

5 7

Re: [patch] proposed patch for managing timeouts on doveadm iteration with -A
by MK 07 Aug '26

07 Aug '26

>On 26. Jul 2026, at 0.17, John Fawcett via dovecot <dovecot(a)dovecot.org> wrote: >> >> Hi >> >> having further investigated the timeouts mentioned in the previous thread, I am submitting this patch as a potential solution. >> >> Basically with -A doveadm parameter, the 155 second timeout is applied to the whole end to end transaction. With -F parameter the 155 second >timeout is applied to each single user operation. Changing the logic of -A to follow -F looks a bit invasive. So this patch does more or less the >same thing, i.e. giving each user operation 155 seconds timeout, done in a different way. Whenever another username is returned, the timeout is >reset. >> >> PS I got help from Claude on this. > >Some further improvements via Fable - does this still fix your problems? > >commit 9deefd88b37d4e9bfecdf6bd1c9fa8a3f9d6b100 >Author: Timo Sirainen <timo.sirainen(a)open-xchange.com> >Date: Wed Aug 5 21:33:42 2026 +0000 > > lib-auth-client: Extend LIST request timeout whenever a username is returned > > The auth lookup timeout (155 seconds by default) was applied to the whole > LIST request, i.e. the entire user enumeration. With large installations > listing all users can easily take longer than that, causing e.g. > doveadm -A to fail with "Auth server request timed out". > > Reset the request's timeout whenever another username is returned. This > turns the timeout into a "no progress for N seconds" stall detector, > matching how independent short-lived per-user requests (e.g. as used when > iterating -F user files) never accumulate a single deadline across the > whole run. > > The timeout handling assumes that requests expire in creation order, so > extending is only allowed while the request is alone on its connection. > This is always the case for LIST, which is asserted at init already. Timo can you say in which upcoming version this patch will be included? Oliver

1 0

[BUG] IMAP hibernation fails on FreeBSD 15.x with "Invalid peer_dev_major value"
by Jordan Ostrev 06 Aug '26

06 Aug '26

Hi, IMAP hibernation compiles and attempts to run on FreeBSD 15.x (tested with Dovecot 2.4.4 and FreeBSD 15.1), but fails at runtime with: plain imap-hibernate: Error: conn unix:imap-hibernate (uid=125): Failed to parse client input: Invalid peer_dev_major value: 18446744073709551615 Jun 24 21:27:08 classic dovecot[36888]: imap-hibernate: Error: conn unix:imap-hibernate (uid=125): Failed to parse client input: Invalid peer_dev_major value: 18446744073709551615 Jun 24 21:27:08 classic dovecot[36888]: imap(jordan(a)ostreff.info)<37120><ExaV/gNVAdt/AAAB>: Error: Couldn't hibernate imap client: /var/run/dovecot/imap-hibernate returned failure: Failed to parse client input: Invalid peer_dev_major value: 18446744073709551615 Root cause: In src/imap/imap-client-hibernate.c, the code serializes peer_dev_major and peer_dev_minor using major() and minor() macros: c str_printfa(cmd, "\tpeer_dev_major=%lu\tpeer_dev_minor=%lu\tpeer_ino=%llu", (unsigned long)major(peer_st.st_dev), (unsigned long)minor(peer_st.st_dev), (unsigned long long)peer_st.st_ino); On FreeBSD, major() and minor() can return UINT64_MAX (18446744073709551615) for file descriptors that are not block/character devices (such as sockets on tmpfs/devfs). This value exceeds UINT_MAX and cannot be parsed by str_to_uint() in imap-master-client.c. Historical context: This was discussed on the Dovecot mailing list in July 2019 (thread: "Hibernation errors") but apparently never resolved upstream. The same failure mode persists in Dovecot 2.4.4. Impact: IMAP hibernation is effectively non-functional on FreeBSD 15.x Processes remain running indefinitely instead of being hibernated Documentation note: The Dovecot 2.4 documentation states that hibernation "is not supported on kqueue based systems currently, such as FreeBSD." However, the code still compiles and attempts to run, producing these errors rather than being disabled at compile time or gracefully skipped at runtime. Would it make sense to either: Disable hibernation at compile time for kqueue systems, or Add validation for major()/minor() return values before serialization? Thanks, Jordan Hi, IMAP hibernation compiles and attempts to run on FreeBSD 15.x (tested with Dovecot 2.4.4 and FreeBSD 15.1), but fails at runtime with: plain imap-hibernate: Error: conn unix:imap-hibernate (uid=125): Failed to parse client input: Invalid peer_dev_major value: 18446744073709551615 Jun 24 21:27:08 classic dovecot[36888]: imap-hibernate: Error: conn unix:imap-hibernate (uid=125): Failed to parse client input: Invalid peer_dev_major value: 18446744073709551615 Jun 24 21:27:08 classic dovecot[36888]: imap(jordan(a)ostreff.info)<37120><ExaV/gNVAdt/AAAB>: Error: Couldn't hibernate imap client: /var/run/dovecot/imap-hibernate returned failure: Failed to parse client input: Invalid peer_dev_major value: 18446744073709551615 Root cause: In src/imap/imap-client-hibernate.c, the code serializes peer_dev_major and peer_dev_minor using major() and minor() macros: c str_printfa(cmd, "\tpeer_dev_major=%lu\tpeer_dev_minor=%lu\tpeer_ino=%llu", (unsigned long)major(peer_st.st_dev), (unsigned long)minor(peer_st.st_dev), (unsigned long long)peer_st.st_ino); On FreeBSD, major() and minor() can return UINT64_MAX (18446744073709551615) for file descriptors that are not block/character devices (such as sockets on tmpfs/devfs). This value exceeds UINT_MAX and cannot be parsed by str_to_uint() in imap-master-client.c. Historical context: This was discussed on the Dovecot mailing list in July 2019 (thread: "Hibernation errors") but apparently never resolved upstream. The same failure mode persists in Dovecot 2.4.4. Impact: o IMAP hibernation is effectively non-functional on FreeBSD 15.x o Processes remain running indefinitely instead of being hibernated Documentation note: The Dovecot 2.4 documentation states that hibernation "is not supported on kqueue based systems currently, such as FreeBSD." However, the code still compiles and attempts to run, producing these errors rather than being disabled at compile time or gracefully skipped at runtime. Would it make sense to either: 1. Disable hibernation at compile time for kqueue systems, or 2. Add validation for major()/minor() return values before serialization? Thanks, Jordan

3 11