Hi Dovecot developers, hi list, I think I found a bug in the new doveadm protocol using OSTREAM_MULTIPLEX_FORMAT_STREAM enabled in 2.4.5 (commit 16226f8021068cda43023982eca45f859164a2d9). For very specific mailboxes (so far I only found 2 users in our setup), the following command runs into a deadlock: doveadm sync -l 30 -u USERNAME tcp:OTHER_SERVER If I start the command with debugging enabled (-D), the command successfully finishes after less than a second (the mailbox is rather small). This looks to me like a race condition which is not triggered when enabling debug output (some call this a heisenbug). Both processes (doveadm and doveadm-server on the OTHER_SERVER) are hanging in epoll_wait(). To create a backtrace, I have sent a SIGSEGV to the processes, once on the source server, once on the other. This resulted in a coredump and termination of the other process due to EOF. Here are the backtraces of those coredumps: PID: 3975605 (doveadm) UID: REDACTED (REDACTED) GID: REDACTED (REDACTED) Signal: 11 (SEGV) Timestamp: Mon 2026-09-21 10:54:13 CEST (2h 19min ago) Command Line: $'doveadm [REDACTED recv_mailbox_tree_deletes]' Executable: /usr/bin/doveadm Control Group: /user.slice/user-0.slice/session-87057.scope Unit: session-87057.scope Slice: user-0.slice Session: 87057 Owner UID: 0 (root) Boot ID: e3d3c33d72164c98b58575bc3ad98ff5 Machine ID: ae0fa1dbcc7d4f4d9bb6ad7fcd1e072f Hostname: typhon Storage: /var/lib/systemd/coredump/core.doveadm.REDACTED.e3d3c33d72164c98b58575bc3ad98ff5.3975605.1789980853000000.zst (present) Size on Disk: 518.3K Message: Process 3975605 (doveadm) of user REDACTED dumped core. Module libuuid.so.1 from deb util-linux-2.41.5-0+deb13u1.amd64 Module libgcc_s.so.1 from deb gcc-14-14.2.0-19.amd64 Module libstdc++.so.6 from deb gcc-14-14.2.0-19.amd64 Module libzstd.so.1 from deb libzstd-1.5.7+dfsg-1.amd64 Stack trace of thread 3975605: #0 0x00007ff829f4f3a7 n/a (libc.so.6 + 0x8f3a7) #1 0x00007ff829f4f3cd n/a (libc.so.6 + 0x8f3cd) #2 0x00007ff829fd07ed epoll_wait (libc.so.6 + 0x1107ed) #3 0x00007ff82a370a85 io_loop_handler_run_internal (libdovecot.so.0 + 0x170a85) #4 0x00007ff82a370be4 io_loop_handler_run (libdovecot.so.0 + 0x170be4) #5 0x00007ff82a370de8 io_loop_run (libdovecot.so.0 + 0x170de8) #6 0x00005f60bd513b3d cmd_dsync_run_remote (/usr/bin/doveadm + 0x3ab3d) #7 0x00005f60bd51484b doveadm_mail_next_user (/usr/bin/doveadm + 0x3b84b) #8 0x00005f60bd516657 doveadm_mail_cmd_exec (/usr/bin/doveadm + 0x3d657) #9 0x00005f60bd52af99 doveadm_cmdline_run (/usr/bin/doveadm + 0x51f99) #10 0x00005f60bd52b122 doveadm_cmdline_try_run (/usr/bin/doveadm + 0x52122) #11 0x00005f60bd50317b main (/usr/bin/doveadm + 0x2a17b) #12 0x00007ff829ee9ca8 n/a (libc.so.6 + 0x29ca8) #13 0x00007ff829ee9d65 __libc_start_main (libc.so.6 + 0x29d65) #14 0x00005f60bd5034b1 _start (/usr/bin/doveadm + 0x2a4b1) ELF object binary architecture: AMD x86-64 PID: 1992428 (doveadm-server) UID: 0 (root) GID: REDACTED (REDACTED) Signal: 11 (SEGV) Timestamp: Mon 2026-09-21 10:58:35 CEST (2h 18min ago) Command Line: $'dovecot/doveadm-server [REDACTED REDACTED slave_recv_mailbox]' Executable: /usr/lib/dovecot/doveadm-server Control Group: /system.slice/dovecot.service Unit: dovecot.service Slice: system.slice Boot ID: fb737a07888046ab9be0656dbe6cb21f Machine ID: 2d9ee85474064e06aa796c67b8f11dd9 Hostname: tyche Storage: /var/lib/systemd/coredump/core.doveadm-server.0.fb737a07888046ab9be0656dbe6cb21f.1992428.1789981115000000.zst (present) Size on Disk: 497.5K Message: Process 1992428 (doveadm-server) of user 0 dumped core. Module libuuid.so.1 from deb util-linux-2.41.5-0+deb13u1.amd64 Module libgcc_s.so.1 from deb gcc-14-14.2.0-19.amd64 Module libstdc++.so.6 from deb gcc-14-14.2.0-19.amd64 Module libzstd.so.1 from deb libzstd-1.5.7+dfsg-1.amd64 Stack trace of thread 1992428: #0 0x000079838fdab3a7 n/a (libc.so.6 + 0x8f3a7) #1 0x000079838fdab3cd n/a (libc.so.6 + 0x8f3cd) #2 0x000079838fe2c7ed epoll_wait (libc.so.6 + 0x1107ed) #3 0x0000798390170a85 io_loop_handler_run_internal (libdovecot.so.0 + 0x170a85) #4 0x0000798390170be4 io_loop_handler_run (libdovecot.so.0 + 0x170be4) #5 0x0000798390170de8 io_loop_run (libdovecot.so.0 + 0x170de8) #6 0x000059f13b7bdcce cmd_dsync_server_run (/usr/lib/dovecot/doveadm-server + 0x39cce) #7 0x000059f13b7bf74b doveadm_mail_next_user (/usr/lib/dovecot/doveadm-server + 0x3b74b) #8 0x000059f13b7c1557 doveadm_mail_cmd_exec (/usr/lib/dovecot/doveadm-server + 0x3d557) #9 0x000059f13b7d55b9 doveadm_cmdline_run (/usr/lib/dovecot/doveadm-server + 0x515b9) #10 0x000059f13b7d8232 doveadm_cmd_server_run_ver2 (/usr/lib/dovecot/doveadm-server + 0x54232) #11 0x000079839016ecdb io_loop_call_io (libdovecot.so.0 + 0x16ecdb) #12 0x0000798390170b1b io_loop_handler_run_internal (libdovecot.so.0 + 0x170b1b) #13 0x0000798390170be4 io_loop_handler_run (libdovecot.so.0 + 0x170be4) #14 0x0000798390170de8 io_loop_run (libdovecot.so.0 + 0x170de8) #15 0x00007983900b84be master_service_run (libdovecot.so.0 + 0xb84be) #16 0x000059f13b7ae307 main (/usr/lib/dovecot/doveadm-server + 0x2a307) #17 0x000079838fd45ca8 n/a (libc.so.6 + 0x29ca8) #18 0x000079838fd45d65 __libc_start_main (libc.so.6 + 0x29d65) #19 0x000059f13b7ae3b1 _start (/usr/lib/dovecot/doveadm-server + 0x2a3b1) ELF object binary architecture: AMD x86-64 The doveconf -n output is attached. As a temp. workaround, I have disabled multiplex stream for the doveadm protocol by: --- dovecot-2.4.5.orig/src/lib-doveadm/doveadm-protocol.h +++ dovecot-2.4.5/src/lib-doveadm/doveadm-protocol.h @@ -16,7 +16,7 @@ #define DOVEADM_PROTOCOL_MIN_VERSION_EXTRA_FIELDS 3 /* Use the (non-deprecated) STREAM multiplex format instead of PACKET for the server -> client output. */ -#define DOVEADM_PROTOCOL_MIN_VERSION_MULTIPLEX_STREAM 4 +#define DOVEADM_PROTOCOL_MIN_VERSION_MULTIPLEX_STREAM 5 #define DOVEADM_EX_NOTFOUND EX_NOHOST #define DOVEADM_EX_NOTPOSSIBLE EX_DATAERR This allows us to continue the service for our users but obviously is not a solution. Unfortunately I cannot provide the data of the affected mailboxes to allow reproducing the bug. But I have saved them on a development system and can reproduce it locally. I'm willing to test suggested patches and provide more data (unless restricted by privacy issues). Can anyone help me debugging this issue? Best regards, -- Patrick Cernko <pcernko@mpi-klsb.mpg.de> +49 681 9325 5815 Joint Scientific IT and Technical Service Max-Planck-Institute für Informatik & Softwaresysteme