Following up to my original mail:
On 18/02/2022 3:59 pm, Reuben Farrelly wrote:
Hi,
I've recently migrated my two VMs across from Linode (who use KVM) onto a local VPS service (which also uses KVM). Since doing so I have started to see some strange problems with Dovecot relating to indexes and replication.
I have copied the configuration files across from old host to new host. The kernel is the same - as this is Gentoo everything was rebuilt and installed from fresh, but with the same options (use flags). Even the Linux kernel is the same version with the exact same options (as is Dovecot). The filesystem is the same EXT4 with the same options too.
No one responded from here (is anyone helping on this list anymore?) but after many hours I found out the problem was to do with replication on the far end host, and not anything to do with either the new VPS or the existing dovecot or linux config.
It turns out that if there is an existing Maildir/ in the user's directory on the remote replica, the initial sync from the master fails. It may fail early on in the sync, or at the end of the initial replication but either way it fails and the user ends up with a mailbox in a half sync'd state. Even if the remote Maildir is completely empty as mine were, it fails - it is the mere presence of the Maildir/ directory on the remote breaks the sync. Typically new users have a new and empty Maildir (copied from /etc/skel) so it fails for them by default.
Once I deleted the Maildir/ from the remote user's home directory and the entire contents of a half replica, then dovecot created a new Maildir and everything was able to sync through on all users to completion.
To reproduce this: create a new user with an empty (Maildir/new Maildir/cur and Maildir/tmp) and then trigger the sync with debug manually: doveadm -v -D sync -u username -f tcp:imap2.reub.net:4814
Here - with a completely empty and brand new Maildir/ on both master and remote replica we can see it already fails:
tornado ~ # doveadm -v -D sync -u testuser -f tcp:imap2.reub.net:4814
Debug: Loading modules from directory: /usr/lib64/dovecot/doveadm
Debug: Skipping module doveadm_acl_plugin, because dlopen() failed:
/usr/lib64/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined
symbol: acl_user_module (this is usually intentional, so just ignore
this message)
Debug: Skipping module doveadm_quota_plugin, because dlopen() failed:
/usr/lib64/dovecot/doveadm/lib10_doveadm_quota_plugin.so: undefined
symbol: quota_user_module (this is usually intentional, so just ignore
this message)
Debug: Module loaded:
/usr/lib64/dovecot/doveadm/lib10_doveadm_sieve_plugin.so
Debug: Skipping module doveadm_fts_plugin, because dlopen() failed:
/usr/lib64/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined
symbol: fts_user_get_language_list (this is usually intentional, so just
ignore this message)
Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen()
failed: /usr/lib64/dovecot/doveadm/libdoveadm_mail_crypt_plugin.so:
undefined symbol: mail_crypt_box_get_pvt_digests (this is usually
intentional, so just ignore this message)
Feb 21 16:31:51 Debug: Loading modules from directory: /usr/lib64/dovecot
Feb 21 16:31:51 Debug: Module loaded:
/usr/lib64/dovecot/lib15_notify_plugin.so
Feb 21 16:31:51 Debug: Module loaded:
/usr/lib64/dovecot/lib20_replication_plugin.so
Feb 21 16:31:51 Debug: Loading modules from directory:
/usr/lib64/dovecot/doveadm
Feb 21 16:31:51 Debug: Skipping module doveadm_acl_plugin, because
dlopen() failed: /usr/lib64/dovecot/doveadm/lib10_doveadm_acl_plugin.so:
undefined symbol: acl_user_module (this is usually intentional, so just
ignore this message)
Feb 21 16:31:51 Debug: Skipping module doveadm_quota_plugin, because
dlopen() failed:
/usr/lib64/dovecot/doveadm/lib10_doveadm_quota_plugin.so: undefined
symbol: quota_user_module (this is usually intentional, so just ignore
this message)
Feb 21 16:31:51 Debug: Skipping module doveadm_fts_plugin, because
dlopen() failed: /usr/lib64/dovecot/doveadm/lib20_doveadm_fts_plugin.so:
undefined symbol: fts_user_get_language_list (this is usually
intentional, so just ignore this message)
Feb 21 16:31:51 Debug: Skipping module doveadm_mail_crypt_plugin,
because dlopen() failed:
/usr/lib64/dovecot/doveadm/libdoveadm_mail_crypt_plugin.so: undefined
symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so
just ignore this message)
Feb 21 16:31:51 doveadm(testuser)<11258><>: Debug: auth-master: userdb
lookup(testuser): Started userdb lookup
Feb 21 16:31:51 doveadm(testuser)<11258><>: Debug: auth-master: conn
unix:/run/dovecot/auth-userdb: Connecting
Feb 21 16:31:51 doveadm(testuser)<11258><>: Debug: auth-master: conn
unix:/run/dovecot/auth-userdb (pid=3070,uid=0): Client connected (fd=9)
Feb 21 16:31:51 doveadm(testuser)<11258><>: Debug: auth-master: userdb
lookup(testuser): auth USER input: testuser system_groups_user=testuser
uid=1006 gid=1006 home=/home/testuser
Feb 21 16:31:51 doveadm(testuser)<11258><>: Debug: auth-master: userdb
lookup(testuser): Finished userdb lookup (username=testuser
system_groups_user=testuser uid=1006 gid=1006 home=/home/testuser)
Feb 21 16:31:51 doveadm(testuser)<11258>
tornado ~ #
That message about the Mailbox changing causing a desync is what screws up the client (in this case testuser) if they happen to be connected to the mater at the time of the attempt. That is why client access was impacted as soon as remote replication was enabled.
If this is run via normal dovecot sync (and not triggered) it results in severe performance problems and disconnections for any clients connected to the master replica pretty much anytime they try and move/delete anything.
I saw a couple of old posts over the years where other people have found the same thing so I know I'm not the first person to run into this problem and I suspect there are a lot more who have run into this problem but not realised what was going on.
What do I need to do to get this fixed properly? Bailing out mid way through replicating and disconnecting clients (and not self-healing) surely can't be the intended behaviour so I'd consider this to be a pretty nasty bug.
Reuben