Replication not working - GUIDs conflict - will be merged later

[Sebastian Marske]

On 8/1/22 11:15, Patrick Westenberg wrote:
> Very interesting new insights:
> 
> When I use imapsync and let it synchronize mails from INBOX to
> INBOX/testfolder, the automatic replication works fine.
> All mails are synchronized between my two backends.
> 
> 
> When I move the mails to the INBOX (doveadm move -u mail at example.com
> INBOX mailbox INBOX/testfolder all), these mails are lost on the
> replica! They are neither in INBOX, nor in INBOX/testfolder
> 
> Regards
> Patrick

Hi,

every now and then I have the same problem on our servers. Currently,
I'm running Dovecot 2.3.19.1 as well, but I upgraded directly from
2.3.16 due to other issues with the versions in between.

Last time I observed a de-sync due to a GUID change, it appeared like
the user had moved a folder around in their mailbox. And indeed, the
output of 'doveadm mailbox status -u someuser guid '*' listed different
GUIDs. Dovecot actually logged some errors for this case:

Dovecot log from replica1:
Jul 27 12:06:08 replica1 dovecot[3431]:
doveadm(someuser)<10206><s1aFMQ8O4WLeJwAAyQQkNg>: Error: Duplicate
mailbox GUID 78c9dc2c0c0ee162c10800000ca22142 for mailboxes
path/to/folder and path/to/folder-temp-1 - giving a new GUID
b0053e390f0ee162de270000c9042436 to path/to/folder
Jul 27 12:06:08 replica1 dovecot[3431]:
doveadm(someuser)<10208><fgWCCRAO4WLgJwAAyQQkNg>: Error: Duplicate
mailbox GUID 78c9dc2c0c0ee162c10800000ca22142 for mailboxes
path/to/folder and path/to/folder-temp-1 - giving a new GUID
5823fe0d100ee162e0270000c9042436 to path/to/folder

Dovecot log from replica2:
Jul 27 12:06:04 replica2 dovecot[47018]:
doveadm(someuser)<2239><TD9EDAwO4WK/CAAADKIhQg>: Warning: Failed to do
incremental sync for mailbox path/to/folder, retry with a full sync
(uidnext 1 < 13)
Jul 27 12:06:04 replica2 dovecot[47018]:
doveadm(someuser)<2241><ix0uKQwO4WLBCAAADKIhQg>: Error: Duplicate
mailbox GUID 0ccaab01079031620e1e00000ca22142 for mailboxes
path/to/folder and some/folder - giving a new GUID
78c9dc2c0c0ee162c10800000ca22142 to path/to/folder

At that time, only replica2 was accepting imap connections.
In this particular case, Dovecot eventually managed to get things back
in sync after way over 24h, but I also had users out of sync for
multiple days.
Running 'doveadm -Dv sync -u someuser -d' manually gave me the same
error message, but didn't change anything.

Other things I've observed:
* it's not limited to a fixed set of users (unlike the
too-many-folders-thing with Dovecot 2.3.1[78])
* it's not limited to newly created users, but also affects users, that
have been in sync for months/years
* it's not limited to mailboxes with lots of imap operations going on
* it's not specific to very large or very small mailboxes (although I've
only seen it for folders with a small number of mails in them)
* in most cases, Dovecot doesn't log any errors
* it does seem to be related to something an imap client can trigger

As of now, my "fix" is to
* make sure that one of the replicas has all mails for that folder
(we're using maildir, so I can just rsync the individual mails/folders)
* create a full copy of the complete folder as backup
* remove the user from replication
* 'doveadm mailbox delete' the folder on one replica to get
rid of one of the conflicting guids (one time, Dovecot replicated the
deletion despite removing the user from replication, so the backup came
in handy)
* alternatively, you might be fine by deleting the folder's index files
* add the user back to replication
* let dsync replicate the user
-> fixed

It's not a very convenient way to resolve this, but maybe it helps. Any
better solutions are greatly appreciated!


Best
Sebastian