Replication not working - GUIDs conflict - will be merged later

[Patrick Westenberg]

Replication is fully setup between the two backend servers and worked 
like a charm for years.

My manual replication was just a desperate trial.





Am 31.07.22 um 14:47 schrieb Paul Kudla (SCOM.CA Internet Services Inc.):
> 
> Ok This is speculation but i understand the issue at a programming level
> 
> what needs to be understood is that imap's uids & ugid's are relative to 
> the host server the email is coming from.
> 
> this is generally not an issue with replication on cyrus or dovecot 
> because the server and the replication is being handled by the same 
> server set (ie the same uids & guids etc are generated as things happen)
> 
> example replicated data :
> 
> -rw-------    1 vmail  vmail  uarch  185K Jul 29 09:30 
> 1659101404.M875201P20192.mail19.scom.ca,S=189252,W=192431:2,S
> -rw-------    1 vmail  vmail  uarch  1.5K Jul 29 09:53 
> 1659102818.M268117P41331.mail18.scom.ca,S=1583,W=1639:2,S
> -rw-------    1 vmail  vmail  uarch  1.0M Jul 29 12:52 
> 1659113530.M841469P58214.mail18.scom.ca,S=1095861,W=1113817:2,S
> -rw-------    1 vmail  vmail  uarch  210K Jul 29 13:15 
> 1659114913.M958008P31982.mail19.scom.ca,S=215405,W=219216:2,S
> 
> you will note the originating server is in the mail file name (mail19 & 
> mail18 in my case)
> 
> this is how dovecot sorts out the uids etc on the fly. (i think)
> 
> If i have read this correctly you are trying to sync to an external imap 
> server that carries its own uids guids etc which will be different.
> 
> where you are saying that you are using imap sync i assume you are using 
> the unix version
> 
> # imapsync
> Name:
> 
>   imapsync - Email IMAP tool for syncing, copying, migrating and archiving
>   email mailboxes between two imap servers, one way, and without 
> duplicates.
> 
> Version:
> 
>   This documentation refers to Imapsync $Revision: 1.977 $
> 
> 
> if so look at the
> 
> --useuid            :
> 
> 
> Use UIDs instead of headers as a criterion to recognize
> messages. Option --usecache is then implied unless
>                          --nousecache is used.
> 
> 
> and the --logfile (ie run a logging file when connecting the the 
> external account), it might help with any errors being generated (run 
> imapsync in debug mode to get full detail)
> 
> basically using useuid deals with sometimes getting a different uid back 
> from the origional server
> 
> i go through this issue more with pop3 as it returns the id list 
> starting at uid 1 (for example) instead of the actual uid against the 
> email on the server.
> 
> uids will force a proper sync (imap or pop3) because the uid on the 
> server will always return the same uid for that email message and 
> increments forward inside the account.
> 
> if so then imap sync should be sorting this out when syncing the imap 
> accounts ? (ie creating new usid guids etc)
> 
> so assuming the above is happening the next question is are you using 
> replication that is fully setup between the two servers or are you doing 
> manual replication (ie running the doveadm command to do the sync?)
> 
> (you mentioned using the backup command which would kinda work but full 
> replication does the changes on the fly and should work)
> 
> if you are running manual replication you should consider going to the 
> live replication, it will sort out stuff as the imap folders sync etc. 
> (or it should)
> 
> The next thing to consider is there were some issues that were fixed in 
> 2.3.19 replication, are you running the same dovecot versions on both 
> servers ?
> 
> I do a ton of emails, reporting etc and find that replication works well 
> on dovecot 2.3.19 bewteen both of my mail servers. ie it does not matter 
> which one receives the email it gets sorted out. If there is an error 
> the replication will sort it out on the next sync run through the 
> replication process running in the background.
> 
> you can set all of the retries etc for replication in the config files.
> 
> the merged later is probably indicating that dovecot will sort stuff out 
> in the background (ie a reindex etc) but that is putting extra stress on 
> the server(s), i used to get the merge or duplicate uids, guids on cyrus 
> and it would try to sort it out on the fly. this would occur when one 
> replicated server was offline and i was forcing a sync update after 
> bringing it back online, this was the case because both servers had 
> received emails into the same account from seperate sources thus the 
> same uid was set for two different messages on each server. (fyi)
> 
> with syrus a rebuild was the only was to sort this out
> 
> dovecot seems way more resiliant in this department.
> 
> 
> again full replication setup would sort these issue out i expect as each 
> server would handle stuff as it happens and adjust uid,guids accordingly.
> 
> 
> 
> 
> 
> 
> 
> Happy Sunday !!!
> Thanks - paul
> 
> Paul Kudla
> 
> 
> Scom.ca Internet Services <http://www.scom.ca>
> 004-1009 Byron Street South
> Whitby, Ontario - Canada
> L1N 4S3
> 
> Toronto 416.642.7266
> Main 1.866.411.7266
> Fax 1.888.892.7266
> Email paul at scom.ca
> 
> On 7/31/2022 8:16 AM, Patrick Westenberg wrote:
>>
>> Hi everyone,
>>
>> I have a weird problem with replication that I'm unable to solve.
>>
>> A new account is sychronized from an external provider via imapsync.
>> The mails end up on my backend1. I see that the folder structure is
>> immediately replicated to backend2.
>>
>> However, a lot of mails are missing and "doveadm replicator status" also
>> states that something failed:
>>
>>
>>                                  priority fast sync full sync success 
>> sync
>> failed
>> mail at example.com none     00:02:36  00:39:47  -            y
>>
>> There are no error log entries regarding this user at all.
>>
>> After some research I tried to start the replication manually with
>> "doveadm -D backup -u mail at example.com -d tcp:x.y.z.11" and finally
>> I got an error message:
>>
>> Jul 31 13:55:37
>> doveadm(mail at example.com)<11341><rm1YCblt5mJNLAAAJFg/Tg>: Debug: brain
>> M: Mailbox INBOX: local=e74590221b6ce6620d29000024583f4e/0/1,
>> remote=20a9ce2b1c6ce662244e0000baba0ddd/0/1: GUIDs conflict - will be
>> merged later
>>
>> As a matter of fact, the mails in the inbox are the ones that are
>> missing on backend2.
>>
>>
>> I always end up with this error, no matter what I do:
>> - I completely removed the folder structure on backend2.
>> - I removed all database entries.
>> - I removed this user from the replication, ran imapsync again followed
>> by "doveadm -D backup ..."
>>
>> I also removed fts-flatcurve to rule this out as a cause of error.
>>
>> I have no idea what to do. This configuration worked for months.
>> I got this error with 2.3.17.1 and 2.3.19.1
>>
>> Any hints would be highly appreciated.
>>
>>
>> Regards
>> Patrick
>>
>>
>>
>>
>>
>> # 2.3.19.1 (9b53102964): /usr/local/etc/dovecot/dovecot.conf
>> # Pigeonhole version 0.5.19 (4eae2f79)
>> auth_master_user_separator = *
>> auth_mechanisms = plain login
>> doveadm_password = # hidden, use -P to show it
>> doveadm_port = 12345
>> listen = x.y.z.12
>> lmtp_save_to_detail_mailbox = yes
>> log_debug = category=fts-flatcurve
>> log_path = /var/log/dovecot.log
>> mail_debug = yes
>> mail_home = /srv/mail/%Ld/%Ln
>> mail_location = maildir:~/Maildir
>> mail_plugins = quota notify replication
>> managesieve_notify_capability = mailto
>> managesieve_sieve_capability = fileinto reject envelope
>> encoded-character vacation subaddress comparator-i;ascii-numeric
>> relational regex imap4flags copy include variables body enotify
>> environment mailbox date index ihave duplicate mime foreverypart
>> extracttext editheader
>> metric imapc_traffic {
>>    fields = bytes_in bytes_out
>>    filter = event=imap_command_finished
>>    group_by = user
>> }
>> namespace inbox {
>>    inbox = yes
>>    location =
>>    mailbox Drafts {
>>      auto = subscribe
>>      special_use = \Drafts
>>    }
>>    mailbox Junk {
>>      special_use = \Junk
>>    }
>>    mailbox Sent {
>>      auto = subscribe
>>      special_use = \Sent
>>    }
>>    mailbox "Sent Messages" {
>>      special_use = \Sent
>>    }
>>    mailbox Spamverdacht {
>>      auto = subscribe
>>      special_use = \Junk
>>    }
>>    mailbox Trash {
>>      auto = subscribe
>>      special_use = \Trash
>>    }
>>    prefix = INBOX/
>>    separator = /
>>    type = private
>> }
>> passdb {
>>    args = /usr/local/etc/dovecot/master-user
>>    driver = passwd-file
>>    master = yes
>>    pass = yes
>> }
>> passdb {
>>    args = /usr/local/etc/dovecot/dovecot-sql.conf.ext
>>    driver = sql
>> }
>> plugin {
>>    mail_replica = tcp:x.y.z.11
>>    quota = count:User quota
>>    quota_grace = 25M
>>    quota_rule2 = INBOX/Trash:storage=+100M
>>    quota_status_nouser = DUNNO
>>    quota_status_success = DUNNO
>>    quota_vsizes = yes
>>    quota_warning = storage=100%% quota-full %u
>>    quota_warning2 = storage=95%% quota-warning 95 %u
>>    quota_warning3 = storage=80%% quota-warning 80 %u
>>    quota_warning4 = -storage=100%% quota-ok %u
>>    sieve = ~/.dovecot.sieve
>>    sieve_after = /usr/local/etc/dovecot/sieve/sieve_after.sieve
>>    sieve_default = /usr/local/etc/dovecot/sieve/default.sieve
>>    sieve_dir = ~/sieve
>>    sieve_extensions = +editheader
>> }
>> protocols = imap pop3 lmtp sieve
>> service aggregator {
>>    fifo_listener replication-notify-fifo {
>>      user = vmail
>>    }
>>    unix_listener replication-notify {
>>      user = vmail
>>    }
>> }
>> service auth {
>>    unix_listener auth-userdb {
>>      group = vmail
>>      mode = 0666
>>      user = vmail
>>    }
>> }
>> service config {
>>    unix_listener config {
>>      mode = 0600
>>      user = vmail
>>    }
>> }
>> service doveadm {
>>    inet_listener {
>>      port = 12345
>>    }
>> }
>> service imap-login {
>>    process_min_avail = 2
>>    service_count = 0
>> }
>> service lmtp {
>>    executable = lmtp -L
>>    inet_listener lmtp {
>>      address = x.y.z.12
>>      port = 24
>>    }
>>    process_min_avail = 20
>> }
>> service managesieve-login {
>>    inet_listener sieve {
>>      port = 4190
>>    }
>> }
>> service quota-full {
>>    executable = script /usr/local/etc/dovecot/quota_full.sh
>>    unix_listener quota-full {
>>      user = vmail
>>    }
>>    user = root
>> }
>> service quota-ok {
>>    executable = script /usr/local/etc/dovecot/quota_ok.sh
>>    unix_listener quota-ok {
>>      user = vmail
>>    }
>>    user = root
>> }
>> service quota-status {
>>    client_limit = 1
>>    executable = quota-status -p postfix
>>    inet_listener {
>>      port = 12340
>>    }
>> }
>> service quota-warning {
>>    executable = script /usr/local/etc/dovecot/quota_warning.sh
>>    unix_listener quota-warning {
>>      user = vmail
>>    }
>>    user = root
>> }
>> service replicator {
>>    process_min_avail = 1
>>    unix_listener replicator-doveadm {
>>      mode = 0600
>>      user = vmail
>>    }
>> }
>> ssl = required
>> ssl_cert = </etc/ssl/certs/bundle.pem
>> ssl_client_ca_dir = /etc/ssl/certs
>> ssl_dh = # hidden, use -P to show it
>> ssl_key = # hidden, use -P to show it
>> userdb {
>>    driver = prefetch
>> }
>> userdb {
>>    args = /usr/local/etc/dovecot/dovecot-sql.conf.ext
>>    driver = sql
>> }
>> verbose_proctitle = yes
>> protocol lmtp {
>>    info_log_path = /var/log/dovecot-lmtp.log
>>    mail_plugins = quota notify replication sieve
>> }
>> protocol imap {
>>    mail_max_userip_connections = 40
>>    mail_plugins = quota notify replication imap_quota
>> }
>>

-- 
Westenberg + Kueppers GbR              Spanische Schanzen 37
---- Buero Koeln ----                  47495 Rheinberg
pwestenberg at wk-serv.de                 Tel.: +49 (0)2843 90369-06
http://www.wk-serv.de                  Fax : +49 (0)2843 90369-07
Gesellschafter: Sebastian Kueppers & Patrick Westenberg