Replication not working - GUIDs conflict - will be merged later
Patrick Westenberg
pw at wk-serv.de
Sun Jul 31 20:47:00 UTC 2022
Replication is fully setup between the two backend servers and worked
like a charm for years.
My manual replication was just a desperate trial.
Am 31.07.22 um 14:47 schrieb Paul Kudla (SCOM.CA Internet Services Inc.):
>
> Ok This is speculation but i understand the issue at a programming level
>
> what needs to be understood is that imap's uids & ugid's are relative to
> the host server the email is coming from.
>
> this is generally not an issue with replication on cyrus or dovecot
> because the server and the replication is being handled by the same
> server set (ie the same uids & guids etc are generated as things happen)
>
> example replicated data :
>
> -rw------- 1 vmail vmail uarch 185K Jul 29 09:30
> 1659101404.M875201P20192.mail19.scom.ca,S=189252,W=192431:2,S
> -rw------- 1 vmail vmail uarch 1.5K Jul 29 09:53
> 1659102818.M268117P41331.mail18.scom.ca,S=1583,W=1639:2,S
> -rw------- 1 vmail vmail uarch 1.0M Jul 29 12:52
> 1659113530.M841469P58214.mail18.scom.ca,S=1095861,W=1113817:2,S
> -rw------- 1 vmail vmail uarch 210K Jul 29 13:15
> 1659114913.M958008P31982.mail19.scom.ca,S=215405,W=219216:2,S
>
> you will note the originating server is in the mail file name (mail19 &
> mail18 in my case)
>
> this is how dovecot sorts out the uids etc on the fly. (i think)
>
> If i have read this correctly you are trying to sync to an external imap
> server that carries its own uids guids etc which will be different.
>
> where you are saying that you are using imap sync i assume you are using
> the unix version
>
> # imapsync
> Name:
>
> imapsync - Email IMAP tool for syncing, copying, migrating and archiving
> email mailboxes between two imap servers, one way, and without
> duplicates.
>
> Version:
>
> This documentation refers to Imapsync $Revision: 1.977 $
>
>
> if so look at the
>
> --useuid :
>
>
> Use UIDs instead of headers as a criterion to recognize
> messages. Option --usecache is then implied unless
> --nousecache is used.
>
>
> and the --logfile (ie run a logging file when connecting the the
> external account), it might help with any errors being generated (run
> imapsync in debug mode to get full detail)
>
> basically using useuid deals with sometimes getting a different uid back
> from the origional server
>
> i go through this issue more with pop3 as it returns the id list
> starting at uid 1 (for example) instead of the actual uid against the
> email on the server.
>
> uids will force a proper sync (imap or pop3) because the uid on the
> server will always return the same uid for that email message and
> increments forward inside the account.
>
> if so then imap sync should be sorting this out when syncing the imap
> accounts ? (ie creating new usid guids etc)
>
> so assuming the above is happening the next question is are you using
> replication that is fully setup between the two servers or are you doing
> manual replication (ie running the doveadm command to do the sync?)
>
> (you mentioned using the backup command which would kinda work but full
> replication does the changes on the fly and should work)
>
> if you are running manual replication you should consider going to the
> live replication, it will sort out stuff as the imap folders sync etc.
> (or it should)
>
> The next thing to consider is there were some issues that were fixed in
> 2.3.19 replication, are you running the same dovecot versions on both
> servers ?
>
> I do a ton of emails, reporting etc and find that replication works well
> on dovecot 2.3.19 bewteen both of my mail servers. ie it does not matter
> which one receives the email it gets sorted out. If there is an error
> the replication will sort it out on the next sync run through the
> replication process running in the background.
>
> you can set all of the retries etc for replication in the config files.
>
> the merged later is probably indicating that dovecot will sort stuff out
> in the background (ie a reindex etc) but that is putting extra stress on
> the server(s), i used to get the merge or duplicate uids, guids on cyrus
> and it would try to sort it out on the fly. this would occur when one
> replicated server was offline and i was forcing a sync update after
> bringing it back online, this was the case because both servers had
> received emails into the same account from seperate sources thus the
> same uid was set for two different messages on each server. (fyi)
>
> with syrus a rebuild was the only was to sort this out
>
> dovecot seems way more resiliant in this department.
>
>
> again full replication setup would sort these issue out i expect as each
> server would handle stuff as it happens and adjust uid,guids accordingly.
>
>
>
>
>
>
>
> Happy Sunday !!!
> Thanks - paul
>
> Paul Kudla
>
>
> Scom.ca Internet Services <http://www.scom.ca>
> 004-1009 Byron Street South
> Whitby, Ontario - Canada
> L1N 4S3
>
> Toronto 416.642.7266
> Main 1.866.411.7266
> Fax 1.888.892.7266
> Email paul at scom.ca
>
> On 7/31/2022 8:16 AM, Patrick Westenberg wrote:
>>
>> Hi everyone,
>>
>> I have a weird problem with replication that I'm unable to solve.
>>
>> A new account is sychronized from an external provider via imapsync.
>> The mails end up on my backend1. I see that the folder structure is
>> immediately replicated to backend2.
>>
>> However, a lot of mails are missing and "doveadm replicator status" also
>> states that something failed:
>>
>>
>> priority fast sync full sync success
>> sync
>> failed
>> mail at example.com none 00:02:36 00:39:47 - y
>>
>> There are no error log entries regarding this user at all.
>>
>> After some research I tried to start the replication manually with
>> "doveadm -D backup -u mail at example.com -d tcp:x.y.z.11" and finally
>> I got an error message:
>>
>> Jul 31 13:55:37
>> doveadm(mail at example.com)<11341><rm1YCblt5mJNLAAAJFg/Tg>: Debug: brain
>> M: Mailbox INBOX: local=e74590221b6ce6620d29000024583f4e/0/1,
>> remote=20a9ce2b1c6ce662244e0000baba0ddd/0/1: GUIDs conflict - will be
>> merged later
>>
>> As a matter of fact, the mails in the inbox are the ones that are
>> missing on backend2.
>>
>>
>> I always end up with this error, no matter what I do:
>> - I completely removed the folder structure on backend2.
>> - I removed all database entries.
>> - I removed this user from the replication, ran imapsync again followed
>> by "doveadm -D backup ..."
>>
>> I also removed fts-flatcurve to rule this out as a cause of error.
>>
>> I have no idea what to do. This configuration worked for months.
>> I got this error with 2.3.17.1 and 2.3.19.1
>>
>> Any hints would be highly appreciated.
>>
>>
>> Regards
>> Patrick
>>
>>
>>
>>
>>
>> # 2.3.19.1 (9b53102964): /usr/local/etc/dovecot/dovecot.conf
>> # Pigeonhole version 0.5.19 (4eae2f79)
>> auth_master_user_separator = *
>> auth_mechanisms = plain login
>> doveadm_password = # hidden, use -P to show it
>> doveadm_port = 12345
>> listen = x.y.z.12
>> lmtp_save_to_detail_mailbox = yes
>> log_debug = category=fts-flatcurve
>> log_path = /var/log/dovecot.log
>> mail_debug = yes
>> mail_home = /srv/mail/%Ld/%Ln
>> mail_location = maildir:~/Maildir
>> mail_plugins = quota notify replication
>> managesieve_notify_capability = mailto
>> managesieve_sieve_capability = fileinto reject envelope
>> encoded-character vacation subaddress comparator-i;ascii-numeric
>> relational regex imap4flags copy include variables body enotify
>> environment mailbox date index ihave duplicate mime foreverypart
>> extracttext editheader
>> metric imapc_traffic {
>> fields = bytes_in bytes_out
>> filter = event=imap_command_finished
>> group_by = user
>> }
>> namespace inbox {
>> inbox = yes
>> location =
>> mailbox Drafts {
>> auto = subscribe
>> special_use = \Drafts
>> }
>> mailbox Junk {
>> special_use = \Junk
>> }
>> mailbox Sent {
>> auto = subscribe
>> special_use = \Sent
>> }
>> mailbox "Sent Messages" {
>> special_use = \Sent
>> }
>> mailbox Spamverdacht {
>> auto = subscribe
>> special_use = \Junk
>> }
>> mailbox Trash {
>> auto = subscribe
>> special_use = \Trash
>> }
>> prefix = INBOX/
>> separator = /
>> type = private
>> }
>> passdb {
>> args = /usr/local/etc/dovecot/master-user
>> driver = passwd-file
>> master = yes
>> pass = yes
>> }
>> passdb {
>> args = /usr/local/etc/dovecot/dovecot-sql.conf.ext
>> driver = sql
>> }
>> plugin {
>> mail_replica = tcp:x.y.z.11
>> quota = count:User quota
>> quota_grace = 25M
>> quota_rule2 = INBOX/Trash:storage=+100M
>> quota_status_nouser = DUNNO
>> quota_status_success = DUNNO
>> quota_vsizes = yes
>> quota_warning = storage=100%% quota-full %u
>> quota_warning2 = storage=95%% quota-warning 95 %u
>> quota_warning3 = storage=80%% quota-warning 80 %u
>> quota_warning4 = -storage=100%% quota-ok %u
>> sieve = ~/.dovecot.sieve
>> sieve_after = /usr/local/etc/dovecot/sieve/sieve_after.sieve
>> sieve_default = /usr/local/etc/dovecot/sieve/default.sieve
>> sieve_dir = ~/sieve
>> sieve_extensions = +editheader
>> }
>> protocols = imap pop3 lmtp sieve
>> service aggregator {
>> fifo_listener replication-notify-fifo {
>> user = vmail
>> }
>> unix_listener replication-notify {
>> user = vmail
>> }
>> }
>> service auth {
>> unix_listener auth-userdb {
>> group = vmail
>> mode = 0666
>> user = vmail
>> }
>> }
>> service config {
>> unix_listener config {
>> mode = 0600
>> user = vmail
>> }
>> }
>> service doveadm {
>> inet_listener {
>> port = 12345
>> }
>> }
>> service imap-login {
>> process_min_avail = 2
>> service_count = 0
>> }
>> service lmtp {
>> executable = lmtp -L
>> inet_listener lmtp {
>> address = x.y.z.12
>> port = 24
>> }
>> process_min_avail = 20
>> }
>> service managesieve-login {
>> inet_listener sieve {
>> port = 4190
>> }
>> }
>> service quota-full {
>> executable = script /usr/local/etc/dovecot/quota_full.sh
>> unix_listener quota-full {
>> user = vmail
>> }
>> user = root
>> }
>> service quota-ok {
>> executable = script /usr/local/etc/dovecot/quota_ok.sh
>> unix_listener quota-ok {
>> user = vmail
>> }
>> user = root
>> }
>> service quota-status {
>> client_limit = 1
>> executable = quota-status -p postfix
>> inet_listener {
>> port = 12340
>> }
>> }
>> service quota-warning {
>> executable = script /usr/local/etc/dovecot/quota_warning.sh
>> unix_listener quota-warning {
>> user = vmail
>> }
>> user = root
>> }
>> service replicator {
>> process_min_avail = 1
>> unix_listener replicator-doveadm {
>> mode = 0600
>> user = vmail
>> }
>> }
>> ssl = required
>> ssl_cert = </etc/ssl/certs/bundle.pem
>> ssl_client_ca_dir = /etc/ssl/certs
>> ssl_dh = # hidden, use -P to show it
>> ssl_key = # hidden, use -P to show it
>> userdb {
>> driver = prefetch
>> }
>> userdb {
>> args = /usr/local/etc/dovecot/dovecot-sql.conf.ext
>> driver = sql
>> }
>> verbose_proctitle = yes
>> protocol lmtp {
>> info_log_path = /var/log/dovecot-lmtp.log
>> mail_plugins = quota notify replication sieve
>> }
>> protocol imap {
>> mail_max_userip_connections = 40
>> mail_plugins = quota notify replication imap_quota
>> }
>>
--
Westenberg + Kueppers GbR Spanische Schanzen 37
---- Buero Koeln ---- 47495 Rheinberg
pwestenberg at wk-serv.de Tel.: +49 (0)2843 90369-06
http://www.wk-serv.de Fax : +49 (0)2843 90369-07
Gesellschafter: Sebastian Kueppers & Patrick Westenberg
More information about the dovecot
mailing list