Ok This is speculation but i understand the issue at a programming level
what needs to be understood is that imap's uids & ugid's are relative to the host server the email is coming from.
this is generally not an issue with replication on cyrus or dovecot because the server and the replication is being handled by the same server set (ie the same uids & guids etc are generated as things happen)
example replicated data :
-rw------- 1 vmail vmail uarch 185K Jul 29 09:30 1659101404.M875201P20192.mail19.scom.ca,S=189252,W=192431:2,S -rw------- 1 vmail vmail uarch 1.5K Jul 29 09:53 1659102818.M268117P41331.mail18.scom.ca,S=1583,W=1639:2,S -rw------- 1 vmail vmail uarch 1.0M Jul 29 12:52 1659113530.M841469P58214.mail18.scom.ca,S=1095861,W=1113817:2,S -rw------- 1 vmail vmail uarch 210K Jul 29 13:15 1659114913.M958008P31982.mail19.scom.ca,S=215405,W=219216:2,S
you will note the originating server is in the mail file name (mail19 & mail18 in my case)
this is how dovecot sorts out the uids etc on the fly. (i think)
If i have read this correctly you are trying to sync to an external imap server that carries its own uids guids etc which will be different.
where you are saying that you are using imap sync i assume you are using the unix version
# imapsync Name:
imapsync - Email IMAP tool for syncing, copying, migrating and archiving email mailboxes between two imap servers, one way, and without duplicates.
Version:
This documentation refers to Imapsync $Revision: 1.977 $
if so look at the
--useuid :
Use UIDs instead of headers as a criterion to recognize messages. Option --usecache is then implied unless --nousecache is used.
and the --logfile (ie run a logging file when connecting the the external account), it might help with any errors being generated (run imapsync in debug mode to get full detail)
basically using useuid deals with sometimes getting a different uid back from the origional server
i go through this issue more with pop3 as it returns the id list starting at uid 1 (for example) instead of the actual uid against the email on the server.
uids will force a proper sync (imap or pop3) because the uid on the server will always return the same uid for that email message and increments forward inside the account.
if so then imap sync should be sorting this out when syncing the imap accounts ? (ie creating new usid guids etc)
so assuming the above is happening the next question is are you using replication that is fully setup between the two servers or are you doing manual replication (ie running the doveadm command to do the sync?)
(you mentioned using the backup command which would kinda work but full replication does the changes on the fly and should work)
if you are running manual replication you should consider going to the live replication, it will sort out stuff as the imap folders sync etc. (or it should)
The next thing to consider is there were some issues that were fixed in 2.3.19 replication, are you running the same dovecot versions on both servers ?
I do a ton of emails, reporting etc and find that replication works well on dovecot 2.3.19 bewteen both of my mail servers. ie it does not matter which one receives the email it gets sorted out. If there is an error the replication will sort it out on the next sync run through the replication process running in the background.
you can set all of the retries etc for replication in the config files.
the merged later is probably indicating that dovecot will sort stuff out in the background (ie a reindex etc) but that is putting extra stress on the server(s), i used to get the merge or duplicate uids, guids on cyrus and it would try to sort it out on the fly. this would occur when one replicated server was offline and i was forcing a sync update after bringing it back online, this was the case because both servers had received emails into the same account from seperate sources thus the same uid was set for two different messages on each server. (fyi)
with syrus a rebuild was the only was to sort this out
dovecot seems way more resiliant in this department.
again full replication setup would sort these issue out i expect as each server would handle stuff as it happens and adjust uid,guids accordingly.
Happy Sunday !!! Thanks - paul
Paul Kudla
Scom.ca Internet Services <http://www.scom.ca> 004-1009 Byron Street South Whitby, Ontario - Canada L1N 4S3
Toronto 416.642.7266 Main 1.866.411.7266 Fax 1.888.892.7266 Email paul@scom.ca
On 7/31/2022 8:16 AM, Patrick Westenberg wrote:
Hi everyone,
I have a weird problem with replication that I'm unable to solve.
A new account is sychronized from an external provider via imapsync. The mails end up on my backend1. I see that the folder structure is immediately replicated to backend2.
However, a lot of mails are missing and "doveadm replicator status" also states that something failed:
priority fast sync full sync success syncfailed mail@example.com none 00:02:36 00:39:47 - y
There are no error log entries regarding this user at all.
After some research I tried to start the replication manually with "doveadm -D backup -u mail@example.com -d tcp:x.y.z.11" and finally I got an error message:
Jul 31 13:55:37 doveadm(mail@example.com)<11341><rm1YCblt5mJNLAAAJFg/Tg>: Debug: brain M: Mailbox INBOX: local=e74590221b6ce6620d29000024583f4e/0/1, remote=20a9ce2b1c6ce662244e0000baba0ddd/0/1: GUIDs conflict - will be merged later
As a matter of fact, the mails in the inbox are the ones that are missing on backend2.
I always end up with this error, no matter what I do:
- I completely removed the folder structure on backend2.
- I removed all database entries.
- I removed this user from the replication, ran imapsync again followed by "doveadm -D backup ..."
I also removed fts-flatcurve to rule this out as a cause of error.
I have no idea what to do. This configuration worked for months. I got this error with 2.3.17.1 and 2.3.19.1
Any hints would be highly appreciated.
Regards Patrick
# 2.3.19.1 (9b53102964): /usr/local/etc/dovecot/dovecot.conf # Pigeonhole version 0.5.19 (4eae2f79) auth_master_user_separator = * auth_mechanisms = plain login doveadm_password = # hidden, use -P to show it doveadm_port = 12345 listen = x.y.z.12 lmtp_save_to_detail_mailbox = yes log_debug = category=fts-flatcurve log_path = /var/log/dovecot.log mail_debug = yes mail_home = /srv/mail/%Ld/%Ln mail_location = maildir:~/Maildir mail_plugins = quota notify replication managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext editheader metric imapc_traffic { fields = bytes_in bytes_out filter = event=imap_command_finished group_by = user } namespace inbox { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Spamverdacht { auto = subscribe special_use = \Junk } mailbox Trash { auto = subscribe special_use = \Trash } prefix = INBOX/ separator = / type = private } passdb { args = /usr/local/etc/dovecot/master-user driver = passwd-file master = yes pass = yes } passdb { args = /usr/local/etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { mail_replica = tcp:x.y.z.11 quota = count:User quota quota_grace = 25M quota_rule2 = INBOX/Trash:storage=+100M quota_status_nouser = DUNNO quota_status_success = DUNNO quota_vsizes = yes quota_warning = storage=100%% quota-full %u quota_warning2 = storage=95%% quota-warning 95 %u quota_warning3 = storage=80%% quota-warning 80 %u quota_warning4 = -storage=100%% quota-ok %u sieve = ~/.dovecot.sieve sieve_after = /usr/local/etc/dovecot/sieve/sieve_after.sieve sieve_default = /usr/local/etc/dovecot/sieve/default.sieve sieve_dir = ~/sieve sieve_extensions = +editheader } protocols = imap pop3 lmtp sieve service aggregator { fifo_listener replication-notify-fifo { user = vmail } unix_listener replication-notify { user = vmail } } service auth { unix_listener auth-userdb { group = vmail mode = 0666 user = vmail } } service config { unix_listener config { mode = 0600 user = vmail } } service doveadm { inet_listener { port = 12345 } } service imap-login { process_min_avail = 2 service_count = 0 } service lmtp { executable = lmtp -L inet_listener lmtp { address = x.y.z.12 port = 24 } process_min_avail = 20 } service managesieve-login { inet_listener sieve { port = 4190 } } service quota-full { executable = script /usr/local/etc/dovecot/quota_full.sh unix_listener quota-full { user = vmail } user = root } service quota-ok { executable = script /usr/local/etc/dovecot/quota_ok.sh unix_listener quota-ok { user = vmail } user = root } service quota-status { client_limit = 1 executable = quota-status -p postfix inet_listener { port = 12340 } } service quota-warning { executable = script /usr/local/etc/dovecot/quota_warning.sh unix_listener quota-warning { user = vmail } user = root } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0600 user = vmail } } ssl = required ssl_cert = </etc/ssl/certs/bundle.pem ssl_client_ca_dir = /etc/ssl/certs ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it userdb { driver = prefetch } userdb { args = /usr/local/etc/dovecot/dovecot-sql.conf.ext driver = sql } verbose_proctitle = yes protocol lmtp { info_log_path = /var/log/dovecot-lmtp.log mail_plugins = quota notify replication sieve } protocol imap { mail_max_userip_connections = 40 mail_plugins = quota notify replication imap_quota }