migration with doveadm backup to new cluster running dovecot 2.2.36 and replicator

[Thomas Winterstein]

we were able to narrow down the cause of the problem.


After the initial dsync migration process the mailbox GUIDs are the same 
for each mailbox-name across all users.
	Is this intended behaviour of dsync?
	If not, how can this be changed?


After the first replication process the Inboxes of ~20% of users get 
different mailbox GUIDs.


During the next incremental dsync process the mailbox GUIDs of these 
~20% get overridden by the inital one.


Then the incremental replication duplicates those Inboxes where the 
mailbox GUIDs don't match.


Any ideas?


thanks
Thomas

On 07.01.2021 16:41, Thomas Winterstein wrote:
>>> dsync is intended to be used to change mailbox format, so it should 
>>> work just fine.
> 
> that's exactly what we thought and why we use dsync to migrate like 
> described here
> 
>    https://wiki2.dovecot.org/Migration/Dsync
> 
> 
> Our replication is configured according to
> 
>    https://wiki.dovecot.org/Replication
> 
> 
> Both processes run separately in time.
> 
> 
> Still on some accounts mails of Inbox or another folder get duplicated. 
> We're currently trying to debug this.
> 
> what are we missing?
> 
> thanks
> Thomas
> 
> On 07.01.2021 10:21, Aki Tuomi wrote:
>> dsync is intended to be used to change mailbox format, so it should 
>> work just fine.
>>
>> Aki
>>
>>> On 07/01/2021 11:17 Andrea Gabellini 
>>> <andrea.gabellini at telecomitalia.sm> wrote:
>>>
>>> Hello,
>>>
>>> I had a similar problem some time ago, and the problem was the mailbox
>>> format change.
>>>
>>> Please try to migrate with the same format.
>>>
>>> Andrea
>>>
>>> Il 05/01/21 15:02, Thomas Winterstein ha scritto:
>>>> No one?
>>>>
>>>> If there are limitations in regards to how dsync in migration and
>>>> replication can operate together these should be stated clearly in the
>>>> documentation.
>>>>
>>>> On 23.12.2020 20:33, Thomas Winterstein wrote:
>>>>> Hello everyone,
>>>>>
>>>>>
>>>>> we are working on migrating from dovecot 2.0.9 (maildir) to 2.2.36
>>>>> (mdbox). The new cluster has two backend mail servers which replicate
>>>>> through doveadm replicator. To move the data initially we use doveadm
>>>>> backup (imapc).
>>>>>
>>>>> arb
>>>>> Our migration command
>>>>>    doveadm -o mail_fsync=never backup -R -u $user imapc:
>>>>>
>>>>>
>>>>> To test the replication of new and purge of old mails with live data
>>>>> changes we ran imapc on a daily basis but encountered the problem
>>>>> that some mailboxes multiplied in size. We then made sure that imapc
>>>>> and replication don't run at the same time but after the first
>>>>> incremental imapc process, we still had the same problems.
>>>>>
>>>>>
>>>>> The doveadm-backup man-page states that it's possible to run it
>>>>> multiple times during migration. But is it also possible to have the
>>>>> replicator running in between? From our understanding the doveadm
>>>>> backup should just work as an imap connection between the servers,
>>>>> synchronizing all changes made on the source to the destination. Or
>>>>> does the conversion from maildir to mdbox format in our case produce
>>>>> the problems?
>>>>>
>>>>>
>>>>> If you're not supposed to run the replicator before having fully
>>>>> migrated, how can we shorten the downtime? rsync? And how can we be
>>>>> sure that similar problems don't occur after the migration if we
>>>>> can't test all mechanisms together with live data?
>>>>>
>>>>>
>>>>> thanks
>>>>
>>>>
>>>
>>> -- 
>>> __________________________
>>> Daddy, why doesn't this magnet pick up this floppy ?
>>> __________________________
>>>
>>> TIM San Marino S.p.A.
>>> Andrea Gabellini
>>> Engineering R&D
>>> TIM San Marino S.p.A. - https://www.telecomitalia.sm
>>> Via Ventotto Luglio, 212 - Piano -2
>>> 47893 - Borgo Maggiore - Republic of San Marino
>>> Tel: (+378) 0549 886237
>>> Fax: (+378) 0549 886188
>>>
>>>
>>> -- 
>>> Informativa Privacy
>>>
>>> Questa email ha per destinatari dei contatti presenti negli archivi 
>>> di TIM San Marino S.p.A.. Tutte le informazioni vengono trattate e 
>>> tutelate nel rispetto della normativa vigente sulla protezione dei 
>>> dati personali (Reg. EU 2016/679). Per richiedere informazioni e/o 
>>> variazioni e/o la cancellazione dei vostri dati presenti nei nostri 
>>> archivi potete inviare una email a privacy at telecomitalia.sm.
>>>
>>> Avviso di Riservatezza
>>>
>>> Il contenuto di questa e-mail e degli eventuali allegati e' 
>>> strettamente confidenziale e destinato alla/e persona/e a cui e' 
>>> indirizzato. Se avete ricevuto per errore questa e-mail, vi preghiamo 
>>> di segnalarcelo immediatamente e di cancellarla dal vostro computer. 
>>> E' fatto divieto di copiare e divulgare il contenuto di questa 
>>> e-mail. Ogni utilizzo abusivo delle informazioni qui contenute da 
>>> parte di persone terze o comunque non indicate nella presente e-mail 
>>> potra' essere perseguito ai sensi di legge.
> 
> 


-- 
Thomas Winterstein .................... http://www.rz.uni-augsburg.de/
Universität Augsburg, Rechenzentrum ............. Tel. (0821) 598-2068
86135 Augsburg .................................. Fax. (0821) 598-2028