migration with doveadm backup to new cluster running dovecot 2.2.36 and replicator

[Aki Tuomi]

> On 11/01/2021 05:18 Thomas Winterstein <thomas.winterstein at rz.uni-augsburg.de> wrote:
> 
>  
> we were able to narrow down the cause of the problem.
> 
> 
> After the initial dsync migration process the mailbox GUIDs are the same 
> for each mailbox-name across all users.
> 	Is this intended behaviour of dsync?
> 	If not, how can this be changed?
> 
> 
> After the first replication process the Inboxes of ~20% of users get 
> different mailbox GUIDs.
> 
> 
> During the next incremental dsync process the mailbox GUIDs of these 
> ~20% get overridden by the inital one.
> 
> 
> Then the incremental replication duplicates those Inboxes where the 
> mailbox GUIDs don't match.
> 
> 
> Any ideas?
> 
> 
> thanks
> Thomas
> 
> On 07.01.2021 16:41, Thomas Winterstein wrote:
> >>> dsync is intended to be used to change mailbox format, so it should 
> >>> work just fine.
> > 
> > that's exactly what we thought and why we use dsync to migrate like 
> > described here
> > 
> >    https://wiki2.dovecot.org/Migration/Dsync
> > 
> > 
> > Our replication is configured according to
> > 
> >    https://wiki.dovecot.org/Replication
> > 
> > 
> > Both processes run separately in time.
> > 
> > 
> > Still on some accounts mails of Inbox or another folder get duplicated. 
> > We're currently trying to debug this.
> > 
> > what are we missing?
> > 
> > thanks
> > Thomas
> > 
> > On 07.01.2021 10:21, Aki Tuomi wrote:
> >> dsync is intended to be used to change mailbox format, so it should 
> >> work just fine.
> >>
> >> Aki
> >>
> >>> On 07/01/2021 11:17 Andrea Gabellini 
> >>> <andrea.gabellini at telecomitalia.sm> wrote:
> >>>
> >>> Hello,
> >>>
> >>> I had a similar problem some time ago, and the problem was the mailbox
> >>> format change.
> >>>
> >>> Please try to migrate with the same format.
> >>>
> >>> Andrea
> >>>
> >>> Il 05/01/21 15:02, Thomas Winterstein ha scritto:
> >>>> No one?
> >>>>
> >>>> If there are limitations in regards to how dsync in migration and
> >>>> replication can operate together these should be stated clearly in the
> >>>> documentation.
> >>>>
> >>>> On 23.12.2020 20:33, Thomas Winterstein wrote:
> >>>>> Hello everyone,
> >>>>>
> >>>>>
> >>>>> we are working on migrating from dovecot 2.0.9 (maildir) to 2.2.36
> >>>>> (mdbox). The new cluster has two backend mail servers which replicate
> >>>>> through doveadm replicator. To move the data initially we use doveadm
> >>>>> backup (imapc).
> >>>>>
> >>>>> arb
> >>>>> Our migration command
> >>>>>    doveadm -o mail_fsync=never backup -R -u $user imapc:
> >>>>>
> >>>>>
> >>>>> To test the replication of new and purge of old mails with live data
> >>>>> changes we ran imapc on a daily basis but encountered the problem
> >>>>> that some mailboxes multiplied in size. We then made sure that imapc
> >>>>> and replication don't run at the same time but after the first
> >>>>> incremental imapc process, we still had the same problems.
> >>>>>
> >>>>>
> >>>>> The doveadm-backup man-page states that it's possible to run it
> >>>>> multiple times during migration. But is it also possible to have the
> >>>>> replicator running in between? From our understanding the doveadm
> >>>>> backup should just work as an imap connection between the servers,
> >>>>> synchronizing all changes made on the source to the destination. Or
> >>>>> does the conversion from maildir to mdbox format in our case produce
> >>>>> the problems?
> >>>>>
> >>>>>
> >>>>> If you're not supposed to run the replicator before having fully
> >>>>> migrated, how can we shorten the downtime? rsync? And how can we be
> >>>>> sure that similar problems don't occur after the migration if we
> >>>>> can't test all mechanisms together with live data?
> >>>>>
> >>>>>
> >>>>> thanks
> >>>>
> >>>>
> >>>


When doing migration, it is very useful to construct a temporary dovecot-migration.conf which lacks *any* automatic mailbox creation and has only the bare minimum you need (sieve needs to be configured, and mailbox attributes). This means no autocreate/autosubscribe loaded, and no auto=create or auto=subscribe in namespace config. It is recommended not to have any mailbox Name entries in your namespace config.

You should use doveadm -c /path/to/dovecot-migration.conf backup command on initial sync. 

The mailboxes must not exist before you do initial sync, if they do, you run into troubles.

Aki