[Dovecot] repeating dsync - questions

[Bob Gustafson]

Ok, see interspersed.

On Mon, 2013-04-08 at 00:53 +0300, Timo Sirainen wrote:
> On 8.4.2013, at 0.10, Bob Gustafson <bobgus at rcn.com> wrote:
> 
> >>> I am still on my quest for a quick way to move mail from a live Maildir
> >>> system to a 'soon to be live' sdbox system.
> >>> 
> >>> I copy Maildir to new system using:
> >>> rsync -ar --times hoho4:/home/bobgus/Maildir/ /home/bobgus/Maildir
> >>> 
> >>> Then I convert from Maildir to sdbox with:
> >>> dsync mirror maildir:~/Maildir
> >>> 
> >>> Then I copy more messages from live system using rsync
> >> 
> >> ^^ that is the mistake
> > 
> > I guess I have a basic misunderstanding of what 'dsync mirror' is doing.
> > 
> > My understanding is that going from Maildir to sdbox, dsync does not
> > mess with the data in Maildir. The Maildir metadata is in one form and
> > the sdbox metadata is in another form (in the sdbox directory).
> 
> dsync does mess with metadata in the maildir. also with dsync mirror (as opposed to dsync backup) it can also modify the contents. The main problem here is:
> 
> 1. dsync sees that a folder A in maildir doesn't have a GUID (because dsync is just about the only tool that uses it right now), and assigns the mailbox a new GUID
> 2. dsync syncs the mailbox to sdbox with that GUID
> 3. rsync comes and wipes out the maildir-uidlist that contained the GUID
> 4. second dsync sees that folder A in maildir doesn't have a GUID, and assigns a new GUID to it
>  - now maildir has folder A with GUID 1, and sdbox has folder A with GUID 2
>  - dsync thinks they are two different folders, and duplicates them as A and A_2. the A_2 gets also copied back to maildir, because you're using dsync mirror. This is why the second dsync is slow, it's actually doing all the work again, and actually doing twice the work since it's copying the mails from sdbox to maildir as well.
> 
> v2.2 dsync is somewhat smarter and can figure out that they are actually the same folder A and it simply changes the other's GUID instead of duplicating all data.
> 

Ok, how is this for a scheme:

1) Initially rsync Maildir to destination system
2) Do initial 'dsync backup' from Maildir to sdbox on destination system
3) Do a 2nd rsync of new Maildir data from live system to destination
system, but don't copy the old maildir-uidlist still in the Maildir of
the live system.
rsync -ar --times --exclude=dovecot-uidlist <live sys Maildir> <dest
system Maildir>

I could also use the --ignore-existing option

In theory, rsync would not touch the dovecot-uidlist file, would not
touch the existing message files, but would copy over the new messages
received during the time 'dsync backup' was doing its previous run.

4) Do a 2nd 'dsync backup' from the Maildir to the sdbox (which hasn't
changed since the 1st 'dsync backup')

> > No new email messages enter the sdbox system to be 'mirrored' to the
> > Maildir system.
> > 
> > I thought of using the 'dsync backup' command, but the sentence "Any
> > changes done in destination are discarded." seems to indicate that each
> > time 'dsync backup' is done, it starts from the beginning. No
> > incremental backup (but this is done in 2.2 ?)
> 
> dsync backup is incremental. it just wipes out any changes done at the other side (if there happens to be any).