[Dovecot] dsync backup questions

[Joe Beaubien]

On Mon, Feb 4, 2013 at 9:01 PM, Timo Sirainen <tss at iki.fi> wrote:

> On Mon, 2013-02-04 at 00:57 +0000, Ben Morrow wrote:
> > I can't give authoratitive answers to either of these, but...
> >
> > At  6PM -0500 on  3/02/13 you (Joe Beaubien) wrote:
> > >
> > > I'm currently trying to setup remote backups of my emails but i'm
> running
> > > into issues (mdbox format, indexes and storage in the same folder
> > > hierarchy).
> > >
> > > Local backup command: dsync -u "my_user" backup /backups/my_user
> > >
> > > (1) Recently, I noticed that the local backup takes up twice the size
> as
> > > the original mail location (8gb vs. 4gb). I purged alot of emails from
> the
> > > original location, so the size shrunk, but the local backup just keeps
> on
> > > getting bigger. I couldn't find any dsync option that would delete
> extra
> > > emails.
> > >
> > > - Question: Why isn't the local backup synced properly and remove the
> extra
> > > emails?
> >
> > Are you running 'doveadm purge' on the backed-up dbox? It looks to me as
> > though dsync doesn't do that. I don't know if there's any (simple) way
> > to do that without a running Dovecot instance attached the dbox
> > directory: it's not entirely clear to me whether doveadm will run
> > locally without contacting a doveadm-server instance running under
> > Dovecot, nor how to point 'doveadm purge' at an arbitrary directory.
>
> Right. doveadm -o mail=mdbox:/backups/my_user purge
>

This worked (at least it seems to, the source and destination are roughly
the same size)..

However, if the the original email location has already been purge, does
the backup email location also need to be purged?


> > It might be easier to dsync to a Maildir instead. This should preserve
> > all the Dovecot-visible metadata, and dsyncing back to the dbox for
> > restore should put it all back.
>
> Better sdbox than maildir.
>

I'd rather stick to mdbox for my remote backups. I have a single email
account with over 1.5 million emails in it. With a 1-file-per-message
storage, this would be slow/hell to run. Unless there is a better way.


>
> > > (2) What is the best why to copy this local backup to a remote location
> > > that does NOT have the possibility to run dsync.
> > >
> > > - Question 1: is rsync safe to use and will this data work for restore?
> > >
> > > - Question 2: Would it be safe to simply rsync the original
> mail_location
> > > to the remote server?
> >
> > AFAICT, if Dovecot is stopped on both sides of the transfer it should be
> > safe. If either side has a currently running Dovecot instance (or any
> > other Dovecot tools, like a concurrent dsync) using that dbox, it's
> > likely rsync will copy an inconsistent snapshot of the data, and the
> > result will be corrupted.
>
> It won't be badly corrupted though. At worst Dovecot will rebuild the
> indexes, which takes a while. And most users probably won't get any
> corruption at all.
>
>
>
I think there was a misunderstanding of the setup. dsync is only running on
the local side, the remote side is a "dumb" rsync server that I don't fully
control.