[Dovecot] dsync backup questions

[Joe Beaubien]

On Tue, Feb 5, 2013 at 10:41 PM, Joe Beaubien <joe.beaubien at gmail.com>wrote:

>
>
>
> On Mon, Feb 4, 2013 at 9:01 PM, Timo Sirainen <tss at iki.fi> wrote:
>
>> On Mon, 2013-02-04 at 00:57 +0000, Ben Morrow wrote:
>> > I can't give authoratitive answers to either of these, but...
>> >
>> > At  6PM -0500 on  3/02/13 you (Joe Beaubien) wrote:
>> > >
>> > > I'm currently trying to setup remote backups of my emails but i'm
>> running
>> > > into issues (mdbox format, indexes and storage in the same folder
>> > > hierarchy).
>> > >
>> > > Local backup command: dsync -u "my_user" backup /backups/my_user
>> > >
>> > > (1) Recently, I noticed that the local backup takes up twice the size
>> as
>> > > the original mail location (8gb vs. 4gb). I purged alot of emails
>> from the
>> > > original location, so the size shrunk, but the local backup just
>> keeps on
>> > > getting bigger. I couldn't find any dsync option that would delete
>> extra
>> > > emails.
>> > >
>> > > - Question: Why isn't the local backup synced properly and remove the
>> extra
>> > > emails?
>> >
>> > Are you running 'doveadm purge' on the backed-up dbox? It looks to me as
>> > though dsync doesn't do that. I don't know if there's any (simple) way
>> > to do that without a running Dovecot instance attached the dbox
>> > directory: it's not entirely clear to me whether doveadm will run
>> > locally without contacting a doveadm-server instance running under
>> > Dovecot, nor how to point 'doveadm purge' at an arbitrary directory.
>>
>> Right. doveadm -o mail=mdbox:/backups/my_user purge
>>
>
> This worked (at least it seems to, the source and destination are roughly
> the same size)..
>
> However, if the the original email location has already been purge, does
> the backup email location also need to be purged?
>
>
>> > It might be easier to dsync to a Maildir instead. This should preserve
>> > all the Dovecot-visible metadata, and dsyncing back to the dbox for
>> > restore should put it all back.
>>
>> Better sdbox than maildir.
>>
>
> I'd rather stick to mdbox for my remote backups. I have a single email
> account with over 1.5 million emails in it. With a 1-file-per-message
> storage, this would be slow/hell to run. Unless there is a better way.
>
>
>>
>> > > (2) What is the best why to copy this local backup to a remote
>> location
>> > > that does NOT have the possibility to run dsync.
>> > >
>> > > - Question 1: is rsync safe to use and will this data work for
>> restore?
>> > >
>> > > - Question 2: Would it be safe to simply rsync the original
>> mail_location
>> > > to the remote server?
>> >
>> > AFAICT, if Dovecot is stopped on both sides of the transfer it should be
>> > safe. If either side has a currently running Dovecot instance (or any
>> > other Dovecot tools, like a concurrent dsync) using that dbox, it's
>> > likely rsync will copy an inconsistent snapshot of the data, and the
>> > result will be corrupted.
>>
>> It won't be badly corrupted though. At worst Dovecot will rebuild the
>> indexes, which takes a while. And most users probably won't get any
>> corruption at all.
>>
>>
>>
> I think there was a misunderstanding of the setup. dsync is only running
> on the local side, the remote side is a "dumb" rsync server that I don't
> fully control.
>
>
>
what i was trying to ask with my last question is the following:

I'm trying to do remote backups of an mdbox setup, and considering that I
only have dsync running on the local side (not on the remote side), is it
safe to simply do an rsync of the mail_location to the remote server, or
should I first do a dsync (make a local duplicate) and then rsync the
duplcate out to the remote server?

(wish i had a whiteboard right about now).

Thanks,

-Joe