[Dovecot] Backing Up

[Ed W]

mikkel at euro123.dk wrote:
>
> Rsync seems to be loading information about each file into memory before
> comparing the lists of files and doing the actual transfer.
> That may be a lot of memory if you have a lot of files.
>
> I sometimes overcome this by rsync’ing each user or domain one at a time.
> That way you will also limit issues of files no longer existing once the
> transfer begins (makes rsync generate errors).
>   

If you are using rsync2 then definitely this is good advice. Massive
memory consumption to backup large mailboxes (and a long time before
anything starts happening, ie snapshot useful)

However, with rsync3 you should look at the options required to do use
the incremental protocol. This trades a bit of efficiency on hardlinked
files for lower memory and perhaps faster sync speeds. I haven't
personally tried this, but reports on the web seem promising. You need
rsync3 at both ends of the link and to examine your sync options a little

However, one thing which is sadly missing on rsync is a fuzzy option
which can spot files moving from /new to /cur... This may well cause
additional load for imap backups which is potentially avoidable with a
simple copy. I suspect it would be easy to patch a custom bit of code to
handle this though..?

> One option that I would prefer if I were to backup the entire store with
> one command would be generating a snapshot of the file system.
> And then rsync or cp that snapshot. That way you’ll always get a
> consistent backup and you won’t have to worry about how long the backup
> takes to finish.
>   

Snapshot seems like an excellent idea to avoid files missing files
moving between /cur and /new. However, it should be pointed out that
this is extra io for the server (with LVM at least) whilst the backup is
running


I should think rsync3 incremental, plus some custom patching to look for
files moving between /cur and /new would be very efficient for backing
up maildir filestores (at least to the extent your filesystem allows
efficient iterating over lots of files)

Ed W