[Dovecot] Backing Up

[mikkel at euro123.dk]

> Dave McGuire wrote:
>> On Oct 29, 2008, at 3:42 PM, Scott Silva wrote:
>>>> What is the best way to do a (server-side) backup of all mail in a
>>>> user's mail?
>>>>
>>> I usually just rsync the /home directories to another server. The
>>> inital sync
>>> can take a while, but it gets faster after there is a base to work
>>> from.
>>
>>   ...and it's much less painful if you're using maildir instead of mbox!
>>
>>            -Dave
>>
> I have to wonder.  I have a mailserver that I do a bootable complete
> image copy of with all files and O/S in two hours to an Ultrium-2 tape,
> 95 GB.  When I switch to maildir, I will go from some 25,000 mbox files
> to 2.5 to 3 million files...I can't believe that isn't going to hurt and
> will force me into incrementals.....
>

My thoughts on rsync.

You may want to consider that incremental backups won’t help you much if
you use Maildir. Incremental or full rsync still has to generate a list of
all the files.

Whether it’ll work for you is impossible to say. I guess you’ll just have
to make a test.
But you're right that the large amount of files will be an issue.

Rsync seems to be loading information about each file into memory before
comparing the lists of files and doing the actual transfer.
That may be a lot of memory if you have a lot of files.

I sometimes overcome this by rsync’ing each user or domain one at a time.
That way you will also limit issues of files no longer existing once the
transfer begins (makes rsync generate errors).

You can estimate the time needed to list all the files.
Try and use iostat to get a rough idea of how many OIPS your system
handles under max stress load and how many it handles under normal
operation.
The difference is the amount available to you during the backup.
Divide the total number of files with the number of available IOPS.

Say you have 100 IOPS available then it will take roughly 8 hours
(3,000,000/100/3600=8.3 hours) to generate the list of 3,000,000 files.
The afterwards transfer will probably be a lot faster.
I'm not sure whether reading information about one file take up one IO
operation. But that way of calculating the time to generate the lists
wasn't much off last time I tried.


One option that I would prefer if I were to backup the entire store with
one command would be generating a snapshot of the file system.
And then rsync or cp that snapshot. That way you’ll always get a
consistent backup and you won’t have to worry about how long the backup
takes to finish.

Regards, Mikkel