Dave McGuire wrote:
On Oct 29, 2008, at 3:42 PM, Scott Silva wrote:
What is the best way to do a (server-side) backup of all mail in a user's mail?
I usually just rsync the /home directories to another server. The inital sync can take a while, but it gets faster after there is a base to work from.
...and it's much less painful if you're using maildir instead of mbox!
-DaveI have to wonder. I have a mailserver that I do a bootable complete image copy of with all files and O/S in two hours to an Ultrium-2 tape, 95 GB. When I switch to maildir, I will go from some 25,000 mbox files to 2.5 to 3 million files...I can't believe that isn't going to hurt and will force me into incrementals.....
My thoughts on rsync.
You may want to consider that incremental backups wont help you much if you use Maildir. Incremental or full rsync still has to generate a list of all the files.
Whether itll work for you is impossible to say. I guess youll just have to make a test. But you're right that the large amount of files will be an issue.
Rsync seems to be loading information about each file into memory before comparing the lists of files and doing the actual transfer. That may be a lot of memory if you have a lot of files.
I sometimes overcome this by rsyncing each user or domain one at a time. That way you will also limit issues of files no longer existing once the transfer begins (makes rsync generate errors).
You can estimate the time needed to list all the files. Try and use iostat to get a rough idea of how many OIPS your system handles under max stress load and how many it handles under normal operation. The difference is the amount available to you during the backup. Divide the total number of files with the number of available IOPS.
Say you have 100 IOPS available then it will take roughly 8 hours (3,000,000/100/3600=8.3 hours) to generate the list of 3,000,000 files. The afterwards transfer will probably be a lot faster. I'm not sure whether reading information about one file take up one IO operation. But that way of calculating the time to generate the lists wasn't much off last time I tried.
One option that I would prefer if I were to backup the entire store with one command would be generating a snapshot of the file system. And then rsync or cp that snapshot. That way youll always get a consistent backup and you wont have to worry about how long the backup takes to finish.
Regards, Mikkel