mikkel@euro123.dk wrote:
Rsync seems to be loading information about each file into memory before comparing the lists of files and doing the actual transfer. That may be a lot of memory if you have a lot of files.
I sometimes overcome this by rsync’ing each user or domain one at a time. That way you will also limit issues of files no longer existing once the transfer begins (makes rsync generate errors).
If you are using rsync2 then definitely this is good advice. Massive memory consumption to backup large mailboxes (and a long time before anything starts happening, ie snapshot useful)
However, with rsync3 you should look at the options required to do use the incremental protocol. This trades a bit of efficiency on hardlinked files for lower memory and perhaps faster sync speeds. I haven't personally tried this, but reports on the web seem promising. You need rsync3 at both ends of the link and to examine your sync options a little
However, one thing which is sadly missing on rsync is a fuzzy option which can spot files moving from /new to /cur... This may well cause additional load for imap backups which is potentially avoidable with a simple copy. I suspect it would be easy to patch a custom bit of code to handle this though..?
One option that I would prefer if I were to backup the entire store with one command would be generating a snapshot of the file system. And then rsync or cp that snapshot. That way you’ll always get a consistent backup and you won’t have to worry about how long the backup takes to finish.
Snapshot seems like an excellent idea to avoid files missing files moving between /cur and /new. However, it should be pointed out that this is extra io for the server (with LVM at least) whilst the backup is running
I should think rsync3 incremental, plus some custom patching to look for files moving between /cur and /new would be very efficient for backing up maildir filestores (at least to the extent your filesystem allows efficient iterating over lots of files)
Ed W