[Dovecot] Semi-OT: Backing up maildir
We switched our mail server over from mbox to maildir a few months ago and couldn't be happier - the performance under normal load is incredible. However we now have a problem with backup.. Typically we would run tar on the mail server, sending its output to another server via ssh.. Like this: tar cfp - /home | gzip | ssh other.server (cd /backups && tar xfp -).. I have tried gzipping the stream on the source side, gzipping on the destination, and running without gzip, all three take an amazingly long time to complete (> 14 hours). Ours is a modest server with about 1300 users, about 300GB of mail total. The whole thing sits on hardware RAID-10, so I'm primarily guarding against a MAJOR hardware failure, a config blunder, or a hacker wiping out files. How do you guys back up your maildir? I've considered exporing the maildir with NFS and backing it up from another server.. Any opinions on that?
Thanks, Shayne
On Wed, 2006-05-10 at 19:29 -0500, Shayne Hardesty wrote:
We switched our mail server over from mbox to maildir a few months ago and couldn't be happier - the performance under normal load is incredible. However we now have a problem with backup.. Typically we would run tar on the mail server, sending its output to another server via ssh.. Like this: tar cfp - /home | gzip | ssh other.server (cd /backups && tar xfp -).. I have tried gzipping the stream on the source side, gzipping on the destination, and running without gzip, all three take an amazingly long time to complete (> 14 hours). Ours is a modest server with about 1300 users, about 300GB of mail total. The whole thing sits on hardware RAID-10, so I'm primarily guarding against a MAJOR hardware failure, a config blunder, or a hacker wiping out files. How do you guys back up your maildir? I've considered exporing the maildir with NFS and backing it up from another server.. Any opinions on that?
I've had great success with rsync and hard link based backup.
I've customised my own solution based on the scripts found at http://www.mikerubel.org/computers/rsync_snapshots/
-- Karl Latiss karl.latiss@atvert.com.au Atvert Systems
On Thu, May 11, 2006 at 08:44:06PM +1000, Karl Latiss wrote:
On Wed, 2006-05-10 at 19:29 -0500, Shayne Hardesty wrote:
How do you guys back up your maildir?
I've had great success with rsync and hard link based backup.
I've customised my own solution based on the scripts found at http://www.mikerubel.org/computers/rsync_snapshots/
Same here, I referred to this just yesterday in the "Recommended FS" thread.
One note : Shayne is also guarding against file disappearance where the client is responsible somehow, which in my experience is much more frequent.
For IMAP the rsync backup means getting back all mail until last backup, except for some percentage of deleted ones, which isn't bad but not quite perfect.
For POP the rsync backup means telling the client "you can get back things you had received but not popped at midnight every day", which isn't very useful.
Ideal protection is copying incoming mail as it arrives (in addition to the periodic backup), and that's an MTA problem.
HTH
On Wed, 2006-05-10 at 19:29 -0500, Shayne Hardesty wrote:
We switched our mail server over from mbox to maildir a few months ago and couldn't be happier - the performance under normal load is incredible. However we now have a problem with backup.. Typically we would run tar on the mail server, sending its output to another server via ssh.. Like this: tar cfp - /home | gzip | ssh other.server (cd /backups && tar xfp -).. I have tried gzipping the stream on the source side, gzipping on the destination, and running without gzip, all three take an amazingly long time to complete (> 14 hours). Ours is a modest server with about 1300 users, about 300GB of mail total. The whole thing sits on hardware RAID-10, so I'm primarily guarding against a MAJOR hardware failure, a config blunder, or a hacker wiping out files. How do you guys back up your maildir? I've considered exporing the maildir with NFS and backing it up from another server.. Any opinions on that?
DRBD + heartbeat works great or real-time mirroring of maildirs between two servers... http://www.drbd.org http://www.linux-ha.org/
This will protect you against hardware failures, but since deletes are mirrored too you will still probably want to also do some sort of snapshot backups or copying of incoming mail as the others have suggested.
Bill
On Thu, 2006-05-11 at 05:44, Karl Latiss wrote:
On Wed, 2006-05-10 at 19:29 -0500, Shayne Hardesty wrote:
We switched our mail server over from mbox to maildir a few months ago and couldn't be happier - the performance under normal load is incredible. However we now have a problem with backup.. Typically we would run tar on the mail server, sending its output to another server via ssh.. Like this: tar cfp - /home | gzip | ssh other.server (cd /backups && tar xfp -).. I have tried gzipping the stream on the source side, gzipping on the destination, and running without gzip, all three take an amazingly long time to complete (> 14 hours). Ours is a modest server with about 1300 users, about 300GB of mail total. The whole thing sits on hardware RAID-10, so I'm primarily guarding against a MAJOR hardware failure, a config blunder, or a hacker wiping out files. How do you guys back up your maildir? I've considered exporing the maildir with NFS and backing it up from another server.. Any opinions on that?
I've had great success with rsync and hard link based backup.
I've customised my own solution based on the scripts found at http://www.mikerubel.org/computers/rsync_snapshots/
Backuppc (http://backuppc.sourceforge.net/) does a great job of automating backups to on-line storage and can use rsync as the transport. However, rsync loads the whole target directory in memory at startup so there is a certain per-file overhead. Be sure you have plenty of RAM on both the target and backup server.
-- Les Mikesell lesmikesell@gmail.com
On Wed, May 10, 2006 at 07:29:02PM -0500, Shayne Hardesty wrote:
We switched our mail server over from mbox to maildir a few months ago and couldn't be happier - the performance under normal load is incredible. However we now have a problem with backup.. Typically we would run tar on the mail server, sending its output to another server via ssh.. Like this: tar cfp - /home | gzip | ssh other.server (cd /backups && tar xfp -).. I have tried gzipping the stream on the source side, gzipping on the destination, and running without gzip, all three take an amazingly long time to complete (> 14 hours). Ours is a modest server with about 1300 users, about 300GB of mail total. The whole thing sits on hardware RAID-10, so I'm primarily guarding against a MAJOR hardware failure, a config blunder, or a hacker wiping out files. How do you guys back up your maildir? I've considered exporing the maildir with NFS and backing it up from another server.. Any opinions on that?
any maildir backup solution that involves reading files from the filesystem will be inherently slower because there are simply lots of files. a filesystem that supports snapshots is quite possibly the most sensible choice for filesystems that contain lots of small files.
I am running maildir on ZFS, each user has their own ZFS file system, and users' mail is backed up using "zfs snapshot" and "zfs send". as these operate at the filesystem-block level, they are significantly faster than any other backup method which reads files from the filesystem.
also, since you mention it, a mirror is not going to help you in the case of a config error, or a hacker trashing files, as you simply end up with two copies of the same (possibly useless) bits on disk.
grant.
On May 10, 2006, at 6:29 PM, Shayne Hardesty wrote:
We switched our mail server over from mbox to maildir a few
months ago and couldn't be happier - the performance under normal
load is incredible. However we now have a problem with backup..
Typically we would run tar on the mail server, sending its output
to another server via ssh.. Like this: tar cfp - /home | gzip |
ssh other.server (cd /backups && tar xfp -).. I have tried
gzipping the stream on the source side, gzipping on the
destination, and running without gzip, all three take an amazingly
long time to complete (> 14 hours). Ours is a modest server with
about 1300 users, about 300GB of mail total. The whole thing sits
on hardware RAID-10, so I'm primarily guarding against a MAJOR
hardware failure, a config blunder, or a hacker wiping out files.
How do you guys back up your maildir? I've considered exporing the
maildir with NFS and backing it up from another server.. Any
opinions on that?
I am on FreeBSD with ufs2. Don't know what you Penguin folks have
that would do similar. ufs2 supports "snapshots" and I just do a
dump/restore with the -L option on dump, which triggers a snapshot.
I dump to a file on another disk and gzip it (in a pipe). 19Gb takes
less than an hour and the snapshot ensures consistency.
I like the ZFS idea and am planning on implementing that when our
backend switches (if it does -- in planning stages now) to Solaris
10... Similar things are available with ZFS.
Chad
Thanks, Shayne
Chad Leigh -- Shire.Net LLC Your Web App and Email hosting provider chad at shire.net
participants (7)
-
Bill Boebel
-
Chad Leigh -- Shire.Net LLC
-
grant beattie
-
Karl Latiss
-
Les Mikesell
-
Lorens
-
Shayne Hardesty