Backups and disaster recovery
Hello all! After reading some of the past threads on backups, I was wondering if I could get a sanity check... I run a Maildir configuration for a small (10 mailboxes) mail server. Using "doveadm backup", for each mailbox I do:
- Weekly full backups and then copy the files to a network-based filesystem.- Daily incremental backups and then copy the files to a network-based filesystem. My recovery assumption is that in case of hardware failure, I would re-set up the mail service and for each mailbox I will recover at the right directory the latest full mailbox backup and on top of it, each incremental backup to the latest day. Does this sound like a sound strategy? Also, I was thinking of setting up a second dovecot server on another server and replicating my primary on an hourly basis to decrease recovery time. But I looked at mbsync and it seems to require mailbox login/password for each mailbox (which I don't have). Is there an alternative? Thank you!
On Sat, Dec 03, 2022 at 11:02:54PM +0000, GDS wrote:
Hello all! After reading some of the past threads on backups, I was wondering if I could get a sanity check... I run a Maildir configuration for a small (10 mailboxes) mail server. Using "doveadm backup", for each mailbox I do:
Weekly full backups and then copy the files to a network-based filesystem.
Daily incremental backups and then copy the files to a network-based filesystem.
My recovery assumption is that in case of hardware failure, I would re-set up the mail service and for each mailbox I will recover at the right directory the latest full mailbox backup and on top of it, each incremental backup to the latest day. Does this sound like a sound strategy?
Assumptions are dangerous things.
Have you tested your assumptions - i.e. simulated recovering from a hardware failure - in order to be sure your backups and procedures are adequate?
Are your server, and your network-based filesystem protected against bit-rot?
Also, I was thinking of setting up a second dovecot server on another server and replicating my primary on an hourly basis to decrease recovery time. But I looked at mbsync and it seems to require mailbox login/password for each mailbox (which I don't have). Is there an alternative?
You might want to consider using ZFS.
https://en.wikipedia.org/wiki/ZFS
Jim Salter has written some fairly accessible tutorials. For instance:
https://arstechnica.com/information-technology/2014/02/ars-walkthrough-using...
https://arstechnica.com/information-technology/2015/12/rsync-net-zfs-replica...
https://arstechnica.com/information-technology/2020/05/zfs-101-understanding...
https://jrs-s.net/category/open-source/zfs/
Sam
Am 04.12.22 um 04:16 schrieb Sam Kuper:
You might want to consider using ZFS.
ZFS has some great advantages, like scheduled scrub runs against bit rot. Not only one can create snapshots and replicate them to a different (even remote) ZFS pool but that is even possible with an encrypted pool without disclosing the encryption password.
So you can synch file system changes to like a remote location with the file system pool in the remote location being encrypted. You don't need to mount the remote file systems and disclose the password, you just send encrpted ZFS data to the remote location. With that approach you can store the remote backups even in less trusted environments (from a confidentiality not availability point of view).
My sdbox mailboxes are stored in ZFS volumes. I run snapshots daily and back them up. I tried to restore single mailboxes or even single mailbox folders. It all worked so far - still I wonder if there might be any issues with snapshotting a file system rather than using 'doveadm backup'. With databases I flush/dump/whatever them so it it safe to snapshot their volume.
-- Cheers spi
On 12/3/22 20:16, Sam Kuper wrote:
You might want to consider using ZFS.
Bystander reply:
I do like ZFS. There are two reasons that I chose btrfs for my deployment:
License incompatibilities with ZFS and Linux. These don't affect end users, but the situation is a little murky if you try to distribute something that uses both.
The ARC cache for ZFS is completely separate from the Linux disk cache, so that memory cannot be reclaimed under memory pressure as the Linux cache can. Because of the license issues, ZFS cannot be a first-class Linux citizen, and I do not know if the module maintainers can have ZFS use Linux cache instead of ARC.
There is a little bit of worry that btrfs is not as mature as ZFS. But I know there are Linux distributions using btrfs for the root filesystem by default, so that's probably a lot less of a worry than it was a few years ago.
Thanks, Shawn
On 12/3/22 16:02, GDS wrote:
After reading some of the past threads on backups, I was wondering if I could get a sanity check... I run a Maildir configuration for a small (10 mailboxes) mail server. Using "/doveadm backup"/, for each mailbox I do:
- Weekly full backups and then copy the files to a network-based filesystem.
- Daily incremental backups and then copy the files to a network-based filesystem.
My recovery assumption is that in case of hardware failure, I would re-set up the mail service and for each mailbox I will recover at the right directory the latest full mailbox backup and on top of it, each incremental backup to the latest day. Does this sound like a sound strategy?
Also, I was thinking of setting up a second dovecot server on another server and replicating my primary on an hourly basis to decrease recovery time. But I looked at mbsync and it seems to require mailbox login/password for each mailbox (which I don't have). Is there an alternative?
At 07:00 every day, my mailserver rsyncs its entire root filesystem to another server with a large btrfs filesystem. All the mail is on that filesystem. I am using Maildir, not sure how to figure out whether that is Maildir++ or not:
00 7 * * * rsync -axH --delete --delete-excluded --exclude=.git / server.domain.tld:/storage0/bilbofull/
The target server has a snapshot maintenance script I wrote that runs every night, as well as weekly and monthly:
45 23 * * * /usr/local/sbin/snapshot-maintenance storage0 daily 50 23 1 * * /usr/local/sbin/snapshot-maintenance storage0 monthly 55 23 * * 7 /usr/local/sbin/snapshot-maintenance storage0 weekly
The script keeps 7 daily snaps, 4 weekly snaps, and 6 monthly snaps. I deleted all the snapshots recently because I was doing significant maintenance on the filesystem and wanted to actually recover the disk space. This shows the snaps that currently exist:
elyograg@smeagol:/usr/local/sbin$ sudo btrfs subvolume list /storage0 ID 68160 gen 711353 top level 5 path .snapshot/2022.11.01.storage0.monthly ID 68163 gen 718255 top level 5 path .snapshot/2022.11.06.storage0.weekly ID 68170 gen 729261 top level 5 path .snapshot/2022.11.13.storage0.weekly ID 68177 gen 746501 top level 5 path .snapshot/2022.11.20.storage0.weekly ID 68181 gen 756805 top level 5 path .snapshot/2022.11.24.storage0.daily ID 68182 gen 759352 top level 5 path .snapshot/2022.11.25.storage0.daily ID 68183 gen 761844 top level 5 path .snapshot/2022.11.26.storage0.daily ID 68184 gen 779474 top level 5 path qemu ID 68185 gen 764384 top level 5 path .snapshot/2022.11.27.storage0.weekly ID 68186 gen 766942 top level 5 path .snapshot/2022.11.28.storage0.daily ID 68187 gen 769388 top level 5 path .snapshot/2022.11.29.storage0.daily ID 68188 gen 771998 top level 5 path .snapshot/2022.11.30.storage0.daily ID 68189 gen 774588 top level 5 path .snapshot/2022.12.01.storage0.monthly ID 68190 gen 777168 top level 5 path .snapshot/2022.12.02.storage0.daily
The end result of this is that I have backups of all the email at many different points in time. I back up a lot of other things into the btrfs filesystem as well.
elyograg@smeagol:~$ df -h /storage0 Filesystem Size Used Avail Use% Mounted on /dev/sdb1 22T 3.5T 19T 17% /storage0
elyograg@smeagol:~$ sudo du -hs /storage0/bilbofull/var/vmail 6.5G /storage0/bilbofull/var/vmail
I also have 10 mailboxes across several domains. There are more than 10 defined, but only 10 of them actually have email. The users are in a postfixadmin database.
You should be able to use rsync to copy Maildirs from one server to another. That is IMHO one of the advantages to Maildir ... each change to the mailbox occurs with a single message file, so it is unlikely the mailbox will be corrupted if it changes during the copy.
Thanks, Shawn
participants (4)
-
GDS
-
Sam Kuper
-
Shawn Heisey
-
spi