Best practices for backing up small mailserver to remote location
My webserver also houses our mailserver. There's about six users on that mail system and I'm thinking it would be good to back up the mailboxes to my always on HTPC computer at home, which is reachable via a dynamic IP service.
I know (or think) I need to use doveadm-backup for this but rather than reinvent the wheel (or use the wrong wheel altogether) I'm wondering if anyone can recommend a good tutorial or wiki entry that shows the best way to loop through the users and send their backups to a remote server.
Thanks.
On 08-08-2018 7:48, Ian Evans wrote:
My webserver also houses our mailserver. There's about six users on that mail system and I'm thinking it would be good to back up the mailboxes to my always on HTPC computer at home, which is reachable via a dynamic IP service.
I know (or think) I need to use doveadm-backup for this but rather than reinvent the wheel (or use the wrong wheel altogether) I'm wondering if anyone can recommend a good tutorial or wiki entry that shows the best way to loop through the users and send their backups to a remote server.
Assuming you're running *nix on your HTPC and can install your own software on it a safe, secure and reliable way of doing it is:
- Since you're on dynamic IP at home, set up a VPN tunnel using the mailserver as server and HTPC as client. OpenVPN is ubiquitous and widely supported.
- rsync your mailboxes using the tunnel connection. This way you can back up your entire server, not only the mailboxes. You can add doveadm in the mix if you want, or use imapsync and so on and so forth.
YMMV
-- Adi Pircalabu
On Wed, 8 Aug 2018, daniel_1983@protonmail.com wrote:
- rsync
may not be the best option depending on the format of mailboxes. If you're using maildir or maildir+ that's fine, but what about mbox or dbox ?
It depends on the situation. I can't speak for dbox, but if the mbox file is not updated, then it's no different than maildir. (It might actually be slightly faster as you don't read a lot of metadata for boxes with many messages. This is a consideration for Maildir if the filesystem format handles metadata poorly.)
For mbox where most of the updates occur at the end of the file (i.e. the latest messages), then you'll have to incur read I/O at the source to calculate rolling checksums, but only the changed blocks will be transferred.
The worst situation is a modification at the beginning of a large mbox (e.g. delete first message), which will trigger a full copy.
So for mostly static mboxes, and moderately sized active mailboxes, rsync will work fine, especially owing to its simplicity.
Joseph Tam <jtam.home@gmail.com>
On 8/7/2018 5:08 PM, Adi Pircalabu wrote:
- Since you're on dynamic IP at home, set up a VPN tunnel using the mailserver as server and HTPC as client. OpenVPN is ubiquitous and widely supported.
- rsync your mailboxes using the tunnel connection. This way you can back up your entire server, not only the mailboxes.
Instead of openvpn, I use openssh. Use compression in the ssh tunnel, not the rsync connection, as rsync compression tends to be buggy and interrupts the download. I run sshd on a non-standard port to keep my logs relatively free of script kiddy noise from people looking for an ssh connection to crack. Run fail2ban to lock out the remaining script kiddies. Use a client certificate to log in with ssh unprompted, making it easy to download in a cron job.
Here's an example of scripting the download. Uncomment the DRYRUN line for testing, then comment for production. Add more rsync commands to back up different partitions. The --one-file-system prevents rsync from trying to back up /dev, /proc, and /sys. The --delete option will remove local files that were deleted on the remote server. Use that set of options once you're happy that the backup is working right.
#!/bin/sh #set -e set -x #DRYRUN=--dry-run #RSYNC_OPTIONS="$DRYRUN --one-file-system -avH --delete" RSYNC_OPTIONS="$DRYRUN --one-file-system -avH" DEST=/home/rsync/Server1
# Allow one hour so we don't burn up our bandwidth allowance # from a command error
time timeout 1h
rsync -e 'ssh -C -p 1234' $RSYNC_OPTIONS example.com:/ ${DEST}/
--exclude tmp
# add more rsync commands here for other partitions
On 09-08-2018 10:05, Kenneth Porter wrote:
On 8/7/2018 5:08 PM, Adi Pircalabu wrote:
- Since you're on dynamic IP at home, set up a VPN tunnel using the mailserver as server and HTPC as client. OpenVPN is ubiquitous and widely supported.
- rsync your mailboxes using the tunnel connection. This way you can back up your entire server, not only the mailboxes.
Instead of openvpn, I use openssh. Use compression in the ssh tunnel, not the rsync connection, as rsync compression tends to be buggy and interrupts the download. I run sshd on a non-standard port to keep my logs relatively free of script kiddy noise from people looking for an ssh connection to crack. Run fail2ban to lock out the remaining script kiddies. Use a client certificate to log in with ssh unprompted, making it easy to download in a cron job.
There's more than one way to skin a cat :) Moving the ssh port and adding fail2ban in the mix is another option. Personally tend to use VPN tunnels for dynamic IP clients for various reasons, such as being able to lock clients out by revoking keys.
-- Adi Pircalabu
On 2018.08.09. 3:21, Adi Pircalabu wrote:
On 09-08-2018 10:05, Kenneth Porter wrote:
On 8/7/2018 5:08 PM, Adi Pircalabu wrote:
- Since you're on dynamic IP at home, set up a VPN tunnel using the mailserver as server and HTPC as client. OpenVPN is ubiquitous and widely supported.
- rsync your mailboxes using the tunnel connection. This way you can back up your entire server, not only the mailboxes.
Instead of openvpn, I use openssh. Use compression in the ssh tunnel, not the rsync connection, as rsync compression tends to be buggy and interrupts the download. I run sshd on a non-standard port to keep my logs relatively free of script kiddy noise from people looking for an ssh connection to crack. Run fail2ban to lock out the remaining script kiddies. Use a client certificate to log in with ssh unprompted, making it easy to download in a cron job.
There's more than one way to skin a cat :) Moving the ssh port and adding fail2ban in the mix is another option. Personally tend to use VPN tunnels for dynamic IP clients for various reasons, such as being able to lock clients out by revoking keys.
I prefer connecting from backup server side (and this will resolve dynamic ip problem in this case), so backups are not accessible from production servers. Another option is rsnapshot (if you need versions), it's rsync over ssh and depending on hard link "magic" it conserves disk space (only one copy of mail, independently how long history you have). But as said before - it better works for maildir.
-- KSB
participants (6)
-
Adi Pircalabu
-
daniel_1983＠protonmail.com
-
Ian Evans
-
Joseph Tam
-
Kenneth Porter
-
KSB