[Dovecot] dsync backup questions
Hi,
I'm currently trying to setup remote backups of my emails but i'm running into issues (mdbox format, indexes and storage in the same folder hierarchy).
Local backup command: dsync -u "my_user" backup /backups/my_user
(1) Recently, I noticed that the local backup takes up twice the size as the original mail location (8gb vs. 4gb). I purged alot of emails from the original location, so the size shrunk, but the local backup just keeps on getting bigger. I couldn't find any dsync option that would delete extra emails.
- Question: Why isn't the local backup synced properly and remove the extra emails?
(2) What is the best why to copy this local backup to a remote location that does NOT have the possibility to run dsync.
Question 1: is rsync safe to use and will this data work for restore?
Question 2: Would it be safe to simply rsync the original mail_location to the remote server?
I searched the net, but it seems that mdbox is tricky to backup.
Thanks in advance,
-Joe
I can't give authoratitive answers to either of these, but...
At 6PM -0500 on 3/02/13 you (Joe Beaubien) wrote:
I'm currently trying to setup remote backups of my emails but i'm running into issues (mdbox format, indexes and storage in the same folder hierarchy).
Local backup command: dsync -u "my_user" backup /backups/my_user
(1) Recently, I noticed that the local backup takes up twice the size as the original mail location (8gb vs. 4gb). I purged alot of emails from the original location, so the size shrunk, but the local backup just keeps on getting bigger. I couldn't find any dsync option that would delete extra emails.
- Question: Why isn't the local backup synced properly and remove the extra emails?
Are you running 'doveadm purge' on the backed-up dbox? It looks to me as though dsync doesn't do that. I don't know if there's any (simple) way to do that without a running Dovecot instance attached the dbox directory: it's not entirely clear to me whether doveadm will run locally without contacting a doveadm-server instance running under Dovecot, nor how to point 'doveadm purge' at an arbitrary directory.
It might be easier to dsync to a Maildir instead. This should preserve all the Dovecot-visible metadata, and dsyncing back to the dbox for restore should put it all back.
(2) What is the best why to copy this local backup to a remote location that does NOT have the possibility to run dsync.
Question 1: is rsync safe to use and will this data work for restore?
Question 2: Would it be safe to simply rsync the original mail_location to the remote server?
AFAICT, if Dovecot is stopped on both sides of the transfer it should be safe. If either side has a currently running Dovecot instance (or any other Dovecot tools, like a concurrent dsync) using that dbox, it's likely rsync will copy an inconsistent snapshot of the data, and the result will be corrupted.
If you have truly atomic filesystem snapshots, rsyncing out of a snapshot also *ought* to be safe, since the situation is the same as if Dovecot crashed at the moment the snapshot was taken, though it might be a good idea to run doveadm force-resync to make sure the result is consistent from Dovecot's point of view.
Ben
On Mon, 2013-02-04 at 00:57 +0000, Ben Morrow wrote:
I can't give authoratitive answers to either of these, but...
At 6PM -0500 on 3/02/13 you (Joe Beaubien) wrote:
I'm currently trying to setup remote backups of my emails but i'm running into issues (mdbox format, indexes and storage in the same folder hierarchy).
Local backup command: dsync -u "my_user" backup /backups/my_user
(1) Recently, I noticed that the local backup takes up twice the size as the original mail location (8gb vs. 4gb). I purged alot of emails from the original location, so the size shrunk, but the local backup just keeps on getting bigger. I couldn't find any dsync option that would delete extra emails.
- Question: Why isn't the local backup synced properly and remove the extra emails?
Are you running 'doveadm purge' on the backed-up dbox? It looks to me as though dsync doesn't do that. I don't know if there's any (simple) way to do that without a running Dovecot instance attached the dbox directory: it's not entirely clear to me whether doveadm will run locally without contacting a doveadm-server instance running under Dovecot, nor how to point 'doveadm purge' at an arbitrary directory.
Right. doveadm -o mail=mdbox:/backups/my_user purge
It might be easier to dsync to a Maildir instead. This should preserve all the Dovecot-visible metadata, and dsyncing back to the dbox for restore should put it all back.
Better sdbox than maildir.
(2) What is the best why to copy this local backup to a remote location that does NOT have the possibility to run dsync.
Question 1: is rsync safe to use and will this data work for restore?
Question 2: Would it be safe to simply rsync the original mail_location to the remote server?
AFAICT, if Dovecot is stopped on both sides of the transfer it should be safe. If either side has a currently running Dovecot instance (or any other Dovecot tools, like a concurrent dsync) using that dbox, it's likely rsync will copy an inconsistent snapshot of the data, and the result will be corrupted.
It won't be badly corrupted though. At worst Dovecot will rebuild the indexes, which takes a while. And most users probably won't get any corruption at all.
On Mon, Feb 4, 2013 at 9:01 PM, Timo Sirainen <tss@iki.fi> wrote:
I can't give authoratitive answers to either of these, but...
At 6PM -0500 on 3/02/13 you (Joe Beaubien) wrote:
I'm currently trying to setup remote backups of my emails but i'm
running
into issues (mdbox format, indexes and storage in the same folder hierarchy).
Local backup command: dsync -u "my_user" backup /backups/my_user
(1) Recently, I noticed that the local backup takes up twice the size as the original mail location (8gb vs. 4gb). I purged alot of emails from
On Mon, 2013-02-04 at 00:57 +0000, Ben Morrow wrote: the
original location, so the size shrunk, but the local backup just keeps on getting bigger. I couldn't find any dsync option that would delete extra emails.
- Question: Why isn't the local backup synced properly and remove the extra emails?
Are you running 'doveadm purge' on the backed-up dbox? It looks to me as though dsync doesn't do that. I don't know if there's any (simple) way to do that without a running Dovecot instance attached the dbox directory: it's not entirely clear to me whether doveadm will run locally without contacting a doveadm-server instance running under Dovecot, nor how to point 'doveadm purge' at an arbitrary directory.
Right. doveadm -o mail=mdbox:/backups/my_user purge
This worked (at least it seems to, the source and destination are roughly the same size)..
However, if the the original email location has already been purge, does the backup email location also need to be purged?
It might be easier to dsync to a Maildir instead. This should preserve all the Dovecot-visible metadata, and dsyncing back to the dbox for restore should put it all back.
Better sdbox than maildir.
I'd rather stick to mdbox for my remote backups. I have a single email account with over 1.5 million emails in it. With a 1-file-per-message storage, this would be slow/hell to run. Unless there is a better way.
(2) What is the best why to copy this local backup to a remote location that does NOT have the possibility to run dsync.
Question 1: is rsync safe to use and will this data work for restore?
Question 2: Would it be safe to simply rsync the original mail_location to the remote server?
AFAICT, if Dovecot is stopped on both sides of the transfer it should be safe. If either side has a currently running Dovecot instance (or any other Dovecot tools, like a concurrent dsync) using that dbox, it's likely rsync will copy an inconsistent snapshot of the data, and the result will be corrupted.
It won't be badly corrupted though. At worst Dovecot will rebuild the indexes, which takes a while. And most users probably won't get any corruption at all.
I think there was a misunderstanding of the setup. dsync is only running on the local side, the remote side is a "dumb" rsync server that I don't fully control.
On Tue, Feb 5, 2013 at 10:41 PM, Joe Beaubien <joe.beaubien@gmail.com>wrote:
On Mon, Feb 4, 2013 at 9:01 PM, Timo Sirainen <tss@iki.fi> wrote:
On Mon, 2013-02-04 at 00:57 +0000, Ben Morrow wrote:
I can't give authoratitive answers to either of these, but...
At 6PM -0500 on 3/02/13 you (Joe Beaubien) wrote:
I'm currently trying to setup remote backups of my emails but i'm
running
into issues (mdbox format, indexes and storage in the same folder hierarchy).
Local backup command: dsync -u "my_user" backup /backups/my_user
(1) Recently, I noticed that the local backup takes up twice the size as the original mail location (8gb vs. 4gb). I purged alot of emails from the original location, so the size shrunk, but the local backup just keeps on getting bigger. I couldn't find any dsync option that would delete extra emails.
- Question: Why isn't the local backup synced properly and remove the extra emails?
Are you running 'doveadm purge' on the backed-up dbox? It looks to me as though dsync doesn't do that. I don't know if there's any (simple) way to do that without a running Dovecot instance attached the dbox directory: it's not entirely clear to me whether doveadm will run locally without contacting a doveadm-server instance running under Dovecot, nor how to point 'doveadm purge' at an arbitrary directory.
Right. doveadm -o mail=mdbox:/backups/my_user purge
This worked (at least it seems to, the source and destination are roughly the same size)..
However, if the the original email location has already been purge, does the backup email location also need to be purged?
It might be easier to dsync to a Maildir instead. This should preserve all the Dovecot-visible metadata, and dsyncing back to the dbox for restore should put it all back.
Better sdbox than maildir.
I'd rather stick to mdbox for my remote backups. I have a single email account with over 1.5 million emails in it. With a 1-file-per-message storage, this would be slow/hell to run. Unless there is a better way.
(2) What is the best why to copy this local backup to a remote location that does NOT have the possibility to run dsync.
Question 1: is rsync safe to use and will this data work for restore?
Question 2: Would it be safe to simply rsync the original mail_location to the remote server?
AFAICT, if Dovecot is stopped on both sides of the transfer it should be safe. If either side has a currently running Dovecot instance (or any other Dovecot tools, like a concurrent dsync) using that dbox, it's likely rsync will copy an inconsistent snapshot of the data, and the result will be corrupted.
It won't be badly corrupted though. At worst Dovecot will rebuild the indexes, which takes a while. And most users probably won't get any corruption at all.
I think there was a misunderstanding of the setup. dsync is only running on the local side, the remote side is a "dumb" rsync server that I don't fully control.
what i was trying to ask with my last question is the following:
I'm trying to do remote backups of an mdbox setup, and considering that I only have dsync running on the local side (not on the remote side), is it safe to simply do an rsync of the mail_location to the remote server, or should I first do a dsync (make a local duplicate) and then rsync the duplcate out to the remote server?
(wish i had a whiteboard right about now).
Thanks,
-Joe
participants (3)
-
Ben Morrow
-
Joe Beaubien
-
Timo Sirainen