Re: [Dovecot] Backup Maildir
- On 06/10/05 13:10 -0400, Geo Carncross wrote:
On Thu, 2005-10-06 at 19:10 +0300, Odhiambo Washington wrote:
- On 06/10/05 16:06 +0100, Gabe Granger wrote:
Hi All,
I'm currently doing nightly full backups of /home. I'm not sure if a
full back up is the best way to do backups. I know with my
fileserver i do weekly full backup and daily incremental which would
file on those type of files. I guess my question to the group is,
how do you guys backup Maildir? whats the best way of doing it?This is an interesting question, especially because it relates to mail! I am a systems admin, but this is one area that I seriously flop in.
How do you backup e-mails for users when during the backup, they are deleting the e-mails from the Maildir (POPping) and at the same, and the SMTP server is also writing to it. This is very real on a busy mail server.
So, how do people make backups of client's e-mails (Maildir)???
The ostrich approach works very well for Maildir. Deleting entries or new entries either show up or they didn't.
Please describe this ostrich approach, or show me how. We only know of Ostriches for meat these sides :-)
Best regards,
Odhiambo Washington
Systems Admin,
Wananchi Online Ltd.Are you hosting your domain name with the leaders??: See http://webhosting.info/webhosts/tophosts/Country/KE
DISCLAIMER : http://ns2.wananchi.com/~wash/Email/disclaimer.txt ----------------------------------+----------------------------------------- Odhiambo WASHINGTON . WANANCHI ONLINE LTD (Nairobi, KE) http://www.wananchi.com/email/ . 1ere Etage, Loita Hse, Loita St., Mobile: (+254) 722 743 223 . # 10286, 00100 NAIROBI ----------------------------------+----------------------------------------- Q: What is the difference between open-source and commercial software? A: If you have a problem with commercial software you can call a phone number and they will tell you it might be solved in a future version. For open-source software there isn't a phone number to call, but you get the solution within a day.
On Thu, 2005-10-06 at 23:54 +0300, Odhiambo Washington wrote:
- On 06/10/05 13:10 -0400, Geo Carncross wrote:
On Thu, 2005-10-06 at 19:10 +0300, Odhiambo Washington wrote:
- On 06/10/05 16:06 +0100, Gabe Granger wrote:
Hi All,
I'm currently doing nightly full backups of /home. I'm not sure if a
full back up is the best way to do backups. I know with my
fileserver i do weekly full backup and daily incremental which would
file on those type of files. I guess my question to the group is,
how do you guys backup Maildir? whats the best way of doing it?This is an interesting question, especially because it relates to mail! I am a systems admin, but this is one area that I seriously flop in.
How do you backup e-mails for users when during the backup, they are deleting the e-mails from the Maildir (POPping) and at the same, and the SMTP server is also writing to it. This is very real on a busy mail server.
So, how do people make backups of client's e-mails (Maildir)???
The ostrich approach works very well for Maildir. Deleting entries or new entries either show up or they didn't.
Please describe this ostrich approach, or show me how. We only know of Ostriches for meat these sides :-)
Ignore the problem. It's a non-problem.
Maildirs can be accessed atomically. The data that dovecot uses/generates that can't be is regenerated by dovecot (or should be) when it needs to be.
Do what an ostrich does and insert head into ground.
If that means clients have to download extra messages, its the least of your worries after a restore.
Nevertheless: since many messages will show up "at least once" (being incremental), this could cause many people problems. Similarly, using mbox, your chances are getting a reasonable "snapshot" of your mailbox, or getting nothing at all.
Backing up mailboxes is like backing up a mail queue. The only way for the backup to really be useful is if you can stop the queue (or the mailbox). But failures that require backups are so infrequent, that is it really worth putting your disk through all that extra work, and slowing down delivery time by so much to get a backup that might not ever be needed?
If you really wanted real backups of something as volatile as a mailbox or a mail queue, you'd use a logging filesystem and replicate your log.
I make naive backups of maildirs, because it's close enough for most things. Any closer than what I get out of the naive approach (and ignoring the problem) slows things down so much that mail becomes useless.
Mbox on the other hand is much easier. Just make sure your backup software locks the mbox before taking a snapshot and make certain your expunge operations NEVER update the mbox file in place (but instead rename() etc)
-- Internet Connection High Quality Web Hosting http://www.internetconnection.net/
On Thu, 2005-10-06 at 17:09 -0400, Geo Carncross wrote:
Mbox on the other hand is much easier. Just make sure your backup software locks the mbox before taking a snapshot and make certain your expunge operations NEVER update the mbox file in place (but instead rename() etc)
Using rename() to overwrite mboxes has two problems:
Uses more disk space so people with filesystem quota wouldn't be able to expunge messages.
A lot of mbox software don't notice the new mbox file, so they keep reading/writing to the old mbox file and that could lose changes. Dovecot checks this though.
Anyway, because of these Dovecot doesn't do the rename()-expunging for mboxes.
On Sun, 2005-10-16 at 18:20 +0300, Timo Sirainen wrote:
On Thu, 2005-10-06 at 17:09 -0400, Geo Carncross wrote:
Mbox on the other hand is much easier. Just make sure your backup software locks the mbox before taking a snapshot and make certain your expunge operations NEVER update the mbox file in place (but instead rename() etc)
Using rename() to overwrite mboxes has two problems:
Agreed, but the problems with not using rename() are also critical:
- Uses more disk space so people with filesystem quota wouldn't be able to expunge messages.
- Cannot safely take a backup since the mbox is never in a safe state
Because there's never a safe state, people _lose_data_. Arranging for quotas to be rescinded while expunging, or making the dovecot/mta combination aware of the quota and lock delivery (see #2) when 1/2 full, solves this problem.
Using an agent that can be immune to quotas that performs the expunge might also be preferable to losing data.
- A lot of mbox software don't notice the new mbox file, so they keep reading/writing to the old mbox file and that could lose changes. Dovecot checks this though.
- No locking mechanism is used that can keep the mbox in a safe state long enough to take a snapshot.
These really are different problems caused by the same root: There needs to be a way to "stop delivery" temporarily, as well as stop breaking the mbox file.
With qmail, one can +t the home directory to halt delivery. dovecot-lda should have a similar mechanism, and other MTAs/MDAs should be providing something similar.
Anyway, because of these Dovecot doesn't do the rename()-expunging for mboxes.
With "2" alone, the mbox could be held in a safe state long enough to take a backup. It would require wiring up the backup software somewhat, but at least it's possible.
If the mbox is always kept in a safe state (!) then existing backup software could be used. I suppose the question is which is easier for administrators to shoehorn the right semantics for their systems: adding some complexity to the backup software, or adding some complexity to the mail delivery process [something that is probably very complicated already :)]
-- Internet Connection High Quality Web Hosting http://www.internetconnection.net/
participants (3)
-
Geo Carncross
-
Odhiambo Washington
-
Timo Sirainen