[Dovecot] Anyone successfully setup Continous Backup of mailboxes using rsync ?
Hi All..
Am working on continous backup of mailboxes using rsync(for e.g by running rsync every 2 min)
Things gets more complicated when users create Subfolders in INBOX , SENT , etc..
If anyone among you did that plz guide & advise.
Thanks CoolAtt
Your E-mail and More On-the-Go. Get Windows Live Hotmail Free. https://signup.live.com/signup.aspx?id=60969
The answer, if you can sell the cost, might be network storage. Most of them provide the capability of take inode snapshots CoolAtt NNA wrote:
Hi All..
Am working on continous backup of mailboxes using rsync(for e.g by running rsync every 2 min)
Things gets more complicated when users create Subfolders in INBOX , SENT , etc..
If anyone among you did that plz guide & advise.
Thanks CoolAtt
Your E-mail and More On-the-Go. Get Windows Live Hotmail Free. https://signup.live.com/signup.aspx?id=60969
-- ==== Once upon a time, the Internet was a friendly, neighbors-helping-neighbors small town, and no one locked their doors. Now it's like an apartment in Bed-Stuy: you need three heavy duty pick-proof locks, one of those braces that goes from the lock to the floor, and bars on the windows.... ==== Stewart Dean, Unix System Admin, Bard College, New York 12504 sdean@bard.edu voice: 845-758-7475, fax: 845-758-7035
On 28.01.2010 09:07, CoolAtt NNA wrote:
Hi All..
Am working on continous backup of mailboxes using rsync(for e.g by running rsync every 2 min)
Things gets more complicated when users create Subfolders in INBOX , SENT , etc..
If anyone among you did that plz guide& advise.
You don't state much on what this runs on, but I do snapshot backups of my IMAP server consistently by using LVM, and just match my cronjob to snapshot it at a few given times, mounting the snapshot and backing up with rdiff-backup.
Very simple, very easy, and very realiable. :)
Thanks CoolAtt
-A
Hi Alex,
I mirror the mailboxes as follows:
rsync -avt --log-file=$log_file /home/vmail/ 10.0.0.x:/home/vmail I run it in a script for e,g every 1 min. By doing so if an email is deleted on the live server it still remain on the backup server. Then I give each mail file a retention period of n days so that it automatically gets deleted from the backup server.
CoolAtt
Date: Thu, 28 Jan 2010 20:57:15 +0100 From: alexh@boxed.no To: dovecot@dovecot.org Subject: Re: [Dovecot] Anyone successfully setup Continous Backup of mailboxes using rsync ?
On 28.01.2010 09:07, CoolAtt NNA wrote:
Hi All..
Am working on continous backup of mailboxes using rsync(for e.g by running rsync every 2 min)
Things gets more complicated when users create Subfolders in INBOX , SENT , etc..
If anyone among you did that plz guide& advise.
You don't state much on what this runs on, but I do snapshot backups of my IMAP server consistently by using LVM, and just match my cronjob to snapshot it at a few given times, mounting the snapshot and backing up with rdiff-backup.
Very simple, very easy, and very realiable. :)
Thanks CoolAtt
-A
Hotmail: Trusted email with Microsoft’s powerful SPAM protection. https://signup.live.com/signup.aspx?id=60969
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
CoolAtt NNA said the following on 29/01/10 06:21:
rsync -avt --log-file=$log_file /home/vmail/ 10.0.0.x:/home/vmail I run it in a script for e,g every 1 min. By doing so if an email is deleted on the live server it still remain on the backup server. Then I give each mail file a retention period of n days so that it automatically gets deleted from the backup server.
I use a similar strategy, with a more relaxed synchronization. But instead of setting an expire on backup system, I simply do a rsync --del every night.
Ciao, luigi
/ +--[Luigi Rosa]-- \
Oo-ta goo-ta, Solo? --Greedo, "Star Wars" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAktidI8ACgkQ3kWu7Tfl6ZRfEACfRfJR4z99TIEgylKSywVAa4px l1kAnRA6xyigbYl+BVZoi6d5Vi5/n4Yk =f8lP -----END PGP SIGNATURE-----
Hi Luigi,
Thats great. Can you please explain how you do the restore?
I plan to do restore of 1 or more mail files. If I simply copy a mailfile e.g "MailDir/cur/my-email-file:2,S" to "MailDir/cur/" on the live system, are there issues associated with it ?
Thanks CoolAtt
Date: Fri, 29 Jan 2010 06:39:31 +0100 From: lists@luigirosa.com To: dovecot@dovecot.org Subject: Re: [Dovecot] Anyone successfully setup Continous Backup of mailboxes using rsync ?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
CoolAtt NNA said the following on 29/01/10 06:21:
rsync -avt --log-file=$log_file /home/vmail/ 10.0.0.x:/home/vmail I run it in a script for e,g every 1 min. By doing so if an email is deleted on the live server it still remain on the backup server. Then I give each mail file a retention period of n days so that it automatically gets deleted from the backup server.
I use a similar strategy, with a more relaxed synchronization. But instead of setting an expire on backup system, I simply do a rsync --del every night.
Ciao, luigi
/ +--[Luigi Rosa]-- \
Oo-ta goo-ta, Solo? --Greedo, "Star Wars" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAktidI8ACgkQ3kWu7Tfl6ZRfEACfRfJR4z99TIEgylKSywVAa4px l1kAnRA6xyigbYl+BVZoi6d5Vi5/n4Yk =f8lP -----END PGP SIGNATURE-----
Hotmail: Trusted email with Microsoft’s powerful SPAM protection. https://signup.live.com/signup.aspx?id=60969
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
CoolAtt NNA said the following on 29/01/10 06:47:
Can you please explain how you do the restore?
I always restore at folder level, not mail level: in this case is the user that choose what to restore and I don't need to read the user's mail.
I create a .Restore directory and copy into it the restored folder files: the users simply sees a Restore folder and do whatevere he/she wants.
If I simply copy a mailfile e.g "MailDir/cur/my-email-file:2,S" to "MailDir/cur/" on the live system, are there issues associated with it ?
With a standard configuration there are no issues at all. Check that the permissions and user/group of restored files are ok.
Ciao, luigi
/ +--[Luigi Rosa]-- \
A government big enough to give you everything you want is big enough to take from you everything you have. --Gerald R. Ford -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAktieHYACgkQ3kWu7Tfl6ZSktwCdHZqyD8+q3hOR43OYhz+qPxiC jisAoKIcZEQ+vL1RQnInjByzUZb5yvEg =riZr -----END PGP SIGNATURE-----
It means if I create a new mailbox and I restore the emails at mail level(individual files) ,will it work correctly? And if I restore the same mail to an existing mailbox ??
Thanks CoolAtt
Date: Fri, 29 Jan 2010 06:56:06 +0100 From: lists@luigirosa.com To: dovecot@dovecot.org Subject: Re: [Dovecot] Anyone successfully setup Continous Backup of mailboxes using rsync ?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
CoolAtt NNA said the following on 29/01/10 06:47:
Can you please explain how you do the restore?
I always restore at folder level, not mail level: in this case is the user that choose what to restore and I don't need to read the user's mail.
I create a .Restore directory and copy into it the restored folder files: the users simply sees a Restore folder and do whatevere he/she wants.
If I simply copy a mailfile e.g "MailDir/cur/my-email-file:2,S" to "MailDir/cur/" on the live system, are there issues associated with it ?
With a standard configuration there are no issues at all. Check that the permissions and user/group of restored files are ok.
Ciao, luigi
/ +--[Luigi Rosa]-- \
A government big enough to give you everything you want is big enough to take from you everything you have. --Gerald R. Ford -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAktieHYACgkQ3kWu7Tfl6ZSktwCdHZqyD8+q3hOR43OYhz+qPxiC jisAoKIcZEQ+vL1RQnInjByzUZb5yvEg =riZr -----END PGP SIGNATURE-----
Hotmail: Trusted email with powerful SPAM protection. https://signup.live.com/signup.aspx?id=60969
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
CoolAtt NNA said the following on 29/01/10 07:18:
It means if I create a new mailbox and I restore the emails at mail level(individual files) ,will it work correctly? And if I restore the same mail to an existing mailbox ??
Not a new mailbox, a new mail folder (maildir directory) of the actual account.
I am assuming you are using IMAP ad maildir, of course.
Ciao, luigi
/ +--[Luigi Rosa]-- \
Scott me up, Beamie! -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAktifjUACgkQ3kWu7Tfl6ZQalACfaEsS0ME6ab27uks49W6ERj59 yAcAnRDmaN3ZCKAEzJrK41CjZK28B3aT =O5hj -----END PGP SIGNATURE-----
Am using Dovecot & the maildir format.
Create a new folder like the following ??
vmail/coolatt/Maildir/restored-emails
Date: Fri, 29 Jan 2010 07:20:37 +0100 From: lists@luigirosa.com To: dovecot@dovecot.org Subject: Re: [Dovecot] Anyone successfully setup Continous Backup of mailboxes using rsync ?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
CoolAtt NNA said the following on 29/01/10 07:18:
It means if I create a new mailbox and I restore the emails at mail level(individual files) ,will it work correctly? And if I restore the same mail to an existing mailbox ??
Not a new mailbox, a new mail folder (maildir directory) of the actual account.
I am assuming you are using IMAP ad maildir, of course.
Ciao, luigi
/ +--[Luigi Rosa]-- \
Scott me up, Beamie! -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAktifjUACgkQ3kWu7Tfl6ZQalACfaEsS0ME6ab27uks49W6ERj59 yAcAnRDmaN3ZCKAEzJrK41CjZK28B3aT =O5hj -----END PGP SIGNATURE-----
Hotmail: Powerful Free email with security by Microsoft. https://signup.live.com/signup.aspx?id=60969
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
CoolAtt NNA said the following on 29/01/10 07:39:
Create a new folder like the following ??
vmail/coolatt/Maildir/restored-emails
Actually vmail/coolatt/Maildir/.restored-emails would be better :) Done that, you copy from the folder to be restored cur, tmp, new.
Some email clients need to shutdown and restart to be aware of the new mail folder.
Ciao, luigi
/ +--[Luigi Rosa]-- \
Moneypenny: In the conference room. Something pretty big. Every double-o man in Europe has been rushed in. And the home secretary too! James Bond: His wife probably lost her dog. --"007 Thunderball" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAktihLQACgkQ3kWu7Tfl6ZSMzQCghKGCWeqflRH4SatM/YItp3HG Ba0AoMFM4CVAdHtvsjNv7k0FLWFeYlw+ =6eIt -----END PGP SIGNATURE-----
On Fri, 2010-01-29 at 10:21 +0500, CoolAtt NNA wrote:
Hi Alex,
I mirror the mailboxes as follows:
rsync -avt --log-file=$log_file /home/vmail/ 10.0.0.x:/home/vmail I run it in a script for e,g every 1 min. By doing so if an email is deleted on the live server it still remain on the backup server. Then I give each mail file a retention period of n days so that it automatically gets deleted from the backup server.
The main potential problem with that is that if a message's flag is changed, the filename changes so it's now stored twice in the backup server. So when restoring mails, be sure not to restore duplicates.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Timo Sirainen said the following on 29/01/10 06:48:
The main potential problem with that is that if a message's flag is changed, the filename changes so it's now stored twice in the backup server. So when restoring mails, be sure not to restore duplicates.
Using a different dedicated directory for restored messages could lead to the same problem?
Ciao, luigi
/ +--[Luigi Rosa]-- \
Dr. Frederick Frankenstein: What a filthy job. Igor: Could be worse. Dr. Frederick Frankenstein: How? Igor: Could be raining. --Young Frankenstein -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAktieNIACgkQ3kWu7Tfl6ZT4CgCgx47smwd+UkL+UsgeAviqj6pq UacAmwa/Fy4zF6FTUc6MoWxvDAorzk2q =xnhw -----END PGP SIGNATURE-----
On 29.1.2010, at 7.57, Luigi Rosa wrote:
Timo Sirainen said the following on 29/01/10 06:48:
The main potential problem with that is that if a message's flag is changed, the filename changes so it's now stored twice in the backup server. So when restoring mails, be sure not to restore duplicates.
Using a different dedicated directory for restored messages could lead to the same problem?
If you didn't give --delete parameter to rsync, there could be duplicates, yes. Dovecot just logs about them and gives new UIDs to them, so you'll end up having duplicate mails. I suppose some day I should just change it to log a warning, but delete if the files are otherwise identical.
On 29.01.2010 08:59, Timo Sirainen wrote:
On 29.1.2010, at 7.57, Luigi Rosa wrote:
Timo Sirainen said the following on 29/01/10 06:48:
The main potential problem with that is that if a message's flag is changed, the filename changes so it's now stored twice in the backup server. So when restoring mails, be sure not to restore duplicates.
Using a different dedicated directory for restored messages could lead to the same problem?
If you didn't give --delete parameter to rsync, there could be duplicates, yes. Dovecot just logs about them and gives new UIDs to them, so you'll end up having duplicate mails. I suppose some day I should just change it to log a warning, but delete if the files are otherwise identical.
Using rsync will have the flags issue and others to be dealt with; lvm (or any other kind) of snapshot and rdiff-backup gives you a consistent snapshot, as well as the option of saving a lot of space and time as only changes gets backed up. As a bonus you can also restore anything to a given point in time, either induvidual messages or whole folders or even the whole mailbox.
-A
For the duplicates , when an email is flagged , am working on a script that will remove the duplicates.
Am thinking about the following files: dovecot.index dovecot.index.cache dovecot.index.log maildirsize subscriptions dovecot-uidlist
Will they affect the restore process ??
========= CoolAtt
Date: Fri, 29 Jan 2010 14:20:05 +0100 From: alexh@boxed.no To: dovecot@dovecot.org CC: tss@iki.fi; lists@luigirosa.com Subject: Re: [Dovecot] Anyone successfully setup Continous Backup of mailboxes using rsync ?
On 29.01.2010 08:59, Timo Sirainen wrote:
On 29.1.2010, at 7.57, Luigi Rosa wrote:
Timo Sirainen said the following on 29/01/10 06:48:
The main potential problem with that is that if a message's flag is changed, the filename changes so it's now stored twice in the backup server. So when restoring mails, be sure not to restore duplicates.
Using a different dedicated directory for restored messages could lead to the same problem?
If you didn't give --delete parameter to rsync, there could be duplicates, yes. Dovecot just logs about them and gives new UIDs to them, so you'll end up having duplicate mails. I suppose some day I should just change it to log a warning, but delete if the files are otherwise identical.
Using rsync will have the flags issue and others to be dealt with; lvm (or any other kind) of snapshot and rdiff-backup gives you a consistent snapshot, as well as the option of saving a lot of space and time as only changes gets backed up. As a bonus you can also restore anything to a given point in time, either induvidual messages or whole folders or even the whole mailbox.
-A
Hotmail: Powerful Free email with security by Microsoft. https://signup.live.com/signup.aspx?id=60969
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
CoolAtt NNA said the following on 02/02/10 08:02:
Am thinking about the following files: dovecot.index dovecot.index.cache dovecot.index.log maildirsize subscriptions dovecot-uidlist
Will they affect the restore process ??
Based on some experience, if you remove dovecot-uidlist AND the user uses POP3 AND the user leaves mail on server, the mail client could download the messages again because it sees them as "unread".
Ciao, luigi
/ +--[Luigi Rosa]-- \
Computers don't actually think. You just think they think. (We think.) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAktnz3EACgkQ3kWu7Tfl6ZQRmgCeI2NJ8Zu9wDsnNK0Qirw4NBEo LJYAoMImNr0Ni/cgt7VQcYCyW4+NuzEA =1MO/ -----END PGP SIGNATURE-----
luigi, can you tell me what will happen if I create a directory in a user's mailbox and restore some emails there. now if the user's quota is x mb & after restoration the mailbox becomes greater than x mb. will this cause any prob ?
Date: Tue, 2 Feb 2010 08:08:33 +0100 From: lists@luigirosa.com To: dovecot@dovecot.org Subject: Re: [Dovecot] Anyone successfully setup Continous Backup of mailboxes using rsync ?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
CoolAtt NNA said the following on 02/02/10 08:02:
Am thinking about the following files: dovecot.index dovecot.index.cache dovecot.index.log maildirsize subscriptions dovecot-uidlist
Will they affect the restore process ??
Based on some experience, if you remove dovecot-uidlist AND the user uses POP3 AND the user leaves mail on server, the mail client could download the messages again because it sees them as "unread".
Ciao, luigi
/ +--[Luigi Rosa]-- \
Computers don't actually think. You just think they think. (We think.) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAktnz3EACgkQ3kWu7Tfl6ZQRmgCeI2NJ8Zu9wDsnNK0Qirw4NBEo LJYAoMImNr0Ni/cgt7VQcYCyW4+NuzEA =1MO/ -----END PGP SIGNATURE-----
Hotmail: Trusted email with powerful SPAM protection. https://signup.live.com/signup.aspx?id=60969
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
CoolAtt NNA said the following on 02/02/10 08:36:
luigi, can you tell me what will happen if I create a directory in a user's mailbox and restore some emails there. now if the user's quota is x mb & after restoration the mailbox becomes greater than x mb. will this cause any prob ?
Unfortunately I do not have any experience with quota.
I think that this scenario is like a situation where an external program (or a MTA) creates files in the maildir structure without using Dovecot API interface.
I think that Dovecot sees the directory change and updates the quota data, but it's just a guess.
Ciao, luigi
/ +--[Luigi Rosa]-- \
A cynic is a man who, when he smells flowers, looks around for a coffin. --H.L. Mencken -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAktn12kACgkQ3kWu7Tfl6ZRAZgCeJIGzAXapzXt6DicafmW3MzPx yMQAn2u8GTjcQrGCBw94YcQBoSrG6Ukk =aqUw -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Tue, 2 Feb 2010, CoolAtt NNA wrote:
luigi, can you tell me what will happen if I create a directory in a user's mailbox and restore some emails there. now if the user's quota is x mb & after restoration the mailbox becomes greater than x mb. will this cause any prob ?
This depends on your quota calculation mechanism.
As you mention "maildirsize", see http://wiki.dovecot.org/Quota/Maildir You should remove that file, unless you use that file to store the user limit, Dovecot will recalculate the size then.
I don't use Maildir++ quota, but Timo usually answers that way :-) the list archive should have several hints.
===
If you use the filesystem backend, the system enforces the quota. There are (had been) systems, that does not enforce quota, if the user gets overquota by non-user actions, e.g. when root "chown"s files.
Regards,
Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux)
iQEVAwUBS2frcL+Vh58GPL/cAQJEdQgApfXH2KQBVO4+N+26gA6asEMBAUhZz1Ux zhHnw6IHmvullfQlXU9d7Q1KYvLglzZtKhfBK9ykk5D70P+QuQY5YdVoL9JA3E5f 25nPSwv9rp56VrJqOaX//rDvmbFCW45rpa/golXsXV9vYo648oN3Z4ivxIbuKWpA +p/raG5llC7yhd41jegZW10U7WooG19DR987blNRSrV4yOlogug7nKvje7/Y22XK wsSPdm0hqX/SZrNlKaPb11iKbL2xprhmLRs9wE6TnlQM+zX2dx/rIosHH4S2+6P4 Jh4owUzvsN4jeY0yq6xrRZ1sqwl9Cewwg+xKKAmY81zfsHV6ZZKtMg== =luFR -----END PGP SIGNATURE-----
On Fri, Jan 29, 2010 at 09:59:22AM +0200, Timo Sirainen wrote:
If you didn't give --delete parameter to rsync, there could be duplicates, yes. Dovecot just logs about them and gives new UIDs to them, so you'll end up having duplicate mails. I suppose some day I should just change it to log a warning, but delete if the files are otherwise identical.
Would this be easy to change? It solves this problem, although perhaps not in the most efficient way...
Geert
-- Geert Hendrickx -=- ghen@telenet.be -=- PGP: 0xC4BB9E9F This e-mail was composed using 100% recycled spam messages!
On Wed, 2010-02-10 at 15:14 +0100, Geert Hendrickx wrote:
On Fri, Jan 29, 2010 at 09:59:22AM +0200, Timo Sirainen wrote:
If you didn't give --delete parameter to rsync, there could be duplicates, yes. Dovecot just logs about them and gives new UIDs to them, so you'll end up having duplicate mails. I suppose some day I should just change it to log a warning, but delete if the files are otherwise identical.
Would this be easy to change? It solves this problem, although perhaps not in the most efficient way...
Sure, it'll be easy to change, for whoever does the change. :) I've a patch somewhere in mailing list archives that just changes the code to assume identical names means identical mails.
Timo Sirainen tss@iki.fi:
The main potential problem with that is that if a message's flag is changed, the filename changes so it's now stored twice in the backup server. So when restoring mails, be sure not to restore duplicates.
I use the --fuzzy flag to handle the renaming issue. Here's my crontab entry on the machine holding the backup: 0 * * * * rsync --archive --fuzzy --delete-after imapserver:Maildir .
As you can see I only sync once an hour. I don't have enough email traffic to justify doing it continously.
The --delete-after is so rsync won't delete files that have been removed on the source, before figuring out that a file has been renamed and not been deleted.
For email email am having the email duplication issue when am doing the backup. Am having multiple copies of a mail file on the backup server.
Does it mean that if I use "--fuzzy --delete-after" only the most recent mailfile will be on the backup server ??
e.g On LIVE server : my-file:2,S On BACKUP server : my-file:2,S + my-file:2, + my-file:2,SF
"my-file:2,S" is the current file. So on the backup only "my-file:2,S" must exist. Will it solve the prob ??
To: dovecot@dovecot.org From: sb@dod.no Date: Fri, 29 Jan 2010 08:58:27 +0100 Subject: Re: [Dovecot] Anyone successfully setup Continous Backup of mailboxes using rsync ?
Timo Sirainen tss@iki.fi:
The main potential problem with that is that if a message's flag is changed, the filename changes so it's now stored twice in the backup server. So when restoring mails, be sure not to restore duplicates.
I use the --fuzzy flag to handle the renaming issue. Here's my crontab entry on the machine holding the backup: 0 * * * * rsync --archive --fuzzy --delete-after imapserver:Maildir .
As you can see I only sync once an hour. I don't have enough email traffic to justify doing it continously.
The --delete-after is so rsync won't delete files that have been removed on the source, before figuring out that a file has been renamed and not been deleted.
Hotmail: Powerful Free email with security by Microsoft. https://signup.live.com/signup.aspx?id=60969
I tried the "--fuzzy --delete-after" options. It works fine when the files are renamed. But the prob is that when a mail is deleted it gets deleted from the backup server as well.
Is there a way to prevent deletion when using these options?
Thanks CoolAtt
To: dovecot@dovecot.org From: sb@dod.no Date: Fri, 29 Jan 2010 08:58:27 +0100 Subject: Re: [Dovecot] Anyone successfully setup Continous Backup of mailboxes using rsync ?
Timo Sirainen tss@iki.fi:
The main potential problem with that is that if a message's flag is changed, the filename changes so it's now stored twice in the backup server. So when restoring mails, be sure not to restore duplicates.
I use the --fuzzy flag to handle the renaming issue. Here's my crontab entry on the machine holding the backup: 0 * * * * rsync --archive --fuzzy --delete-after imapserver:Maildir .
As you can see I only sync once an hour. I don't have enough email traffic to justify doing it continously.
The --delete-after is so rsync won't delete files that have been removed on the source, before figuring out that a file has been renamed and not been deleted.
Hotmail: Trusted email with Microsoft’s powerful SPAM protection. https://signup.live.com/signup.aspx?id=60969
On Fri, Jan 29, 2010 at 10:21:39AM +0500, CoolAtt NNA wrote:
Hi Alex,
I mirror the mailboxes as follows:
rsync -avt --log-file=$log_file /home/vmail/ 10.0.0.x:/home/vmail I run it in a script for e,g every 1 min.
If running this from cron every minute, make sure to use a lockfile or something to make sure you don't end up with multiple rsync jobs running at the same time...
Geert
-- Geert Hendrickx -=- ghen@telenet.be -=- PGP: 0xC4BB9E9F This e-mail was composed using 100% recycled spam messages!
On 1/28/10 12:07 AM, CoolAtt NNA wrote:
Hi All..
Am working on continous backup of mailboxes using rsync(for e.g by running rsync every 2 min)
Things gets more complicated when users create Subfolders in INBOX , SENT , etc..
How?
If anyone among you did that plz guide & advise.
Sure:
rsync -e 'ssh -ax -i mykey'
-a --delete --stats
mailserver.host:/var/mail/vmail
/localbackup
~Seth
On Thu, 28 Jan 2010 13:07:32 +0500 CoolAtt NNA coolatt@live.com wrote:
Hi All..
Am working on continous backup of mailboxes using rsync(for e.g by running rsync every 2 min)
Things gets more complicated when users create Subfolders in INBOX , SENT , etc..
If anyone among you did that plz guide & advise.
Thanks CoolAtt
Take a look on ccollect project http://unix.schottelius.org/ccollect/. It made incremental backups using rsync. That will be working for maildirs perfectly.
Your E-mail and More On-the-Go. Get Windows Live Hotmail Free. https://signup.live.com/signup.aspx?id=60969
Le Thu, 28 Jan 2010 13:07:32 +0500, CoolAtt NNA coolatt@live.com a écrit :
Am working on continous backup of mailboxes using rsync(for e.g by running rsync every 2 min) Things gets more complicated when users create Subfolders in INBOX , SENT , etc.. If anyone among you did that plz guide & advise.
I'm going to share what I've did. This is not rocket science but enough for my SOHO needs. Each user mail boxes are stored in ~HOME/mail/
# cat /opt/crontab-scripts/rsync-local-email-backup.sh [...] for i in /home/*/mail/ rsync -a --delete $i ${PATHTOSAFEPLACE}/$(date +%u)/${i}/ RETURNCODE=$? [...] done [...]
In clear, each day, I copy all mailboxes to a different (date +%u -> 0 to 6) safe sub-directory. I admit, it's very simple but enough for my needs and certainly could be improved..
Best regards,
-- LMJ "May the source be with you my young padawan" http://sites.google.com/site/imatruelinuxmasterjedi/
"L.M.J" linuxmasterjedi@free.fr:
...
rsync -a --delete $i ${PATHTOSAFEPLACE}/$(date +%u)/${i}/
...
The downside to not using --fuzzy, is that the entire article is transferred again if a message names changes slightly (ie. if we're talking maildir here), as a result of status change.
Note if you decide to use --fuzzy and want to delete articles, you should chose --delete-after. Otherwise rsync will delete the candidates for fuzzy name matching, before having a chance to make use of them.
Le Sun, 31 Jan 2010 21:04:12 +0100, Steinar Bang sb@dod.no a écrit :
The downside to not using --fuzzy, is that the entire article is transferred again if a message names changes slightly (ie. if we're talking maildir here), as a result of status change.
Note if you decide to use --fuzzy and want to delete articles, you should chose --delete-after. Otherwise rsync will delete the candidates for fuzzy name matching, before having a chance to make use of them.
Unfortunately, I'm using mbox :-/ In that case, --fuzzy got less interest but --delete-after is definitely a good one.
Thanks
-- LMJ "May the source be with you my young padawan" http://sites.google.com/site/imatruelinuxmasterjedi/
"L.M.J" linuxmasterjedi@free.fr:
Unfortunately, I'm using mbox :-/ In that case, --fuzzy got less interest but --delete-after is definitely a good one.
If you're talking mbox then -fuzzy has no interest, that's right. But rsync is still a good match, actually a better match than with maildir. The rsync program is very good at only transferring the changed part of a file.
On January 31, 2010 9:04:12 PM +0100 Steinar Bang sb@dod.no wrote:
"L.M.J" linuxmasterjedi@free.fr:
...
rsync -a --delete $i ${PATHTOSAFEPLACE}/$(date +%u)/${i}/
...
The downside to not using --fuzzy, is that the entire article is transferred again if a message names changes slightly (ie. if we're talking maildir here), as a result of status change.
zfs really shines here. Only the filename change would be transmitted.
And if you're using opensolaris you can add in AVS to get real time continous synchronous replication if you want. (I would also consider having the MTA duplicate the mail to a 2nd mail server and use another method to synchronize deletions and flag changes.)
-frank
CoolAtt NNA wrote:
Hi All..
Am working on continous backup of mailboxes using rsync(for e.g by running rsync every 2 min)
Things gets more complicated when users create Subfolders in INBOX , SENT , etc..
If anyone among you did that plz guide & advise.
We have a slightly different method of doing a back-up, at least of people's inboxes (which is all I "archive").
I've written a program, which is run by cron once a minute, which searches for messages within all the users' inboxes and then hard-links them into a backup store elsewhere on the disk. Then, once a day a reaper process runs which checks for all files in that store with only one link and have not been accessed for a set amount of time (in our case 90 days) and then deletes those files. In other words, the messages are kept for 90 days after they have been deleted or moved out of the user's inbox.
I've attached the source code tar file as you or someone else may find it useful.
In your case you could rsync the back-up store instead of the user's main directory structure and maybe only do it once per day.
Steve
IT Systems Administrator, E-Mail:- steve@earth.ox.ac.uk Department of Earth Sciences, Tel:- +44 (0)1865 282110 University of Oxford, Parks Road, Oxford, UK. Fax:- +44 (0)1865 272072
On Wed, 2010-02-10 at 13:57 +0000, Stephen Usher wrote:
I've written a program, which is run by cron once a minute, which searches for messages within all the users' inboxes and then hard-links them into a backup store elsewhere on the disk. Then, once a day a reaper process runs which checks for all files in that store with only one link and have not been accessed for a set amount of time (in our case 90 days) and then deletes those files. In other words, the messages are kept for 90 days after they have been deleted or moved out of the user's inbox.
http://wiki.dovecot.org/Plugins/Lazyexpunge also does pretty much that, although for all mailboxes, not just for INBOX. And it doesn't automatically expire them.
On Feb 10, 2010, at 6:08 AM, Timo Sirainen wrote:
On Wed, 2010-02-10 at 13:57 +0000, Stephen Usher wrote:
I've written a program, which is run by cron once a minute, which
searches for messages within all the users' inboxes and then hard-links them
into a backup store elsewhere on the disk. Then, once a day a reaper process runs
which checks for all files in that store with only one link and have not been
accessed for a set amount of time (in our case 90 days) and then deletes those
files. In other words, the messages are kept for 90 days after they have been
deleted or moved out of the user's inbox.http://wiki.dovecot.org/Plugins/Lazyexpunge also does pretty much
that, although for all mailboxes, not just for INBOX. And it doesn't automatically expire them.
There's also rsnapshot which I've been using for hardlinked,
incremental backups for a couple years now. Very flexible and has
saved us a couple times.
-Terry
Terry Barnum digital OutPost San Diego, CA
http://www.dop.com 800/464-6434
On 2010-02-10 2:22 PM, Terry Barnum wrote:
There's also rsnapshot which I've been using for hardlinked, incremental backups for a couple years now. Very flexible and has saved us a couple times.
+1 for rsnapshot - although it doesn't answer the 'continuous' backup aspect of this thread...
--
Best regards,
Charles
On Feb 10, 2010, at 11:53 AM, Charles Marcus wrote:
On 2010-02-10 2:22 PM, Terry Barnum wrote:
There's also rsnapshot which I've been using for hardlinked,
incremental backups for a couple years now. Very flexible and has saved us a
couple times.+1 for rsnapshot - although it doesn't answer the 'continuous' backup aspect of this thread...
I'm pretty sure rsnapshot could be configured to run every x minutes.
-Terry
Terry Barnum digital OutPost San Diego, CA
http://www.dop.com 800/464-6434
from the wiki of Lazy Expunge: "The plugin is configured by defining namespaces where the mails are moved. You can decide if you want the namespaces to be visible to clients, or if you want to show them only via some special webmail interface. You can use either one or three namespaces"
How do I access the namespaces(the back up mailboxes) ? We are using Thunderbird,Microsoft Outlook & RoundCube Webmail as email clients.
From: terry@dop.com To: dovecot@dovecot.org Date: Wed, 10 Feb 2010 11:22:06 -0800 Subject: Re: [Dovecot] Anyone successfully setup Continous Backup of mailboxes using rsync ?
On Feb 10, 2010, at 6:08 AM, Timo Sirainen wrote:
On Wed, 2010-02-10 at 13:57 +0000, Stephen Usher wrote:
I've written a program, which is run by cron once a minute, which
searches for messages within all the users' inboxes and then hard-links them
into a backup store elsewhere on the disk. Then, once a day a reaper process runs
which checks for all files in that store with only one link and have not been
accessed for a set amount of time (in our case 90 days) and then deletes those
files. In other words, the messages are kept for 90 days after they have been
deleted or moved out of the user's inbox.http://wiki.dovecot.org/Plugins/Lazyexpunge also does pretty much
that, although for all mailboxes, not just for INBOX. And it doesn't automatically expire them.There's also rsnapshot which I've been using for hardlinked,
incremental backups for a couple years now. Very flexible and has
saved us a couple times.-Terry
Terry Barnum digital OutPost San Diego, CA
http://www.dop.com 800/464-6434
Hotmail: Trusted email with powerful SPAM protection. https://signup.live.com/signup.aspx?id=60969
On Thu, 2010-02-11 at 10:46 +0500, CoolAtt NNA wrote:
from the wiki of Lazy Expunge: "The plugin is configured by defining namespaces where the mails are moved. You can decide if you want the namespaces to be visible to clients, or if you want to show them only via some special webmail interface. You can use either one or three namespaces"
How do I access the namespaces(the back up mailboxes) ?
Well,
a) You can make the namespaces visible (list=yes) and they'll show up just like regular mailboxes. Might confuse users.
b) You make the namespaces hidden (list=no, hidden=yes) and the only way to access them is by explicitly configuring client's namespace path to that. That's of course a pretty painful thing to do. Maybe by using two different accounts that would be slightly better.
But the main reason for lazy_expunge is to allow a) custom-built webmails or b) sysadmins to do the unexpunging.
On 10/02/2010 19:22, Terry Barnum wrote:
There's also rsnapshot which I've been using for hardlinked, incremental backups for a couple years now. Very flexible and has saved us a couple times.
It's rarely mentioned, but I prefer snapback2 instead - basically the same idea, but it has a rotation schedule which is X daily, Y weekly and Z Monthly backups - more efficient in my mind
Ed W
i use drbd to keep the mailstore redundant and rsnapshot for backup.
participants (17)
-
alex handle
-
Alexander Hoogerhuis
-
Charles Marcus
-
CoolAtt NNA
-
Ed W
-
Frank Cusack
-
Geert Hendrickx
-
L.M.J
-
Luigi Rosa
-
Nikita Koshikov
-
Seth Mattinen
-
Steffen Kaiser
-
Steinar Bang
-
Stephen Usher
-
Stewart Dean
-
Terry Barnum
-
Timo Sirainen