Hi Alex,

I mirror the mailboxes as follows:

rsync -avt --log-file=$log_file /home/vmail/ 10.0.0.x:/home/vmail I run it in a script for e,g every 1 min. By doing so if an email is deleted on the live server it still remain on the backup server. Then I give each mail file a retention period of n days so that it automatically gets deleted from the backup server.

CoolAtt

Date: Thu, 28 Jan 2010 20:57:15 +0100 From: alexh@boxed.no To: dovecot@dovecot.org Subject: Re: [Dovecot] Anyone successfully setup Continous Backup of mailboxes using rsync ?

On 28.01.2010 09:07, CoolAtt NNA wrote:

...

Hi All..

Am working on continous backup of mailboxes using rsync(for e.g by running rsync every 2 min)

Things gets more complicated when users create Subfolders in INBOX , SENT , etc..

If anyone among you did that plz guide& advise.

You don't state much on what this runs on, but I do snapshot backups of my IMAP server consistently by using LVM, and just match my cronjob to snapshot it at a few given times, mounting the snapshot and backing up with rdiff-backup.

Very simple, very easy, and very realiable. :)

...

Thanks CoolAtt

-A

Hotmail: Trusted email with Microsoft’s powerful SPAM protection. https://signup.live.com/signup.aspx?id=60969

Hi Luigi,

Thats great. Can you please explain how you do the restore?

I plan to do restore of 1 or more mail files. If I simply copy a mailfile e.g "MailDir/cur/my-email-file:2,S" to "MailDir/cur/" on the live system, are there issues associated with it ?

Thanks CoolAtt

Date: Fri, 29 Jan 2010 06:39:31 +0100 From: lists@luigirosa.com To: dovecot@dovecot.org Subject: Re: [Dovecot] Anyone successfully setup Continous Backup of mailboxes using rsync ?

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

CoolAtt NNA said the following on 29/01/10 06:21:

...

rsync -avt --log-file=$log_file /home/vmail/ 10.0.0.x:/home/vmail I run it in a script for e,g every 1 min. By doing so if an email is deleted on the live server it still remain on the backup server. Then I give each mail file a retention period of n days so that it automatically gets deleted from the backup server.

I use a similar strategy, with a more relaxed synchronization. But instead of setting an expire on backup system, I simply do a rsync --del every night.

Ciao, luigi

---

/ +--[Luigi Rosa]-- \

Oo-ta goo-ta, Solo? --Greedo, "Star Wars" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAktidI8ACgkQ3kWu7Tfl6ZRfEACfRfJR4z99TIEgylKSywVAa4px l1kAnRA6xyigbYl+BVZoi6d5Vi5/n4Yk =f8lP -----END PGP SIGNATURE-----

Hotmail: Trusted email with Microsoft’s powerful SPAM protection. https://signup.live.com/signup.aspx?id=60969

It means if I create a new mailbox and I restore the emails at mail level(individual files) ,will it work correctly? And if I restore the same mail to an existing mailbox ??

Thanks CoolAtt

Date: Fri, 29 Jan 2010 06:56:06 +0100 From: lists@luigirosa.com To: dovecot@dovecot.org Subject: Re: [Dovecot] Anyone successfully setup Continous Backup of mailboxes using rsync ?

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

CoolAtt NNA said the following on 29/01/10 06:47:

...

Can you please explain how you do the restore?

I always restore at folder level, not mail level: in this case is the user that choose what to restore and I don't need to read the user's mail.

I create a .Restore directory and copy into it the restored folder files: the users simply sees a Restore folder and do whatevere he/she wants.

...

If I simply copy a mailfile e.g "MailDir/cur/my-email-file:2,S" to "MailDir/cur/" on the live system, are there issues associated with it ?

With a standard configuration there are no issues at all. Check that the permissions and user/group of restored files are ok.

Ciao, luigi

---

/ +--[Luigi Rosa]-- \

A government big enough to give you everything you want is big enough to take from you everything you have. --Gerald R. Ford -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAktieHYACgkQ3kWu7Tfl6ZSktwCdHZqyD8+q3hOR43OYhz+qPxiC jisAoKIcZEQ+vL1RQnInjByzUZb5yvEg =riZr -----END PGP SIGNATURE-----

Hotmail: Trusted email with powerful SPAM protection. https://signup.live.com/signup.aspx?id=60969

Am using Dovecot & the maildir format.

Create a new folder like the following ??

vmail/coolatt/Maildir/restored-emails

Date: Fri, 29 Jan 2010 07:20:37 +0100 From: lists@luigirosa.com To: dovecot@dovecot.org Subject: Re: [Dovecot] Anyone successfully setup Continous Backup of mailboxes using rsync ?

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

CoolAtt NNA said the following on 29/01/10 07:18:

...

It means if I create a new mailbox and I restore the emails at mail level(individual files) ,will it work correctly? And if I restore the same mail to an existing mailbox ??

Not a new mailbox, a new mail folder (maildir directory) of the actual account.

I am assuming you are using IMAP ad maildir, of course.

Ciao, luigi

---

/ +--[Luigi Rosa]-- \

Scott me up, Beamie! -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAktifjUACgkQ3kWu7Tfl6ZQalACfaEsS0ME6ab27uks49W6ERj59 yAcAnRDmaN3ZCKAEzJrK41CjZK28B3aT =O5hj -----END PGP SIGNATURE-----

Hotmail: Powerful Free email with security by Microsoft. https://signup.live.com/signup.aspx?id=60969

On Fri, 2010-01-29 at 10:21 +0500, CoolAtt NNA wrote:

Hi Alex,

I mirror the mailboxes as follows:

rsync -avt --log-file=$log_file /home/vmail/ 10.0.0.x:/home/vmail I run it in a script for e,g every 1 min. By doing so if an email is deleted on the live server it still remain on the backup server. Then I give each mail file a retention period of n days so that it automatically gets deleted from the backup server.

The main potential problem with that is that if a message's flag is changed, the filename changes so it's now stored twice in the backup server. So when restoring mails, be sure not to restore duplicates.

On 29.1.2010, at 7.57, Luigi Rosa wrote:

Timo Sirainen said the following on 29/01/10 06:48:

...

The main potential problem with that is that if a message's flag is changed, the filename changes so it's now stored twice in the backup server. So when restoring mails, be sure not to restore duplicates.

Using a different dedicated directory for restored messages could lead to the same problem?

If you didn't give --delete parameter to rsync, there could be duplicates, yes. Dovecot just logs about them and gives new UIDs to them, so you'll end up having duplicate mails. I suppose some day I should just change it to log a warning, but delete if the files are otherwise identical.

For the duplicates , when an email is flagged , am working on a script that will remove the duplicates.

Am thinking about the following files: dovecot.index dovecot.index.cache dovecot.index.log maildirsize subscriptions dovecot-uidlist

Will they affect the restore process ??

========= CoolAtt

Date: Fri, 29 Jan 2010 14:20:05 +0100 From: alexh@boxed.no To: dovecot@dovecot.org CC: tss@iki.fi; lists@luigirosa.com Subject: Re: [Dovecot] Anyone successfully setup Continous Backup of mailboxes using rsync ?

On 29.01.2010 08:59, Timo Sirainen wrote:

...

On 29.1.2010, at 7.57, Luigi Rosa wrote:

...

Timo Sirainen said the following on 29/01/10 06:48:

...

The main potential problem with that is that if a message's flag is changed, the filename changes so it's now stored twice in the backup server. So when restoring mails, be sure not to restore duplicates.

Using a different dedicated directory for restored messages could lead to the same problem?

If you didn't give --delete parameter to rsync, there could be duplicates, yes. Dovecot just logs about them and gives new UIDs to them, so you'll end up having duplicate mails. I suppose some day I should just change it to log a warning, but delete if the files are otherwise identical.

Using rsync will have the flags issue and others to be dealt with; lvm (or any other kind) of snapshot and rdiff-backup gives you a consistent snapshot, as well as the option of saving a lot of space and time as only changes gets backed up. As a bonus you can also restore anything to a given point in time, either induvidual messages or whole folders or even the whole mailbox.

-A

Hotmail: Powerful Free email with security by Microsoft. https://signup.live.com/signup.aspx?id=60969

luigi, can you tell me what will happen if I create a directory in a user's mailbox and restore some emails there. now if the user's quota is x mb & after restoration the mailbox becomes greater than x mb. will this cause any prob ?

Date: Tue, 2 Feb 2010 08:08:33 +0100 From: lists@luigirosa.com To: dovecot@dovecot.org Subject: Re: [Dovecot] Anyone successfully setup Continous Backup of mailboxes using rsync ?

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

CoolAtt NNA said the following on 02/02/10 08:02:

...

Am thinking about the following files: dovecot.index dovecot.index.cache dovecot.index.log maildirsize subscriptions dovecot-uidlist

Will they affect the restore process ??

Based on some experience, if you remove dovecot-uidlist AND the user uses POP3 AND the user leaves mail on server, the mail client could download the messages again because it sees them as "unread".

Ciao, luigi

---

/ +--[Luigi Rosa]-- \

Computers don't actually think. You just think they think. (We think.) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAktnz3EACgkQ3kWu7Tfl6ZQRmgCeI2NJ8Zu9wDsnNK0Qirw4NBEo LJYAoMImNr0Ni/cgt7VQcYCyW4+NuzEA =1MO/ -----END PGP SIGNATURE-----

Hotmail: Trusted email with powerful SPAM protection. https://signup.live.com/signup.aspx?id=60969

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Tue, 2 Feb 2010, CoolAtt NNA wrote:

luigi, can you tell me what will happen if I create a directory in a user's mailbox and restore some emails there. now if the user's quota is x mb & after restoration the mailbox becomes greater than x mb. will this cause any prob ?

This depends on your quota calculation mechanism.

As you mention "maildirsize", see http://wiki.dovecot.org/Quota/Maildir You should remove that file, unless you use that file to store the user limit, Dovecot will recalculate the size then.

I don't use Maildir++ quota, but Timo usually answers that way :-) the list archive should have several hints.

===

If you use the filesystem backend, the system enforces the quota. There are (had been) systems, that does not enforce quota, if the user gets overquota by non-user actions, e.g. when root "chown"s files.

Regards,

Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux)

iQEVAwUBS2frcL+Vh58GPL/cAQJEdQgApfXH2KQBVO4+N+26gA6asEMBAUhZz1Ux zhHnw6IHmvullfQlXU9d7Q1KYvLglzZtKhfBK9ykk5D70P+QuQY5YdVoL9JA3E5f 25nPSwv9rp56VrJqOaX//rDvmbFCW45rpa/golXsXV9vYo648oN3Z4ivxIbuKWpA +p/raG5llC7yhd41jegZW10U7WooG19DR987blNRSrV4yOlogug7nKvje7/Y22XK wsSPdm0hqX/SZrNlKaPb11iKbL2xprhmLRs9wE6TnlQM+zX2dx/rIosHH4S2+6P4 Jh4owUzvsN4jeY0yq6xrRZ1sqwl9Cewwg+xKKAmY81zfsHV6ZZKtMg== =luFR -----END PGP SIGNATURE-----

On Wed, 2010-02-10 at 15:14 +0100, Geert Hendrickx wrote:

On Fri, Jan 29, 2010 at 09:59:22AM +0200, Timo Sirainen wrote:

...

If you didn't give --delete parameter to rsync, there could be duplicates, yes. Dovecot just logs about them and gives new UIDs to them, so you'll end up having duplicate mails. I suppose some day I should just change it to log a warning, but delete if the files are otherwise identical.

Would this be easy to change? It solves this problem, although perhaps not in the most efficient way...

Sure, it'll be easy to change, for whoever does the change. :) I've a patch somewhere in mailing list archives that just changes the code to assume identical names means identical mails.

For email email am having the email duplication issue when am doing the backup. Am having multiple copies of a mail file on the backup server.

Does it mean that if I use "--fuzzy --delete-after" only the most recent mailfile will be on the backup server ??

e.g On LIVE server : my-file:2,S On BACKUP server : my-file:2,S + my-file:2, + my-file:2,SF

"my-file:2,S" is the current file. So on the backup only "my-file:2,S" must exist. Will it solve the prob ??

To: dovecot@dovecot.org From: sb@dod.no Date: Fri, 29 Jan 2010 08:58:27 +0100 Subject: Re: [Dovecot] Anyone successfully setup Continous Backup of mailboxes using rsync ?

...

...

...

...

...

Timo Sirainen <tss@iki.fi>:

...

The main potential problem with that is that if a message's flag is changed, the filename changes so it's now stored twice in the backup server. So when restoring mails, be sure not to restore duplicates.

I use the --fuzzy flag to handle the renaming issue. Here's my crontab entry on the machine holding the backup: 0 * * * * rsync --archive --fuzzy --delete-after imapserver:Maildir .

As you can see I only sync once an hour. I don't have enough email traffic to justify doing it continously.

The --delete-after is so rsync won't delete files that have been removed on the source, before figuring out that a file has been renamed and not been deleted.

Hotmail: Powerful Free email with security by Microsoft. https://signup.live.com/signup.aspx?id=60969

On Fri, Jan 29, 2010 at 10:21:39AM +0500, CoolAtt NNA wrote:

Hi Alex,

I mirror the mailboxes as follows:

rsync -avt --log-file=$log_file /home/vmail/ 10.0.0.x:/home/vmail I run it in a script for e,g every 1 min.

If running this from cron every minute, make sure to use a lockfile or something to make sure you don't end up with multiple rsync jobs running at the same time...

Geert

-- Geert Hendrickx -=- ghen@telenet.be -=- PGP: 0xC4BB9E9F This e-mail was composed using 100% recycled spam messages!

...

...

...

...

"L.M.J" <linuxmasterjedi@free.fr>:

...

rsync -a --delete $i ${PATHTOSAFEPLACE}/$(date +%u)/${i}/

...

The downside to not using --fuzzy, is that the entire article is transferred again if a message names changes slightly (ie. if we're talking maildir here), as a result of status change.

Note if you decide to use --fuzzy and want to delete articles, you should chose --delete-after. Otherwise rsync will delete the candidates for fuzzy name matching, before having a chance to make use of them.

...

...

...

...

"L.M.J" <linuxmasterjedi@free.fr>:

Unfortunately, I'm using mbox :-/ In that case, --fuzzy got less interest but --delete-after is definitely a good one.

If you're talking mbox then -fuzzy has no interest, that's right. But rsync is still a good match, actually a better match than with maildir. The rsync program is very good at only transferring the changed part of a file.

On Feb 10, 2010, at 6:08 AM, Timo Sirainen wrote:

On Wed, 2010-02-10 at 13:57 +0000, Stephen Usher wrote:

...

I've written a program, which is run by cron once a minute, which
searches for messages within all the users' inboxes and then hard-links them
into a backup store elsewhere on the disk. Then, once a day a reaper process runs
which checks for all files in that store with only one link and have not been
accessed for a set amount of time (in our case 90 days) and then deletes those
files. In other words, the messages are kept for 90 days after they have been
deleted or moved out of the user's inbox.

http://wiki.dovecot.org/Plugins/Lazyexpunge also does pretty much
that, although for all mailboxes, not just for INBOX. And it doesn't automatically expire them.

There's also rsnapshot which I've been using for hardlinked,
incremental backups for a couple years now. Very flexible and has
saved us a couple times.

<http://www.rsnapshot.org>

-Terry

Terry Barnum digital OutPost San Diego, CA

http://www.dop.com 800/464-6434

On Feb 10, 2010, at 11:53 AM, Charles Marcus wrote:

On 2010-02-10 2:22 PM, Terry Barnum wrote:

...

There's also rsnapshot which I've been using for hardlinked,
incremental backups for a couple years now. Very flexible and has saved us a
couple times.

<http://www.rsnapshot.org>

+1 for rsnapshot - although it doesn't answer the 'continuous' backup aspect of this thread...

I'm pretty sure rsnapshot could be configured to run every x minutes.

-Terry

Terry Barnum digital OutPost San Diego, CA

http://www.dop.com 800/464-6434

On Thu, 2010-02-11 at 10:46 +0500, CoolAtt NNA wrote:

from the wiki of Lazy Expunge: "The plugin is configured by defining namespaces where the mails are moved. You can decide if you want the namespaces to be visible to clients, or if you want to show them only via some special webmail interface. You can use either one or three namespaces"

How do I access the namespaces(the back up mailboxes) ?

Well,

a) You can make the namespaces visible (list=yes) and they'll show up just like regular mailboxes. Might confuse users.

b) You make the namespaces hidden (list=no, hidden=yes) and the only way to access them is by explicitly configuring client's namespace path to that. That's of course a pretty painful thing to do. Maybe by using two different accounts that would be slightly better.

But the main reason for lazy_expunge is to allow a) custom-built webmails or b) sysadmins to do the unexpunging.