On 4.12.2011, at 19.41, Terry Carmen wrote:

...

...

If we can not safely restore from backup a user's mailbox with SiS enabled, we can not enable SiS. Any plan to include this backup recovery tool in dovecot 2.0.X or 2.1?

I'd first have to design it. And before designing it I'd need to look into how the backup softwares usually work.. If anyone has any ideas about this, I'd like to hear.

BackupPC uses rsync by default for *nix boxes.

No idea what SiS is, but I'm guessing you're running into the same problem as backing up any other open file with changing internal data that may be inconsistent.

Inconsistency is an issue, but it's not the biggest problem. It would be possible to write a tool that scans through all mails and makes sure everything is consistent after a restore. dbox mostly does this automatically already, but SIS would need a separate program to ensure its consistency.

SIS is anyway a single instance attachment storage. So that lets say you send one 100 MB pdf to 10 people, and it's stored only once in disk under /attachments/aa/bb/aabccddeeff-etc. Each people would have their own unique link under /attachments/, but all of them would be hard linked to a common file.

So the problem is mainly about restoring a single user's mails. The mail files are simple to restore, but then you need to figure out which attachments to restore. There's no simple way to know which attachment files belong to which users, so you need to scan through the mail files and see what attachments are referred to.

Also backing up the attachment links could be problematic if the backup system doesn't support hard links. Each attachment always has at least 2 links, so if the backup doesn't realize that it at minimum duplicates the space used by attachments.

So I was thinking that there probably could be some tool that during a user's backup it would write the attachments among the user's other files, so it would be easy to find all of the files needed for a restore. This would of course mean that backups can take a lot more space, because there's no SIS. Perhaps there could be some other alternatives .. like maybe not storing the attachments directly to backups, but add symlinks to them so they can be used to figure out what to restore. Or maybe the backing up wouldn't need a special tool, but the restoring tool could just read through the dbox files to see what attachments are also needed and write a list of them somewhere so they can be taken from backups as well.

I'm not really sure what is the best way.

El 04/12/11 21:16, Terry Carmen escribió:

...

So I was thinking that there probably could be some tool that during a user's backup it would write the attachments among the user's other files, so it would be easy to find all of the files needed for a restore. This would of course mean that backups can take a lot more space, because there's no SIS. Perhaps there could be some other

I see.

Instead of writing the links directly to the filesystem, why not keep a links list (not a linked list 8-)) file in each directory that contains the information for the links that should be there (source, dest, attributes), then add an inotify hook in Dovecot to create/update/delete the hard links in the directory so they match the links list?

The links list would only need to be opened when there's a change and could remain closed (and backup-able) at all other times, and restoring a links list would immediately trigger the inotify hook and regenerate all the required links.

Terry

Sorry for joining late this thread, but this is a very important issue for us.

Terry's solution feels great: just a little modification of the mdbox, adding a ¿text? file with the list of attachment files, that is modified every time an attachment is added/deleted.

With that, it seems quite easy to modify our mailbox recovery script to something like:

• recover the mailbox as now
• recover every attachment file that file points to.

Other option: a new doveadm option that could generate this list, and then recover the mailbox, generate the list, recover the attachments.

Also: no change needed to the backup process itself, and that's good news.

Timo Sirainen wrote:

And before designing it I'd need to look into how the backup softwares usually work.. If anyone has any ideas about this, I'd like to hear.

Simple or even moderately efficient backup programs like rsync copy all the files. Of course, if the program copies directories A B C in that order, then while B is being backed up, a file A/x could be created, and a file C/index could be modified to reflect that A/x exists. C/index would be backed up, but A/x would not be.

The way around that is with snapshots. Linux supports this with LVM, FreeBSD with UFS and ZFS file systems, Windows with VSS, and all selfcontained disk arrays I know of implement snapshots. The application quiesces its disk writes, flushes buffers to disk, triggers a snapshot, and resumes work as usual.

Also backing up the attachment links could be problematic if the backup system doesn't support hard links. Each attachment always has at least 2 links, so if the backup doesn't realize that it at minimum duplicates the space used by attachments.

rsync recognizes hard links with option -H, but at a very noticeable performance cost when dealing with millions of files. If the aa/bb/aabccddeeff-etc is unique across the whole mailstore, it would be easy to replace the hard link with a symlink, as you said:

maybe not storing the attachments directly to backups, but add symlinks to them so they can be used to figure out what to restore. Or maybe the backing up wouldn't need a special tool, but the restoring tool could just read through the dbox files to see what attachments are also needed and write a list of them somewhere so they can be taken from backups as well.

In the second way, you would have a separate hierarchy for multiple-recipient attachments, or would the attachment be "really" stored in the box of a recipient chosen at random?

Just some random thoughts: professionally, I use Zimbra. Messages are stored in Maildir-equivalents. The time it takes to backup is a quite severe constraint on the backup technique. For example, compressing the backup files takes too long, so the zip files are not compressed. Instead, the individual mails are stored compressed on disk. Each backup zips up the mails in a few big backup files. An improvement could be to sort mails into backup zip files so that once a zip file is made, it stays the same. After all, if a mail is not deleted a month after it is read, then it will probably stay in the same state forever, or at least until the user starts a keep-me-under-quota cleaning-up spree. During this time, backing up that big zip file can just be a check to see if it is already OK in the backup, which is much quicker. I have no idea if this could be applied to Dovecot, but who knows.

On Mon, Dec 05, 2011 at 01:51:05AM +0200, Timo Sirainen wrote:

I'm mainly wondering if it's common for backup programs to support using a separate program to generate the backups. For example if there was a "dovecot-backup" binary that just dumps all (or new-since-last-backup) of the users' mails into stdout, which the backup program can use. Or perhaps in that case there wouldn't really be much of anything for the backup to do except to write it to tape..

For databases, most commercial programs use some kind of application-specific plugins. Bakula has "Client Run Before Job" which permits indicating a command to be run before proceeding with the backup, maybe there is more. However I have not heard of any standardized way of doing it so that the application provider could provide a way to interact with multiple backup programs. It's more like each backup vendor boasting that his backup software can backup application XXX without downtime.

SIS was designed to work with hard links. They couldn't be replaced with symlinks without a redesign (which would be less efficient in normal operation).

Right, but if the backup program recognized this then maybe the replacement by a symlink could be done only in the backup. rsync has to keep in memory all the inodes and check the list every time a hard link is found. If it knew that "a hard link in attachments should link to an identical file name in the common attachment SiS store.

[Zimbra]

You mean you first create uncompressed zip files (why not just tar?) of all the mails to the filesystem and the backup software then backups those zip files?

Well, not I, this is Zimbra's backup system :-) The backups are the destination zips. Maybe zip is used because the extension and extraction method is the same whether compressed or not; that way compression is just an option to the backup program to be turned on or off.

Dovecot's mdbox files already contain multiple messages in each file, so it should be a lot more efficient to do backups on those. And each message in an mdbox file can be compressed if zlib plugin is enabled. So I think that sounds quite a lot like what you propose.

Is that combined or combinable with SiS? If attachments are in separate files, that means they are aligned on block boundaries, which make block-level SiS (like NetApp's) much more efficient. Think of an attachment sent to all department heads, all of whom forward the attachment to all their subordinates.

On 12/4/2011 12:16 PM, Timo Sirainen wrote:

On 4.12.2011, at 19.41, Terry Carmen wrote:

...

...

...

If we can not safely restore from backup a user's mailbox with SiS enabled, we can not enable SiS. Any plan to include this backup recovery tool in dovecot 2.0.X or 2.1?

I'd first have to design it. And before designing it I'd need to look into how the backup softwares usually work.. If anyone has any ideas about this, I'd like to hear.

BackupPC uses rsync by default for *nix boxes.

No idea what SiS is, but I'm guessing you're running into the same problem as backing up any other open file with changing internal data that may be inconsistent.

Inconsistency is an issue, but it's not the biggest problem. It would be possible to write a tool that scans through all mails and makes sure everything is consistent after a restore. dbox mostly does this automatically already, but SIS would need a separate program to ensure its consistency.

SIS is anyway a single instance attachment storage. So that lets say you send one 100 MB pdf to 10 people, and it's stored only once in disk under /attachments/aa/bb/aabccddeeff-etc. Each people would have their own unique link under /attachments/, but all of them would be hard linked to a common file.

So the problem is mainly about restoring a single user's mails. The mail files are simple to restore, but then you need to figure out which attachments to restore. There's no simple way to know which attachment files belong to which users, so you need to scan through the mail files and see what attachments are referred to.

Also backing up the attachment links could be problematic if the backup system doesn't support hard links. Each attachment always has at least 2 links, so if the backup doesn't realize that it at minimum duplicates the space used by attachments.

So I was thinking that there probably could be some tool that during a user's backup it would write the attachments among the user's other files, so it would be easy to find all of the files needed for a restore. This would of course mean that backups can take a lot more space, because there's no SIS. Perhaps there could be some other alternatives .. like maybe not storing the attachments directly to backups, but add symlinks to them so they can be used to figure out what to restore. Or maybe the backing up wouldn't need a special tool, but the restoring tool could just read through the dbox files to see what attachments are also needed and write a list of them somewhere so they can be taken from backups as well.

I'm not really sure what is the best way.

Backup and restore user hard links to SiS files, but not the files themselves. A single user "Oops" isn't going to delete the shared SiS files (they're already read only until the "last user" aren't they?). Thus we can assume the SiS files are always intact but for catastrophic failure.

To cope with catastrophic failure, create a "special" Dovecot administrator only mailbox (real/virtual/whatever) that "contains" all of the SiS files, a special Dovecot index. Back this mailbox up separately, just like you would any mailbox--i.e. backup the SiS mailbox independently from individual user backups. In the event of catastrophic failure, restore the SiS mailbox first, then user mailboxes, so user mailbox hard link creation during restore succeeds.

I would think something like this should maintain compatibility with standard IMAP backup software. I don't know if this could be done directly at the filesystem level. If this special mailbox simply contained hard links to the SiS files, would a restore operation at the filesystem level simply attempt to (re)create the hard links, or would it actually writes the files to their original filesystem locations? If only the former, then this proposed solution would only work with IMAP based backup/restore.

Just thinking out loud. There are probably better ways to do this.

-- Stan

Le 05/12/2011 02:45, Timo Sirainen a écrit :

On 5.12.2011, at 3.03, Stan Hoeppner wrote:

...

To cope with catastrophic failure, create a "special" Dovecot administrator only mailbox (real/virtual/whatever) that "contains" all of the SiS files, a special Dovecot index.

I'm not thinking about a catastrophe. For that a regular full filesystem backup+restore would work mostly okay (a snapshot would be perfect, without snapshot some extra work would be needed). The problem is that people want to recover only one specific user's mails from some older backup, because they accidentally deleted the mails.. This needs to be somewhat easy to implement with SIS, but it isn't.

Other problem I'm thinking of, because I'd like to use SIS on our production servers ; but right now I think I can't .

We have lots of users (+5000 teachers/engineers) on our first setup, +70000 students on our 2nd setup.

The user base is on LDAP, and move on a daily basis. When a user leave the university, he has right to use mailbox for a certain time and then we close the account.

Right now, we archive and then delete the mailbox directories (we don't use special dovecot mecanism : we migrated from another system not long time ago and we had special scripts for that).

If we use SIS, what happen to the attachements ? The usage count will never go to 0, and the attachements will stay there forever.

In that situation, I think we have no means to correct the attachement usage count ?

-- Yann Dupont - Service IRTS, DSI Université de Nantes Tel : 02.53.48.49.20 - Mail/Jabber : Yann.Dupont@univ-nantes.fr

On 7.12.2011, at 12.25, Yann Dupont wrote:

Other problem I'm thinking of, because I'd like to use SIS on our production servers ; but right now I think I can't .

We have lots of users (+5000 teachers/engineers) on our first setup, +70000 students on our 2nd setup.

The user base is on LDAP, and move on a daily basis. When a user leave the university, he has right to use mailbox for a certain time and then we close the account.

Right now, we archive and then delete the mailbox directories (we don't use special dovecot mecanism : we migrated from another system not long time ago and we had special scripts for that).

If we use SIS, what happen to the attachements ? The usage count will never go to 0, and the attachements will stay there forever.

In that situation, I think we have no means to correct the attachement usage count ?

You'll need to change the deletion script then. Run:

doveadm expunge -u user mailbox '*' all

before doing rm -rf for the user's mails. And in the archiving step you should do it with dsync with mail_attachment_dir disabled in the destination storage, so the the attachments get written to the archive directly instead of only referencing SIS.

On Wed, 2011-12-07 at 17:02 +0100, Yann Dupont wrote:

...

before doing rm -rf for the user's mails. And in the archiving step you should do it with dsync with mail_attachment_dir disabled in the destination storage, so the the attachments get written to the archive directly instead of only referencing SIS.

Yes, I understand, it will work. But, if case of any error (even our fault : premature end of script, for example) you can still end up with attachement forever lost on the filesystem.

Right, it SHOULD not happen, and it probably won't represent a big volume. But Still, it could happen under specific circonstances. In that case, I don't see any simple way to detect that kind of files ?

Do you see how a script could detect some orphaned links ??

It wouldn't be simple. The only safe way would be to:

1. Scan through all the attachment HASH-GUID names and save them. This scanning step could already detect some orphaned attachments, where the hashes/HASH file exists with nlink=1 (i.e. HASH-GUID* files have been deleted, but the HASH itself hasn't been for some reason).

2. Read through all users' all dboxes contents and get a list of all referenced attachment HASH-GUIDs.

3. Delete all attachments that exist in list 1, but not in list 2.

I guess there should be a "doveadm sis rescan" command that does this.