[Dovecot] Questions about single intance storage

[Stan Hoeppner]

On 12/4/2011 12:16 PM, Timo Sirainen wrote:
> On 4.12.2011, at 19.41, Terry Carmen wrote:
> 
>>>> If we can not safely restore from backup a user's mailbox with SiS enabled, we can not enable SiS. Any plan to include this backup recovery tool in dovecot 2.0.X or 2.1?
>>>
>>> I'd first have to design it. And before designing it I'd need to look into how the backup softwares usually work.. If anyone has any ideas about this, I'd like to hear.
>>
>> BackupPC uses rsync by default for *nix boxes.
>>
>> No idea what SiS is, but I'm guessing you're running into the same problem as backing up any other open file with changing internal data that may be inconsistent.
> 
> Inconsistency is an issue, but it's not the biggest problem. It would be possible to write a tool that scans through all mails and makes sure everything is consistent after a restore. dbox mostly does this automatically already, but SIS would need a separate program to ensure its consistency.
> 
> SIS is anyway a single instance attachment storage. So that lets say you send one 100 MB pdf to 10 people, and it's stored only once in disk under /attachments/aa/bb/aabccddeeff-etc. Each people would have their own unique link under /attachments/, but all of them would be hard linked to a common file.
> 
> So the problem is mainly about restoring a single user's mails. The mail files are simple to restore, but then you need to figure out which attachments to restore. There's no simple way to know which attachment files belong to which users, so you need to scan through the mail files and see what attachments are referred to.
> 
> Also backing up the attachment links could be problematic if the backup system doesn't support hard links. Each attachment always has at least 2 links, so if the backup doesn't realize that it at minimum duplicates the space used by attachments.
> 
> So I was thinking that there probably could be some tool that during a user's backup it would write the attachments among the user's other files, so it would be easy to find all of the files needed for a restore. This would of course mean that backups can take a lot more space, because there's no SIS. Perhaps there could be some other alternatives .. like maybe not storing the attachments directly to backups, but add symlinks to them so they can be used to figure out what to restore. Or maybe the backing up wouldn't need a special tool, but the restoring tool could just read through the dbox files to see what attachments are also needed and write a list of them somewhere so they can be taken from backups as well.
> 
> I'm not really sure what is the best way.

Backup and restore user hard links to SiS files, but not the files
themselves.  A single user "Oops" isn't going to delete the shared SiS
files (they're already read only until the "last user" aren't they?).
Thus we can assume the SiS files are always intact but for catastrophic
failure.

To cope with catastrophic failure, create a "special" Dovecot
administrator only mailbox (real/virtual/whatever) that "contains" all
of the SiS files, a special Dovecot index.  Back this mailbox up
separately, just like you would any mailbox--i.e. backup the SiS mailbox
independently from individual user backups.  In the event of
catastrophic failure, restore the SiS mailbox first, then user
mailboxes, so user mailbox hard link creation during restore succeeds.

I would think something like this should maintain compatibility with
standard IMAP backup software.  I don't know if this could be done
directly at the filesystem level.  If this special mailbox simply
contained hard links to the SiS files, would a restore operation at the
filesystem level simply attempt to (re)create the hard links, or would
it actually writes the files to their original filesystem locations?  If
only the former, then this proposed solution would only work with IMAP
based backup/restore.

Just thinking out loud.  There are probably better ways to do this.

-- 
Stan