[Dovecot] Questions about single intance storage

[Timo Sirainen]

On 4.12.2011, at 19.41, Terry Carmen wrote:

>>> If we can not safely restore from backup a user's mailbox with SiS enabled, we can not enable SiS. Any plan to include this backup recovery tool in dovecot 2.0.X or 2.1?
>> 
>> I'd first have to design it. And before designing it I'd need to look into how the backup softwares usually work.. If anyone has any ideas about this, I'd like to hear.
> 
> BackupPC uses rsync by default for *nix boxes.
> 
> No idea what SiS is, but I'm guessing you're running into the same problem as backing up any other open file with changing internal data that may be inconsistent.

Inconsistency is an issue, but it's not the biggest problem. It would be possible to write a tool that scans through all mails and makes sure everything is consistent after a restore. dbox mostly does this automatically already, but SIS would need a separate program to ensure its consistency.

SIS is anyway a single instance attachment storage. So that lets say you send one 100 MB pdf to 10 people, and it's stored only once in disk under /attachments/aa/bb/aabccddeeff-etc. Each people would have their own unique link under /attachments/, but all of them would be hard linked to a common file.

So the problem is mainly about restoring a single user's mails. The mail files are simple to restore, but then you need to figure out which attachments to restore. There's no simple way to know which attachment files belong to which users, so you need to scan through the mail files and see what attachments are referred to.

Also backing up the attachment links could be problematic if the backup system doesn't support hard links. Each attachment always has at least 2 links, so if the backup doesn't realize that it at minimum duplicates the space used by attachments.

So I was thinking that there probably could be some tool that during a user's backup it would write the attachments among the user's other files, so it would be easy to find all of the files needed for a restore. This would of course mean that backups can take a lot more space, because there's no SIS. Perhaps there could be some other alternatives .. like maybe not storing the attachments directly to backups, but add symlinks to them so they can be used to figure out what to restore. Or maybe the backing up wouldn't need a special tool, but the restoring tool could just read through the dbox files to see what attachments are also needed and write a list of them somewhere so they can be taken from backups as well.

I'm not really sure what is the best way.