[Dovecot] (Single instance) attachment storage with single-dbox

[William Blunn]

On 24/09/2010 19:57, Timo Sirainen wrote:
> Just a note to myself and whoever else cares, should be added to wiki once it has its own page about this:
>    

I have been meaning to say we should have a wiki page about this.

> With single-dbox messages can be copied with hard linking. This means that there can be multiple files that point to the same attachment file. The attachment is now deleted only once the mail file's link count drops to zero, so this works fine..
>
> ..until someone goes and starts manually copying files or maybe restoring from backups or whatever, causing the hard links to be replaced with separate files having link count=1, even when there are other files pointing to the same attachment file. Now deleting this one mail will delete the attachment too early.
>
> I can't think of any other reasonable way to handle this though, so unless someon has some great ideas, I think the solution is to simply
> add enough warnings that message store shouldn't be accessed directly. Maybe add some import/export commands to doveadm which can be used to add a bunch of mails to storage without doing it directly on filesystem.

Conceptually there is an attachment table with a primary key, and a 
message table which has foreign keys referring into the attachment 
table. In database theory, you could set up a foreign key constraint, 
and then it would not be possible to remove attachments which were still 
referenced by messages.

I'm not saying all this data should be under *SQL; just thinking aloud.

Can we do something /like/ that but only using a filesystem?

Perhaps each message could have its own hard links to the attachment 
file? That way a message's attachments would be kept in existence by the 
filesystem itself.

When you store a message with attachments, you could store the message file:

u.123

and have its attachments stored as hard links under names which appear 
adjacent to the message file's name when the directory listing is sorted.

u.123.a.1
u.123.a.2

Because the attachment files are adjacently-named to the message file, 
people accessing the sdbox mailstore as a filesystem should realise that 
they must backup/restore the message file and its associated attachment 
files.

Bill