[Dovecot] (Single instance) attachment storage

[William Blunn]

Timo Sirainen wrote:
> I was thinking it would be nice to be able to compress attachments after they've already been delivered. Like maybe keep the attachments decoded for a few weeks and then compress them. Similar to how some people do it with Maildir. This can't work without a small header, otherwise you can't know if the attachment was originally compressed or not.

What about doing what people do with files and codify the compression 
status in the filename?

Assume the attachment file has a base name of "foo".

If you want to compress it, create a file "foo.gz" (or "foo.bz2" or 
"foo.xz"), and remove the original file "foo".

This assumes that if an open file is unlinked it will remain in 
existence until closed.

Code which wishes to read attachment files should first try to open the 
file using the base name. If the base name does not exist then it can 
try suffixes until it finds a file which opens. The suffix which worked 
then determines the decompression algorithm to be used.

The cost of an extra directory search (on messages which are understood 
to be historical in any case) should be dwarfed by the cost of 
decompressing the data.

This also solves the problem of how you compress a file in place. By 
creating the compressed version alongside, you never upset any readers.

This also helps out sysadmins because they would then be able to examine 
compressed attachment files straight off (zcat, bzcat, xzcat, et. al.) 
without having to think about any Dovecot-specific way of how the files 
might be laid out.

Bill