[Dovecot] Compressed mail archives

[Will Palmer]

On Thu, 2010-05-13 at 10:11 -0400, Charles Marcus wrote:
> Also forgot:
> 
> http://wiki.dovecot.org/Plugins/Zlib?highlight=%28compress%29

Yeah, that's one of the things I was referring to when I mentioned
solutions which apply only to single mails (though read-only archives
would possibly benefit from inter-mail compression, I don't know how I'd
set up "live" mails to use Maildir while archives use mbox)

> > Single-instance attachment storage sounds very useful too, though I
> > can't help but wonder if it can be extended to message bodies in
> > general.
> 
> It *could*, but personally I think that complicates the task
> dramatically while providing much less benefit. The vast majority of
> storage for email is due to binary attachments, not email body text.
> 

That's just not the case here, as none of the messages I'm wishing to
compress have an attachments. I'm talking about high-volume mails which
sometimes need to be looked back on for reference, weeks or months
later.The mails themselves are all often very similar, and are sent to
several people.

> Also, with compression, you get the best of both worlds (since text
> compresses so well...

Not sure what this is supposed to mean. Over the course of the past two
weeks I've received 213MB of plain-text Emails, with no attachments.
Compressing each of these results in 60% savings (du -h shows 85mb), but
compressing the entire directory (tar.gz) results in 90% savings (ls -lh
shows 22mb) And of course, any way you slice it, 20,000 mails,
compressed or not, sent to five people, take up 100,000 mails worth of
space if you don't share them between accounts. (perhaps they could even
simply be hard-linked to the same file?)

I'm not saying this is a serious problem and we should all make it a
priority to solve it: it's most certainly not. Worst case scenario we
archive them somewhere else, this is purely about laziness and
convenience, and of course my desire to play with a new toy to see what
I can get it to do. But to hand-wave with "text compresses well" seems
mathematically absurd. 5X > X.

Feel free to ignore this message, as I've passed the point of practical
necessity long ago.


-- 
-- Will