[Dovecot] mdbox compression

[Matt Reimer]

On Fri, Feb 5, 2010 at 5:46 PM, Timo Sirainen <tss at iki.fi> wrote:

> On 6.2.2010, at 3.23, Timo Sirainen wrote:
>
> > I was thinking that the compression would be delayed so that it would be
> done only after mdbox already decided that it wouldn't write any more data
> to it.
>
> Oh, and this is actually why I was thinking that maybe it could be a good
> idea. If it's only done for older mails, they aren't accessed that often. So
> maybe a hybrid solution would be a good idea for mdbox users with alt
> storage:
>
>  * primary storage: SSD disks, mdbox file size = 100k, compress each mail
> separately
>  * alt storage: spinning disks, mdbox file size = 2 MB, compress the entire
> file
>
> Mails would be moved to alt storage after n days, perhaps dynamically
> depending on available SSD disk space.
>
> SSDs can read data pretty fast though, so it would be nice to look at some
> benchmarks that read tons of emails concurrently compressed vs.
> uncompressed. Is the bottleneck CPU or I/O? Hmm. A quick test with my Intel
> SSD shows that it can read 243 MB/s from a single large file, while zlib
> input is only 100 MB/s with Macbook's one CPU core. Faster CPUs and more
> cores would make zlib faster though.
>

Nice!

Matt