[Dovecot] Scalability plans: Abstract out filesystem and make it someone else's problem

[Charles Marcus]

On 8/12/2009, Daniel L. Miller (dmiller at amfes.com) wrote:
> Under the structure I've proposed, net storage consumed by the
> attachments should be one copy of attachment 1, and one copy of
> attachment two, plus headers and any comments in the messages times
> the number of recipients.  Domino would store one copy of attachment
> 1, then a copy of attachment 1 + attachment 2, then another copy of
> attachment 1.

Personally, I only care about binary attachments over a certain size.

I have said before, I don't see the value in doing this for every
message and for every mime-part. That said, if it doesn't really cost
anything extra to do the entire message and all mime-parts, then fine, I
don't really have anything against it, as long as it is robust and reliable.

But for example, what I'd really like to be able to do is say something
like:

SiS_mode = binary,64K

So only binary attachments over 64KB in size would be checksummed and
single instance stored. I don't care bout the headers, or body text, or
tiny (< 64KB) sig attachments, or text attachments (PGP sigs, etc).

Again - for shops that must deal with large binary attachments, this
would be a god-send.

Our max allowed message size is 50MB, and we typically get anywhere from
2-10 messages a day containing 20, 30, or even 40MB attachments sent to
our distribution lists - so these would go to 50+ people, who then
forward them to others, etc, etc ad nauseum.

Currently, I have mailman set to hold these, then I go in and strip off
the attachment, put it in a shared location, then let the message (minus
the attachment) through. But we still have a *lot* of messages like this
that don't go through our lists, but are sent to 2, 3, or 10 of our reps
individually.

I did a manual approximation on one persons mail store once, abd
determined that our total storage requirements, if SiS was implemented
for large attachments, would be reduced by about 90-95%. So, from about
2TB currently, to about 100-200GB. That is HUGE, from both a storage
*and* backup standpoint.

-- 

Best regards,

Charles