[Dovecot] Scalability plans: Abstract out filesystem and make it someone else's problem

[Steve]

-------- Original-Nachricht --------
> Datum: Wed, 12 Aug 2009 12:34:40 -0700
> Von: "Daniel L. Miller" <dmiller at amfes.com>
> An: Dovecot Mailing List <dovecot at dovecot.org>
> Betreff: Re: [Dovecot] Scalability plans: Abstract out filesystem and make it someone else\'s problem

> Timo Sirainen wrote:
> > On Wed, 2009-08-12 at 11:35 -0700, Daniel L. Miller wrote:
> >   
> >> Timo Sirainen wrote:
> >>     
> >>>> Also the mime structure could be torn apart to store 
> >>>> attachments individually - the motivation being single instance
> storage 
> >>>> of large attachments with identical content...  Anyway, these seem
> like 
> >>>> very speculative directions...
> >>>>     
> >>>>         
> >>> Yes, this is also something in dbox's far future plans.
> >>>   
> >>>       
> >> Speaking as a pathetic little admin of a small site of 20 users, my 
> >> needs for replication & scalability are quite minor.  However, 
> >> single-instance storage would be a miracle of biblical proportions. 
> Has 
> >> any progress been made on this?  
> >>     
> >
> > Do you need per-MIME part single instance storage, or would per-email be
> > enough? Since the per-email can already done with hard links.
> >   
> Definitely per MIME part.
> >> Do you have a roadmap for how you plan on implementing it?
> >>     
> >
> > I've written about it a couple of times I think, but no specific plans.
> > Something about using hashes anyway.
> >
> >   
> >> I don't know if you've considered this at all - this was my first
> thought:
> >>
> >> If you're able to store a message with the attachments separately, then
> >> you can come up with an attachment database (not meaning to imply SQL 
> >> backend).  Then after breaking the message up into message + 
> >> attachments, you scan the attachment database to see if it is already 
> >> present prior to saving it.  This could mean that not only could we
> save 
> >> on the huge space wasted by idiots merrily forwarding large attachments
> >> to multiple people, but even received mails with embedded graphical 
> >> signatures would benefit.
> >>     
> >
> > Yes, that's pretty much how I thought about it. It's anyway going to be
> > dbox-only feature. Would be way too much trouble with other formats.
> >   
> dbox-only is fine.  I could care less about the storage method chosen - 
> filesystem, db, encrypted, whatever - but I believe the impact on 
> storage - and possibly indexes & searching - would be huge.
> 
> On the personal greedy side, if you want to see a mass corporate 
> migration to Dovecot, with potential service contracts - that would be a 
> feature worth talking about.  I can see IT manager's eyes light up at 
> hearing about such a item - and I've never heard of any other mail 
> server supporting such a thing.
>
IBM Lotus Domino has that feature since ages (they call it shared mail). And they don't have that just for normal mails but for archives as well (called single instance store). This feature was first introduced in cc:Mail and then got integrated into Domino and is still there and even extended to work with various backends (like the new DB2 backend). Microsoft copied that concept from them (from my viewpoint the way how MS has done it in the past was horrible. I think newer versions work better but I am not sure).

>From my experience in doing messaging since 2 decades I can tell you that it is not worth doing single instance store (or how ever you call it). Storage is ultra cheep these days and backup systems are so fast that all the benefits which where valid some years ago are gone today.

It might rock your geek heart to implement something like that but doing the math on costs versus benefits will soon or later show you that today it's not worth doing it.


> --
> Daniel
>
Steve

-- 
Neu: GMX Doppel-FLAT mit Internet-Flatrate + Telefon-Flatrate
für nur 19,99 Euro/mtl.!* http://portal.gmx.net/de/go/dsl02