[Dovecot] Scalability plans: Abstract out filesystem and make it someone else's problem

[Timo Sirainen]

On Mon, 2009-09-28 at 18:35 +0100, Ed W wrote:
> I would have thought that your API will prefer to request message parts 
> where it can (eg header, body, mime part), and just issue a read_bytes, 
> where that's what the client is asking for otherwise.  This would allow 
> the storage engine to optimise where it can and sadly for the dumb 
> client we just stream bytes since that's all they asked for...

In my mind this is more about what lib-storage API was supposed to
abstract out, whereas my filesystem API would be used simply for binary
data storage. The same FS API could be used to store both dbox files and
index files.

> Perhaps the API should also request specific headers from the storage 
> engine where possible and ask for all headers only where it's 
> necessary?  This would allow an sql database to be heavily normalised 
> (I'm sure performance is iffy, but we have to pre-suppose some reason 
> why this design is useful for other reasons)

This is really going towards what lib-storage API is supposed to do
already.. It's not even horribly difficult to write a new backend for
it. For example in v2.0 the fully functional Cydir backend code looks
like:

% wc *[ch]
  152   357  3740 cydir-mail.c
  319   783  8420 cydir-save.c
  402  1087 10806 cydir-storage.c
   35    82  1085 cydir-storage.h
  187   465  4798 cydir-sync.c
   24    54   615 cydir-sync.h
 1119  2828 29464 total

There is still a bit of code duplication between backends that could
reduce the line count by maybe 100-200 lines. Anyway I think the only
good way to implement support for normalized SQL database in Dovecot
would be to implement a new lib-storage backend, and it shouldn't be a
hugely difficult job.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url : http://dovecot.org/pipermail/dovecot/attachments/20090928/ebc823be/attachment.bin