[Dovecot] Scalability plans: Abstract out filesystem and make it someone else's problem

[Ed W]

Timo Sirainen wrote:
> On Mon, 2009-09-28 at 18:35 +0100, Ed W wrote:
>   
>> I would have thought that your API will prefer to request message parts 
>> where it can (eg header, body, mime part), and just issue a read_bytes, 
>> where that's what the client is asking for otherwise.  This would allow 
>> the storage engine to optimise where it can and sadly for the dumb 
>> client we just stream bytes since that's all they asked for...
>>     
>
> In my mind this is more about what lib-storage API was supposed to
> abstract out, whereas my filesystem API would be used simply for binary
> data storage. The same FS API could be used to store both dbox files and
> index files.
>   

I guess in this case it would be interesting to hear the kind of use 
cases you imagine that the storage API will be used for in practice?  I 
think I might be kind of overthinking the problem?

Seems like it's a very thin shim between a real file system and dovecot 
and would be mainly useful for supporting filesystems with non posix 
protocols, eg someone wants to store their mail files on mogile or DAV, 
but it doesn't address anything lower or higher than blocks of data?

I can see how this would be useful in certain scenarios, but kind of 
interested to hear where you think it will go?

Seems like it would be useful for:

- implementing very specific optimisations for example for NFS
- optimisation for filesystems with unusual strengths/weaknesses, eg GFS 
or Gluster?
- non posix file system storage (but without trying to leverage 
particular features of that storage)

Actually, it mainly seems like a way for you to break out the access 
paths for NFS versus local storage when I write it down like that?

Ed W