[Dovecot] Scalability plans: Abstract out filesystem and make it someone else's problem

[Ed W]

>> Oh, have you considered some "optional" api calls in the storage API?
>> The logic might be to assume that someone wanted to do something
>> clever and split the message up in some way, eg store headers
>> separately to bodies or bodies carved up into mime parts.  The
>> motivation would be if there was a certain access pattern to optimise.
>> Eg for an SQL database it may well be sensible to split headers and
>> the message body in order to optimise searching - the current API may
>> not take advantage of that?  
>>     
>
> Well, files have paths. I think the storage backend can determine from
> that what type the data is. So if you're writing to mails/foo/bar/123 it
> means you're storing a message with ID 123 to mailbox "foo/bar". It
> could then internally parse the message and store its header/body/mime
> separately.
>   


But would the storage be used optimally if there was a requirement to 
read in all headers from all emails, say in order to build the cache of 
messages on "Subject", or what about a backend which has some sort of 
search capability that we could usefully leverage? It's worth 
considering anyway because this looks like a design to remote the main 
storage from the imap server side and scale out (massively) so network 
capacity might be worth planning for being a limited resource?

Does it make sense to push some of the understanding of a message 
structure down to the storage backend? Perhaps it could be in some way 
optional with a more bruteforce option available on the dovecot side?  
ie like fuse, implement what you need and not more?

Ed W