[Dovecot] Scalability plans: Abstract out filesystem and make it someone else's problem

[Timo Sirainen]

On Mon, 2009-09-28 at 19:21 +0100, Ed W wrote:
> > In my mind this is more about what lib-storage API was supposed to
> > abstract out, whereas my filesystem API would be used simply for binary
> > data storage. The same FS API could be used to store both dbox files and
> > index files.
> >   
> 
> I guess in this case it would be interesting to hear the kind of use 
> cases you imagine that the storage API will be used for in practice?  I 
> think I might be kind of overthinking the problem?

lib-storage API has existed since Dovecot v1.0 and it's used to abstract
out access to maildir, mbox, dbox, cydir, etc. SQL would fit right there
with those.

Or did you mean FS API? For that my plans are to implement backends for:

 - POSIX (just the way it works now)
 - Async I/O (once Dovecot can do more things in parallel)
 - Some kind of proxying to support shared mailboxes between different
servers (or within same server when users are using different UIDs and
don't have a common group)
 - Massively distributed database storage for mails
 - In-memory cache for index files, which permanently writes them using
another storage. This is useful for any kind of multi-master setup like
distributed database, NFS, clusterfs.

> Seems like it's a very thin shim between a real file system and dovecot 
> and would be mainly useful for supporting filesystems with non posix 
> protocols, eg someone wants to store their mail files on mogile or DAV, 
> but it doesn't address anything lower or higher than blocks of data?

Right, path/filename (or "key") -> binary byte stream.

> Seems like it would be useful for:
> 
> - implementing very specific optimisations for example for NFS
> - optimisation for filesystems with unusual strengths/weaknesses, eg GFS 
> or Gluster?

In both of these I think the primary problem is that Dovecot tries to do
IPC via filesystem (index files). So accessing the indexes via the
in-memory cache that is guaranteed to be always up-to-date would get rid
of all these ugly NFS cache flushing attempts etc.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url : http://dovecot.org/pipermail/dovecot/attachments/20090928/758da3fc/attachment.bin