[Dovecot] Scalability plans: Abstract out filesystem and make it someone else's problem

[Seth Mattinen]

Timo Sirainen wrote:
> On Aug 11, 2009, at 12:41 AM, Seth Mattinen wrote:
> 
>>> Nothing forces you to switch from maildir, if you're happy with it :)
>>> But if you want to support millions of users, it's simpler to distribute
>>> the storage and disk I/O evenly across hundreds of servers using a
>>> database that was designed for it. And by databases I mean here some of
>>> those key/value-like databases, not SQL. (What's a good collective name
>>> for those dbs anyway? BASE and NoSQL are a couple names I've seen.)
>>>
>>
>>
>> Why is a database a better choice than a clustered filesystem?
> 
> Show me a clustered filesystem that can guarantee that each file is
> stored in at least 3 different data centers and can scale linearly by
> simply adding more servers (let's say at least up to thousands).

Easy, AFS. It is known to support tens of thousands of clients [1] and
it's not exactly new. Like supporting the quirks of NFS, the quirks of a
clustered filesystem could be found and dealt with, too.

Key/value databases are hardly a magic bullet for redundancy. You don't
get 3 copies in different datacenters by simply switching to a
database-style storage.

[1]
http://www-conf.slac.stanford.edu/AFSBestPractices/Slides/MorganStanley.pdf


> Clustered filesystems are also complex. They're much more complex than
> what Dovecot really requires.
> 

I mention it because you stated wanting to outsource the storage
portion. The complexity of whatever database engine you choose or
supporting a clustered filesystem (like NFS) is a wash since you're not
maintaining either one personally.

~Seth