10 Aug
2009
10 Aug
'09
8:10 p.m.
I just wrote a "Scalability plans" mail, which explains what I've been recently thinking about. I think that would solve most of your problems.
On Mon, 2009-08-10 at 11:20 +0100, Mark Zealey wrote:
- One major issue we have with dovecot is the number of iops it requires to keep its files in sync on our big disk arrays. If this were a mysql database, we'd have better ways of keeping these in sync, for example switching the speed at which it flushes the updates to disk, growth using partitioning/sharding, read-only slave instances etc, specific ssd storage for indexes/caches. I know that we could create a separate ssd disk array and export that over nfs too, but see issue (2) below. If indexes were stored in a mysql database we could choose to have a different reliability for the indexes (ie if the database server crashes we have 1 second of index changes lost, whereas for the email itself we can ensure it is always kept in sync on our other disk arrays).
Seems pretty complex to me, but it should be possible to implement MySQL FS backend and keep indexes/mails there. They would anyway pretty much have to be just blobs, but I guess that's not an issue for you?
- nfs locking issues. No matter how hard we try, we always find there are some issues with concurrency over nfs.
Solved again.
- As we store all of our mail systems over nfs, effectively each export is a single-threaded connection to the server. No matter how fast the central nfs server you can realistically not go above 10k nfs ops/sec on the fastest of local networks due to latency issues. With mysql you could have multiple parallel connections to the central database store (1 per process?) which would mean more nfs ops available for actually serving the files.
lib-storage parallelization with async I/O backend should also solve this. Even when using mysql the lib-storage parallelization would be required to take advantage of multiple connections.
- Caching. Much more tuneable on a database server than in a filesystem. Eg in mysql there are loads of options to tune the various innodb/query/key caches, on nfs there are very few even on advanced netapps etc
I'm not sure about this. I think the in-memory index cache would solve the worst problem.