[Dovecot] Webmail architecture : questions about cache and storage backend

[Timo Sirainen]

On Wed, 2011-09-21 at 14:04 +0200, forumer at smartmobili.com wrote:

> 1) Do not use a cache and rely on dovecot performance to retrieve 
> email's header in real time.
>     Actually don't know if it's that simple because we might need to 
> have a transformed version
>     of emails to be able to display them more easily in the webmail. In 
> this case this option is
>     not really possible.

I'm not really sure what you mean by this. Dovecot can cache individual
headers and make fetching them fast.

> 2) Use new dovecot-2.1 imap backend and develop a kind of dummy imap 
> server listening to a custom port
>     and storing emails inside a mongoDB database. Advantage is it allow 
> replication very easily.

I suppose that would be the easiest way to do it currently. But note
that you can't get all of Dovecot's features to work through imapc
backend. Especially the concept of message and mailbox GUIDs doesn't
exist in IMAP protocol (some kind of a new X-DOVECOT extension could be
added for those of course).

Also implementing a bug-free IMAP server may not be as easy as you
think, even though Dovecot uses only a small subset of IMAP commands. Be
sure to stress test Dovecot using imaptest: http://imapwiki.org/ImapTest
(I've noticed this week that Exchange totally screws up its internal
IMAP state when stress testing.)

> 3) Wait for dovecot2.x to implement a NoSQL storage. Will it happen 
> soon ?

I hope so! :) But I can't really say when.. My priorities currently are:

1. Do whatever paying customers want (not that much currently)
2. Finish v2.1.0 (still quite a lot of small things to do)
3. Create v2.2 branch where lib-storage API is changed to support more
asynchronous lookups, and modify imapc backend to use those (= support
handling commands in parallel)
4. Probably change dbox code to use lib-fs, or maybe create a new
backend using lib-fs
5. Implement NoSQL backend for lib-fs

Actually 3 isn't really necessary for 4-5, but it improves the
performance.