RFC: HTTP based storage API

[Nagy, Attila]

Hi,

Nearly every popular programming language has an LMTP/POP/IMAP 
implementation, most of them suck in many different ways.
I don't know any server or library which provides a well-established, 
compatible protocol frontend with an open backend API, which could be 
used to easily make a custom storage backend for the LMTP/POP/IMAP 
frontend in any language, in any programming paradigm, irregardless of 
the frontend implementation.

Dovecot is very close to this: it speaks IMAP to the backend, but you 
still have to make a nearly complete IMAP implementation, which is a PITA.

Therefore I am curious about your opinion: what about an HTTP-based open 
backend (lib-storage) API to Dovecot?

Some thoughts on this:
1. it should be a simple REST API
2. it should support efficient operations on both sides (client, 
server), for example listing a multi-million folder shouldn't be a 
single JSON response, but a JSON stream, which could be produced and 
parsed individually or in smaller batches efficiently
3. it should support capabilities, somewhat like the current imapc 
backend, so if the backend service suppports search, it should announce 
this capability and dovecot could offload these requests to it, but if 
it doesn't, dovecot could build its indices.
4. only the needed amount of POP/IMAP legacy should appear in the protocol
5. the operations should be asynchronous on Dovecot's side (to be scalable)
6. maybe even Dovecot's index-queries could be offloaded to the backend 
service (this way the local file system wouldn't be used at all and all 
caches could be stored in a distributed dictionary), this could also be 
a capability
7. it could support some kind of push (for IDLE-ing and the like)
8. it should be stateless

If Dovecot could make this right, it could be the FUSE of the mail 
protocols, acting as a glue for experimental or production ready mail 
storage backends, without the need to write these backends in "Dovecot 
C" (I mean it's process and API structure).

Of course HTTP here is not a strict requirement. It could be anything, 
which can be easy to implement and understand and is able to fulfill the 
needs.
An example could be Dovecot's own dictionary protocol.

What do you think about this?