RFC: HTTP based storage API

[Timo Sirainen]

> On 08 Feb 2016, at 13:28, Nagy, Attila <bra at fsn.hu> wrote:
> 
> On 02/08/16 12:07, Timo Sirainen wrote:
>> On 08 Feb 2016, at 12:56, Nagy, Attila <bra at fsn.hu> wrote:
>>> On 02/08/16 11:16, Timo Sirainen wrote:
>>>> Oh, you were thinking about ability to provide IMAP/etc support for other random servers, and have Dovecot act as kind of a middleware and translate the requests. Maybe the answer is still jmap though? It would require jmap lib-storage backend similar to imapc, which would be doable, although not really something we're right now planning to implement.
>>>> 
>>> Yeah, the opposite, in this case a jmap backend to Dovecot.
>>> BTW, I think jmap is too high level and implementing a jmap server is very much like implementing an IMAP one.
>>> 
>>> I much more think of a pluggable, easy (remote) storage API, which has much less to do with IMAP, but can offer capabilities, which can help Dovecot (like the search and indexes).
>> What kind of use cases are you thinking for this remote storage API? What kind of remote storages would implement it, and what kind of installations would use them?
>> 
> Interfacing non-email systems to e-mail protocols and implementing other storage options (I can't say a better analogue than what FUSE is).
> Using Dovecot as a protocol implementation, but not for the underlying storage method, which may be custom to the given solution.
> A flexible way of implementing and adapting anything to IMAP.
> 
> For example if I want to store really old and archived e-mails in a costly compression format somewhere in the cloud, but hot mails locally, based on my custom policies.
> In an international company there may be some regulations about what should go where, so for example if I can handle object placement myself, I can use the same installation to store non-EU mails in non-EU countries and others in US or EU clouds if law permits.

You could already implement these with mail-filter plugin. The locally stored mail would be just an object pointer, which the mail-filter plugin reads from the remote storage. Although I think mail-filter is lacking error handling right now.

> Intermixing several systems into one (like merging two IMAP accounts into one) etc.

If I bothered to implement per-namespace imapc-settings, this could be done with virtual plugin.

> Some of these are very costly to develop in Dovecot, while just a few lines in -say- Python or go.

I think it might not be that easy. Although I've made lib-storage backends easier and easier to implement, it's still not exactly trivial. Implementing it behind some API could simplify it somewhat, but it wouldn't really remove all the difficult work that needs to be done. For example some backends might want to be read-only, others read-write. Some might want Dovecot to store all the message flags and other metadata locally, while others might want to store it themselves (and that would require two-way syncing between them). Some backends would support searching, fetching some headers fast and in general support different kinds of optimizations, while others wouldn't. The imapc backend is of course already implementing a lot of this functionality, but it's also getting complex. The backend would also have to guarantee some things to be compatible with IMAP, mainly never modifying existing mails.

One alternative might be to add scripting support to Dovecot. I've been thinking about that several times over the years. So instead of implementing plugins with C, you could implement them with Python or some other language. Long time ago I tried to do this with SWIG, but it didn't work out. The main problem was function pointers in structs, but nowadays those aren't directly called and I think it could be implemented a bit differently.