RFC: HTTP based storage API
Hi,
Nearly every popular programming language has an LMTP/POP/IMAP implementation, most of them suck in many different ways. I don't know any server or library which provides a well-established, compatible protocol frontend with an open backend API, which could be used to easily make a custom storage backend for the LMTP/POP/IMAP frontend in any language, in any programming paradigm, irregardless of the frontend implementation.
Dovecot is very close to this: it speaks IMAP to the backend, but you still have to make a nearly complete IMAP implementation, which is a PITA.
Therefore I am curious about your opinion: what about an HTTP-based open backend (lib-storage) API to Dovecot?
Some thoughts on this: parsed individually or in smaller batches efficiently
- it should be a simple REST API
- it should support efficient operations on both sides (client, server), for example listing a multi-million folder shouldn't be a single JSON response, but a JSON stream, which could be produced and
- it should support capabilities, somewhat like the current imapc backend, so if the backend service suppports search, it should announce this capability and dovecot could offload these requests to it, but if it doesn't, dovecot could build its indices.
- only the needed amount of POP/IMAP legacy should appear in the protocol
- the operations should be asynchronous on Dovecot's side (to be scalable)
- maybe even Dovecot's index-queries could be offloaded to the backend service (this way the local file system wouldn't be used at all and all caches could be stored in a distributed dictionary), this could also be a capability
- it could support some kind of push (for IDLE-ing and the like)
- it should be stateless
If Dovecot could make this right, it could be the FUSE of the mail protocols, acting as a glue for experimental or production ready mail storage backends, without the need to write these backends in "Dovecot C" (I mean it's process and API structure).
Of course HTTP here is not a strict requirement. It could be anything, which can be easy to implement and understand and is able to fulfill the needs. An example could be Dovecot's own dictionary protocol.
What do you think about this?
On 08 Feb 2016, at 11:01, Nagy, Attila <bra@fsn.hu> wrote:
Hi,
Nearly every popular programming language has an LMTP/POP/IMAP implementation, most of them suck in many different ways. I don't know any server or library which provides a well-established, compatible protocol frontend with an open backend API, which could be used to easily make a custom storage backend for the LMTP/POP/IMAP frontend in any language, in any programming paradigm, irregardless of the frontend implementation.
Dovecot is very close to this: it speaks IMAP to the backend, but you still have to make a nearly complete IMAP implementation, which is a PITA.
Therefore I am curious about your opinion: what about an HTTP-based open backend (lib-storage) API to Dovecot?
We're planning on implementing http://jmap.io/ to Dovecot v2.3. It's close enough to how Dovecot works internally (although we'd still need the threading support) and also at least attempting to become a standard. We haven't really started it yet though, so now would be a good time to give any alternative suggestions or complaints about jmap :)
On 08 Feb 2016, at 11:59, Timo Sirainen <tss@iki.fi> wrote:
On 08 Feb 2016, at 11:01, Nagy, Attila <bra@fsn.hu> wrote:
Hi,
Nearly every popular programming language has an LMTP/POP/IMAP implementation, most of them suck in many different ways. I don't know any server or library which provides a well-established, compatible protocol frontend with an open backend API, which could be used to easily make a custom storage backend for the LMTP/POP/IMAP frontend in any language, in any programming paradigm, irregardless of the frontend implementation.
Dovecot is very close to this: it speaks IMAP to the backend, but you still have to make a nearly complete IMAP implementation, which is a PITA.
Therefore I am curious about your opinion: what about an HTTP-based open backend (lib-storage) API to Dovecot?
We're planning on implementing http://jmap.io/ to Dovecot v2.3. It's close enough to how Dovecot works internally (although we'd still need the threading support) and also at least attempting to become a standard. We haven't really started it yet though, so now would be a good time to give any alternative suggestions or complaints about jmap :)
Oh, you were thinking about ability to provide IMAP/etc support for other random servers, and have Dovecot act as kind of a middleware and translate the requests. Maybe the answer is still jmap though? It would require jmap lib-storage backend similar to imapc, which would be doable, although not really something we're right now planning to implement.
On 02/08/16 11:16, Timo Sirainen wrote:
On 08 Feb 2016, at 11:59, Timo Sirainen <tss@iki.fi> wrote:
On 08 Feb 2016, at 11:01, Nagy, Attila <bra@fsn.hu> wrote:
Hi,
Nearly every popular programming language has an LMTP/POP/IMAP implementation, most of them suck in many different ways. I don't know any server or library which provides a well-established, compatible protocol frontend with an open backend API, which could be used to easily make a custom storage backend for the LMTP/POP/IMAP frontend in any language, in any programming paradigm, irregardless of the frontend implementation.
Dovecot is very close to this: it speaks IMAP to the backend, but you still have to make a nearly complete IMAP implementation, which is a PITA.
Therefore I am curious about your opinion: what about an HTTP-based open backend (lib-storage) API to Dovecot? We're planning on implementing http://jmap.io/ to Dovecot v2.3. It's close enough to how Dovecot works internally (although we'd still need the threading support) and also at least attempting to become a standard. We haven't really started it yet though, so now would be a good time to give any alternative suggestions or complaints about jmap :) Oh, you were thinking about ability to provide IMAP/etc support for other random servers, and have Dovecot act as kind of a middleware and translate the requests. Maybe the answer is still jmap though? It would require jmap lib-storage backend similar to imapc, which would be doable, although not really something we're right now planning to implement.
Yeah, the opposite, in this case a jmap backend to Dovecot. BTW, I think jmap is too high level and implementing a jmap server is very much like implementing an IMAP one.
I much more think of a pluggable, easy (remote) storage API, which has much less to do with IMAP, but can offer capabilities, which can help Dovecot (like the search and indexes).
On 08 Feb 2016, at 12:56, Nagy, Attila <bra@fsn.hu> wrote:
On 02/08/16 11:16, Timo Sirainen wrote:
Oh, you were thinking about ability to provide IMAP/etc support for other random servers, and have Dovecot act as kind of a middleware and translate the requests. Maybe the answer is still jmap though? It would require jmap lib-storage backend similar to imapc, which would be doable, although not really something we're right now planning to implement.
Yeah, the opposite, in this case a jmap backend to Dovecot. BTW, I think jmap is too high level and implementing a jmap server is very much like implementing an IMAP one.
I much more think of a pluggable, easy (remote) storage API, which has much less to do with IMAP, but can offer capabilities, which can help Dovecot (like the search and indexes).
What kind of use cases are you thinking for this remote storage API? What kind of remote storages would implement it, and what kind of installations would use them?
On 02/08/16 12:07, Timo Sirainen wrote:
On 08 Feb 2016, at 12:56, Nagy, Attila <bra@fsn.hu> wrote:
On 02/08/16 11:16, Timo Sirainen wrote:
Oh, you were thinking about ability to provide IMAP/etc support for other random servers, and have Dovecot act as kind of a middleware and translate the requests. Maybe the answer is still jmap though? It would require jmap lib-storage backend similar to imapc, which would be doable, although not really something we're right now planning to implement.
Yeah, the opposite, in this case a jmap backend to Dovecot. BTW, I think jmap is too high level and implementing a jmap server is very much like implementing an IMAP one.
I much more think of a pluggable, easy (remote) storage API, which has much less to do with IMAP, but can offer capabilities, which can help Dovecot (like the search and indexes). What kind of use cases are you thinking for this remote storage API? What kind of remote storages would implement it, and what kind of installations would use them?
Interfacing non-email systems to e-mail protocols and implementing other storage options (I can't say a better analogue than what FUSE is). Using Dovecot as a protocol implementation, but not for the underlying storage method, which may be custom to the given solution. A flexible way of implementing and adapting anything to IMAP.
For example if I want to store really old and archived e-mails in a costly compression format somewhere in the cloud, but hot mails locally, based on my custom policies. In an international company there may be some regulations about what should go where, so for example if I can handle object placement myself, I can use the same installation to store non-EU mails in non-EU countries and others in US or EU clouds if law permits.
Intermixing several systems into one (like merging two IMAP accounts into one) etc.
Some of these are very costly to develop in Dovecot, while just a few lines in -say- Python or go.
On 08 Feb 2016, at 13:28, Nagy, Attila <bra@fsn.hu> wrote:
On 02/08/16 12:07, Timo Sirainen wrote:
On 08 Feb 2016, at 12:56, Nagy, Attila <bra@fsn.hu> wrote:
On 02/08/16 11:16, Timo Sirainen wrote:
Oh, you were thinking about ability to provide IMAP/etc support for other random servers, and have Dovecot act as kind of a middleware and translate the requests. Maybe the answer is still jmap though? It would require jmap lib-storage backend similar to imapc, which would be doable, although not really something we're right now planning to implement.
Yeah, the opposite, in this case a jmap backend to Dovecot. BTW, I think jmap is too high level and implementing a jmap server is very much like implementing an IMAP one.
I much more think of a pluggable, easy (remote) storage API, which has much less to do with IMAP, but can offer capabilities, which can help Dovecot (like the search and indexes). What kind of use cases are you thinking for this remote storage API? What kind of remote storages would implement it, and what kind of installations would use them?
Interfacing non-email systems to e-mail protocols and implementing other storage options (I can't say a better analogue than what FUSE is). Using Dovecot as a protocol implementation, but not for the underlying storage method, which may be custom to the given solution. A flexible way of implementing and adapting anything to IMAP.
For example if I want to store really old and archived e-mails in a costly compression format somewhere in the cloud, but hot mails locally, based on my custom policies. In an international company there may be some regulations about what should go where, so for example if I can handle object placement myself, I can use the same installation to store non-EU mails in non-EU countries and others in US or EU clouds if law permits.
You could already implement these with mail-filter plugin. The locally stored mail would be just an object pointer, which the mail-filter plugin reads from the remote storage. Although I think mail-filter is lacking error handling right now.
Intermixing several systems into one (like merging two IMAP accounts into one) etc.
If I bothered to implement per-namespace imapc-settings, this could be done with virtual plugin.
Some of these are very costly to develop in Dovecot, while just a few lines in -say- Python or go.
I think it might not be that easy. Although I've made lib-storage backends easier and easier to implement, it's still not exactly trivial. Implementing it behind some API could simplify it somewhat, but it wouldn't really remove all the difficult work that needs to be done. For example some backends might want to be read-only, others read-write. Some might want Dovecot to store all the message flags and other metadata locally, while others might want to store it themselves (and that would require two-way syncing between them). Some backends would support searching, fetching some headers fast and in general support different kinds of optimizations, while others wouldn't. The imapc backend is of course already implementing a lot of this functionality, but it's also getting complex. The backend would also have to guarantee some things to be compatible with IMAP, mainly never modifying existing mails.
One alternative might be to add scripting support to Dovecot. I've been thinking about that several times over the years. So instead of implementing plugins with C, you could implement them with Python or some other language. Long time ago I tried to do this with SWIG, but it didn't work out. The main problem was function pointers in structs, but nowadays those aren't directly called and I think it could be implemented a bit differently.
Zitat von Timo Sirainen <tss@iki.fi>:
On 08 Feb 2016, at 11:01, Nagy, Attila <bra@fsn.hu> wrote:
Hi,
Nearly every popular programming language has an LMTP/POP/IMAP
implementation, most of them suck in many different ways. I don't know any server or library which provides a
well-established, compatible protocol frontend with an open backend
API, which could be used to easily make a custom storage backend
for the LMTP/POP/IMAP frontend in any language, in any programming
paradigm, irregardless of the frontend implementation.Dovecot is very close to this: it speaks IMAP to the backend, but
you still have to make a nearly complete IMAP implementation, which
is a PITA.Therefore I am curious about your opinion: what about an HTTP-based
open backend (lib-storage) API to Dovecot?We're planning on implementing http://jmap.io/ to Dovecot v2.3. It's
close enough to how Dovecot works internally (although we'd still
need the threading support) and also at least attempting to become a
standard. We haven't really started it yet though, so now would be a
good time to give any alternative suggestions or complaints about
jmap :)
That would be a cool feature, Thanks.
Andreas
that sounds like s3 API based backend.
also note that dovecot comes with some "non local mail storage plugins" in the commercial part:
In this release, we support Windows Azure, Amazon S3, Scality and
Dropbox systems.
so that should come close to what you want. for a local version you could use e.g. radosdb/ceph or riak. both can provide the s3 API for you.
hth
darix-- openSUSE - SUSE Linux is my linux openSUSE is good for you www.opensuse.org
On 02/08/16 11:44, Marcus Rueckert wrote:
that sounds like s3 API based backend.
also note that dovecot comes with some "non local mail storage plugins" in the commercial part:
In this release, we support Windows Azure, Amazon S3, Scality and Dropbox systems.so that should come close to what you want. for a local version you could use e.g. radosdb/ceph or riak. both can provide the s3 API for you.
Close, but this may be too low level, for example it may hide whose e-mail do you store (I don't know the exact implementation, it's not open source AFAIK).
On 2016-02-08 12:31:57 +0100, Nagy, Attila wrote:
On 02/08/16 11:44, Marcus Rueckert wrote:
that sounds like s3 API based backend.
also note that dovecot comes with some "non local mail storage plugins" in the commercial part:
In this release, we support Windows Azure, Amazon S3, Scality and Dropbox systems.so that should come close to what you want. for a local version you could use e.g. radosdb/ceph or riak. both can provide the s3 API for you.
Close, but this may be too low level, for example it may hide whose e-mail do you store (I don't know the exact implementation, it's not open source AFAIK).
True ... but that might be the hook point you can use for your more fancy implementation. and some of the mail backends are public. like mdbox and so on. the api towards dovecot should be the same.
darix-- openSUSE - SUSE Linux is my linux openSUSE is good for you www.opensuse.org
participants (4)
-
lst_hoe02＠kwsoft.de
-
Marcus Rueckert
-
Nagy, Attila
-
Timo Sirainen