[Dovecot] Smart IMAP proxying with imapc storage
I just committed a very early initial implementation of "imapc" storage backend to v2.1 hg: http://hg.dovecot.org/dovecot-2.1
You can't really do anything except open INBOX and read mails from it, so it's currently only intended for initial testing. It sucks in many ways right now, but I'll be improving it.
The idea is that you could set for example:
mail_location = imapc:imap.gmail.com
And then Dovecot could act as a proxy to gmail. It won't actually work currently with gmail, because there's no SSL support for outgoing connections.
Currently index files are forcibly disabled, but it would be easy to enable them, allowing Dovecot to do caching locally to improve performance. In future perhaps it will also support caching message headers/bodies to avoid unnecessary traffic.
Besides the mail_location setting, you'll need to also forward the user's password to imap process in "pass" userdb extra field. How to do that depends on what passdb/userdb you're using. While testing you could simply set a static password:
plugin { pass = yourpassword }
On 16/01/2011 16:18, Timo Sirainen wrote:
I just committed a very early initial implementation of "imapc" storage backend to v2.1 hg: http://hg.dovecot.org/dovecot-2.1
I know "me too" posts aren't that useful, but this does sound like a terrific direction.
Thanks for development
Ed W
On Sun, 2011-01-16 at 18:18 +0200, Timo Sirainen wrote:
I just committed a very early initial implementation of "imapc" storage backend to v2.1 hg: http://hg.dovecot.org/dovecot-2.1
This is "fully functional" now, at least in theory. TODO:
- SSL/TLS support to remote server
- Don't read entire message bodies into a string in memory
- Fetch more than one message at a time when it's known that more messages are going to be fetched
- Support locally stored indexes
- Error handling could be better
- Eventually make lib-storage API more asynchronous -> make imapc more asynchronous
Some potential (future) uses for imapc include:
- Locally caching IMAP proxy
- Maybe even fully transparent proxy to client, so at office it would be using Dovecot proxy while at home it would connect directly to remote server. (A bit tricky with SSL connections, but possible I'd think.)
- Filtering proxy that modifies messages on the fly
- Fixing Exchange's (and everyone else's) IMAP support by being in front of it
- Could be used for sharing mailboxes across multiple Dovecot servers without shared storage
- Easy "perfect migration" from any IMAP server to Dovecot with dsync
On 01/28/2011 11:17 AM, Timo Sirainen wrote:
On Sun, 2011-01-16 at 18:18 +0200, Timo Sirainen wrote:
I just committed a very early initial implementation of "imapc" storage backend to v2.1 hg: http://hg.dovecot.org/dovecot-2.1
This is "fully functional" now, at least in theory. TODO:
- SSL/TLS support to remote server
- Don't read entire message bodies into a string in memory
- Fetch more than one message at a time when it's known that more messages are going to be fetched
- Support locally stored indexes
- Error handling could be better
- Eventually make lib-storage API more asynchronous -> make imapc more asynchronous
Some potential (future) uses for imapc include:
- Locally caching IMAP proxy
- Maybe even fully transparent proxy to client, so at office it would be using Dovecot proxy while at home it would connect directly to remote server. (A bit tricky with SSL connections, but possible I'd think.)
- Filtering proxy that modifies messages on the fly
- Fixing Exchange's (and everyone else's) IMAP support by being in front of it
- Could be used for sharing mailboxes across multiple Dovecot servers without shared storage
- Easy "perfect migration" from any IMAP server to Dovecot with dsync
+100! This is great!
On Fri, 2011-01-28 at 20:17 +0200, Timo Sirainen wrote:
I just committed a very early initial implementation of "imapc" storage backend to v2.1 hg: http://hg.dovecot.org/dovecot-2.1
This is "fully functional" now, at least in theory. TODO:
- Don't read entire message bodies into a string in memory
- Fetch more than one message at a time when it's known that more messages are going to be fetched
The above features are done now. So the performance shouldn't be too horrible anymore.
- SSL/TLS support to remote server
- Support locally stored indexes
- Error handling could be better
- Eventually make lib-storage API more asynchronous -> make imapc more asynchronous
These are not. Another TODO item is:
- Add support for multiple connections to IMAP server. This is required for e.g. THREAD command to work correctly.
On Mon, 2011-01-31 at 04:05 +0200, Timo Sirainen wrote:
On Fri, 2011-01-28 at 20:17 +0200, Timo Sirainen wrote:
I just committed a very early initial implementation of "imapc" storage backend to v2.1 hg: http://hg.dovecot.org/dovecot-2.1
- SSL/TLS support to remote server
Implemented. I tested with gmail:
mail_location = imapc:imap.gmail.com plugin { pass = secret imapc_user = user@gmail.com imapc_port = 993 imapc_ssl_ca_dir = /etc/ssl/certs
# If you want to use STARTTLS (with port 143), set: #imapc_ssl_starttls = yes }
So in above config /etc/ssl/certs/ contains all the valid CA certs. The code requires a valid certificate to exist, but doesn't currently verify that the name actually matches imap.gmail.com. I really need to put that code there soon so I won't forget it..
imapc settings have moved away from plugin {} section and mail_location. Now instead use:
mail_location = imapc:
imapc_host = imap.example.com #imapc_port = 143 # default
#imapc_user = %u # default imapc_password = secret
imapc_ssl = no # or imaps or starttls imapc_ssl_ca_dir = /etc/ssl/certs
Note the imapc_password change also. If passdb/userdb returned userdb_pass/pass previously, return now instead userdb_imapc_password/imapc_password.
On Tue, 2011-02-01 at 12:25 -0700, Willie Gillespie wrote:
On 02/01/2011 10:20 AM, Timo Sirainen wrote:
imapc settings have moved away from plugin {} section and mail_location.
#imapc_user = %u # default imapc_password = secret
Is there a way to pass through a password like is done with the user name?
Depends on what passdb and userdb is used. With sql, ldap, passwd-file, checkpassword it's possible by returning userdb_imapc_password=%w and using userdb prefetch to pass it through. This works of course only with plaintext authentication.
On 01/-10/-28163 08:59 PM, Timo Sirainen wrote:
imapc settings have moved away from plugin {} section and mail_location. Now instead use:
mail_location = imapc:
imapc_host = imap.example.com #imapc_port = 143 # default
#imapc_user = %u # default imapc_password = secret
imapc_ssl = no # or imaps or starttls imapc_ssl_ca_dir = /etc/ssl/certs
Note the imapc_password change also. If passdb/userdb returned userdb_pass/pass previously, return now instead userdb_imapc_password/imapc_password. I get the following error in the log: Feb 6 00:17:44 hostname dovecot: auth: static(user.name@domain.ch,127.0.0.1): No password Feb 6 00:17:44 hostname dovecot: imap-login: Login: user=user.name@domain.ch, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=20283, secured Feb 6 00:17:44 hostname dovecot: master: Error: service(imap): child 20283 killed with signal 11 (core dumps disabled)
dovecot -n # 2.1.UNSTABLE (4e4c7f982fd5): /usr/local/etc/dovecot/dovecot.conf # OS: Linux 2.6.32-27-generic i686 Ubuntu 10.04.1 LTS auth_debug = yes auth_debug_passwords = yes auth_verbose = yes debug_log_path = /home/local_user/dovecot.log default_login_user = nobody imapc_host = mail.domain.ch mail_debug = yes mail_gid = local_user mail_home = /var/run/dovecot/empty mail_location = imapc: mail_plugins = mail_filter mail_uid = local_user passdb { args = nopassword=y userdb_imapc_password=%w driver = static } plugin { imapc_password = secret mail_filter = mail-filter %u mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid box msgid size from subject } protocols = imap service mail-filter { executable = script /home/local_user/mydata/workspace_perl/sec_dovecot_filter/sec_dovecot_filter.pl unix_listener mail-filter { mode = 0666 user = root } user = local_user } ssl = no userdb { driver = prefetch }
Andy idea how to resolve. Thanks, Mike
On 6.2.2011, at 1.23, Mike Korizek wrote:
Feb 6 00:17:44 hostname dovecot: master: Error: service(imap): child 20283 killed with signal 11 (core dumps disabled) .. # 2.1.UNSTABLE (4e4c7f982fd5): /usr/local/etc/dovecot/dovecot.conf
4e4c7f982fd5 was a buggy version that always crashed at startup. I fixed it a couple of days ago.
Timo Sirainen tss@iki.fi wrote:
- Fixing Exchange's (and everyone else's) IMAP support by being in front of it
Davmail makes a nice job of this. I see no reason why you couldn't inspect its code, should you want to do so.
Chris
On 19.2.2011, at 1.03, Chris Davies wrote:
Timo Sirainen tss@iki.fi wrote:
- Fixing Exchange's (and everyone else's) IMAP support by being in front of it
Davmail makes a nice job of this. I see no reason why you couldn't inspect its code, should you want to do so.
Interesting project. However, I find it doubtful that it can implement a fully compliant IMAP server in just 1700 lines of code. A quick look shows that its LIST/LSUB handling code has been written to just handle a couple of IMAP clients, not a full correct support. I don't see it handling FETCH BODY[HEADER.FIELDS (x y z)] at all. Probably other things too.
On Fri, 2011-01-28 at 20:17 +0200, Timo Sirainen wrote:
Some potential (future) uses for imapc include:
- Locally caching IMAP proxy
- Maybe even fully transparent proxy to client, so at office it would be using Dovecot proxy while at home it would connect directly to remote server. (A bit tricky with SSL connections, but possible I'd think.)
- Filtering proxy that modifies messages on the fly
- Fixing Exchange's (and everyone else's) IMAP support by being in front of it
Anyone have good ideas what this thing could be called? :) Calling it "proxy" would confuse it with the dummy proxying feature of Dovecot. And calling it "smart proxy" isn't very clear either what it does, especially since I don't think it even is a proxy really.
So far the best I've come up with is a buzzwordy "IMAP accelerator".
On 2/28/2011 9:20 AM, Timo Sirainen wrote:
On Fri, 2011-01-28 at 20:17 +0200, Timo Sirainen wrote:
Some potential (future) uses for imapc include:
- Locally caching IMAP proxy
- Maybe even fully transparent proxy to client, so at office it would be using Dovecot proxy while at home it would connect directly to remote server. (A bit tricky with SSL connections, but possible I'd think.)
- Filtering proxy that modifies messages on the fly
- Fixing Exchange's (and everyone else's) IMAP support by being in front of it
Anyone have good ideas what this thing could be called? :) Calling it "proxy" would confuse it with the dummy proxying feature of Dovecot. And calling it "smart proxy" isn't very clear either what it does, especially since I don't think it even is a proxy really.
So far the best I've come up with is a buzzwordy "IMAP accelerator".
Accelerator would work since a good use could be at the edge of a company branch network... especially if it had a "prefetch = yes" option and was given sufficient disk space.
On 28/02/2011 16:20, Timo Sirainen wrote:
On Fri, 2011-01-28 at 20:17 +0200, Timo Sirainen wrote:
Some potential (future) uses for imapc include:
- Locally caching IMAP proxy
- Maybe even fully transparent proxy to client, so at office it would be using Dovecot proxy while at home it would connect directly to remote server. (A bit tricky with SSL connections, but possible I'd think.)
- Filtering proxy that modifies messages on the fly
- Fixing Exchange's (and everyone else's) IMAP support by being in front of it Anyone have good ideas what this thing could be called? :) Calling it "proxy" would confuse it with the dummy proxying feature of Dovecot. And calling it "smart proxy" isn't very clear either what it does, especially since I don't think it even is a proxy really.
So far the best I've come up with is a buzzwordy "IMAP accelerator".
"Cache" is a good word for this kind of application? Perhaps something like "IMAP Cache"?
Ed W
On 2011-02-28 11:20 AM, Timo Sirainen wrote:
On Fri, 2011-01-28 at 20:17 +0200, Timo Sirainen wrote:
Some potential (future) uses for imapc include:
- Locally caching IMAP proxy
- Maybe even fully transparent proxy to client, so at office it would be using Dovecot proxy while at home it would connect directly to remote server. (A bit tricky with SSL connections, but possible I'd think.)
- Filtering proxy that modifies messages on the fly
- Fixing Exchange's (and everyone else's) IMAP support by being in front of it
Anyone have good ideas what this thing could be called? :) Calling it "proxy" would confuse it with the dummy proxying feature of Dovecot. And calling it "smart proxy" isn't very clear either what it does, especially since I don't think it even is a proxy really.
So far the best I've come up with is a buzzwordy "IMAP accelerator".
IMAP collector? proxyfetch? proxyconnect?
--
Best regards,
Charles
participants (6)
-
Charles Marcus
-
Chris Davies
-
Ed W
-
Mike Korizek
-
Timo Sirainen
-
Willie Gillespie