[Dovecot] Thinking Outside the Box - Extending IMAP
Here's some thoughts I'd like to throw out there. I know it's not standard IMAP protocol but someone has to try new ideas first and I want to see what people (Timo) think of this.
IMAP establishes a connection between the client and the server. Wouldn't it be great if it could be a conduit to let custom Thunderbird plugins talk to custom server application over the IMAP interface? For example, personalized server settings. Suppose for example I want Thunderbird to edit my server side white lists or black lists or any other setting? Wouldn't it be nice if IMAP supported these changes? The connection is made. It's a secure connection that's been authenticated. Lets use it!
Here's my initial thoughts on this. Suppose we extended IMAP to include an EXECUTE command as follows:
EXECUTE command parameter, parameter ....
On the server side is a config file that has the commands that execute will allow and what programs they run. When the execute command is seen by Dovecot then Dovecot runs the program in the list with the parameters passed. For example, suppose there is a command to add a user to a server side blacklist.
100 execute blacklist add joe@smith.com 100 ok
Dovecot would open a two way connection to the server application allowing the client to talk to any application that is configured and can send and receive text. The connection persists until the server end terminates or the client closes the connection.
With a tool like this one can write generic applications easily that would greatly expand what email clients can do interacting with the server. Not only can setting be changes but you could interact with server side calendars, pick up voice messages from phone systems, run any sort of groupware, all over a generic IMAP connection with this simple extension.
Example:
100 EXECUTE calendar 100 ok 100 list schedule today 8:00 10:00 100 8:00 make coffee 100 9:00 meeting with boss 100 9:30 Call Joe Blow at 415-555-1212 100 ok 100 quit 100 ok
One thing I'd like to use it for is an outgoing SMTP connection to send outgoing email over IMAP. A session might look like this:
999 EXECUTE smtp 999 220 darwin.ctyme.com ESMTP Exim 4.67 Sun, 13 May 2007 06:52:26 -0700 999 helo ctyme.com 999 250 darwin.ctyme.com Hello localhost [127.0.0.1] 999 mail from:marc@perkel.com 999 250 OK 999 rcpt to:dovecot@dovecot.org 999 250 Accepted ...... 999 quit 999 OK
Who likes this idea?
I love this idea! I know it wouldn't be part of the standard IMAP specification, and you'd probably have to have an extra capability such as CUSTOMEXECUTE or something in the IMAP capabilities, which Thunderbird would have to look for before it allows the use of the plugins.
As you suggested, this would make it so much easier to create an Exchange-style account with Thunderbird/other client software.
Anyone else?
Andy.
Marc Perkel wrote:
Here's some thoughts I'd like to throw out there. I know it's not standard IMAP protocol but someone has to try new ideas first and I want to see what people (Timo) think of this.
IMAP establishes a connection between the client and the server. Wouldn't it be great if it could be a conduit to let custom Thunderbird plugins talk to custom server application over the IMAP interface? For example, personalized server settings. Suppose for example I want Thunderbird to edit my server side white lists or black lists or any other setting? Wouldn't it be nice if IMAP supported these changes? The connection is made. It's a secure connection that's been authenticated. Lets use it!
Here's my initial thoughts on this. Suppose we extended IMAP to include an EXECUTE command as follows:
EXECUTE command parameter, parameter ....
On the server side is a config file that has the commands that execute will allow and what programs they run. When the execute command is seen by Dovecot then Dovecot runs the program in the list with the parameters passed. For example, suppose there is a command to add a user to a server side blacklist.
100 execute blacklist add joe@smith.com 100 ok
Dovecot would open a two way connection to the server application allowing the client to talk to any application that is configured and can send and receive text. The connection persists until the server end terminates or the client closes the connection.
With a tool like this one can write generic applications easily that would greatly expand what email clients can do interacting with the server. Not only can setting be changes but you could interact with server side calendars, pick up voice messages from phone systems, run any sort of groupware, all over a generic IMAP connection with this simple extension.
Example:
100 EXECUTE calendar 100 ok 100 list schedule today 8:00 10:00 100 8:00 make coffee 100 9:00 meeting with boss 100 9:30 Call Joe Blow at 415-555-1212 100 ok 100 quit 100 ok
One thing I'd like to use it for is an outgoing SMTP connection to send outgoing email over IMAP. A session might look like this:
999 EXECUTE smtp 999 220 darwin.ctyme.com ESMTP Exim 4.67 Sun, 13 May 2007 06:52:26 -0700 999 helo ctyme.com 999 250 darwin.ctyme.com Hello localhost [127.0.0.1] 999 mail from:marc@perkel.com 999 250 OK 999 rcpt to:dovecot@dovecot.org 999 250 Accepted ...... 999 quit 999 OK
Who likes this idea?
!DSPAM:37,4647190f343946551815980!
Andy Shellam wrote:
I love this idea! I know it wouldn't be part of the standard IMAP specification, and you'd probably have to have an extra capability such as CUSTOMEXECUTE or something in the IMAP capabilities, which Thunderbird would have to look for before it allows the use of the plugins.
As you suggested, this would make it so much easier to create an Exchange-style account with Thunderbird/other client software.
Anyone else?
Andy.
Thanks Andy. You literally could do anything with it. All you have to do is create a server and a client and let IMAP establish a connection. Seems to me it would be pretty easy to code up. The executed program would need environment variables set so that the server side would know the login name and IP address of the connection in case you wanted to restrict or log that.
So - who wants to code it up? I'll write a few server end pieces to say edit text files to be used for server side block lists with Exim. Need someone to code it into Dovecot, and need someone to write a Thunderbird plugin that would add a "blacklist sender" button to send the command.
How hard can it be? Right?
I vote for making a custom Thunderbird plugin that will have a standard "blacklist/whitelist" for the exact sender or the whole domain (like Outlook 2003's junk handling.) E.g. I could black/whitelist "marc@perkel.com" or "perkel.com." This would then send the "execute blacklist <data>" command and the server-side will then be custom to whatever you need.
E.g. you say your black/whitelists are in text files, mine are in a PostgreSQL database, but the front-end client interface would be exactly the same regardless.
It depends if Thunderbird plugins can interact with the IMAP connection, as most of the plugins I've seen are all client-side functions.
Andy.
Marc Perkel wrote:
Andy Shellam wrote:
I love this idea! I know it wouldn't be part of the standard IMAP specification, and you'd probably have to have an extra capability such as CUSTOMEXECUTE or something in the IMAP capabilities, which Thunderbird would have to look for before it allows the use of the plugins.
As you suggested, this would make it so much easier to create an Exchange-style account with Thunderbird/other client software.
Anyone else?
Andy.
Thanks Andy. You literally could do anything with it. All you have to do is create a server and a client and let IMAP establish a connection. Seems to me it would be pretty easy to code up. The executed program would need environment variables set so that the server side would know the login name and IP address of the connection in case you wanted to restrict or log that.
So - who wants to code it up? I'll write a few server end pieces to say edit text files to be used for server side block lists with Exim. Need someone to code it into Dovecot, and need someone to write a Thunderbird plugin that would add a "blacklist sender" button to send the command.
How hard can it be? Right?
!DSPAM:37,46471e50343942291814979!
OK - many of you responded very positively to this idea and I'd really like to see it happen. My gut level feeling is that this wouldn't be too hard to get working. So I'm willing to kick in $500 to make it happen. And if any of you who like this idea agree then maybe others could pledge something.
If this works I'm going to find a Thunderbird programmer to write some simple plugins that will put up some buttons to send simple messages to a server to do a few simple things like one line messages to add or remove email addresses from white/black lists.
I believe this feature is the next big revolution in expanding email functionality and I'm willing to put some bucks behind making it happen. I'm not a rich guy but with my $500 maybe I can get someone interested. Maybe other will be willing to kick in a few dollars or euros to make it happen.
Thoughts?
Andy Shellam wrote:
I love this idea! I know it wouldn't be part of the standard IMAP specification, and you'd probably have to have an extra capability such as CUSTOMEXECUTE or something in the IMAP capabilities, which Thunderbird would have to look for before it allows the use of the plugins.
As you suggested, this would make it so much easier to create an Exchange-style account with Thunderbird/other client software.
Anyone else?
Andy.
Marc Perkel wrote:
Here's some thoughts I'd like to throw out there. I know it's not standard IMAP protocol but someone has to try new ideas first and I want to see what people (Timo) think of this.
IMAP establishes a connection between the client and the server. Wouldn't it be great if it could be a conduit to let custom Thunderbird plugins talk to custom server application over the IMAP interface? For example, personalized server settings. Suppose for example I want Thunderbird to edit my server side white lists or black lists or any other setting? Wouldn't it be nice if IMAP supported these changes? The connection is made. It's a secure connection that's been authenticated. Lets use it!
Here's my initial thoughts on this. Suppose we extended IMAP to include an EXECUTE command as follows:
EXECUTE command parameter, parameter ....
On the server side is a config file that has the commands that execute will allow and what programs they run. When the execute command is seen by Dovecot then Dovecot runs the program in the list with the parameters passed. For example, suppose there is a command to add a user to a server side blacklist.
100 execute blacklist add joe@smith.com 100 ok
Dovecot would open a two way connection to the server application allowing the client to talk to any application that is configured and can send and receive text. The connection persists until the server end terminates or the client closes the connection.
With a tool like this one can write generic applications easily that would greatly expand what email clients can do interacting with the server. Not only can setting be changes but you could interact with server side calendars, pick up voice messages from phone systems, run any sort of groupware, all over a generic IMAP connection with this simple extension.
Example:
100 EXECUTE calendar 100 ok 100 list schedule today 8:00 10:00 100 8:00 make coffee 100 9:00 meeting with boss 100 9:30 Call Joe Blow at 415-555-1212 100 ok 100 quit 100 ok
One thing I'd like to use it for is an outgoing SMTP connection to send outgoing email over IMAP. A session might look like this:
999 EXECUTE smtp 999 220 darwin.ctyme.com ESMTP Exim 4.67 Sun, 13 May 2007 06:52:26 -0700 999 helo ctyme.com 999 250 darwin.ctyme.com Hello localhost [127.0.0.1] 999 mail from:marc@perkel.com 999 250 OK 999 rcpt to:dovecot@dovecot.org 999 250 Accepted ...... 999 quit 999 OK
Who likes this idea?
!DSPAM:37,4647190f343946551815980!
On Sat, 2007-05-19 at 07:06 -0700, Marc Perkel wrote:
I believe this feature is the next big revolution in expanding email functionality and I'm willing to put some bucks behind making it happen. I'm not a rich guy but with my $500 maybe I can get someone interested. Maybe other will be willing to kick in a few dollars or euros to make it happen.
Thoughts?
My thoughts:
It should be pretty easy add new commands with plugins.
Unless the commands are at least becoming an accepted standard (or possibly otherwise widely used), I won't include them in the main Dovecot tarball. But they can be listed in the Wiki of course.
I'm not really interested in implementing it. I'd rather get all the other things done that I already have in my TODO list. Also it looks like I'm going to get paid for many of them (I'll probably write something about this later when everything is clear).
Timo Sirainen wrote:
On Sat, 2007-05-19 at 07:06 -0700, Marc Perkel wrote:
I believe this feature is the next big revolution in expanding email functionality and I'm willing to put some bucks behind making it happen. I'm not a rich guy but with my $500 maybe I can get someone interested. Maybe other will be willing to kick in a few dollars or euros to make it happen.
Thoughts?
My thoughts:
It should be pretty easy add new commands with plugins.
Unless the commands are at least becoming an accepted standard (or possibly otherwise widely used), I won't include them in the main Dovecot tarball. But they can be listed in the Wiki of course.
I'm not really interested in implementing it. I'd rather get all the other things done that I already have in my TODO list. Also it looks like I'm going to get paid for many of them (I'll probably write something about this later when everything is clear).
My offer is for anyone who will write it. Who wants to make $500?
Timo Sirainen wrote:
On Sat, 2007-05-19 at 07:06 -0700, Marc Perkel wrote:
I believe this feature is the next big revolution in expanding email functionality and I'm willing to put some bucks behind making it happen. I'm not a rich guy but with my $500 maybe I can get someone interested. Maybe other will be willing to kick in a few dollars or euros to make it happen.
Thoughts?
My thoughts:
It should be pretty easy add new commands with plugins.
Unless the commands are at least becoming an accepted standard (or possibly otherwise widely used), I won't include them in the main Dovecot tarball. But they can be listed in the Wiki of course.
I'm not really interested in implementing it. I'd rather get all the other things done that I already have in my TODO list. Also it looks like I'm going to get paid for many of them (I'll probably write something about this later when everything is clear).
That's acceptable not being a documented feature. Probably better. This is somewhat just totally experimental with the idea that if it gets exciting then the idea will become a standard and probably anything developed for it could be changed to meet the standard.
On May 13, 2007, at 9:55 AM, Marc Perkel wrote:
Here's some thoughts I'd like to throw out there. I know it's not
standard IMAP protocol but someone has to try new ideas first and I
want to see what people (Timo) think of this.IMAP establishes a connection between the client and the server.
Wouldn't it be great if it could be a conduit to let custom
Thunderbird plugins talk to custom server application over the IMAP
interface? For example, personalized server settings. Suppose for
example I want Thunderbird to edit my server side white lists or
black lists or any other setting? Wouldn't it be nice if IMAP
supported these changes? The connection is made. It's a secure
connection that's been authenticated. Lets use it!Here's my initial thoughts on this. Suppose we extended IMAP to
include an EXECUTE command as follows:EXECUTE command parameter, parameter ....
On the server side is a config file that has the commands that
execute will allow and what programs they run. When the execute
command is seen by Dovecot then Dovecot runs the program in the
list with the parameters passed. For example, suppose there is a
command to add a user to a server side blacklist.100 execute blacklist add joe@smith.com 100 ok ...
I think this is a fantastic idea! I'm always opposed to
proprietary extensions to standardized protocols, but in this case,
it may be useful enough to gain acceptance. With Dovecot (as far as
I can tell) quickly becoming the dominant IMAP server, it probably
has the "push" to be able to pull this off.
Of course to make it useful you'd need some client
support...that's even more work than implementing it in the server,
multiple clients 'n all..
-Dave-- Dave McGuire Port Charlotte, FL
On Sun, 2007-05-13 at 06:55 -0700, Marc Perkel wrote:
IMAP establishes a connection between the client and the server. Wouldn't it be great if it could be a conduit to let custom Thunderbird plugins talk to custom server application over the IMAP interface? For example, personalized server settings. Suppose for example I want Thunderbird to edit my server side white lists or black lists or any other setting? Wouldn't it be nice if IMAP supported these changes? The connection is made. It's a secure connection that's been authenticated. Lets use it!
Here's my initial thoughts on this. Suppose we extended IMAP to include an EXECUTE command as follows:
EXECUTE command parameter, parameter ....
I think this could be done in almost standard way with METADATA extension (http://www.ietf.org/internet-drafts/draft-daboo-imap-annotatemore-11.txt). It allows setting and getting any key=value pairs in the server (also for mailboxes). Then you could have a Dovecot plugin that listens for some specific keys and handles them in whatever way you want.
100 execute blacklist add joe@smith.com 100 ok
METADATA doesn't really support adding/removing, but it could be kludged something like this:
1 setmetadata "" "/x-blacklist/joe@smith.com" (value.priv "1")
Although the joe@smith.com can't contain "*" or "%" characters. If that's needed then the string could be base64 encoded. Or the address could be in the value and the key would contain some kind of a unique ID.
Listing blacklist would work like:
1 getmetadata "/x-blacklist/* value.priv"
Anyway all of this would mean that if client implements a standard METADATA browser you could edit these kinds of lists without any special support from the client.
Timo,
Think outside the box on this. Create a new standard. Sometimes someone has to make the move and lead the standard instead of following it. This is new territory. Let's just try it and see where it goes.
On Sun, 2007-05-13 at 06:55 -0700, Marc Perkel wrote:
One thing I'd like to use it for is an outgoing SMTP connection to send outgoing email over IMAP. A session might look like this:
999 EXECUTE smtp 999 220 darwin.ctyme.com ESMTP Exim 4.67 Sun, 13 May 2007 06:52:26 -0700 999 helo ctyme.com 999 250 darwin.ctyme.com Hello localhost [127.0.0.1] 999 mail from:marc@perkel.com 999 250 OK 999 rcpt to:dovecot@dovecot.org 999 250 Accepted ...... 999 quit 999 OK
And as for this, I think embedding SMTP inside IMAP session only complicates things. LEMONADE support (http://www.ietf.org/html.charters/lemonade-charter.html http://www.lemonadeformobiles.com/index.html) is apparently being implemented in all kinds of mobile clients and probably will get used a lot more.
They're not using port 25 for sending messages, they're using port 587 which requires authentication (RFC 4409). So instead of trying to create a new non-standard implementation, why not just use the submission port, which is pretty much guaranteed to be the standard future way of sending messages?
On 5/13/07, Marc Perkel <marc@perkel.com> wrote:
Here's some thoughts I'd like to throw out there. I know it's not standard IMAP protocol but someone has to try new ideas first and I want to see what people (Timo) think of this.
Somewhat off topic.
I long time ago rather than making just a intranet/extranet website that dealt out reports I made a imap daemon in perl that would also serve the same report. Easily navigated virtual folders and subject lines that helped with the content and attachments.
The backend would handle the authentication, new/read/flag, caching the reports if needed, automatic folder subscription.
This was very useful for people that live in outlook, need to get price lists on the road. The phrase 'awesome the noon call center report is done' isn't something that you often think you would hear, but is priceless.
I also made a imap front end to bugzilla in much the same manner for people that couldn't handle bugzilla but that I wanted to be involved (project managers / executives). Product Folder->Component Subfolder->Threaded Bugs all coupled with a smtp2bugzilla gateway.
I find imap's ability to be secure and easy to integrate into the personalized desktop. For some things its better than the browser.
Only thing that will get you in some of the above examples is a maniac anti-virus administrator that wants all the outlook copies to rip through your server frequently ... ouch.
-- Gabriel Millerd
On Sun, 13 May 2007 06:55:59 -0700 Marc Perkel <marc@perkel.com> wrote:
Here's some thoughts I'd like to throw out there. I know it's not standard IMAP protocol but someone has to try new ideas first and I want to see what people (Timo) think of this.
IMAP establishes a connection between the client and the server. Wouldn't it be great if it could be a conduit to let custom Thunderbird plugins talk to custom server application over the IMAP interface? For example, personalized server settings. Suppose for example I want Thunderbird to edit my server side white lists or black lists or any other setting? Wouldn't it be nice if IMAP supported these changes? The connection is made. It's a secure connection that's been authenticated. Lets use it!
Here's my initial thoughts on this. Suppose we extended IMAP to include an EXECUTE command as follows:
EXECUTE command parameter, parameter ....
On the server side is a config file that has the commands that execute will allow and what programs they run. When the execute command is seen by Dovecot then Dovecot runs the program in the list with the parameters passed. For example, suppose there is a command to add a user to a server side blacklist.
100 execute blacklist add joe@smith.com 100 ok
I like the idea of having to and from added to the w/b list. That way it would be way nicer to use with SpamAssassin.
Dovecot would open a two way connection to the server application allowing the client to talk to any application that is configured and can send and receive text. The connection persists until the server end terminates or the client closes the connection.
With a tool like this one can write generic applications easily that would greatly expand what email clients can do interacting with the server. Not only can setting be changes but you could interact with server side calendars, pick up voice messages from phone systems, run any sort of groupware, all over a generic IMAP connection with this simple extension.
Example:
100 EXECUTE calendar 100 ok 100 list schedule today 8:00 10:00 100 8:00 make coffee 100 9:00 meeting with boss 100 9:30 Call Joe Blow at 415-555-1212 100 ok 100 quit 100 ok
One thing I'd like to use it for is an outgoing SMTP connection to send outgoing email over IMAP. A session might look like this:
999 EXECUTE smtp 999 220 darwin.ctyme.com ESMTP Exim 4.67 Sun, 13 May 2007 06:52:26 -0700 999 helo ctyme.com 999 250 darwin.ctyme.com Hello localhost [127.0.0.1] 999 mail from:marc@perkel.com 999 250 OK 999 rcpt to:dovecot@dovecot.org 999 250 Accepted ...... 999 quit 999 OK
Who likes this idea?
I love that idea. Especially if it could run it through a specified program for deliver as well as passing the username, from, and password to the program. It may seem like a silly idea to pass that to the program, but it does have a awesome usage. That way the backend program can fetch the outgoing SMTP server through some method and figure out how to send it. Been kicking around writing something like that for QPSMTPD. It would be nice for with squirrelmail.
Vulpes Velox wrote:
On Sun, 13 May 2007 06:55:59 -0700 Marc Perkel <marc@perkel.com> wrote:
Here's some thoughts I'd like to throw out there. I know it's not standard IMAP protocol but someone has to try new ideas first and I want to see what people (Timo) think of this.
IMAP establishes a connection between the client and the server. Wouldn't it be great if it could be a conduit to let custom Thunderbird plugins talk to custom server application over the IMAP interface? For example, personalized server settings. Suppose for example I want Thunderbird to edit my server side white lists or black lists or any other setting? Wouldn't it be nice if IMAP supported these changes? The connection is made. It's a secure connection that's been authenticated. Lets use it!
Here's my initial thoughts on this. Suppose we extended IMAP to include an EXECUTE command as follows:
EXECUTE command parameter, parameter ....
On the server side is a config file that has the commands that execute will allow and what programs they run. When the execute command is seen by Dovecot then Dovecot runs the program in the list with the parameters passed. For example, suppose there is a command to add a user to a server side blacklist.
100 execute blacklist add joe@smith.com 100 ok
I like the idea of having to and from added to the w/b list. That way it would be way nicer to use with SpamAssassin.
Dovecot would open a two way connection to the server application allowing the client to talk to any application that is configured and can send and receive text. The connection persists until the server end terminates or the client closes the connection.
With a tool like this one can write generic applications easily that would greatly expand what email clients can do interacting with the server. Not only can setting be changes but you could interact with server side calendars, pick up voice messages from phone systems, run any sort of groupware, all over a generic IMAP connection with this simple extension.
Example:
100 EXECUTE calendar 100 ok 100 list schedule today 8:00 10:00 100 8:00 make coffee 100 9:00 meeting with boss 100 9:30 Call Joe Blow at 415-555-1212 100 ok 100 quit 100 ok
One thing I'd like to use it for is an outgoing SMTP connection to send outgoing email over IMAP. A session might look like this:
999 EXECUTE smtp 999 220 darwin.ctyme.com ESMTP Exim 4.67 Sun, 13 May 2007 06:52:26 -0700 999 helo ctyme.com 999 250 darwin.ctyme.com Hello localhost [127.0.0.1] 999 mail from:marc@perkel.com 999 250 OK 999 rcpt to:dovecot@dovecot.org 999 250 Accepted ...... 999 quit 999 OK
Who likes this idea?
I love that idea. Especially if it could run it through a specified program for deliver as well as passing the username, from, and password to the program. It may seem like a silly idea to pass that to the program, but it does have a awesome usage. That way the backend program can fetch the outgoing SMTP server through some method and figure out how to send it. Been kicking around writing something like that for QPSMTPD. It would be nice for with squirrelmail.
I would think that when Dovecot executed the server end program that they would pass through in environment variables all the user name and connection info so that the server had that info for lookups.
Quoting Marc Perkel <marc@perkel.com>:
Here's some thoughts I'd like to throw out there. I know it's not standard IMAP protocol but someone has to try new ideas first and I want to see what people (Timo) think of this.
I don't think you will find many supporters, other than the usual crowd who always wants SMTP over IMAP support.
Here's my initial thoughts on this. Suppose we extended IMAP to include an EXECUTE command as follows:
You'd be better off either making these "generic action verbs" like "add" and "remove" (if each is meant to be self-sufficient, one-line actions), or calling them "modes" (if the idea is to switch contexts to some other protocol such as smtp, calendars, etc).
On the server side is a config file
Yes, that would be critical IMHO, or at least that there was some centralized way to control it, and not Dovecot-specific if you wanted the idea to spread past dovecot.
parameters passed. For example, suppose there is a command to add a user to a server side blacklist.
100 execute blacklist add joe@smith.com 100 ok
You'd be better off with a generic "add" which took paramters like:
100 add blacklist joe@smith.com 101 add user joe@mydomain.com 102 add whitelist joe@jones.com
You'd then of course have a "remove" or similar named function to undo the add. You could also do a "set" or "modify" and so on.
The first argument to "add" (or whatever) would say what to execute (via a map file/table, for example). This makes the implementation a bit more generic and independent of any executable.
Dovecot would open a two way connection to the server application allowing the client to talk to any application that is configured and can send and receive text. The connection persists until the server end terminates or the client closes the connection.
I'd rather see these as "plugin" type applications, but that certainly limits their adoption...
With a tool like this one can write generic applications easily that would greatly expand what email clients can do interacting with the server.
Sure, but why not just use a protocol for each, rather than layering them on top of IMAP? Do you really think the overhead of opening a new connection is so great? Debugging sure would be easier if they are separate...
Not only can setting be changes but you could interact with server side calendars, pick up voice messages from phone systems, run any sort of groupware, all over a generic IMAP connection with this simple extension.
This is more like the "mode" idea I mention above, IMHO.
Example:
100 EXECUTE calendar 100 ok 100 list schedule today 8:00 10:00 100 8:00 make coffee 100 9:00 meeting with boss 100 9:30 Call Joe Blow at 415-555-1212 100 ok 100 quit 100 ok
And when "quit" happens, what mode are you in? What if the task executed changed the IMAP state, how would IMAP know that? Seems like keeping track of state would be next to impossible, and would require the IMAP session to reinitialize after the execute command, which would be about the same overhead as just using another connection in the first place (a bit less, but...).
In other words, unless the task executed was specified as to never change the IMAP universe (couldn't deal with SPAM filtering, password changing, sorting/deleting/adding mail/folders, etc) then when it was done executing the IMAP server would have to re-initialize to catch any changes, and this kind of defeats the purpose of the proposal IMHO.
One thing I'd like to use it for is an outgoing SMTP connection to send outgoing email over IMAP. A session might look like this:
Yeah, I knew that was coming. :) And of course, this very well could change the IMAP environment, and hence would require an IMAP session reset, which means why do it?
Who likes this idea?
Not me. I think using different protocols for things...
Now, that doesn't mean dovecot couldn't support more protocols. Just as it already supports multiple protocols (pop3 and imap) it could add others... No reason not to, and not reason to piggy back them through the IMAP session, IMHO.
-- Eric Rostetter The Department of Physics The University of Texas at Austin
Go Longhorns!
Eric Rostetter wrote:
Here's my initial thoughts on this. Suppose we extended IMAP to include an EXECUTE command as follows:
You'd be better off either making these "generic action verbs" like "add" and "remove" (if each is meant to be self-sufficient, one-line actions), or calling them "modes" (if the idea is to switch contexts to some other protocol such as smtp, calendars, etc).
Generic action verbs are handy, but a LOT less flexible. They limit me to that set of actions for _all_ possible extensions... and whilst a good set can be found for many cases, the contortions it will force people into for cases that don't fit... well...
On the server side is a config file
Yes, that would be critical IMHO, or at least that there was some centralized way to control it, and not Dovecot-specific if you wanted the idea to spread past dovecot.
A general config form I like. My biggest concern on reading this was it opens one enormous can of security worms - albeit it self-inflicted by admins, essentially, Dovecot will still get blamed for making it easy.
With a tool like this one can write generic applications easily that would greatly expand what email clients can do interacting with the server.
Sure, but why not just use a protocol for each, rather than layering them on top of IMAP? Do you really think the overhead of opening a new connection is so great? Debugging sure would be easier if they are separate...
Whilst it does sort of make a step towards "IMAP as RPC", I think the nice thing about IMAP is the basic protocol level is quite solid - how to send/parse various data types, etc, is clear and clean, and is independent, pretty much, of the command set you implement. Take ACAP as an example - same low-level protocol, whole different command set.
Add to that the IMAP connection has already dealt with handshaking, authentication, and encryption...
Now, from what I've seen of Dovecot's innards, it probably wouldn't be too difficult to write your own protocol using the imap base, and build your own daemon...
Who likes this idea?
Not me. I think using different protocols for things...
Now, that doesn't mean dovecot couldn't support more protocols. Just as it already supports multiple protocols (pop3 and imap) it could add others... No reason not to, and not reason to piggy back them through the IMAP session, IMHO.
As you've probably guessed, I'm somewhat in favour of this approach. Dovecot as a base for more protocols. I like the IMAP base protocols, and Dovecot already provides the parsing and command dispatch for such.
One day... One day I'll have time... and in that time I'll try to implement a Calendar protocol, because I'm sick of all these polling-based implementations. But that will have to be after I hook LigHTTPd and Apache into Dovecot Auth, and .... all the billion other things I want to do :(
-- Curtis Maloney cmaloney@cardgate.net
On Sun, 2007-05-13 at 06:55 -0700, Marc Perkel wrote:
Here's some thoughts I'd like to throw out there. I know it's not standard IMAP protocol but someone has to try new ideas first and I want to see what people (Timo) think of this.
IMAP establishes a connection between the client and the server. Wouldn't it be great if it could be a conduit to let custom Thunderbird plugins talk to custom server application over the IMAP interface?
Why do you always want to stuff everything into IMAP?
For example, personalized server settings.
Isn't there some protocol similar to IMAP that solves this?
Who likes this idea?
I strongly disagree with this idea. Too little definition, too much server dependence, not portable across installations.
IMHO defining some behaviour that is so little related to the original purpose of IMAP is counterproductive.
Besides, why do you need to do this with IMAP? There's a protocol that supports all this already, it's called ssh. You can even tunnel pre-auth IMAP tunnels through the same ssh connection :)
johannes
I disagree about SSH.
Firstly, how do virtual users fit into your proposed setup? Secondly, as a service provider to the general public, the absolute LAST thing I want to be doing is opening up SSH access to my servers.
Mark has a valid point in that you have to connect to the server via IMAP to get your mail, why should you have to have a second protocol to do other things with the same mailbox? And why worry about a whole second set of authentication when you've got a pre-authenticated connection ready and waiting?
I agree it's not portable, and not ideal (ie. look at M$ Exchange's handling of custom server features), but Timo's suggestion of using the METADATA extension may strike the ideal balance between an extensible feature and the IMAP standard.
Andy.
Johannes Berg <johannes@sipsolutions.net> wrote:
On Sun, 2007-05-13 at 06:55 -0700, Marc Perkel wrote:
Here's some thoughts I'd like to throw out there. I know it's not standard IMAP protocol but someone has to try new ideas first and I want to see what people (Timo) think of this.
IMAP establishes a connection between the client and the server. Wouldn't it be great if it could be a conduit to let custom Thunderbird plugins talk to custom server application over the IMAP interface?
Why do you always want to stuff everything into IMAP?
For example, personalized server settings.
Isn't there some protocol similar to IMAP that solves this?
Who likes this idea?
I strongly disagree with this idea. Too little definition, too much server dependence, not portable across installations.
IMHO defining some behaviour that is so little related to the original purpose of IMAP is counterproductive.
Besides, why do you need to do this with IMAP? There's a protocol that supports all this already, it's called ssh. You can even tunnel pre-auth IMAP tunnels through the same ssh connection :)
johannes
DISCLAIMER
This e-mail was sent through a Mail Network server. The Mail Network accepts no liability for it's content.
Quoting andy.shellam-lists@mailnetwork.co.uk:
I disagree about SSH.
Good for you.
Firstly, how do virtual users fit into your proposed setup?
You can setup a ssh tunnel on the server on any port. The user then sets up to connect to that port. The authentication can be done anyway you want, or not at all. We're not talking ssh logins to the server, we're talking ssh tunneling.
Secondly, as a service provider to the general public, the absolute
LAST thing I want to be doing is opening up SSH access to my servers.
Why? An SSH tunnel can be useful, and can increase rather than decrease your security.
Now, if you meant logins, that would be a different story.
Mark has a valid point in that you have to connect to the server via IMAP to get your mail, why should you have to have a second protocol to do other things with the same mailbox?
Because some of these things may not involve the same mailbox? And because it makes for several smaller code bases which are then easier to secure, audit, debug, etc. (instead of one massive code base that is harder to secure, audit, debug, etc). And because then the authors only need to know and understand one protocol, instead of trying to know and understand any number of protocols that they may not use, have any interest in, etc. And so that those who only want 1 protocol can install only that protocol, and not have to worry about all the others? And, well, you should get the idea by now.
And why worry about a whole second set of authentication when you've got a pre-authenticated connection ready and waiting?
That was what the SSH connection was about I think.
I agree it's not portable, and not ideal (ie. look at M$ Exchange's handling of custom server features), but Timo's suggestion of using the METADATA extension may strike the ideal balance between an extensible feature and the IMAP standard.
Yes, but it only would handle some situations (maybe whitelist/blacklist) and not others (like SMTP) so it isn't what the original question was about really.
Andy.
-- Eric Rostetter The Department of Physics The University of Texas at Austin
Go Longhorns!
Hi Eric,
SSH tunnelling requires an SSH login though, right? Even if that's via PAM or other authentication method, it's another set of authentication to set up, why not keep it all in the same system (i.e. Dovecot?) I wouldn't allow SSH access connections to a public-facing server, mainly for the fact I'd have to disclose the details to my user-base, which the majority are non-techies, and it'd be harder to audit logins (there's only my team that login to each server so it's easy to spot if someone's intruding/attempting to.)
SSH is for tunneling a connection from the user's client machine to the server which encrypts data over the connection. Why should the average user have to download a piece of software (such as PuTTY), configure an SSH session and tunnel, connect to SSH, then run their usual mail client just to blacklist/whitelist a sender? It'd open up a world of interesting problems for our support guys and is a massive overhead that's unnecessary.
Why would a user logged into their mailbox want to make changes to another user's mailbox? More importantly, why would I, as the administrator, want to let them? If one user owns more than 1 mailbox, then they just switch over to the other account - like myself, my Thunderbird has both my -lists account and my normal account in the same window. It takes one click to switch accounts.
I also disagree with your comment about code security. The fact it requires another system, in whatever fashion, means there's a completely different code-base which could pose even more of a security problem rather than extending an existing code-base which is already well-known (and guaranteed) for it's security anyway? As Timo is offering 1,000 EUR to find a security hole, I very much doubt he'd release an extension that isn't secure. The way it's been suggested, Dovecot would run an external program to set/get the metadata anyway.
The comment about another set of authentication was, if you've already established an IMAP connection, why not use it for something else? If you take it as you said with SSH authentication, the user has to have authenticated via SSH first (to open the tunnel), then authenticates via IMAP. Sure if your IMAP component recognises an SSH tunnelled connection, and doesn't require authentication for it (not sure if Dovecot can be set up like this), then that closes the door for using your mail account on a different machine without having to set up the SSH tunnel as well. Using the suggested metadata extension, any IMAP-capable client could be configured to use it with no dependencies on anything else.
The original question was suggesting that Dovecot could extend the IMAP protocol to do other things (SMTP was an example, as was black- and white-listing.)
Andy.
Eric Rostetter wrote:
Quoting andy.shellam-lists@mailnetwork.co.uk:
I disagree about SSH.
Good for you.
Firstly, how do virtual users fit into your proposed setup?
You can setup a ssh tunnel on the server on any port. The user then sets up to connect to that port. The authentication can be done anyway you want, or not at all. We're not talking ssh logins to the server, we're talking ssh tunneling.
Secondly, as a service provider to the general public, the absolute
LAST thing I want to be doing is opening up SSH access to my servers.Why? An SSH tunnel can be useful, and can increase rather than decrease your security.
Now, if you meant logins, that would be a different story.
Mark has a valid point in that you have to connect to the server via IMAP to get your mail, why should you have to have a second protocol to do other things with the same mailbox?
Because some of these things may not involve the same mailbox? And because it makes for several smaller code bases which are then easier to secure, audit, debug, etc. (instead of one massive code base that is harder to secure, audit, debug, etc). And because then the authors only need to know and understand one protocol, instead of trying to know and understand any number of protocols that they may not use, have any interest in, etc. And so that those who only want 1 protocol can install only that protocol, and not have to worry about all the others? And, well, you should get the idea by now.
And why worry about a whole second set of authentication when you've got a pre-authenticated connection ready and waiting?
That was what the SSH connection was about I think.
I agree it's not portable, and not ideal (ie. look at M$ Exchange's handling of custom server features), but Timo's suggestion of using the METADATA extension may strike the ideal balance between an extensible feature and the IMAP standard.
Yes, but it only would handle some situations (maybe whitelist/blacklist) and not others (like SMTP) so it isn't what the original question was about really.
Andy.
Quoting Andy Shellam <andy.shellam-lists@mailnetwork.co.uk>:
Hi Eric,
SSH tunnelling requires an SSH login though, right?
No. SSH tunneling connects to a service, and that service can require authentication if it so desires. SSH tunneling is just port forwarding via SSH. It can be used to tunnel (and hence encrypt) pop3, imap, smtp, popassd, telnet, ftp, etc.
The client/server applications using an SSH tunnel will carry out their own authentication procedures (if any) the same way they would without the encrypted tunnel.
Even if that's via PAM or other authentication method, it's another set of authentication to set up
No, it uses the existing authentication already in the service, if any, and no additional authentication is needed.
why not keep it all in the same system (i.e. Dovecot?) I
I never moved it out. I only moved the encryption and some session managment out.
wouldn't allow SSH access connections to a public-facing server, mainly for the fact I'd have to disclose the details to my user-base, which
If you are already running IMAP, then you are already exposing this. Tunneling the IMAP via SSH exposes nothing more than the IMAP exposes, except that you are using an SSH tunnel (meaning if they can find an SSH vulnerability, then can crack you).
the majority are non-techies, and it'd be harder to audit logins (there's only my team that login to each server so it's easy to spot if someone's intruding/attempting to.)
No. It may be harder to diagnose problems, but not harder to audit logins. The logins are no different than before, only the transport is.
SSH is for tunneling a connection from the user's client machine to the server which encrypts data over the connection. Why should the average user have to download a piece of software (such as PuTTY), configure an SSH session and tunnel, connect to SSH, then run their usual mail client just to blacklist/whitelist a sender? It'd open up a world of interesting problems for our support guys and is a massive overhead that's unnecessary.
Yes, that would be fairly rediculous. You'd have to have support in your mail client or OS to make it worth doing.
Why would a user logged into their mailbox want to make changes to another user's mailbox?
Hopefully they wouldn't be allowed to. But one user might own multiple mailboxes, and want to manipulate them all. Plus there could be shared mailboxes, etc.
More importantly, why would I, as the administrator, want to let them? If one user owns more than 1 mailbox, then they just switch over to the other account - like myself, my Thunderbird has both my -lists account and my normal account in the same window. It takes one click to switch accounts.
Not sure why you bring this up actually. But there are of course obscure reasons (want to blacklist across all accounts, and not do it manually for each one, etc).
I also disagree with your comment about code security. The fact it requires another system, in whatever fashion, means there's a completely different code-base which could pose even more of a security problem rather than extending an existing code-base which is already well-known (and guaranteed) for it's security anyway? As Timo is offering 1,000 EUR to find a security hole, I very much doubt he'd release an extension that isn't secure. The way it's been suggested, Dovecot would run an external program to set/get the metadata anyway.
Yes, and so, the external program would be another code base not controlled by Timo, so... Well, you do the math.
There are various theories about small vs large code bases. But if you look at security statistics, you will see small code bases almost always win over big code bases, if all other things are held equal. Of course, people can debate that until the cows come home, can you can't really prove it one way or the other, and there are always exceptions.
The comment about another set of authentication was, if you've already established an IMAP connection, why not use it for something else?
That was the whole point of the SSH connection.
If you take it as you said with SSH authentication, the user has to have authenticated via SSH first (to open the tunnel), then
First, it was not my idea to use SSH. I was simply trying to correct your assumption that SSH tunnels would require an authentication. Second, they don't have to authenticate via SSH first, so your assumption is wrong.
the SSH tunnel as well. Using the suggested metadata extension, any IMAP-capable client could be configured to use it with no dependencies on anything else.
As I pointed out, that only addresses issues like black/whitelist, not issues like SMTP via IMAP.
The original question was suggesting that Dovecot could extend the IMAP protocol to do other things (SMTP was an example, as was black- and white-listing.)
Yes, and no solution provided so far has addressed both of those in any real way. The SSH one at least has potential to do so, whereas I see no way the METADATA solution could possible address both of those issues.
METADATA is for storing and retrieving data, not for performing an action. So, yes, it is a possible solution for things like updating a blacklist or whitelist, but not for things like sendmail an e-mail (unless you want to create a batch mechanism for doing so).
Andy.
-- Eric Rostetter The Department of Physics The University of Texas at Austin
Go Longhorns!
On Mon, 2007-05-14 at 11:39 -0500, Eric Rostetter wrote:
You can setup a ssh tunnel on the server on any port. The user then sets up to connect to that port. The authentication can be done anyway you want, or not at all. We're not talking ssh logins to the server, we're talking ssh tunneling.
Actually, I was thinking ssh logins :)
What I was thinking is this:
- user logs in via ssh
- ssh exposes a bunch of services like sftp and doesn't expose shell (iirc this is possible), except the service wouldn't be sftp but 'imap'
- the imap service you provide is a pre-authenticated imap session so that authentication/encryption is in ssh. I read my mail this way all the time.
- the ssh also provides a few other services that you can use
Yes, this'll need a new client, but no, it won't require installing putty, it'll require adding an ssh-transport to the imap protocol in your mail client. Evolution is capable of doing that already by a preauth tunnel command. But you need to modify the client anyway, so this isn't a concern.
Thus, what you get is exactly what you want: a service that provides multiple virtual services within a single existing connection. You ought to be able to partition users within ssh, maybe not with openssh but surely with some other implementation, virtual users should be possible too, there's nothing in the ssh protocol that forbids this.
Now, at this point, you might want to support imap-only clients. But that's probably easy too, just set up *dovecot* to do the authentication (might sound odd first) and then write a dovecot PAM module that allows ssh to authenticate against dovecot's auth service. Trivial, really. Then you can *also* expose plain imap (with TLS) on another port and therefore support legacy clients.
johannes
Quoting Johannes Berg <johannes@sipsolutions.net>:
On Mon, 2007-05-14 at 11:39 -0500, Eric Rostetter wrote:
You can setup a ssh tunnel on the server on any port. The user then sets up to connect to that port. The authentication can be done anyway you want, or not at all. We're not talking ssh logins to the server, we're talking ssh tunneling.
Actually, I was thinking ssh logins :)
Huh... Not sure why, but... This sounds like it would require both ssh server modifications and e-mail client modifications. As such, you may not get a lot of buy in to your idea. At that point, you're almost half way to creating a new protocol anyway...
- the imap service you provide is a pre-authenticated imap session so that authentication/encryption is in ssh. I read my mail this way all the time.
- the ssh also provides a few other services that you can use
Seems to me that instead of adding plugins to dovecot and the e-mail client, you've added "subsystems" and plugins to the ssh server and e-mail client. So you've just traded one server/client combination for another.
Thus, what you get is exactly what you want: a service that provides multiple virtual services within a single existing connection.
But since you've had to modify the client and server, why not just do this with any old client/server protocol? What is so special about ssh in this case?
I'd rather just tunnel the imap via ssh, and use the existing ssh tunnel to do pre-auth for other services... Seems more trivial, as we're only modifying the client, not the server... But what do I know/care. I've always been happy with multiple protocols.
One reason I like multiple protocols, each with their own server code, is that it scales well. I can put each service on a separate machine if I need to, I can re-prioritize them individually, I can proxy them with ease, etc. When you start jamming lots of protocols into one code base, not only is it harder to audit and debug, it is harder to scale. Yes, you can still scale with load balancers and such, but that introduces additional cost and complexity which isn't needed when the services are isolated.
But, I guess not every one needs to scale, and not everyone is on the server end (and yes, things always look different from the client end).
johannes
-- Eric Rostetter The Department of Physics The University of Texas at Austin
Go Longhorns!
participants (10)
-
Andy Shellam
-
andy.shellam-lists＠mailnetwork.co.uk
-
Curtis Maloney
-
Dave McGuire
-
Eric Rostetter
-
Gabriel Millerd
-
Johannes Berg
-
Marc Perkel
-
Timo Sirainen
-
Vulpes Velox