[Dovecot] Webmail app ... again.
There was a thread several weeks (or months) ago on webmail applications.
I didn't like any of the options and only found few that I did but they were either expensive or not maintained.
So ... I raised the question on the Catalyst Framework list about Perl based webmail apps and got nothing usable. It came down to what would it take to build a Perl (Cat) webmail app.
Several others on the list got interested so we decided to build it. The main issue I run into is my lack of understanding of the ins-and-outs of the IMAP specification and the features that come with using Dovecot. (Oh yeah. Dovecot was the preferred IMAP server!)
Another project I'm working on uses IMAP and what I found was there is a great quantity of Perl modules for dealing with IMAP. A great quantity! Unfortunately many of them assume a deeper knowledge of IMAP than I have so are very confusing.
What I'm looking for is a good reference, besides the RFC, on IMAP. Anything out there? Electronic or dead-tree is fine. (A book of Dovecot would be neat too.)
TIA, Rod
On Aug 13, 2008, at 4:05 PM, Roderick A. Anderson wrote:
What I'm looking for is a good reference, besides the RFC, on IMAP.
Anything out there? Electronic or dead-tree is fine. (A book of
Dovecot would be neat too.)
I wrote this a while ago: http://imapwiki.org/ClientImplementation
One thing that would be nice, that pretty much no webmail does, is to
keep a stateful connection open all the time (or at least some of the
time) instead of creating tons of short-lived connections that ask the
same stuff over and over again. With a stateful connection you could
basically run IDLE and wait for changes there instead of asking all
the time "is there new mail?" "is there new mail now?" "what about
now?".
Timo Sirainen wrote:
On Aug 13, 2008, at 4:05 PM, Roderick A. Anderson wrote:
What I'm looking for is a good reference, besides the RFC, on IMAP. Anything out there? Electronic or dead-tree is fine. (A book of Dovecot would be neat too.)
I wrote this a while ago: http://imapwiki.org/ClientImplementation
WOW! Thanks. I will use it. It is interesting to see this from the server view point.
One thing that would be nice, that pretty much no webmail does, is to keep a stateful connection open all the time (or at least some of the time) instead of creating tons of short-lived connections that ask the same stuff over and over again. With a stateful connection you could basically run IDLE and wait for changes there instead of asking all the time "is there new mail?" "is there new mail now?" "what about now?".
I've seen several messages on the list that seem to indicate some clients don't maintain the connection. It will be interesting to see what the implications of this. Not sure if it can be done from a HTTP connection as it is suppose to be stateless.
Again thanks.
Rod
On Aug 13 2008, Roderick A. Anderson wrote:
I've seen several messages on the list that seem to indicate some clients don't maintain the connection. It will be interesting to see what the implications of this. Not sure if it can be done from a HTTP connection as it is suppose to be stateless.
Have you investigated Prayer? That's what we use, in a modded version, because it maintains persistent IMAP connections (among other reasons). You could think of it as an IMAP client that happens to be using a web browser to draw back to your screen, as if it were X Windows. It's a sort of clumsy analogy but in the end that's more or less what it amounts to.
-- Brian Hayden UMN OIT Internet Services
Brian Hayden wrote:
Have you investigated Prayer? That's what we use, in a modded version, because it maintains persistent IMAP connections (among other reasons). You could think of it as an IMAP client that happens to be using a web browser to draw back to your screen, as if it were X Windows. It's a sort of clumsy analogy but in the end that's more or less what it amounts to.
Beat me to it, and me having spent most of the day porting my patches to Prayer 1.0.x to the new version 1.2.3 :)
When we chose Prayer, the persistent IMAP connections were the main selling point (as they were with "IMHO" a Roxen-webserver-based client written in Pike, a language nobody else seems to have heard of).
Prayer keeps an IMAP connection open for up to 10 minutes, AFAIR, but remembers the users session (session id is stored either in a cookie or the URL) for 30 minutes (or two hours on the compose screen).
Prayer's user interface is a bit old-fashioned, but is undergoing some serious rewriting at the moment (hence the need for porting my patches). It's server load is very low both on the webmail server and on the IMAP server.
I think Dovecot's cacheing may help the more common Webmail apps out there such as IMP and Squirrelmail and probably Roundcube (which does some cacheing of its own), plus there's imapproxy or Dovecot's auth cache if it's authentication that's expensive.
Another persistent IMAP Webmail app may be Web-Alpine from UW, but I haven't tried it out yet. If it's expecting to be talking to UW-IMAP it'll need to use persistent connections!
Chris
-- --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- Christopher Wakelin, c.d.wakelin@reading.ac.uk IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439 Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094
Chris Wakelin wrote:
Another persistent IMAP Webmail app may be Web-Alpine from UW, but I haven't tried it out yet. If it's expecting to be talking to UW-IMAP it'll need to use persistent connections!
While it's not free, a really nice webmail that does a lot of smart things (persistent imap connections, ldap connection pooling, and one of the best interfaces I've seen) is Nitido's PIM http://www.nitido.com/products/index.shtml?web_pim . It's used by a few of the larger US/Canadian ISPs for their webmail, as well as some big hosted email resellers. A bright group of guys too.
On Thursday, August 14 at 07:01 AM, quoth Eric Toczek:
While it's not free, a really nice webmail that does a lot of smart things (persistent imap connections, ldap connection pooling, and one of the best interfaces I've seen) is Nitido's PIM http://www.nitido.com/products/index.shtml?web_pim . It's used by a few of the larger US/Canadian ISPs for their webmail, as well as some big hosted email resellers. A bright group of guys too.
Do they have some better screenshots or a live demo anywhere?
~Kyle
Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it. -- Brian Kernighan
On Aug 13, 2008, at 4:59 PM, Roderick A. Anderson wrote:
One thing that would be nice, that pretty much no webmail does, is
to keep a stateful connection open all the time (or at least some
of the time) instead of creating tons of short-lived connections
that ask the same stuff over and over again. With a stateful
connection you could basically run IDLE and wait for changes there
instead of asking all the time "is there new mail?" "is there new
mail now?" "what about now?".I've seen several messages on the list that seem to indicate some
clients don't maintain the connection. It will be interesting to
see what the implications of this. Not sure if it can be done from
a HTTP connection as it is suppose to be stateless.
It doesn't have to rely on a stateful HTTP connection. You already
most likely use some server-stored state based on cookies/whatever. So
among that server state you could also keep the opened IMAP
connection. I guess the main issue here is that it doesn't work all
that nicely if the HTTP requests get redirected to different servers
randomly.
On Wed, Aug 13, 2008 at 04:37:11PM -0400, Timo Sirainen wrote:
One thing that would be nice, that pretty much no webmail does, is to keep a stateful connection open all the time (or at least some of the time) instead of creating tons of short-lived connections that ask the same stuff over and over again. With a stateful connection you could basically run IDLE and wait for changes there instead of asking all the time "is there new mail?" "is there new mail now?" "what about now?".
AlphaMail has this: http://alphamail.sourceforge.net/
It's a webmail written in Perl that uses a C++ "middleware" which keeps a persistent connection to the backend IMAP server.
But I'm afraid the project is dead...
GeertGeert Hendrickx wrote:
On Wed, Aug 13, 2008 at 04:37:11PM -0400, Timo Sirainen wrote:
One thing that would be nice, that pretty much no webmail does, is to keep a stateful connection open all the time (or at least some of the time) instead of creating tons of short-lived connections that ask the same stuff over and over again. With a stateful connection you could basically run IDLE and wait for changes there instead of asking all the time "is there new mail?" "is there new mail now?" "what about now?".
I'm sure "native" support would be better, but how is this different from using Squirrelmail with IMAPProxy?
-- Daniel
On Aug 13, 2008, at 6:53 PM, Daniel L. Miller wrote:
Geert Hendrickx wrote:
On Wed, Aug 13, 2008 at 04:37:11PM -0400, Timo Sirainen wrote:
One thing that would be nice, that pretty much no webmail does, is
to keep a stateful connection open all the time (or at least some of
the time) instead of creating tons of short-lived connections that ask
the same stuff over and over again. With a stateful connection you could basically run IDLE and wait for changes there instead of asking
all the time "is there new mail?" "is there new mail now?" "what about
now?".I'm sure "native" support would be better, but how is this different
from using Squirrelmail with IMAPProxy?
Squirrelmail still keeps asking for new mail etc. The only thing
imapproxy helps for is that the IMAP process stays alive longer. Maybe
helps a bit since there's less forking, but that's about it.
Daniel L. Miller wrote:
Geert Hendrickx wrote:
On Wed, Aug 13, 2008 at 04:37:11PM -0400, Timo Sirainen wrote:
One thing that would be nice, that pretty much no webmail does, is to keep a stateful connection open all the time (or at least some of the time) instead of creating tons of short-lived connections that ask the same stuff over and over again. With a stateful connection you could basically run IDLE and wait for changes there instead of asking all the time "is there new mail?" "is there new mail now?" "what about now?".
I'm sure "native" support would be better, but how is this different from using Squirrelmail with IMAPProxy?
Very interesting. I was thinking of hos to do something similar to IMAPProxy. Now I may not have to reinvent that wheel.
Thanks, Rod
On Aug 13, 2008, at 4:03 PM, Roderick A. Anderson wrote:
Daniel L. Miller wrote:
Geert Hendrickx wrote:
On Wed, Aug 13, 2008 at 04:37:11PM -0400, Timo Sirainen wrote:
One thing that would be nice, that pretty much no webmail does,
is to keep a stateful connection open all the time (or at least some of
the time) instead of creating tons of short-lived connections that
ask the same stuff over and over again. With a stateful connection you
could basically run IDLE and wait for changes there instead of asking
all the time "is there new mail?" "is there new mail now?" "what about
now?".I'm sure "native" support would be better, but how is this
different from using Squirrelmail with IMAPProxy?Very interesting. I was thinking of hos to do something similar to
IMAPProxy. Now I may not have to reinvent that wheel.
Seems to me you could use something like mod_perl to have state-
keeping processes running that can keep the connections alive, going
IDLE after 10 seconds or so after the last request they got. You can
limit the number of open connections from any given process with LRU
queuing, but I don't have suggestions on how to tie which process gets
which request. Perhaps by writing a middle-layer service that all the
processes talk to?
But the big killer is scaleability and handling multiple servers,
which is why some sort of front end like IMAPProxy are attractive.
Good luck
Sean
On Aug 14, 2008, at 1:26 AM, Sean Kamath wrote:
But the big killer is scaleability and handling multiple servers,
which is why some sort of front end like IMAPProxy are attractive.
I've heard that imapproxy isn't all that useful with Dovecot once auth
cache is enabled and set large enough. It'll then just basically
replace Dovecot's process fork(s) with the overhead of its own.
On Aug 13, 2008, at 10:32 PM, Timo Sirainen wrote:
On Aug 14, 2008, at 1:26 AM, Sean Kamath wrote:
But the big killer is scaleability and handling multiple servers,
which is why some sort of front end like IMAPProxy are attractive.I've heard that imapproxy isn't all that useful with Dovecot once
auth cache is enabled and set large enough. It'll then just
basically replace Dovecot's process fork(s) with the overhead of its
own.
Oops, good point, I'd forgotten about that whole discussion from a few
months ago.
So the only real benefit to keeping cached connections would be in
saving the TCP overhead, I guess...
Sean
Timo Sirainen escreveu:
I've heard that imapproxy isn't all that useful with Dovecot once auth cache is enabled and set large enough. It'll then just basically replace Dovecot's process fork(s) with the overhead of its own.
If I understand Dovecot's auth caching, it will save DB lookups (or sequential passwd-file lookups, etc), but it will still need to spawn a new imap process for each connection the webmail does.
With imapproxy, the process will be kept running for a couple minutes after the webmail disconnects that connection and will be reused if a new request is made in sequence.
On Aug 15, 2008, at 2:22 PM, Eduardo M KALINOWSKI wrote:
Timo Sirainen escreveu:
I've heard that imapproxy isn't all that useful with Dovecot once
auth cache is enabled and set large enough. It'll then just
basically replace Dovecot's process fork(s) with the overhead of
its own.If I understand Dovecot's auth caching, it will save DB lookups (or
sequential passwd-file lookups, etc), but it will still need to
spawn a new imap process for each connection the webmail does.With imapproxy, the process will be kept running for a couple
minutes after the webmail disconnects that connection and will be
reused if a new request is made in sequence.
Yes, but is it worth it to keep an extra daemon proxying all TCP
connetions to IMAP server just to save a some imap process creations?
With Dovecot v1.0/v1.1 I'm thinking "not really". v1.2 adds a bit more
state tracking that's a bit more expensive to calculate at startup.
v2.0 hopefully allows this kind of "wait a couple of minutes before
dying" natively.
Timo Sirainen wrote:
On Aug 15, 2008, at 2:22 PM, Eduardo M KALINOWSKI wrote:
If I understand Dovecot's auth caching, it will save DB lookups (or sequential passwd-file lookups, etc), but it will still need to spawn a new imap process for each connection the webmail does.
With imapproxy, the process will be kept running for a couple minutes after the webmail disconnects that connection and will be reused if a new request is made in sequence.
Yes, but is it worth it to keep an extra daemon proxying all TCP connetions to IMAP server just to save a some imap process creations?
Good question, only benchmarking could tell, and the results would probably vary between different machines, operating systems and architectures.
v2.0 hopefully allows this kind of "wait a couple of minutes before dying" natively.
That would be really nice.
-- BOFH excuse #52:
Smell from unhygienic janitorial staff wrecked the tape heads
Eduardo M KALINOWSKI eduardo@kalinowski.com.br http://move.to/hpkb
participants (10)
-
Brian Hayden
-
Chris Wakelin
-
Daniel L. Miller
-
Eduardo M KALINOWSKI
-
Eric Toczek
-
Geert Hendrickx
-
Kyle Wheeler
-
Roderick A. Anderson
-
Sean Kamath
-
Timo Sirainen