[Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend
We've been collecting some stats to see what kind of benefits UP/SquirrelMail's IMAP Proxy in for our SOGo webmail users. Dovecot is running in High-performance mode http://wiki2.dovecot.org/LoginProcess with authentication caching http://wiki2.dovecot.org/Authentication/Caching
During the weekend two servers (webmail3 and webmail4) has been running with local imapproxy and two servers without (webmail1 and webmail2). Each server has served about 1 million http requests, over 3 days.
server avg. response time # requests
webmail1.example.net 0.370411 1092386 webmail2.example.net 0.374227 1045141 webmail3.example.net 0.378097 1043919 imapproxy webmail4.example.net 0.378593 1028653 imapproxy
ONLY requests that took more than 5 seconds to process:
server avg. response time # requests
webmail1.example.net 26.048 1125 webmail2.example.net 26.2997 1080 webmail3.example.net 28.5596 808 imapproxy webmail4.example.net 27.1004 964 imapproxy
ONLY requests that took more than 10 seconds to process:
server avg. response time # requests
webmail1.example.net 49.1407 516 webmail2.example.net 53.0139 459 webmail3.example.net 59.7906 333 imapproxy webmail4.example.net 58.167 384 imapproxy
The responstimes are not very fast, but they do seem to support the claim that an imapproxy isn't needed for dovecot.
-jf
On 13.2.2012, at 15.54, Jan-Frode Myklebust wrote:
The responstimes are not very fast, but they do seem to support the claim that an imapproxy isn't needed for dovecot.
That's what I always suspected, but good to have someone actually test it. :) This is with Maildir?
Other things that would be interesting to try out (both from latency and disk IO usage point of view):
- maildir_very_dirty_syncs
- mail_prefetch_count (Linux+maildir only, v2.1+)
On Mon, Feb 13, 2012 at 04:14:22PM +0200, Timo Sirainen wrote:
The responstimes are not very fast, but they do seem to support the claim that an imapproxy isn't needed for dovecot.
That's what I always suspected, but good to have someone actually test it. :) This is with Maildir?
Yes, this is maildirs (on GPFS).
Other things that would be interesting to try out (both from latency and disk IO usage point of view):
- maildir_very_dirty_syncs
We already have
$ doveconf maildir_very_dirty_syncs
maildir_very_dirty_syncs = yesbut I don't think this gave the advantage I was expecting.. Was expecting this to move most iops to the index-luns, but the maildir luns seems just as busy.
- mail_prefetch_count (Linux+maildir only, v2.1+)
Will look into if this works with GPFS when we upgrade to v2.1. It has it's own page cache, so I have no idea if it will respect POSIX_FADV_WILLNEED or if one will need to use it's own API's for hinting:
http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=%2Fcom.ibm.cluster.gpfs.v3r4.gpfs300.doc%2Fbl1adm_mlacrge.html-jf
On 13.2.2012, at 21.36, Jan-Frode Myklebust wrote:
Other things that would be interesting to try out (both from latency and disk IO usage point of view):
- maildir_very_dirty_syncs
We already have
$ doveconf maildir_very_dirty_syncs maildir_very_dirty_syncs = yes
but I don't think this gave the advantage I was expecting.. Was expecting this to move most iops to the index-luns, but the maildir luns seems just as busy.
This setting should get rid of almost all readdir() calls. If it doesn't, something's not working right.
- mail_prefetch_count (Linux+maildir only, v2.1+)
Will look into if this works with GPFS when we upgrade to v2.1. It has it's own page cache, so I have no idea if it will respect POSIX_FADV_WILLNEED or if one will need to use it's own API's for hinting:
http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=%2Fco...
I guess if there's an easy way to lookup filename or fd -> blockNumber that wouldn't be difficult to implement with a plugin.
On Mon, Feb 13, 2012 at 09:57:31PM +0200, Timo Sirainen wrote:
$ doveconf maildir_very_dirty_syncs maildir_very_dirty_syncs = yes
but I don't think this gave the advantage I was expecting.. Was expecting this to move most iops to the index-luns, but the maildir luns seems just as busy.
This setting should get rid of almost all readdir() calls. If it doesn't, something's not working right.
With maildir_very_dirty_syncs = yes:
ReadMB/s WriteMB/s F_open f_close reads writes rdir inode 1.5 0.0 96 92 514 73 9 7 1.2 0.0 59 43 367 18 4 76 1.7 0.0 66 61 477 67 2 6 1.2 0.0 54 50 348 31 1 145 3.0 0.0 113 90 860 59 7 8 2.9 0.0 107 99 840 58 5 11 4.0 0.0 131 101 1117 77 2 65
With maildir_very_dirty_syncs = no (same node, a few seconds later):
ReadMB/s WriteMB/s F_open f_close reads writes rdir inode 4.6 0.9 125 91 1161 1096 41 6 2.3 0.7 200 170 697 127 86 16 1.1 0.6 124 99 406 61 48 109 2.7 0.1 212 144 755 114 74 15 2.7 0.0 159 133 818 70 78 194 0.8 1.2 86 73 225 60 16 9 1.9 0.0 124 116 573 53 30 6
So it seems to be working, good :-)
-jf
On Mon, Feb 13, 2012 at 5:54 AM, Jan-Frode Myklebust <janfrode@tanso.net> wrote:
We've been collecting some stats to see what kind of benefits UP/SquirrelMail's IMAP Proxy in for our SOGo webmail users. Dovecot is running in High-performance mode http://wiki2.dovecot.org/LoginProcess with authentication caching http://wiki2.dovecot.org/Authentication/Caching
During the weekend two servers (webmail3 and webmail4) has been running with local imapproxy and two servers without (webmail1 and webmail2). Each server has served about 1 million http requests, over 3 days.
server avg. response time # requests
webmail1.example.net 0.370411 1092386 webmail2.example.net 0.374227 1045141 webmail3.example.net 0.378097 1043919 imapproxy webmail4.example.net 0.378593 1028653 imapproxy
ONLY requests that took more than 5 seconds to process:
server avg. response time # requests
webmail1.example.net 26.048 1125 webmail2.example.net 26.2997 1080 webmail3.example.net 28.5596 808 imapproxy webmail4.example.net 27.1004 964 imapproxy
ONLY requests that took more than 10 seconds to process:
server avg. response time # requests
webmail1.example.net 49.1407 516 webmail2.example.net 53.0139 459 webmail3.example.net 59.7906 333 imapproxy webmail4.example.net 58.167 384 imapproxy
The responstimes are not very fast, but they do seem to support the claim that an imapproxy isn't needed for dovecot.
Out of curiosity, are you running dovecot locally on those webmail servers as well, or is it talking to remote dovecot servers? I ask because I'm looking at moving our webmail from an on-box setup to a remote pool to support director and was going to look into whether running imapproxyd would help there. We don't bother with it in the local setup, since dovecot is so fast, but remote (but still on a LAN) might be different. Though imapproxyd seems to make (wait for it...) squirrelmail unhappy (complains about IMAP errors, when sniffing shows none), though I've not bothered to debug it yet.
On Mon, Feb 13, 2012 at 11:08:48AM -0800, Mark Moseley wrote:
Out of curiosity, are you running dovecot locally on those webmail servers as well, or is it talking to remote dovecot servers?
The webmail servers are talking with dovecot director servers which in turn are talking with the backend dovecot servers. Each service running on different servers.
Webmail-servers -> director-servers -> backend-serversI ask because I'm looking at moving our webmail from an on-box setup to a remote pool to support director and was going to look into whether running imapproxyd would help there. We don't bother with it in the local setup, since dovecot is so fast, but remote (but still on a LAN) might be different.
Doesn't seem so to us...
Though imapproxyd seems to make (wait for it...) squirrelmail unhappy (complains about IMAP errors, when sniffing shows none), though I've not bothered to debug it yet.
:-)
-jf
On 13/02/2012 19:43, Jan-Frode Myklebust wrote:
On Mon, Feb 13, 2012 at 11:08:48AM -0800, Mark Moseley wrote:
Out of curiosity, are you running dovecot locally on those webmail servers as well, or is it talking to remote dovecot servers? The webmail servers are talking with dovecot director servers which in turn are talking with the backend dovecot servers. Each service running on different servers.
Webmail-servers -> director-servers -> backend-servers
I think the original question was still sensible. In your case it seems like the ping times are identical between: webmail -> imap-proxy webmail -> imap server
I think your results show that a proxy has little (or negative) benefit in this situation, but it seems feasible that a proxy could eliminate several RTT trips in the event that the proxy is "closer" than the imap server? This might happen if say the imap server is in a different datacenter (webmail on an office server machine?)
I'm also pleased to see that there is little negative cost in using a proxy... I recently added imap-proxy to our webmail setup because I wanted to log "last login + logout" times. I haven't quite figured out how to best log "logout time" (Timo, any chance of a post logout script? Or perhaps it's possible with the current login scripting?). However, using imap-proxy has the benefit of "clustering" logins a little and this makes log files a little easier to understand in the face of users with desktop mail clients plus webmail users. Possibly this idea useful to someone else...
Thanks for measuring this!
Ed W
On 21.2.2012, at 16.33, Ed W wrote:
I'm also pleased to see that there is little negative cost in using a proxy... I recently added imap-proxy to our webmail setup because I wanted to log "last login + logout" times. I haven't quite figured out how to best log "logout time" (Timo, any chance of a post logout script? Or perhaps it's possible with the current login scripting?).
You could of course grep the logs, but other than that you'd need to write a Dovecot plugin. Luckily it's really simple to write a plugin. Basically:
void postlogout_init(struct module *module) { } void postlogout_deinit(void) { system("/usr/local/bin/dovecot-postlogout.sh"); }
Add a few missing #includes and compile and enable for imap/pop3 and that should be it.
On 21/02/2012 20:36, Timo Sirainen wrote:
On 21.2.2012, at 16.33, Ed W wrote:
I'm also pleased to see that there is little negative cost in using a proxy... I recently added imap-proxy to our webmail setup because I wanted to log "last login + logout" times. I haven't quite figured out how to best log "logout time" (Timo, any chance of a post logout script? Or perhaps it's possible with the current login scripting?). You could of course grep the logs, but other than that you'd need to write a Dovecot plugin. Luckily it's really simple to write a plugin. Basically:
void postlogout_init(struct module *module) { } void postlogout_deinit(void) { system("/usr/local/bin/dovecot-postlogout.sh"); }
Add a few missing #includes and compile and enable for imap/pop3 and that should be it.
Thanks - that's really obvious and quite interesting. I guess a simple "log" plugin makes sense.
Quick followup question - the logout log file currently logs a bunch of statistics such as mails read/deleted, bytes sent/received. How might I access these from the _deinit context as above? Apologies if this is a RTFM question?
Finally, do you see it feasible to offer a "scriptable" plugin
interface, eg perhaps using some high performance scripting language
such as lua? Such a plugin might itself be simply a standard plugin..?
The motivation being to offer the ability to create plugins to those who
are nervous of using a compiler, and of course to reduce the ability of
a badly written plugin to kill dovecot?
Cheers
Ed W
On 22.2.2012, at 11.38, Ed W wrote:
void postlogout_init(struct module *module) { } void postlogout_deinit(void) { system("/usr/local/bin/dovecot-postlogout.sh"); }
Add a few missing #includes and compile and enable for imap/pop3 and that should be it.
Thanks - that's really obvious and quite interesting. I guess a simple "log" plugin makes sense.
Quick followup question - the logout log file currently logs a bunch of statistics such as mails read/deleted, bytes sent/received. How might I access these from the _deinit context as above? Apologies if this is a RTFM question?
You'd have to build separate plugins for POP3 and IMAP, and even then it becomes tricky since there's no simple hook for catching when client gets destroyed.
Finally, do you see it feasible to offer a "scriptable" plugin interface, eg perhaps using some high performance scripting language such as lua? Such a plugin might itself be simply a standard plugin..? The motivation being to offer the ability to create plugins to those who are nervous of using a compiler, and of course to reduce the ability of a badly written plugin to kill dovecot?
I've been thinking about adding a scripting language plugin to Dovecot. Perhaps even using one of the existing generators that are supposed to make this easy for multiple languages, such as SWIG. But this is pretty low priority currently..
On 22/02/2012 19:49, Timo Sirainen wrote:
On 22.2.2012, at 11.38, Ed W wrote:
void postlogout_init(struct module *module) { } void postlogout_deinit(void) { system("/usr/local/bin/dovecot-postlogout.sh"); }
Add a few missing #includes and compile and enable for imap/pop3 and that should be it. Thanks - that's really obvious and quite interesting. I guess a simple "log" plugin makes sense.
Quick followup question - the logout log file currently logs a bunch of statistics such as mails read/deleted, bytes sent/received. How might I access these from the _deinit context as above? Apologies if this is a RTFM question? You'd have to build separate plugins for POP3 and IMAP, and even then it becomes tricky since there's no simple hook for catching when client gets destroyed.
Do you think you could keep something similar on your low priority backlog? Clearly parsing log files or hacking the code is possible, but I think the interest in the login scripting shows there is general interest, and having a full log of logon/logoff/bytes is clearly interesting to more minority users?
Finally, do you see it feasible to offer a "scriptable" plugin interface, eg perhaps using some high performance scripting language such as lua? Such a plugin might itself be simply a standard plugin..? The motivation being to offer the ability to create plugins to those who are nervous of using a compiler, and of course to reduce the ability of a badly written plugin to kill dovecot? I've been thinking about adding a scripting language plugin to Dovecot. Perhaps even using one of the existing generators that are supposed to make this easy for multiple languages, such as SWIG. But this is pretty low priority currently..
I think SWIG is for wrapping dovecot's api into the scripting language?
(ie you could call dovecot methods from say perl/python/etc). What I had
in mind was the reverse, ie embed LUA inside dovecot. Whenever dovecot
normally calls a plugin method it will also run any [lua] scripts.
I'm sure you know how to use google, but just so we are on the same page, top hit (below) from google shows how straight forward this is (lua has been built to be extremely fast and easy to embed, ie it's not an arbitrary choice) http://heavycoder.com/tutorials/lua_embed.php
Cheers
Ed W
On Wed, 2012-02-22 at 22:54 +0000, Ed W wrote:
Quick followup question - the logout log file currently logs a bunch of statistics such as mails read/deleted, bytes sent/received. How might I access these from the _deinit context as above? Apologies if this is a RTFM question? You'd have to build separate plugins for POP3 and IMAP, and even then it becomes tricky since there's no simple hook for catching when client gets destroyed.
Do you think you could keep something similar on your low priority backlog? Clearly parsing log files or hacking the code is possible, but I think the interest in the login scripting shows there is general interest, and having a full log of logon/logoff/bytes is clearly interesting to more minority users?
What do you need the statistics for? I could make imap_client and pop3_client support some virtual methods, like user.destroy() initially, which would be enough for your use. I guess I could add that for v2.2.
Finally, do you see it feasible to offer a "scriptable" plugin interface, eg perhaps using some high performance scripting language such as lua? Such a plugin might itself be simply a standard plugin..? The motivation being to offer the ability to create plugins to those who are nervous of using a compiler, and of course to reduce the ability of a badly written plugin to kill dovecot? I've been thinking about adding a scripting language plugin to Dovecot. Perhaps even using one of the existing generators that are supposed to make this easy for multiple languages, such as SWIG. But this is pretty low priority currently..
I think SWIG is for wrapping dovecot's api into the scripting language?
(ie you could call dovecot methods from say perl/python/etc). What I had in mind was the reverse, ie embed LUA inside dovecot. Whenever dovecot normally calls a plugin method it will also run any [lua] scripts.
Yes, but I think SWIG can do that too. You'll need Dovecot's API implemented for scripting language if you want to do anything useful in the embedded script. That's the biggest job actually. Actually embeddeding some scripting language to Dovecot processes would be simple, they just couldn't really do anything useful.
Also the nice thing about generically implementing Dovecot's APIs means that they could be used to build independent Dovecot applications, not just plugins.
I'm sure you know how to use google, but just so we are on the same page, top hit (below) from google shows how straight forward this is (lua has been built to be extremely fast and easy to embed, ie it's not an arbitrary choice) http://heavycoder.com/tutorials/lua_embed.php
I've heard LUA being a commonly used embedded language, but I'd prefer to instead support several very widely used languages, such as Perl/Python.
On 22/02/2012 23:41, Timo Sirainen wrote:
I've heard LUA being a commonly used embedded language, but I'd prefer to instead support several very widely used languages, such as Perl/Python.
I'm a perl/ruby fan myself, but I would still recommend a good look at lua (or python) simply because they seem to be performant, easy to use, and on the surface seem to have had some thought about making them "embeddable".
My new favourite editor "Sublime Text 2" has python as it's scripting language. Lua has been used for some big name games amongst other things.
Perl has some memory management issues if you leave it long running, also writing XS code looks ok on the surface, but is an exercise in hair pulling in practice.... Ruby is a beautiful language, but unsure of how easy to embed and speed + memory management is an unknown (for high performance applications)
I think it has potential though. I think a lot of the current plugins on the website could easily be rewritten, likely without performance concerns, using a scripting based plugin system. I could see that some other big picture pieces could potentially benefit also
Thanks for considering it
Ed W
On Wed, 2012-02-22 at 23:56 +0000, Ed W wrote:
On 22/02/2012 23:41, Timo Sirainen wrote:
I've heard LUA being a commonly used embedded language, but I'd prefer to instead support several very widely used languages, such as Perl/Python.
I'm a perl/ruby fan myself, but I would still recommend a good look at lua (or python) simply because they seem to be performant, easy to use, and on the surface seem to have had some thought about making them "embeddable".
SWIG appears to generate Lua bindings just as well. But yes, I noticed Lua 5.1 has a feature called FFI that makes calling C functions quite easy (but then again, SWIG is supposed to do the same thing).
Perl has some memory management issues if you leave it long running, also writing XS code looks ok on the surface, but is an exercise in hair pulling in practice....
I've written a Perl plugin to irssi before directly with XS, and that's my main motivation for using some kind of an easy generator this time. It's way too much trouble to write any glue functions, especially when it works for only a single scripting language.
On 22/02/2012 23:56, Ed W wrote:
I think it has potential though. I think a lot of the current plugins on the website could easily be rewritten, likely without performance concerns, using a scripting based plugin system. I could see that some other big picture pieces could potentially benefit also
One interesting test case for such a scripting hooks solution might be login restrictions. There seem to be regular requests for the ability to setup arbitrarily complicated restrictions on users per IP, attempts per second, etc (and my logging interest is kind of related also).
Not trying to bump the item up any todo lists, just trying to chuck in some concrete ideas for actually testing a specific implementation...
I guess a substantially more performance orientated area that seems to get some interest would be various spam, expunge, delete ideas and the hooks needed for those. These seem much more tricky to implement a scripting hook and still stay performant. Again just ideas for real things people might want to do?
Cheers
Ed W
On Thu, 2012-02-23 at 01:41 +0200, Timo Sirainen wrote:
What do you need the statistics for? I could make imap_client and pop3_client support some virtual methods, like user.destroy() initially, which would be enough for your use. I guess I could add that for v2.2.
On 27/02/2012 08:34, Timo Sirainen wrote:
On Thu, 2012-02-23 at 01:41 +0200, Timo Sirainen wrote:
What do you need the statistics for? I could make imap_client and pop3_client support some virtual methods, like user.destroy() initially, which would be enough for your use. I guess I could add that for v2.2. http://dovecot.org/patches/2.2/imap-logout-plugin.c
Thanks - can I assume that a pop-logout would be basically the same?
Also, how might I access the bytes in/out statistics from that context?
Thanks
Ed W
On Mon, 2012-02-27 at 08:46 +0000, Ed W wrote:
On 27/02/2012 08:34, Timo Sirainen wrote:
On Thu, 2012-02-23 at 01:41 +0200, Timo Sirainen wrote:
What do you need the statistics for? I could make imap_client and pop3_client support some virtual methods, like user.destroy() initially, which would be enough for your use. I guess I could add that for v2.2. http://dovecot.org/patches/2.2/imap-logout-plugin.c
Thanks - can I assume that a pop-logout would be basically the same?
Yes, basically s/imap/pop3.
Also, how might I access the bytes in/out statistics from that context?
input: i_stream_get_absolute_offset(client->input) output: client->output->offset
On Tue, Feb 21, 2012 at 02:33:24PM +0000, Ed W wrote:
I think the original question was still sensible. In your case it seems like the ping times are identical between: webmail -> imap-proxy webmail -> imap server
I think your results show that a proxy has little (or negative) benefit in this situation, but it seems feasible that a proxy could eliminate several RTT trips in the event that the proxy is "closer" than the imap server? This might happen if say the imap server is in a different datacenter (webmail on an office server machine?)
The webmail/imapproxy were actually running in a different datacenter to the dovecot director/backend servers, but only about 20KM away.
Ping tests:
webmail->director:
rtt min/avg/max/mdev = 0.933/1.061/2.034/0.183 msdirector->backend:
rtt min/avg/max/mdev = 0.104/0.108/0.127/0.005 mswebmail->localhost:
rtt min/avg/max/mdev = 0.020/0.062/1.866/0.257 ms-jf
On 22/02/2012 08:25, Jan-Frode Myklebust wrote:
I think the original question was still sensible. In your case it seems like the ping times are identical between: webmail -> imap-proxy webmail -> imap server
I think your results show that a proxy has little (or negative) benefit in this situation, but it seems feasible that a proxy could eliminate several RTT trips in the event that the proxy is "closer" than the imap server? This might happen if say the imap server is in a different datacenter (webmail on an office server machine?) The webmail/imapproxy were actually running in a different datacenter to
On Tue, Feb 21, 2012 at 02:33:24PM +0000, Ed W wrote: the dovecot director/backend servers, but only about 20KM away.
Ping tests:
webmail->director:
rtt min/avg/max/mdev = 0.933/1.061/2.034/0.183 ms
director->backend:
rtt min/avg/max/mdev = 0.104/0.108/0.127/0.005 ms
webmail->localhost:
rtt min/avg/max/mdev = 0.020/0.062/1.866/0.257 ms
-jf
Hmm, not sure I understand the original numbers then?
It seems intuitive that the proxy installed locally could save you 2x RTT increment, which is about 0.8ms in your case. So I might expect the proxy to reduce rendering times by around 1.6ms simply because it reduces the number of round trips to login? Kind of curious why that's not achieved..?
Cheers
Ed W
On Wed, Feb 22, 2012 at 09:31:55AM +0000, Ed W wrote:
It seems intuitive that the proxy installed locally could save you 2x RTT increment, which is about 0.8ms in your case. So I might expect the proxy to reduce rendering times by around 1.6ms simply because it reduces the number of round trips to login? Kind of curious why that's not achieved..?
Each http-request can probably trigger several IMAP requests. Maybe these work better in parallel directly to dovecot, than serialized (?) trough the imapproxy ? No idea if that's what's happening... or maybe the imapproxy just adds more overhead than the 2xRTT + imap logins it's supposed to save us ?
-jf
Quoting Jan-Frode Myklebust <janfrode@tanso.net>:
We've been collecting some stats to see what kind of benefits UP/SquirrelMail's IMAP Proxy in for our SOGo webmail users. Dovecot is running in High-performance mode http://wiki2.dovecot.org/LoginProcess with authentication caching http://wiki2.dovecot.org/Authentication/Caching
During the weekend two servers (webmail3 and webmail4) has been running with local imapproxy and two servers without (webmail1 and webmail2). Each server has served about 1 million http requests, over 3 days.
server avg. response time # requests
webmail1.example.net 0.370411 1092386 webmail2.example.net 0.374227 1045141 webmail3.example.net 0.378097 1043919 imapproxy webmail4.example.net 0.378593 1028653 imapproxy
ONLY requests that took more than 5 seconds to process:
server avg. response time # requests
webmail1.example.net 26.048 1125 webmail2.example.net 26.2997 1080 webmail3.example.net 28.5596 808 imapproxy webmail4.example.net 27.1004 964 imapproxy
ONLY requests that took more than 10 seconds to process:
server avg. response time # requests
webmail1.example.net 49.1407 516 webmail2.example.net 53.0139 459 webmail3.example.net 59.7906 333 imapproxy webmail4.example.net 58.167 384 imapproxy
The responstimes are not very fast, but they do seem to support the claim that an imapproxy isn't needed for dovecot.
Except you are most likely NOT leveraging the truly interesting part
of imapproxy - the ability to restore the IMAP connection state via
the XPROXYREUSE status response. This is a significant performance
improvement since it also reduces processing load on the client side
(everything before/including authentication needs to be done whether
using imapproxy or not, so there is no client-side savings for these
commands).
For further information, see, e.g.:
http://lists.horde.org/archives/imp/Week-of-Mon-20110523/052316.html http://lists.horde.org/archives/imp/Week-of-Mon-20110523/052317.html
These posts neglect the fact that you don't need to issue a CAPABILITY
command if the connection is reused either, so that's an additional
advantage. Note that the XPROXYREUSE-enabled MUA must be the exclusive
user of the imapproxy instance for this feature to work correctly.
Somewhat topical, since Timo was just mentioning support for some sort
of IMAP state save/restore feature possibly making it into 2.2.
michael
On Mon, Feb 13, 2012 at 01:24:25PM -0700, Michael M Slusarz wrote:
Except you are most likely NOT leveraging the truly interesting part of imapproxy - the ability to restore the IMAP connection state via the XPROXYREUSE status response. This is a significant performance improvement since it also reduces processing load on the client side (everything before/including authentication needs to be done whether using imapproxy or not, so there is no client-side savings for these commands).
Thanks for this info, good to know. I'll check with inverse/sogo if this is something they use/intend to use..
additional advantage. Note that the XPROXYREUSE-enabled MUA must be the exclusive user of the imapproxy instance for this feature to work correctly.
Not a problem. Assuming it doesn't also need to be the only imap user of the account/folder.
BTW: do you also have information on the state of select caching in the up-imapproxy? I got some very negative comments when googling it, and the changelog didn't suggest there had been any improvements since..
-jf
Quoting Jan-Frode Myklebust <janfrode@tanso.net>:
BTW: do you also have information on the state of select caching in the up-imapproxy? I got some very negative comments when googling it, and the changelog didn't suggest there had been any improvements since..
I wouldn't trust it. IIRC, it was added years ago and given the
syntax changes to the SELECT/EXAMINE call since then (e.g. QRESYNC
extentsion), it is doubtful the code has been updated to handle these
situations. And in a server-caching IMAP server like dovecot, all
that SELECT information is not expensive anyway, so any gains would be
miniscule.
michael
On 13.2.2012, at 22.24, Michael M Slusarz wrote:
http://lists.horde.org/archives/imp/Week-of-Mon-20110523/052316.html http://lists.horde.org/archives/imp/Week-of-Mon-20110523/052317.html
These posts neglect the fact that you don't need to issue a CAPABILITY command if the connection is reused either, so that's an additional advantage. Note that the XPROXYREUSE-enabled MUA must be the exclusive user of the imapproxy instance for this feature to work correctly.
Somewhat topical, since Timo was just mentioning support for some sort of IMAP state save/restore feature possibly making it into 2.2.
Perhaps a way for (trusted) clients to be able to do this? :)
a logout save
- OK [SAVEDSTATE 1234567890]
- BYE logged out a OK
...
b login (SAVEDSTATE 1234567890) user pass
- OK [RESTOREDSTATE 1234567890] b ok
vs.
b login (SAVEDSTATE 1234567890) user pass
- NO [UNKNOWNSTATE 1234567890] b ok
On 13.2.2012, at 23.32, Timo Sirainen wrote:
Perhaps a way for (trusted) clients to be able to do this? :)
a logout save
- OK [SAVEDSTATE 1234567890]
- BYE logged out a OK
Actually, this could probably be safely implemented by sending all of the state to the client as a string:
- OK [SAVEDSTATE base64-encoded-state]
There isn't a whole lot of state to be saved really. Mailbox GUID, UIDVALIDITY, HIGHESTMODSEQ gives the mailbox state. Then you have the language/etc. states. Clients could restore their earlier state from days ago, as long as Dovecot still has the necessary .log records available (similar to how QRESYNC works).
Quoting Timo Sirainen <tss@iki.fi>:
On 13.2.2012, at 23.32, Timo Sirainen wrote:
Perhaps a way for (trusted) clients to be able to do this? :)
a logout save
- OK [SAVEDSTATE 1234567890]
- BYE logged out a OK
Actually, this could probably be safely implemented by sending all
of the state to the client as a string:
- OK [SAVEDSTATE base64-encoded-state]
There isn't a whole lot of state to be saved really. Mailbox GUID,
UIDVALIDITY, HIGHESTMODSEQ gives the mailbox state. Then you have
the language/etc. states. Clients could restore their earlier state
from days ago, as long as Dovecot still has the necessary .log
records available (similar to how QRESYNC works).
Given that it is not *that* expensive to re-create the state, I don't
think the ability to recreate state from several days ago would be
worth the effort of storing in the log/index files. For a
disconnected client (e.g. webmail), there's going to necessarily need
to be startup costs of initializing the session anyway so re-creating
the state can be buried in this other work.
Re-creating becomes more important when users are doing actions they
expect immediate (or at least very fast) responses for. Things like
listing messages in a mailbox, viewing a message, or polling
mailboxes. Saving 50ms per request becomes important in these
situations since re-creating state now takes 10% of the total request
time.
So I don't think states don't need a terribly long lifetime. I almost
think of something like an internal Dovecot IDLE queue - after a
logout is received, state is stored for 30 minutes after which it is
discarded. (Although not knowing anything about internal Dovecot
state, don't know if this is overly resource intensive.)
As for base-64-encoded state: if other IMAP servers ever wanted to
implement a similar protocol trying to coordinate the data structure
would be a nightmare. Keeping it to something like a MODSEQ value
would hide the state ID -> data abstraction entirely within dovecot.
And would allow you to change your mind in the future if
you come up with a better way to represent state.
michael
On 14.2.2012, at 5.30, Michael M Slusarz wrote:
Actually, this could probably be safely implemented by sending all of the state to the client as a string:
- OK [SAVEDSTATE base64-encoded-state]
There isn't a whole lot of state to be saved really. Mailbox GUID, UIDVALIDITY, HIGHESTMODSEQ gives the mailbox state. Then you have the language/etc. states. Clients could restore their earlier state from days ago, as long as Dovecot still has the necessary .log records available (similar to how QRESYNC works).
Given that it is not *that* expensive to re-create the state, I don't think the ability to recreate state from several days ago would be worth the effort of storing in the log/index files.
There wouldn't be *any* additional state stored. All the necessary state is already stored to .log files for other purposes. The base64-encoded-state would be a self-contained description of the IMAP connection's entire state.
As for base-64-encoded state: if other IMAP servers ever wanted to implement a similar protocol trying to coordinate the data structure would be a nightmare. Keeping it to something like a MODSEQ value would hide the state ID -> data abstraction entirely within dovecot. And would allow you to change your mind in the future if you come up with a better way to represent state.
The client doesn't need to care about the data structure. The client simply reads a string and sends it later back to server if it wants to restore that state. Only the server needs to verify that the string looks reasonable. The same string wouldn't be sent to a different server implementation, so there wouldn't be any interoperability issues. Each server can implement it in whatever way they want to (although there should be some checks in case the same string is sent to different versions of the same server).
Quoting Timo Sirainen <tss@iki.fi>:
On 13.2.2012, at 22.24, Michael M Slusarz wrote:
http://lists.horde.org/archives/imp/Week-of-Mon-20110523/052316.html http://lists.horde.org/archives/imp/Week-of-Mon-20110523/052317.html
These posts neglect the fact that you don't need to issue a
CAPABILITY command if the connection is reused either, so that's an
additional advantage. Note that the XPROXYREUSE-enabled MUA must be
the exclusive user of the imapproxy instance for this feature to
work correctly.Somewhat topical, since Timo was just mentioning support for some
sort of IMAP state save/restore feature possibly making it into 2.2.Perhaps a way for (trusted) clients to be able to do this? :)
a logout save
- OK [SAVEDSTATE 1234567890]
- BYE logged out a OK
...
b login (SAVEDSTATE 1234567890) user pass
- OK [RESTOREDSTATE 1234567890] b ok
vs.
b login (SAVEDSTATE 1234567890) user pass
- NO [UNKNOWNSTATE 1234567890] b ok
I guess the drawback for this approach is that you are explicitly
breaking the LOGIN definition. And you don't allow reviving the state
if using the AUTHENTICATE command.
The alternative would be to have an additional pre-authentication
command that sets the desired state. But that adds the overhead of
sending/parsing another command.
michael
On 14.2.2012, at 5.19, Michael M Slusarz wrote:
b login (SAVEDSTATE 1234567890) user pass
I guess the drawback for this approach is that you are explicitly breaking the LOGIN definition.
No breaking, extending :)
And you don't allow reviving the state if using the AUTHENTICATE command.
Just as easy:
b authenticate (savedstate ..) plain ..
The alternative would be to have an additional pre-authentication command that sets the desired state. But that adds the overhead of sending/parsing another command.
Yeah. Although not bad either, since you can pipeline that command + login/authenticate.
Quoting Timo Sirainen <tss@iki.fi>:
On 14.2.2012, at 5.19, Michael M Slusarz wrote:
b login (SAVEDSTATE 1234567890) user pass
I guess the drawback for this approach is that you are explicitly
breaking the LOGIN definition.No breaking, extending :)
Heh. I should know better after reading about 1000 RFCs last year...
Anyway, to play devil's advocate, another drawback of this approach is
that in the (fairly remote) chance that the LOGIN command is extended
in the future by an RFC, there is no guarantee that this syntax would
be used.
michael
participants (5)
-
Ed W
-
Jan-Frode Myklebust
-
Mark Moseley
-
Michael M Slusarz
-
Timo Sirainen