[Dovecot] dovecot evaluation on a 30 gb mailbox
hi all
i use qmail toaster. i recently changed over to dovecot.
posting below my evaluation on the same
my machine dual core dual xeon 5140 processor, with 4 gb ram, centos linux with qmailtoaster
i had setup a qmail tap account which backups up data for a client of mine
mailbox size over 30 gb
setup dovecot with squirrelmail and logged in
it had over one hundred and thirty nine thousand emails
139019 emails in the 30 gb mailbox
Reports
- first login : took 7 minutes to parse 139019 mails but never timed out.
this was simply impossible with courier
- the next login it took 5 seconds by which time the mail numbers
increased to 139045
- next i changed squirrelmail display preference to list 500 mails per page
and then clicked on inbox ... expecting dovecot to take atleast around 1-2 mins to show the mails
... the mails were displayed in just 8 seconds .. awesome performance
qmailtoaster is simply waiting for a dovecot integration
i am evaluating the stability of dovecot now
in the next 2-3 weeks i post stability report
rajesh
On 20.6.2010, at 3.33, Rajesh M wrote:
- next i changed squirrelmail display preference to list 500 mails per page
and then clicked on inbox ... expecting dovecot to take atleast around 1-2 mins to show the mails
... the mails were displayed in just 8 seconds .. awesome performance
Fetching metadata for 500 messages took 8 seconds? Still sounds slow. Although maybe it's because of maildir. I'm anyway not happy until Dovecot can open a mailbox with millions of messages and access them in nearly constant time (few milliseconds, with SSD).
Timo Sirainen put forth on 6/19/2010 9:43 PM:
On 20.6.2010, at 3.33, Rajesh M wrote:
- next i changed squirrelmail display preference to list 500 mails per page
and then clicked on inbox ... expecting dovecot to take atleast around 1-2 mins to show the mails
... the mails were displayed in just 8 seconds .. awesome performance
Fetching metadata for 500 messages took 8 seconds? Still sounds slow. Although maybe it's because of maildir. I'm anyway not happy until Dovecot can open a mailbox with millions of messages and access them in nearly constant time (few milliseconds, with SSD).
Roundcube 0.3.1 seems to have a hard coded limit of 200 mails per page. On a dual CPU 500 MHz Celeron with 384MB and a single 500GB SATA drive, running both RC and Dovecot 1.2.11 on the box, it takes about 5 seconds to pull metadata for each set of 200 messages on a 13k+ message mailbox. This is using mbox.
I agree with Timo, 8 seconds for 500 messages on that hardware seems a bit slow given what my lowly setup can do. Was the box under load when you tested or idle? What disk subsystem is the maildir stored on? Single disk or high performance RAID? Was the disk subsystem under load or idle when you tested?
Timo Sirainen put forth on 6/19/2010 9:43 PM:
On 20.6.2010, at 3.33, Rajesh M wrote:
- next i changed squirrelmail display preference to list 500 mails per
page and then clicked on inbox ... expecting dovecot to take atleast around 1-2 mins to show the mails
... the mails were displayed in just 8 seconds .. awesome performance
Fetching metadata for 500 messages took 8 seconds? Still sounds slow. Although maybe it's because of maildir. I'm anyway not happy until Dovecot can open a mailbox with millions of messages and access them in nearly constant time (few milliseconds, with SSD).
Roundcube 0.3.1 seems to have a hard coded limit of 200 mails per page. On a dual CPU 500 MHz Celeron with 384MB and a single 500GB SATA drive, running both RC and Dovecot 1.2.11 on the box, it takes about 5 seconds to pull metadata for each set of 200 messages on a 13k+ message mailbox. This is using mbox.
I agree with Timo, 8 seconds for 500 messages on that hardware seems a bit slow given what my lowly setup can do. Was the box under load when you tested or idle? What disk subsystem is the maildir stored on? Single disk or high performance RAID? Was the disk subsystem under load or idle when you tested?
-- Stan
stan
my machine is dual core dual processor xeon 1.6 gig proc 4 gb ram 1 tb sata for centos linux 1 tb sata for vpopmail domains data, mysql db and log data my hdd usage is at 60 percent now.
i accessed the large email box (this is qmail tap account in which emails are continuously added) yesterday 10 pm at night login took around 5 mins Around 169000 emails in the box
I logged out and logged in again after around 5 mins -- access took just around 4-5 seconds. Around 10 emails had been added.
at 10 am in the morning today i logged in again ie around 12 hours after above login.
it again took around 4-5 mins
i have currently over 170000 emails in the mailbox ie overnight around 1000 emails were added i logged off and access again -- the mailbox opened in around 5 seconds
so what i can conclude is that if relogin after a few minutes then the access is very fast whereas if i do the same after a few hours then it takes time.
i am new to dovecot and a few questions
- is it ok that dovecot takes 5 mins every time i login if there is
significant time difference between the two logins
- as per what timo said the metadata is cached so the data pertaining to
the extra 1000 emails were to be added to the cache .. does dovecot go thru all the 170000 emails again to form the meta data ? or is there some cache expiry period ?
thanks rajesh
Rajesh M put forth on 6/23/2010 12:12 AM:
- is it ok that dovecot takes 5 mins every time i login if there is
significant time difference between the two logins
No, 5 minutes is unacceptable to pretty much anyone.
- as per what timo said the metadata is cached so the data pertaining to
the extra 1000 emails were to be added to the cache .. does dovecot go thru all the 170000 emails again to form the meta data ? or is there some cache expiry period ?
It sounds like the entire mailbox is being read, and that would explain the 5 minute issue. I had a very similar problem not long ago with large mbox files which was solved with the mbox version of the parameter below.
Try setting:
maildir_very_dirty_syncs = yes
in dovecot.conf and see if that helps. Please read the documentation before doing so. If you do this it would probably be a really good idea to use Dovecot LDA for the mailbox delivery instead of having qmail write the mail. Having both qmail and dovecot writing and reading the same maildir could be problematic.
http://wiki.dovecot.org/LDA/Qmail
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Wed, 23 Jun 2010, Rajesh M wrote:
Roundcube 0.3.1 seems to have a hard coded limit of 200 mails per page.
it again took around 4-5 mins
I do not know nothing about Roundcube, therefore I wonder where the 5min delay come from:
a) Do you keep index files on Dovecot side?
b) Do you deliver / manipulate the mailbox via Dovecot only? E.g. do you use Dovecot deliver?
c) If you rawlog or sniff the connection and issue the same commands as roundcube to Dovecot via, say, telnet next morning, is the delay the same? http://wiki.dovecot.org/Debugging/Rawlog http://wiki.dovecot.org/TestInstallation?highlight=%28telnet%29
a) and b) should avoid rereading of the mailbox. c) is to ensure that Roundcube does not add the delay itself.
the extra 1000 emails were to be added to the cache .. does dovecot go thru all the 170000 emails again to form the meta data ? or is there some cache expiry period ?
Usually no.
Regards,
Steffen Kaiser
Il 23/06/2010 07:12, Rajesh M ha scritto:
stan
my machine is dual core dual processor xeon 1.6 gig proc 4 gb ram 1 tb sata for centos linux 1 tb sata for vpopmail domains data, mysql db and log data my hdd usage is at 60 percent now.
Hello, dunno what filesystem you are using. If it's ext3/4 I suggest you to mount it with noatime option.
Accessing a large Maildir folder with atime it's a pain. On an old server that in the peak time was at 12-15 load average I've lowered the load to < 5 only remounting with noatime option.
On 24.6.2010, at 5.45, Daniel L. Miller wrote:
On 6/19/2010 7:33 PM, Rajesh M wrote:
setup dovecot with squirrelmail and logged in
I'd recommend also installing and configuring imapproxy - it can be beneficial with squirrelmail.
Do you have any about a real world numbers about installation with and without imapproxy?
On 6/24/2010 4:23 AM, Timo Sirainen wrote:
I'd recommend also installing and configuring imapproxy - it can be beneficial with squirrelmail.
Do you have any about a real world numbers about installation with and without imapproxy?
What, you want me to actually back up that statement with data?! Who do you think I am?! Never mind - don't answer that.
I know when I was playing web clients - particularly squirrelmail - there was a definite perceived improvement - but I never measured it.
On Thu, 2010-06-24 at 16:58 -0700, Daniel L. Miller wrote:
On 6/24/2010 4:23 AM, Timo Sirainen wrote:
I'd recommend also installing and configuring imapproxy - it can be beneficial with squirrelmail.
Do you have any about a real world numbers about installation with and without imapproxy?
What, you want me to actually back up that statement with data?! Who do you think I am?! Never mind - don't answer that.
I know when I was playing web clients - particularly squirrelmail - there was a definite perceived improvement - but I never measured it.
Since every request (enter message, delete, forward, reply, etc etc etc) is a login -> action -> logout, it stands to reason on busy servers this will impact greatly, I saw a remarkable increase when using flat file years ago when changing to using imapproxy, but for many years now, using MySQL, it also was very noticeable improvement using it since it did not need to make all those extra concurrent requests.
If its running on a SOHO, you probably wouldn't even be able to measure the difference, but for an ISP/Telco or large institution, you will certainly notice the reduction in I/O (or loads on your database server). This goes for any imap webmail software, not just SM.
The advantage to this combination since about 2 months ago, the Squirrelmail team now also look after the imapproxy project :)
On 25.6.2010, at 1.37, Noel Butler wrote:
If its running on a SOHO, you probably wouldn't even be able to measure the difference, but for an ISP/Telco or large institution, you will certainly notice the reduction in I/O (or loads on your database server). This goes for any imap webmail software, not just SM.
But with auth cache enabled, there is no extra database load. The index files are also most likely in OS's cache (assuming local disk), so no extra disk I/O to read them either. I'm sure it's a bit more extra CPU usage, but I'm not all that certain that it's really that a big of a difference with Dovecot.
In my measurements I did years ago, I found imapproxy to give a small
improvement compared to 0.9x dovecot, I believe that went away in 1.x
versions.
I did use imapproxy on a few systems. But over time I finally gave up
on it, mainly cause it kept crashing, causing webmail to stop working,
and I didn't feel adding a watcher or looping it to keep it running to
be the correct fix. And given the very small improvment it gave, felt
it wasn't worth the hassle of dealing with.
Quoting Timo Sirainen tss@iki.fi:
On 25.6.2010, at 1.37, Noel Butler wrote:
If its running on a SOHO, you probably wouldn't even be able to measure the difference, but for an ISP/Telco or large institution, you will certainly notice the reduction in I/O (or loads on your database server). This goes for any imap webmail software, not just SM.
But with auth cache enabled, there is no extra database load. The
index files are also most likely in OS's cache (assuming local
disk), so no extra disk I/O to read them either. I'm sure it's a bit
more extra CPU usage, but I'm not all that certain that it's really
that a big of a difference with Dovecot.
On Fri, 2010-06-25 at 02:18 +0100, Timo Sirainen wrote:
On 25.6.2010, at 1.37, Noel Butler wrote:
If its running on a SOHO, you probably wouldn't even be able to measure the difference, but for an ISP/Telco or large institution, you will certainly notice the reduction in I/O (or loads on your database server). This goes for any imap webmail software, not just SM.
But with auth cache enabled, there is no extra database load. The index files are also most likely in OS's cache (assuming local disk), so no extra disk I/O to read them either. I'm sure it's a bit more extra CPU usage, but I'm not all that certain that it's really that a big of a difference with Dovecot.
auth_cache_size = 512 auth_cache_ttl = 3600 auth_cache_negative_ttl = 0
So, we use it at 512KB, perhaps this value was just too small, but admittedly, this change was made back in the early 0.9 days, and since imapproxy cured the problems, we (ok.. I :->) decided it shall be mandatory with all webmail servers since... I'll drop it off on one of them for a day or so and keep an eye on it.
On 2010-06-24 9:18 PM, Timo Sirainen wrote:
But with auth cache enabled, there is no extra database load. The index files are also most likely in OS's cache (assuming local disk), so no extra disk I/O to read them either. I'm sure it's a bit more extra CPU usage, but I'm not all that certain that it's really that a big of a difference with Dovecot.
We have very few web users, so I really didn't see anything noticeable performance wise, but I'll always use it just for the much cleaner logs.
On Friday 25 June 2010 wrote Charles Marcus:
On 2010-06-24 9:18 PM, Timo Sirainen wrote:
But with auth cache enabled, there is no extra database load. The index files are also most likely in OS's cache (assuming local disk), so no extra disk I/O to read them either. I'm sure it's a bit more extra CPU usage, but I'm not all that certain that it's really that a big of a difference with Dovecot.
We have very few web users, so I really didn't see anything noticeable performance wise, but I'll always use it just for the much cleaner logs.
For whatever it is worth, we use imapproxy, because it allows us to use one-time passwords for webmail users. One login, not many.
Amon Ott
Dr. Amon Ott - m-privacy GmbH Am Köllnischen Park 1, 10179 Berlin Tel: +49 30 24342334 Fax: +49 30 24342336 Web: http://www.m-privacy.de Handelsregister: Amtsgericht Charlottenburg HRB 84946 Geschäftsführer: Dipl.-Kfm. Holger Maczkowsky, Roman Maczkowsky GnuPG-Key-ID: EA898571
On Fri, 2010-06-25 at 14:44 +0200, Amon Ott wrote:
For whatever it is worth, we use imapproxy, because it allows us to use one-time passwords for webmail users. One login, not many.
With large enough auth cache, I think that should also work with Dovecot directly (although if the session takes longer than auth_cache_ttl, the user needs to re-login).
On Fri, 2010-06-25 at 13:49 +0100, Timo Sirainen wrote:
On Fri, 2010-06-25 at 14:44 +0200, Amon Ott wrote:
For whatever it is worth, we use imapproxy, because it allows us to use one-time passwords for webmail users. One login, not many.
With large enough auth cache, I think that should also work with Dovecot directly (although if the session takes longer than auth_cache_ttl, the user needs to re-login).
As I mentioned earlier It doesnt with squirrelmail, I tried it on production and the I/O increased a bit, watching logging alone was giving me a headache (no, not in debug mode either)..
So I thought I'd try it on a dev box, in case the cache wasnt large enough, but nope, incidentally imap with evolution works as you indicate it should, but webmail, not a hope in ..... it is login -action logout constantly, imapproxy avoids all this crud, so it obviously does something differently, maybe you find that and incorporate it in dovecot :) that way all this login logout stuff is only seen in say, debug mod? Though the overhead for running imapproxy is non existent it is still an needed devil in a busy environment.
On 26.6.2010, at 0.57, Noel Butler wrote:
As I mentioned earlier It doesnt with squirrelmail, I tried it on production and the I/O increased a bit, watching logging alone was giving me a headache (no, not in debug mode either)..
How much is "a bit" that the I/O increased? Less than 10%?
So I thought I'd try it on a dev box, in case the cache wasnt large enough, but nope, incidentally imap with evolution works as you indicate it should, but webmail, not a hope in ..... it is login -action logout constantly, imapproxy avoids all this crud, so it obviously does something differently, maybe you find that and incorporate it in dovecot :) that way all this login logout stuff is only seen in say, debug mod?
It still sounds like the only problem you have is the excessive log messages?.. Sure, Dovecot doesn't even try to get rid of those.
On Sat, 2010-06-26 at 01:02 +0100, Timo Sirainen wrote:
On 26.6.2010, at 0.57, Noel Butler wrote:
As I mentioned earlier It doesnt with squirrelmail, I tried it on production and the I/O increased a bit, watching logging alone was giving me a headache (no, not in debug mode either)..
How much is "a bit" that the I/O increased? Less than 10%?
perhaps around the 10-15% (see below)
So I thought I'd try it on a dev box, in case the cache wasnt large enough, but nope, incidentally imap with evolution works as you indicate it should, but webmail, not a hope in ..... it is login -action logout constantly, imapproxy avoids all this crud, so it obviously does something differently, maybe you find that and incorporate it in dovecot :) that way all this login logout stuff is only seen in say, debug mod?
It still sounds like the only problem you have is the excessive log messages?.. Sure, Dovecot doesn't even try to get rid of those.
this likely is the sole reason for the I/O increase, but yes. Oh and the local (replicated) DB was seeing maybe 40 requests a second rather than only a few, no big deal there though.
On 2010-06-25 8:44 AM, Amon Ott wrote:
For whatever it is worth, we use imapproxy, because it allows us to use one-time passwords for webmail users. One login, not many.
That's what I meant by cleaner logs... ;)
On 2010-06-25 8:49 AM, Timo Sirainen wrote:
With large enough auth cache, I think that should also work with Dovecot directly (although if the session takes longer than auth_cache_ttl, the user needs to re-login).
Great! Now to test this... thanks Timo... :)
hi,
possible to setup a imap server without smtp? just want to store some old emails.
Angelo
On 6/25/2010 10:05 AM, Charles Marcus wrote:
On 2010-06-25 9:58 AM, Angelo Chen wrote:
possible to setup a imap server without smtp? just want to store some old emails.
Of course, one has nothing to do with the other.
A lot of trouble, more than just importing them into your e-mail client! You would require no configuration, or very little, other than putting the e-mails in a home/usernameofpersononlinux/.mail folder and just logging into imap with your usernameofpersononlinux's password.
Quoting Timo Sirainen tss@iki.fi:
But with auth cache enabled, there is no extra database load. The
index files are also most likely in OS's cache (assuming local
disk), so no extra disk I/O to read them either. I'm sure it's a bit
more extra CPU usage, but I'm not all that certain that it's really
that a big of a difference with Dovecot.
Just the amount of logs generated could be significant for disk load... Not to mention network overhead and all the rest...
Again, your millage may vary...
Quoting "Daniel L. Miller" dmiller@amfes.com:
On 6/24/2010 4:23 AM, Timo Sirainen wrote:
I'd recommend also installing and configuring imapproxy - it can
be beneficial with squirrelmail.Do you have any about a real world numbers about installation with
and without imapproxy?What, you want me to actually back up that statement with data?!
Who do you think I am?! Never mind - don't answer that.
I don't have any numbers either, but...
I know when I was playing web clients - particularly squirrelmail -
there was a definite perceived improvement - but I never measured it.
Webmail clients are basically stateless. Over time, they have "cached" some info for performance, but basically they are still stateless. So almost any time you click on a link dealing with an email (delete, purge, next message, forward, etc) it has to open a _new_ connection to the imap (or pop) server, which means a new login.
Simple proxies (like say perdition) don't help, as each connection will still be a new login.
Some proxies (like imapproxy) however can keep a login session open, with the proxy caching the authentication issue. For each connection after the first, it can do the authentication itself and use the existing login connection to the pop/imap server, avoiding a login for each operation after the first (until a timeout is reached). By avoiding the constant login/logout cycle, it will generally perform at least slightly better with the proxy than without (no new connection overhead, no login and logout overhead, same over head most everywhere else unless the proxy's authentication is slow).
That's why a connection/authentication caching proxy is generally recommended for webmail setups. It also keeps the log files from filling up on the pop/imap server with the constant login/logout log lines.
This is also why you can get a speedup even if the proxy is on the same host as the pop/imap server (assume sufficient memory and other resources).
Daniel
Quoting Eric Rostetter rostetter@mail.utexas.edu:
Quoting "Daniel L. Miller" dmiller@amfes.com:
I know when I was playing web clients - particularly squirrelmail -
there was a definite perceived improvement - but I never measured it.Webmail clients are basically stateless. Over time, they have "cached" some info for performance, but basically they are still stateless. So almost any time you click on a link dealing with an email (delete, purge, next message, forward, etc) it has to open a _new_ connection to the imap (or pop) server, which means a new login.
Simple proxies (like say perdition) don't help, as each connection will still be a new login.
Some proxies (like imapproxy) however can keep a login session open, with the proxy caching the authentication issue. For each connection after the first, it can do the authentication itself and use the existing login connection to the pop/imap server, avoiding a login for each operation after the first (until a timeout is reached). By avoiding the constant
login/logout cycle, it will generally perform at least slightly better with the proxy than without (no new connection overhead, no login and logout overhead, same over head most everywhere else unless the proxy's authentication is slow).
However, imapproxy (at least as of v1.2.6) is still less than
desirable. Even though you are saving on the authentication overhead,
every time your stateless client (i.e. webmail) connects it still
needs to potentially issue a CAPABILITY command and/or other enabling
commands (e.g. 'ENABLE QRESYNC'). So that's still a bunch of overhead
that could potentially be saved.
Solution: make the imapproxy layer visible to the client. Patches
were added to imapproxy v1.2.7 to do just that. When IMAPPROXY is
being used, XIMAPPROXY is announced via the IMAP capability string.
More important, if the cached proxy connection is successfully reused,
a XPROXYREUSE status response is returned after the authentication
commands are completed. If this status return is observed by the
client, it knows that it can reuse its cached information about the
connection (if it exists locally on the client) without needing to
check on CAPABILITIES or issuing ENABLING commands. Additionally, it
can prevent unnecessary initialization code on the client side that
only needs to be done when a user logs into the server.
IMP 5 (more specifically, the Horde 4: Horde_Imap_Client socket
driver) does just this to prevent unnecessary traffic to the IMAP
server.
michael
Timo Sirainen put forth on 6/24/2010 6:23 AM:
On 24.6.2010, at 5.45, Daniel L. Miller wrote:
On 6/19/2010 7:33 PM, Rajesh M wrote:
setup dovecot with squirrelmail and logged in
I'd recommend also installing and configuring imapproxy - it can be beneficial with squirrelmail.
Do you have any about a real world numbers about installation with and without imapproxy?
IIRC the OP is running SM and Dovecot on the same host. Does running imapproxy in this scenario yield any benefit, given there is no network traffic involved?
Am Donnerstag, den 24.06.2010, 22:30 -0500 schrieb Stan Hoeppner:
Does running imapproxy in this scenario yield any benefit, given there is no network traffic involved?
it does. IMAPPRoxy caches for example all directories. If the user open a folder, then you have a new connection to the IMAP server, that can take a while (auth ...). After a few seconds the connection is closed (that's quite normal for webapplications like webmail etc.) and than the user requests a new folder ... . So the imapproxy can cache all these things and make the Webmail faster. The only problem is, if there are new folders, so maybe it can take a few minutes, before the folder is visible.
cu denny
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Fri, 25 Jun 2010, Denny Schierz wrote:
it does. IMAPPRoxy caches for example all directories. If the user open a folder, then you have a new connection to the IMAP server, that can take a while (auth ...). After a few seconds the connection is closed (that's quite normal for webapplications like webmail etc.) and than the user requests a new folder ... . So the imapproxy can cache all these things and make the Webmail faster. The only problem is, if there are
Hmm, IMHO Dovecot is to cache most of the stuff, isn't it? Esp. "LOGIN user pass" is cheap, because the password is cached.
new folders, so maybe it can take a few minutes, before the folder is visible.
imapproxy caches the output of LIST? On the website the FAQ only talks about "connections", for me it reads that imapproxy saves to open another TCP connection, the possible creation of other Dovecot processes and the LOGIN command.
Regards,
Steffen Kaiser
On 25.6.2010, at 8.04, Denny Schierz wrote:
Am Donnerstag, den 24.06.2010, 22:30 -0500 schrieb Stan Hoeppner:
Does running imapproxy in this scenario yield any benefit, given there is no network traffic involved?
it does. IMAPPRoxy caches for example all directories. If the user open a folder, then you have a new connection to the IMAP server, that can take a while (auth ...). After a few seconds the connection is closed (that's quite normal for webapplications like webmail etc.) and than the user requests a new folder ... . So the imapproxy can cache all these things and make the Webmail faster. The only problem is, if there are new folders, so maybe it can take a few minutes, before the folder is visible.
Last I looked, imapproxy had some kind of a SELECT cache and possibly something else. They were implemented in an unreliable way and unless something has changed, I wouldn't recommend enabling them. Also I doubt any of them help with performance with Dovecot.
Most people seem to be saying imapproxy helps a lot, because they noticed it helped so much with Courier/UW-IMAP..
Timo,
On 6/24/10 4:23 AM, "Timo Sirainen" tss@iki.fi wrote:
I'd recommend also installing and configuring imapproxy - it can be beneficial with squirrelmail.
Do you have any about a real world numbers about installation with and without imapproxy?
We run imapproxy behind our Roundcube instance, and our old in-house Perl mail system has a custom equivalent written in C that also does some caching of folder metadata and message headers.
We run a proxy instance on each of the webmail hosts, with the communication between the web application add the proxy being done in cleartext, but with the proxy -> Dovecot communication secured over SSL. Besides preventing a lot of extra SSL handshakes and login/logout actions, it also helps tie a user session to a single backend node in our pool of IMAP servers. It seems like there might also be other benefits to having Dovecot not tear down all of the user session state between page loads.
A lot of this stuff might be nice to see in the Director some day. If there was an option to not immediately close the Director proxy's backend connections when the user logs out, (ie leave the connection active and logged in for X seconds, and reuse it if the user logs in to the Director again), and if the auth caching works as well as you say, then I could definitely see a day where we replace imapproxy with a director instance on the webmail host.
-Brad
On 26.6.2010, at 3.08, Brandon Davidson wrote:
We run a proxy instance on each of the webmail hosts, with the communication between the web application add the proxy being done in cleartext, but with the proxy -> Dovecot communication secured over SSL. Besides preventing a lot of extra SSL handshakes and login/logout actions,
Yes, SSL handshakes are extra. Although SSL supports some kind of quick renegotiation too, but Dovecot doesn't support that yet. No one's ever requested it..
it also helps tie a user session to a single backend node in our pool of IMAP servers. It seems like there might also be other benefits to having Dovecot not tear down all of the user session state between page loads.
Some, but I suspect mostly CPU performance related (which usually doesn't matter).
A lot of this stuff might be nice to see in the Director some day. If there was an option to not immediately close the Director proxy's backend connections when the user logs out, (ie leave the connection active and logged in for X seconds, and reuse it if the user logs in to the Director again),
It hasn't really ever been in my plans to do anything like this. A separate imapproxy program for this is probably better. At least I don't really see any benefits compared to it if it was done by Dovecot.
I mainly just wish someone would give some kind of real numbers when running with and without imapproxy with Dovecot (and not some other server they used to run years ago) when they're talking about imapproxy helping the performance a lot. So far the best reports I've heard are "imapproxy improves a little bit, but adds too much complexity/fragility". So I'm a bit sceptical about using it.
I've also recently heard from a large installation who was complaining about Dovecot's memory usage (compared to Courier). imapproxy would definitely make their memory usage a lot higher, since the connections would stay around for longer. Although I'm still wondering why Dovecot would take so much more memory with large maildirs .. should look into that some day.
Timo Sirainen tss@iki.fi:
Yes, SSL handshakes are extra. Although SSL supports some kind of quick renegotiation too, but Dovecot doesn't support that yet. No one's ever requested it..
Hum... this article (in Norwegian) http://www.digi.no/881186/skrekkverktoy-slaar-ut-%ABsikre%BB-servere addresses the SSL renegotiation vulnerability, and how it can be used to DOS servers using SSL from a single machine with low bandwidth.
At the end the article is discussing how to configure off the SSL renegotiate in different servers, and that the author had been unable to find a setting for disabling SSL renegotiate in dovecot (and if anyone knows how, please inform him).
Could the reason he hasn't found such a setting be that SSL renegotiate isn't supported at all in dovecot...?
Thanks!
- Steinar
On 25.10.2011, at 14.38, Steinar Bang wrote:
Timo Sirainen tss@iki.fi:
Yes, SSL handshakes are extra. Although SSL supports some kind of quick renegotiation too, but Dovecot doesn't support that yet. No one's ever requested it..
Looks like it's not "renegotiation" but more like session resume/resumption/cache or something that I was thinking about.
Hum... this article (in Norwegian) http://www.digi.no/881186/skrekkverktoy-slaar-ut-%ABsikre%BB-servere addresses the SSL renegotiation vulnerability, and how it can be used to DOS servers using SSL from a single machine with low bandwidth.
At the end the article is discussing how to configure off the SSL renegotiate in different servers, and that the author had been unable to find a setting for disabling SSL renegotiate in dovecot (and if anyone knows how, please inform him).
Could the reason he hasn't found such a setting be that SSL renegotiate isn't supported at all in dovecot...?
Looking at the OpenSSL code, I don't see any way to disable it. Or possibly with some undocumented kludgy way, but I don't really know enough about OpenSSL to implement it.
Anyway, I'd think fail2ban should mostly solve this problem.
On 25.10.2011, at 21.13, Timo Sirainen wrote:
Could the reason he hasn't found such a setting be that SSL renegotiate isn't supported at all in dovecot...?
Looking at the OpenSSL code, I don't see any way to disable it. Or possibly with some undocumented kludgy way, but I don't really know enough about OpenSSL to implement it.
Actually, the attached patch works for v2.0. I'm not really sure yet if I should add a setting for it, force it always or just wait for SSL people to figure out something else. I think I'll do the last option for now.
In any case, I noticed there was some memory "leaking" when doing SSL renegotiation and that definitely needs to be fixed: http://hg.dovecot.org/dovecot-2.0/rev/ad2ebc237570
On 25.10.2011, at 21.51, Timo Sirainen wrote:
On 25.10.2011, at 21.13, Timo Sirainen wrote:
Could the reason he hasn't found such a setting be that SSL renegotiate isn't supported at all in dovecot...?
Looking at the OpenSSL code, I don't see any way to disable it. Or possibly with some undocumented kludgy way, but I don't really know enough about OpenSSL to implement it.
Actually, the attached patch works for v2.0. I'm not really sure yet if I should add a setting for it, force it always or just wait for SSL people to figure out something else. I think I'll do the last option for now.
In any case, I noticed there was some memory "leaking" when doing SSL renegotiation and that definitely needs to be fixed: http://hg.dovecot.org/dovecot-2.0/rev/ad2ebc237570
I don't know if I'm doing something wrong, but I can't even cause a DoS. Even while all imap-login processes are eating 100% CPU (almost 500 handshakes/second), I can successfully log in with another client.
Timo Sirainen tss@iki.fi:
I don't know if I'm doing something wrong, but I can't even cause a DoS. Even while all imap-login processes are eating 100% CPU (almost 500 handshakes/second), I can successfully log in with another client.
Are you using the tool linked to in the article, to stress the server? http://www.thc.org/thc-ssl-dos/
Steinar Bang sb@dod.no: Timo Sirainen tss@iki.fi:
I don't know if I'm doing something wrong, but I can't even cause a DoS. Even while all imap-login processes are eating 100% CPU (almost 500 handshakes/second), I can successfully log in with another client.
Are you using the tool linked to in the article, to stress the server? http://www.thc.org/thc-ssl-dos/
Here's what the article says about stressing dovecot: "Alle servertjenester benytter SSL kan i utgangspunktet være berørt. Digi.no har testet verktøyet mot en eldre, intern server som kjører Linux. Angrepet mot Apache/HTTPD var mislykket, fordi SSL Renegotiation var deaktivert som standard. Men en angrep mot en POP3S-basert (kryptert e-post) tjeneste levert av serverprogramvaren Dovecot, kjørte CPU-lasten i taket med over tusen «handshakes» i sekundet. Angrepet førte ikke til at hele maskinen ble utilgjengelig, men POP3S-tjenesten ble i praksis ubrukelig så lenge angrepet varte."
A quick translate: All services using SSL can be affected. Digi.no has tested the tool against an old, internal server running Linux. The attach against Apache httpd failed, because SSL Renegotiation was deactivated by default. But an attach against a POP3S (encrypted email) service delivered by the server program Dovecot, ran the CPU-load into the roof with over a thousand "Handshakes" per second. The attack didn't cause the computer to be inaccessible, but the POP3S-service was unusable for the duration of the attack.
So it looks like they didn't test IMAPS access, only POP3S.
Am 26.10.2011 10:43, schrieb Steinar Bang:
Steinar Bang sb@dod.no: Timo Sirainen tss@iki.fi:
I don't know if I'm doing something wrong, but I can't even cause a DoS. Even while all imap-login processes are eating 100% CPU (almost 500 handshakes/second), I can successfully log in with another client.
Are you using the tool linked to in the article, to stress the server? http://www.thc.org/thc-ssl-dos/
Here's what the article says about stressing dovecot: "Alle servertjenester benytter SSL kan i utgangspunktet være berørt. Digi.no har testet verktøyet mot en eldre, intern server som kjører Linux. Angrepet mot Apache/HTTPD var mislykket, fordi SSL Renegotiation var deaktivert som standard. Men en angrep mot en POP3S-basert (kryptert e-post) tjeneste levert av serverprogramvaren Dovecot, kjørte CPU-lasten i taket med over tusen «handshakes» i sekundet. Angrepet førte ikke til at hele maskinen ble utilgjengelig, men POP3S-tjenesten ble i praksis ubrukelig så lenge angrepet varte."
A quick translate: All services using SSL can be affected. Digi.no has tested the tool against an old, internal server running Linux. The attach against Apache httpd failed, because SSL Renegotiation was deactivated by default. But an attach against a POP3S (encrypted email) service delivered by the server program Dovecot, ran the CPU-load into the roof with over a thousand "Handshakes" per second. The attack didn't cause the computer to be inaccessible, but the POP3S-service was unusable for the duration of the attack.
So it looks like they didn't test IMAPS access, only POP3S.
however wasnt it possible ever to stress any service via ddos ? this tool may only very effective in doing that
the most problem is see , not everybody can use fail2ban on his servers by keeping out dummy auth users over nat ( I have such case )
anyway ,firewalls should slow down ddos attacks, which might cause other problems then *g, but for sure not from one ip
just a few thoughts..,for sure ,best way would be, getting it fixed
Best Regards
MfG Robert Schetterer
Germany/Munich/Bavaria
On 26/10/2011 10:01, Robert Schetterer wrote:
the most problem is see , not everybody can use fail2ban on his servers by keeping out dummy auth users over nat ( I have such case )
anyway ,firewalls should slow down ddos attacks, which might cause other problems then *g, but for sure not from one ip
...
just a few thoughts..,for sure ,best way would be, getting it fixed
If you google (I think it was on slashdot), I saw a couple of posts with a simple iptables rule with some rate limits attached to it. Clearly you could also read the iptables instructions and figure it out for yourself, but just highlighting that even the footwork has been done if you want copy/paste
I think it's generally not such a bad idea to say limit tcp connections per second from a source IPs. There are plenty of big services that might not be able to implement this as a blanket, but for many shops it could probably be just added as a default for the server...
Cheers
Ed W
Am 27.10.2011 10:25, schrieb Ed W:
On 26/10/2011 10:01, Robert Schetterer wrote:
the most problem is see , not everybody can use fail2ban on his servers by keeping out dummy auth users over nat ( I have such case )
anyway ,firewalls should slow down ddos attacks, which might cause other problems then *g, but for sure not from one ip
...
just a few thoughts..,for sure ,best way would be, getting it fixed
If you google (I think it was on slashdot), I saw a couple of posts with a simple iptables rule with some rate limits attached to it. Clearly you could also read the iptables instructions and figure it out for yourself, but just highlighting that even the footwork has been done if you want copy/paste
i just read it, but its my understanding, that this isnt solving the real Problem, also these rules cant used everywhere by tec layout reasons however youre right, this might help where using it is possible
I think it's generally not such a bad idea to say limit tcp connections per second from a source IPs. There are plenty of big services that might not be able to implement this as a blanket, but for many shops it could probably be just added as a default for the server...
we have a big firewall before all server, it does rate con, but in heavy attacks, this can take off the whole farm, cause every firewall has its limits too, also the problem may involve core routers etc every big attack has to be analysed and reacted, there is reason to do something better ever, but there never be a safe world in www *g
Cheers
Ed W
http://vincent.bernat.im/en/blog/2011-ssl-dos-mitigation.html -> "Things get worse" shows that it's easier to DoS the server with multiple connections than with renegotiations, so I don't know if there's much point in disabling renegotiations. Perhaps Dovecot could allow e.g. one renegotiation per minute, but is that really worth the trouble?.. Perhaps there even are some clients that do renegotiations and Dovecot would break them.
participants (19)
-
Amon Ott
-
Angelo Chen
-
Brandon Davidson
-
Charles Marcus
-
Daniel L. Miller
-
Denny Schierz
-
Ed W
-
Eric Rostetter
-
John S
-
mailing＠securitylabs.it
-
Michael M. Slusarz
-
Noel Butler
-
Patrick Domack
-
Rajesh M
-
Robert Schetterer
-
Stan Hoeppner
-
Steffen Kaiser
-
Steinar Bang
-
Timo Sirainen