On Thu, 2012-06-21 at 13:05 -0700, email builder wrote:

Thank you very much for the fast reply.

...

...

We are building a new system that will support a large number of users

...

...

(high volume, high concurrent usage, etc). We have played with Dovecot, but in most serious applications we have traditionally used Courier IMAP. It's my (lay) understanding that with indexing and perhaps other things in Dovecot, it might perform better than Courier in larger environments like this. Am I correct or is it less clear-cut?

If you disable index index files in Dovecot, its performance should be slightly better than Courier. With index files the performance is typically much better in Dovecot, especially if you use a (non-caching) webmail.

Interesting. What would be the motivations for disabling indexing? Indexing is by default enabled?

Yes, enabled by default. There aren't many good reasons for disabling indexing.

Do you know what webmails are caching vs. non-caching?

Nearly all of them are non-caching. (I don't know of any caching ones.)

Am I correct that what you're pointing out is that with non-caching webmails you will notice IMAP performance differences more readily but that a caching webmail application might be better no matter which IMAP server because it reduces the need for webmail to make IMAP connections?

It's not about the IMAP connections themselves, but how often they fetch message (meta)data. http://www.imapwiki.org/Benchmarking should explain this better. Dovecot's indexing can lower the disk I/O usage perhaps by 10x compared to Courier.

...

...

Any tips on making the migration (not migrating an existing system, I mean migrating our paradigm - things to consider, things to watch out for)?

If you don't migrate any existing users, I guess this doesn't differ much from any other optimized Dovecot installation. Usually large installations (>1M users) use NetApp NFS + Dovecot director. You might also want to enable full text searches. http://wiki2.dovecot.org/PerformanceTuning lists some other things.

Ah, I didn't know about Director. That looks very nice. I had in mind that we would have to use Perdition, but an integrated solution might be good.

Anyone have any thoughts or opinions considering Perdition vs. Director?

Dovecot proxy has several Dovecot-specific features that make it work better than perdition (forwards client IP address to backend, handle CAPABILITY stuff better, maybe other things).

Full text searches don't hurt performance too bad?

They should improve the performance, at least from the user's point of view when doing a search on webmail. But yes, the indexing itself does cost CPU cycles, disk I/O and disk usage (perhaps 30% more disk space).

On 21.6.2012, at 23.34, Reindl Harald wrote:

Am 21.06.2012 22:22, schrieb Timo Sirainen:

...

...

Do you know what webmails are caching vs. non-caching?

Nearly all of them are non-caching. (I don't know of any caching ones.)

roundcube can if configured

additionally you should install imapproxy on the webserver wehre your webmail is running and configure the webmail for using 127.0.0.1 - so only one connection per user is persistent instead make a new one for each ajax-request

Someone benchmarked Dovecot a while ago in this list with and without imapproxy and the results showed that imapproxy simply slowed things down by adding extra latency. This probably isn't true for all installations, but I don't think there's much of a difference either way.

On 21.6.2012, at 23.48, Reindl Harald wrote:

...

Someone benchmarked Dovecot a while ago in this list with and without imapproxy and the results showed that imapproxy simply slowed things down by adding extra latency. This probably isn't true for all installations, but I don't think there's much of a difference either way.

depends on network-latency, parallel users and last but not least count of folders - if you have 30 folders and roundcube refreshs every 20 seconds it will make in the worst case 180 connections for one user per minute

Really? Doesn't it simply connect once every 20 seconds and send 30 STATUS requests in one connection?

On 21/06/2012 21:54, Reindl Harald wrote:

and last but not least i have lesser entries in maillog which goes to a central mysql-server for self-developed web-interfaces

I recently added imapproxy to my Roundcube installation. Benchmarks showed a very slight slowdown, but as you point out it reduced the login count from dovecot and I use a login script to kind of report last login / length of session and this tallies better with an imap desktop user now

I think the conclusion is that imapproxy is not necessary. There are some advantages (eg with high network latency between web and imap server, and reducing apparent login count), and some disadvantages (extra complexity, slowdown)

On average I think few users should use it.. Or at least benchmark and add it reluctantly...

Ed

On 22.6.2012, at 0.58, Michael M Slusarz wrote:

...

I think the conclusion is that imapproxy is not necessary. There are some advantages (eg with high network latency between web and imap server, and reducing apparent login count), and some disadvantages (extra complexity, slowdown)

Not entirely true. See this thread:

http://markmail.org/thread/z7ctwle2go6zafas

Thread in short: imapproxy provides benefits for more MUAs that take advantage of the XIMAPPROXY feature (only IMP, AFAIK), and Timo is/was considering adding a similar state saving feature to Dovecot 2.2.

Well, I had completely forgotten about it :) Reading my old mail:

There isn't a whole lot of state to be saved really. Mailbox GUID, UIDVALIDITY, HIGHESTMODSEQ gives the mailbox state. Then you have the language/etc. states. Clients could restore their earlier state from days ago, as long as Dovecot still has the necessary .log records available (similar to how QRESYNC works).

Yeah .. Perhaps something like:

1. if client issues LOGOUT XSTATE

2. And server sees that it can actually save all of the state (some things are a bit tricky, and probably not worth the trouble in initial implementation)

3. Then the server server sends

• OK XSTATE <string>
• BYE

4. The client can pipeline after LOGIN/AUTHENTICATE: a XSTATERESTORE <string> a OK Yeah! or a NO Not gonna work.

Perhaps even a real RFC for this thing? .. If it's worth it.. Would save at least a few X bytes from network traffic :)

On 21.6.2012, at 23.48, Reindl Harald wrote:

...

Someone benchmarked Dovecot a while ago in this list with and without imapproxy and the results showed that imapproxy simply slowed things down by adding extra latency. This probably isn't true for all installations, but I don't think there's much of a difference either way.

depends on network-latency, parallel users and last but not least count of folders - if you have 30 folders and roundcube refreshs every 20 seconds it will make in the worst case 180 connections for one user per minute

maybe a bechmark with high load shows other values

but felt performance in our setup is much better with imapproxy in front - roundcube feels like a desktop client

Oh, and of course it also depends on Dovecot configuration :) Authentication cache is needed and login processes must be in high performance mode. There is still the extra work of forking a new imap process (could also be avoided with yet another config option) and some other extra CPU usage, but those shouldn't cause much of a difference.

The extra network latency during login is a good point though.

On 22.6.2012, at 5.28, email builder wrote:

...

Oh, and of course it also depends on Dovecot configuration :) Authentication cache is needed and login processes must be in high performance mode.

I.e., I think:

http://wiki2.dovecot.org/LoginProcess http://wiki2.dovecot.org/Authentication/Caching

Yes.

...

There is still the extra work of forking a new imap process (could also be avoided with yet another config option)

Are you referring to client_limit or service_count or something else as yet undeveloped?

service imap { service_count = 0 } (default=1) allows imap processes to be reused for more than 1 connection. The downside is that if there are any bugs in Dovecot, they might accidentally expose another user's email data to the wrong user. That's very unlikely to happen but since this isn't a performance problem in most (if any) systems I don't want to enable it by default. Dovecot code is written so that write buffer overflows (= arbitrary code execution) is minimized to be as zero possibility as I could think of, but read buffer overflows (= exposing data within the process) isn't treated nearly as much with paranoia.

Speaking of which, I cannot understand the different between those two. Hints in the configuration file (10-master.conf) and the wiki make them sound like they do the same thing -- ??

service_count limits the maximum of client_limit. One connection = one service. Once a process has serviced "service_count" number of connections it disconnects itself. There can never be more than "client_limit" number of simultaneous connections. The important stuff to understand about these are:

• service_count=1: The most secure setting for a process. The process serves a single connection and kills itself. No possibility of data leaking to unintended connection.
• service_count=0, client_limit=1: The process does blocking operations (e.g. blocking disk IO). You don't want one connection's blocking operation to affect other connections. But you're not paranoid about security, since in case of some bugs some data might leak to unintended connection.
• service_count>0: Restart process ever N connections, just in case it leaks some memory.
• client_limit>1: Limit the amount of CPU/memory a single process takes. The process should never be blocking on disk I/O or locks or anything else. This means it shouldn't be used for imap/pop3/lmtp processes. For CPU bound processes it's fine.

Maybe these could be copy&pasted to the wiki2/Services.

On 22.6.2012, at 8.27, email builder wrote:

So really, a new process is created under *two* circumstances? 1. when a process reaches client_limit number of *simultaneous* connections or 2. when a process has serviced service_count number of connections. Is this correct?

Yes.

So for service *-login, is it OK to do something like service_count=5000, client_limit=2000

It would work, but for login processes the service_count can be 0. I haven't seen them leaking any memory recently.

One somewhat annoying thing with service_count>1 is that the processes have to wait until all of the connections have disconnected before shutting down. For processes handling long running connections (especially IMAP) this can mean that you'll end up with a lot of processes that are ready to shutdown but a couple of connections prevent it from doing this.

...

...

wehre your webmail is running and configure the webmail for using 127.0.0.1 - so only one connection per user is persistent instead make a new one for each ajax-request

Someone benchmarked Dovecot a while ago in this list with and without imapproxy and the results showed that imapproxy simply slowed things down by adding extra latency. This probably isn't true for all installations, but I don't think there's much of a difference either way. nothing strange. I really wonder if there are available FASTER implementations of imap service. Quite probably not.

It's stupid how webmail works but dovecot doesn't have a problem to get new connections every now and then. just make sure you didn't set up SSL by accident.

On Jun 21, 2012, at 3:22 PM, Timo Sirainen wrote:

On Thu, 2012-06-21 at 13:05 -0700, email builder wrote:

...

Thank you very much for the fast reply.

...

...

We are building a new system that will support a large number of users

...

...

(high volume, high concurrent usage, etc). We have played with Dovecot, but in most serious applications we have traditionally used Courier IMAP. It's my (lay) understanding that with indexing and perhaps other things in Dovecot, it might perform better than Courier in larger environments like this. Am I correct or is it less clear-cut?

If you disable index index files in Dovecot, its performance should be slightly better than Courier. With index files the performance is typically much better in Dovecot, especially if you use a (non-caching) webmail.

Interesting. What would be the motivations for disabling indexing? Indexing is by default enabled?

Yes, enabled by default. There aren't many good reasons for disabling indexing.

...

Do you know what webmails are caching vs. non-caching?

Nearly all of them are non-caching. (I don't know of any caching ones.)

Prayer, from University of Cambridge, or Chickadee, a fork of it. It's essentially a proper IMAP client in C that runs on a server, and uses HTTPS (via an embedded server, no external dependency on apache or etc.) to the end user just to deliver the display.

When I was on the email project for the University of Minnesota, I modified it heavily for interface and to add some features that admins are used to having in systems where apache is involved (virtual hosts, things like that). I have it available (GPL) as a vanilla, de-branded package--Chickadee. Website is currently offline as I've been switching hosts, anyone who's interested can feel free to drop me a line.

-Brian

On 21/06/2012 21:37, René Neumann wrote:

Am 21.06.2012 22:22, schrieb Timo Sirainen:

...

On Thu, 2012-06-21 at 13:05 -0700, email builder wrote:

...

Do you know what webmails are caching vs. non-caching? Nearly all of them are non-caching. (I don't know of any caching ones.) At least roundcube (v0.7.1 here) has some caching options:

------------------[excerpt from roundcubes main.inc.php]------------- // Type of IMAP indexes cache. Supported values: 'db', 'apc' and 'memcache'. $rcmail_config['imap_cache'] = null;

// Enables messages cache. Only 'db' cache is supported. $rcmail_config['messages_cache'] = false; -------------------------[end]----------------------------------------

But I don't know, whether this is the sort of caching you are referring to.

• René

It is caching, but unless your mysql / memcache server is lower latency than your dovecot server, then the caching does very little.

I tested it very briefly and it added a lot of latency to my results when adding a mysql cache. However, my setup has the mysql/dovecot/roundcube all on the same machine, so latency is minimal.

Roughly I found that the amount of caching is absolutely massive, eg roughly subject headers, message ids and more for every message in every folder. This meant multiple seconds of latency on first login and then slight additional latency on every folder view. I guess this might breakeven in the situation of a roundcube installation in an office and dovecot on the far end of an ADSL line with 60-100ms+ of latency and bandwidth constraints, but it's really, really hard to see it's sensible for two machines in the same datacenter with an uncontended network connection between them

This isn't to say that the caching isn't sensible for use with other mail servers, but I don't see it offers any benefit for most Dovecot installations?

However, very clever and full featured webmail client!

Ed W

P.S. Sogo has a kind of caching in that it has a clientside javascript cache. Not what was meant, but for all practical purposes much more useful...

On 23/06/2012 09:22, Wojciech Puchar wrote:

...

...

Nearly all of them are non-caching. (I don't know of any caching ones.)

At least roundcube (v0.7.1 here) has some caching options:

------------------[excerpt from roundcubes main.inc.php]------------- // Type of IMAP indexes cache. Supported values: 'db', 'apc' and 'memcache'. $rcmail_config['imap_cache'] = null;

// Enables messages cache. Only 'db' cache is supported. $rcmail_config['messages_cache'] = false; -------------------------[end]----------------------------------------

But I don't know, whether this is the sort of caching you are referring to.

what's a point of caching imap, except your webmail service is not locally connected (localhost or LAN) to imap server?

Asking for items 600-615 from a threaded list, sorted by something, can be an expensive operation, especially if you just asked for items 585-600 a moment ago?

Ed

On 23.6.2012, at 13.21, Ed W wrote:

...

...

But I don't know, whether this is the sort of caching you are referring to.

what's a point of caching imap, except your webmail service is not locally connected (localhost or LAN) to imap server?

Asking for items 600-615 from a threaded list, sorted by something, can be an expensive operation, especially if you just asked for items 585-600 a moment ago?

Can be, but is it? :) Dovecot attempts to cache/index stuff as well. Normally there shouldn't be a need for extra caching layer except in cases of higher network latency.

Quoting Timo Sirainen tss@iki.fi:

On 23.6.2012, at 13.21, Ed W wrote:

...

...

...

But I don't know, whether this is the sort of caching you are
referring to.

what's a point of caching imap, except your webmail service is not
locally connected (localhost or LAN) to imap server?

Asking for items 600-615 from a threaded list, sorted by something,
can be an expensive operation, especially if you just asked for
items 585-600 a moment ago?

Can be, but is it? :) Dovecot attempts to cache/index stuff as well.
Normally there shouldn't be a need for extra caching layer except in
cases of higher network latency.

Timo: I'm not sure if you are saying that all client-side caching is
wrong. If so, I'm going to disagree with you, especially when dealing
with more complex data structures.

Let me first say that I don't take IMAP response parsing to be a
computationally easy action. So it's not just network latency you are
worrying about; parsing a line can be the limiting factor in many
cases. For example, a deeply threaded 400 message mailbox will return
a THREAD response line that will take quite a bit of recursive parsing
to decode.

And various FETCH criteria most definitely benefit from local caching
above/beyond what dovecot provides. An example: BODYSTRUCTURE. This
may be cached on the dovecot side, but when received by the MUA you
have to parse the IMAP BODYSTRUCTURE response (not trivial). You also
have to potentially handle IMAP response codes in the server command
completion line. And the bodystructure data is probably not all that
useful until converted to a usable object on the MUA side, which may
be another relatively expensive operation. So a locally cached
bodystructure object is a substantial performance benefit over having
to recreate this data from the cached data on the dovecot side.

ENVELOPE is similar. Most likely this will be converted to an object
representation in the MUA so you have the same benefits as
BODYSTRUCTURE. Additionally, in IMP we do things like scan for broken
charset headers (e.g. Subject headers that contain non-ASCII
characters) and have some algorithms to fix these issues. This
"value-added" code would be prohibitively expensive if we have to do
it on every mailbox access.

Message flags are another benefit to caching. The list of flags may
be cached on dovecot, but not having to issue a flag FETCH every time
you access a mailbox can be a substantial benefit.

But I will heartily agree that nobody should be caching things like
headertext or bodypart data. There is little/no benefit you receive
from caching this locally. This is where you should be leveraging the
storage on the IMAP server.

As an MUA author you can't rely on the fact that you are connecting to
a competent IMAP server. You just as likely are going to be
connecting to a server that implements base RFC 3501, and most likely
implements that incorrectly. Not all of us are lucky to connect to
Dovecot (or Cyrus).

So smart caching most definitely can and will increase performance of
an MUA, regardless of caching performed by the IMAP server.

michael

Am 27.06.2012 07:44, schrieb Wojciech Puchar:

...

Timo: I'm not sure if you are saying that all client-side caching is wrong. If so, I'm going to disagree with you, especially when dealing with more complex data structures.

it is always good - on WAN links.

Hi, i dont wanna flame into this thread, cause its heavy tec stuff which i dont really fit in

but for some webmail you can use http://imapproxy.org/ its running here fine with squirrelmail and roundcube

-- Best Regards MfG Robert Schetterer

Am 27.06.2012 09:32, schrieb Wojciech Puchar:

...

Hi, i dont wanna flame into this thread, cause its heavy tec stuff which i dont really fit in

but for some webmail you can use http://imapproxy.org/

the discussion was about if running proxy at all make sense.

Proxies are to reduce traffic or server load by avoiding repetitive requests.

With dovecot it's unlikely proxy itself will be faster, so second reason doesn't exist.

With same computer or fast lan or virtual lan (==normal way of running webmail) first reason doesn't exist.

Hi, sorry ,only my meaning, beside coding layout questions about dovecot etc which is clearly not my case

for questions like: "does a proxy make sense" there will never be an uni right answer

the answer may ever depend on what fits best at your side general setup/layout

-- Best Regards MfG Robert Schetterer

Nearly all of them are non-caching. (I don't know of any caching ones.)

which is definite adventage in spite of it's numerous security holes.