[Dovecot] the most serius problems [part 1]
hi, I try to collect out most serious problem with dovecot. so I write a series of mails:-) the first on is the load. has anybody else use dovecot in a real enviromnt with at least hundreds of users and at least 50 concurrent user at the same time? it seems to me no one else just me. the weekend I move the whole mail system to a quad P4 with 1Gb ram and this server do nothing just imap. yesterday evening I have to delete all .imap.index and other dovecot specific files. and as I wrote today morning the load was about 20-30 !!!. but even in an average day the load is over 2-3 white the cpu load is 1-2% in each of the 4 processor. the system is totaly unusable. what more today I've got other errors. of course all the time when I can't find the colution I try to escape to the latest cvs. today finally I compile a new courier install it to another port and those clients which can't connect to dovecot reconfigure to courier. but they are not compatible!!! I have to set root directory to INBOX in courier while not for dovecot and this is a real big problem. is there any way to compile dovecot to use INBOX as a root directory? since in this case at least there is a chance to switch to courier without reconfigure 300 OE6. I try to help to find out the reason for the load in any way, but it seems to me than it's better to change to courier or cyrus since our users getting angry not to mention my boss.
-- Levente "Si vis pacem para bellum!"
On Wed, 2003-05-28 at 14:46, Farkas Levente wrote:
has anybody else use dovecot in a real enviromnt with at least hundreds of users and at least 50 concurrent user at the same time?
Well, I know 0.99.9 has been in use with some 40 concurrent users (asked two people). Didn't 0.99.9 actually work with you too without high load?
I have to set root directory to INBOX in courier while not for dovecot and this is a real big problem. is there any way to compile dovecot to use INBOX as a root directory? since in this case at least there is a chance to switch to courier without reconfigure 300 OE6.
Haven't you configured them now to non-INBOX?
You could kludge Dovecot to use INBOX. prefix, or Courier to not use it, but there's no pretty way to do it really.
..but i'm late now, have to look these again later today.
Timo Sirainen wrote:
On Wed, 2003-05-28 at 14:46, Farkas Levente wrote:
has anybody else use dovecot in a real enviromnt with at least hundreds of users and at least 50 concurrent user at the same time?
Well, I know 0.99.9 has been in use with some 40 concurrent users (asked two people). Didn't 0.99.9 actually work with you too without high load?
unfortunately not.
I have to set root directory to INBOX in courier while not for dovecot and this is a real big problem. is there any way to compile dovecot to use INBOX as a root directory? since in this case at least there is a chance to switch to courier without reconfigure 300 OE6.
Haven't you configured them now to non-INBOX?
You could kludge Dovecot to use INBOX. prefix, or Courier to not use it, but there's no pretty way to do it really.
how can I do that? do you mean there is a way to configure courier not to use INBOX as a prefix? or is there any way to configure dovecot to use INBOX as a prefix? I would be very happy if you tel me! I'd prefer the first one. since in that case I always has a chance to simple stop dovecot and start courier in any serious situation. as I said this's our production server and I can play with it a bit, but it's not acceptable to stop the whole mail system for a day. and without the above solution it's not possible, since currently if we sould have to reconfigure about 300 clients settings (unfortunately this can't be automatized since OE6 keep this setting in a registry key which is depend on the users mailbox number:-() so if I decide to switch to courier than probably we won't shitch back soon.
ps. actualy I'm the head of the it dep. and in normal circumstances there are sysadms for the linux systems and for the windows system I don't use to login to the servers as root. but since dovecot was my decision I try to protect it, but it seems to me problems are getting too fast.
-- Levente "Si vis pacem para bellum!"
On Thu, 2003-05-29 at 00:27, Farkas Levente wrote:
You could kludge Dovecot to use INBOX. prefix, or Courier to not use it, but there's no pretty way to do it really.
how can I do that? do you mean there is a way to configure courier not to use INBOX as a prefix?
No, I mean you can do it by messing enough with the sources. Hmm. I don't think it would be too difficult. OK, here's patch against 1.7.0. I tested a few commands manually and they seemed to work, but maybe I missed something.
If you plan on switching between Courier <-> Dovecot, you should keep courierimapuiddb and dovecot-uidlist files synchronized or clients will download the whole mailbox again every time you do the switch. Easiest way to do this would be to just change Dovecot to use courierimapuiddb file directly, you can change that from src/lib-index/maildir/maildir-uidlist.h
Subscription file would be also nice to keep synced. You set Dovecot to use courierimapsubscribed from src/lib-storage/subscription-file/subscription-file.c
or is there any way to configure dovecot to use INBOX as a prefix?
Not yet, but it's in TODO along with other namespace configuration.
Timo Sirainen wrote:
On Thu, 2003-05-29 at 00:27, Farkas Levente wrote:
You could kludge Dovecot to use INBOX. prefix, or Courier to not use it, but there's no pretty way to do it really.
how can I do that? do you mean there is a way to configure courier not to use INBOX as a prefix?
No, I mean you can do it by messing enough with the sources. Hmm. I don't think it would be too difficult. OK, here's patch against 1.7.0. I tested a few commands manually and they seemed to work, but maybe I missed something.
thanks!!!
If you plan on switching between Courier <-> Dovecot, you should keep courierimapuiddb and dovecot-uidlist files synchronized or clients will download the whole mailbox again every time you do the switch. Easiest
that's not a problem since I won't like to switch to often and it's automatic.
-- Levente "Si vis pacem para bellum!"
On Wed, May 28, 2003 at 01:46:33PM +0200, Farkas Levente wrote:
the first on is the load. has anybody else use dovecot in a real enviromnt with at least hundreds of users and at least 50 concurrent user at the same time? it seems to me no one else just me.
That seems like an awfully high percentage of users connecting- if I assume "hundreds" might mean 300, 1 out of every 6 users is connected at any given time?
One of the servers running dovecot here has thousands of users. During the middle of the day I typically see 17-20 users connected at any given time. This has been known to increase to 50+ especially during prime time (evening). Over a period of 24 hours dovecot logins occur at the rate of around 2.5 logins per second. Now- most of our users are using POP rather than IMAP, but there is still a reasonable amount of IMAP activity. Plus I'm not sure that detail matters: POP logins use resources too, perhaps repeated POP logins use more than constantly-connected imap sessions.
The load on the server due to dovecot is negligable. Most of the load is due to handling incoming mail. Obviously you are seeing something different- perhaps you are using features that we are not.
I rather wish dovecot would log some more statistics, as I think has been mentioned already- like a record for logout as well as one for login, including number of messages scanned, left on server, downloaded, and deleted, plus total bytes transferred). These numbers would give a better picture of what's going on.
-mm-
Mark E. Mallett wrote:
On Wed, May 28, 2003 at 01:46:33PM +0200, Farkas Levente wrote:
the first on is the load. has anybody else use dovecot in a real enviromnt with at least hundreds of users and at least 50 concurrent user at the same time? it seems to me no one else just me.
That seems like an awfully high percentage of users connecting- if I assume "hundreds" might mean 300, 1 out of every 6 users is connected at any given time?
One of the servers running dovecot here has thousands of users. During the middle of the day I typically see 17-20 users connected at any given time. This has been known to increase to 50+ especially during prime time (evening). Over a period of 24 hours dovecot logins occur at the rate of around 2.5 logins per second. Now- most of our users are using POP rather than IMAP, but there is still a reasonable amount of IMAP activity. Plus I'm not sure that detail matters: POP logins use resources too, perhaps repeated POP logins use more than constantly-connected imap sessions.
The load on the server due to dovecot is negligable. Most of the load is due to handling incoming mail. Obviously you are seeing something different- perhaps you are using features that we are not.
I rather wish dovecot would log some more statistics, as I think has been mentioned already- like a record for logout as well as one for login, including number of messages scanned, left on server, downloaded, and deleted, plus total bytes transferred). These numbers would give a better picture of what's going on.
the reason is simple there are about 300 users when they come in at the morning thay simple start OE6 and never exit from it just when they go home (why do they do so?). so in the prime I saw 126 imap connection at the same time. and most user has a real big mailboxes without any organization/folders (just inbox, sent and drafts). incoming mails are handled by postfix which is realy fast, reliable and almost no load at all. here everybody use imap since most people doesn't have an own desk (only 50 has) so everybody just sit down to the first free desk. in this case only the imap is the solution. the real load comes when indexing or reindexing is running (eg. if we delete all index file for same reason). in this case even 10 imap connections are able to create a load about 10 (while the cpu are 95% idle). and there is something wrong with 0.99.9 series. since now when I've to switch back to 0.99.8 it's a lot faster.
-- Levente "Si vis pacem para bellum!"
what's more we plan to centralize two more companies mailbox (about 50+200 mailbox) so we need a system which able to handle about 500 mailbox and about 200 concurrent users.
Mark E. Mallett wrote:
On Wed, May 28, 2003 at 01:46:33PM +0200, Farkas Levente wrote:
the first on is the load. has anybody else use dovecot in a real enviromnt with at least hundreds of users and at least 50 concurrent user at the same time? it seems to me no one else just me.
That seems like an awfully high percentage of users connecting- if I assume "hundreds" might mean 300, 1 out of every 6 users is connected at any given time?
One of the servers running dovecot here has thousands of users. During the middle of the day I typically see 17-20 users connected at any given time. This has been known to increase to 50+ especially during prime time (evening). Over a period of 24 hours dovecot logins occur at the rate of around 2.5 logins per second. Now- most of our users are using POP rather than IMAP, but there is still a reasonable amount of IMAP activity. Plus I'm not sure that detail matters: POP logins use resources too, perhaps repeated POP logins use more than constantly-connected imap sessions.
The load on the server due to dovecot is negligable. Most of the load is due to handling incoming mail. Obviously you are seeing something different- perhaps you are using features that we are not.
I rather wish dovecot would log some more statistics, as I think has been mentioned already- like a record for logout as well as one for login, including number of messages scanned, left on server, downloaded, and deleted, plus total bytes transferred). These numbers would give a better picture of what's going on.
-mm-
-- Levente "Si vis pacem para bellum!"
participants (3)
-
Farkas Levente
-
Mark E. Mallett
-
Timo Sirainen