On Sun, 30 Sep 2007, Timo Sirainen wrote:

On Fri, 2007-09-28 at 17:22 -0400, Alan Ferrency wrote:

...

Hello,

We are running dovecot 1.0.5 on a test server, with FreeBSD 6.2 (though I have noticed the same problem since dovecot versions in the 0.99 range).

We don't have very many simultaneous pop/imap users, but we have a proliferation of pop3-login processes.

Currently we have 128 such processes. We have 11 imap-login processes, but only a few actual imap processes running.

Is this normal? Can we stop it?

Do you have lots of login attempts? If you have more than 64 concurrent POP3 login attempts, you'll get 128 pop3-login processes.

We don't have many concurrent login attempts, but we do have many in sequence, in the form of automatic mail checking by clients.

http://wiki.dovecot.org/LoginProcess explains more.

I'll check this and report back anything I find. Thanks.

Alan

I notice in the logs, that the number of imap-login Login: lines matches the number of Disconnect: lines in the log. For pop3-login, there are 111 Login: lines and only 8 Disconnect: lines.

Nevermind. A closer look at the logs shows this is a complete red herring, the logs are unrelated.

Still digging.

Alan

Thanks! Alan Ferrency pair Networks, Inc. alan@pair.com

On Sun, 30 Sep 2007, Timo Sirainen wrote:

...

On Fri, 2007-09-28 at 17:22 -0400, Alan Ferrency wrote:

...

We are running dovecot 1.0.5 on a test server, with FreeBSD 6.2 (though I have noticed the same problem since dovecot versions in the 0.99 range).

We don't have very many simultaneous pop/imap users, but we have a proliferation of pop3-login processes.

Currently we have 128 such processes. We have 11 imap-login processes, but only a few actual imap processes running.

Is this normal? Can we stop it?

Do you have lots of login attempts? If you have more than 64 concurrent POP3 login attempts, you'll get 128 pop3-login processes. http://wiki.dovecot.org/LoginProcess explains more.

On Sat, 27 Oct 2007, Timo Sirainen wrote:

On Thu, 2007-10-11 at 14:36 -0400, Alan Ferrency wrote:

...

On this particular server we have exactly 2 different users who log in via POP, and they seem to be set up to log in automatically once every 10 minutes.

imap-login process count is reasonable: there are 11 processes running and it doesn't seem to increase. pop3-login is the issue.

I notice in the logs, that the number of imap-login Login: lines matches the number of Disconnect: lines in the log. For pop3-login, there are 111 Login: lines and only 8 Disconnect: lines. However, there are no pop3 processes currently running, as far as I can tell.

Looking further at the logs, I don't think I saw an actual problem there, I was just misinterpreting the logs.

In any case, I was able to recreate the 128 pop3-login processes by hammering the server with many simultaneous connection attempts even without doing anything on those connections.

I expect this is the reality of the situation: some port-scanner or something somewhere is occasionally hitting the server enough to increase the pop3-login process count, but nothing ever uses them after that.

The processes aren't doing anything or using many resources, it's just annoying that they hang around forever without serving any purpose. Once the process count goes up, do these login processes ever give up and start dying off?

Thanks,

Alan

My system is very slow, the Maildir is mounted via NFS, the index is on a local disk system.

In the config, there is:

max_mail_processes = 3000 login_processes_count = 6 login_max_processes_count = 1500

I can see a large number of processes named "imap", some users have more than 12 such processes running. Is such a large number per user a "normal" behaviour ?

What is wrong here ?

Thanks a lot ! Claude

Timo Sirainen wrote:

...

I can see a large number of processes named "imap", some users have more than 12 such processes running. Is such a large number per user a "normal" behaviour ?

It's probably normal. Each connection is handled by a separate imap process currently, and many clients can create a lot of connections.

OK, Timo ! Many thanks for your help !

At the present time, I understand better what is happening here, causing problems.

At the present time, the maidirs are mounted via NFS v3 on a NetApp system. The index is on a local RAID file system. mboxes are not used. Incoming messages are inserted in the maildirs using sendmail and maildrop. The usual behavior is OK: the CPU usage and Load Average are moderate. There are many processes sleeping.

When a large number of messages for many different users arrive nearly at the same time (e.g. a message to all the users), a large number of sleeping processes become active, the CPU usage increases to more than 80 or 90 % and the LA can be many hundreds for a long time. The machine is not more really operational from user's point of view. NFS is probably a problem here, but some NFS tests have not allowed to discover problems at this level. Now, I'm trying to find a "solution" in the way to delay the delivery of the messages to the maildirs. I think, it is not really a nice solution.

I think that the large number of now running processes is the rebuilding of the index. I am right here, Timo ? Please note that many users have a large number of messages accumulated in the incoming maildir. Unfortunately, I cannot change that. In the past, this was the reason to migrate to maildirs. At the present time, the mailfilter language of maildrop (and Courier) is used here in place of procmail used in the past. I ignore if using dovecot's LDA could be a solution in this environment. But the mailfilter language could be a problem. A migration to sieve is not really a choice, in my opinion. I have never seen a good sieve script testing software.

Thanks a lot ! Claude

Timo Sirainen wrote:

...

I think that the large number of now running processes is the rebuilding of the index. I am right here, Timo ?

Index updates typically don't take long. Although if client fetches some metadata that isn't in cache file, Dovecot needs to open the messages and parse them. That takes some I/O. This can be avoided with v1.1 by using deliver, since it updates the cache file immediately.

I have tried it using deliver in the version dovecot-1.0.10-0_66.fc7. Each delivery run was longer when using maildrop, but the load was too high too. Do you think that in the version 1.1 the behavior would be different ?

Although I'm still not sure if that would help, because a lot of clients just download the entire message body automatically, which in any case causes I/O.

Most users here use thunderbird, outlook (and express) or a webmailer using squirrelmail.

...

At the present time, the mailfilter language of maildrop (and Courier) is used here in place of procmail used in the past. I ignore if using dovecot's LDA could be a solution in this environment. But the mailfilter language could be a problem. A migration to sieve is not really a choice, in my opinion. I have never seen a good sieve script testing software.

Can you tell maildrop to deliver the message to a specific mailbox using an external command (i.e. calling deliver) instead of delivering itself?

Yes, it is possible and I have tried it. It is then necessary to include the code in a mailfilter which finally calls deliver when the decision has been made. Here again, the result was not really better. In fact, it is difficult to make such a delivery via maidrop calling deliver as a "standard" in a productive environment.

In order to circumvent the problem, I have now configured sendmail, the MTA, in a little bit special manner: Incoming messages are stored in the queue first and the delivery occurs only in the queue run. These are run every 2 minutes, each run processes max. 150 messages. Incoming messages are split in multiple ones having 4 recipients max. In this manner the load can be limited at the expense of pass through time, if necessary. Further, there are more (split) message in the queue. This is not really a good "solution", but it will limit the problems a little bit. I'm still looking for a better solution. NFS is still a problem.

Thanks a lot !

Claude

Timo Sirainen wrote:

Yes, the main benefit of deliver with Maildir is when it updates the cache file, but v1.0 deliver doesn't do that.

Oh ! Very interesting. I did think, that this function was already available in the version 1.0.

...

NFS is still a problem.

Do you use a single Dovecot server or multiple? v1.1's NFS support for multiple servers is a lot better than with v1.0.

At the present time, there is only one server but it is intended to have multiple ones in the future.

Please allow me to report that it is difficult to predict what the setting of "mmap_disable" will produce, from the performance point of view. In the case of the system about which I was reporting from, the "yes" setting performs really better. It is a large system, having 4 processors, 8 GB of RAM, a large local RAID system attached and a rather large external file store via NFS. There are more than 5000 accounts and usually 600 to 1200 users using IMAP, POP3 and squirrelmail, at the same time. In contrast, on a very small system, having few accounts, small RAM and small local disk space, the "no" setting performs better.

Thanks a lot, Timo ! Claude