[Dovecot] High level of pop3 popping causing server to become unresponsive

[Root Kev]

Thierry,

Had a chance to test this change this morning, and in my test environment,
this does drastically improve the ability to ssh and su during heavy pop3
load (in test environment, change of 10-15sec to 1-2sec login).  While this
is better, the popping is still slowing down authentication on the box
during load.  Is there anything else that might be changed to increase
speed?

Just a recap, I basically have between 2-8 mailboxes on each server, and an
application that creates two connections to each mailbox, processes each
message that comes in, and then deleted each message from the pop3
mailbox.  During high load times (500+ messages) with the settings that I
had, I was unable to reconnect to the server, nor su to root.  Ended up
having to stop the dovecot process and revert to popa3d (luckily had a ssh
session still open to the box).

This is the config from my test environment:
root at devsmtp:~# dovecot -c /usr/local/etc/dovecot/dovecot-test.conf -n
# 2.1.4: /usr/local/etc/dovecot/dovecot-test.conf
# OS: Linux 2.6.32-33-generic-pae i686 Ubuntu 10.04.4 LTS ext4
auth_verbose = yes
base_dir = /var/run/dovecot-test/
disable_plaintext_auth = no
instance_name = dovecot-test
listen = <removed>
login_greeting = Dovecot-Test for removing index
mail_location = mbox:/var/empty:INBOX=/var/mail/%u
mail_privileged_group = mail
namespace inbox {
  inbox = yes
  location =
  mailbox Drafts {
    special_use = \Drafts
  }
  mailbox Junk {
    special_use = \Junk
  }
  mailbox Sent {
    special_use = \Sent
  }
  mailbox "Sent Messages" {
    special_use = \Sent
  }
  mailbox Trash {
    special_use = \Trash
  }
  prefix =
}
passdb {
  driver = shadow
}
protocols = pop3
service pop3-login {
  inet_listener pop3 {
    port = 130
  }
  service_count = 0
}
ssl = no
userdb {
  driver = passwd
}
protocol pop3 {
  pop3_uidl_format = %08Xu%08Xv
}


And from one of my production servers that has the issue:
# 2.1.4: /usr/local/etc/dovecot/dovecot.conf-ml
# OS: Linux 2.6.18-194.17.1.el5 x86_64 Red Hat Enterprise Linux Server
release 5.6 (Tikanga) ext3
auth_verbose = yes
base_dir = /var/run/dovecotml/
disable_plaintext_auth = no
instance_name = Popper
listen = <removed>
login_greeting = Popper
mail_location =
mbox:/var/empty:INBOX=/opt/mailstore/spool/mail/%u:INDEX=MEMORY
mail_privileged_group = mail
namespace inbox {
  inbox = yes
  location =
  mailbox Drafts {
    special_use = \Drafts
  }
  mailbox Junk {
    special_use = \Junk
  }
  mailbox Sent {
    special_use = \Sent
  }
  mailbox "Sent Messages" {
    special_use = \Sent
  }
  mailbox Trash {
    special_use = \Trash
  }
  prefix =
}
passdb {
  driver = shadow
}
protocols = pop3
ssl = no
userdb {
  args = blocking=yes
  driver = passwd
}
protocol pop3 {
  pop3_uidl_format = %08Xu%08Xv
}


On Fri, May 25, 2012 at 2:24 PM, Thierry de Montaudry <thierry at odry.net>wrote:

> On 25 May 2012, at 20:00, Root Kev <root.kev at gmail.com> wrote:
>
> So the best way to would be to remove it that part completely, or should
> it be stored somewhere on disk?
>
> Thanks again,
>
> Kevin
>
> yes, best is to remove ":INDEX=MEMORY" part, and it will store indexes in
> your INBOX path, which is fine.
>
> If you want to put it in a different path, you can have something like
> INDEX=/path/to/indexes/%u
>
> Regards,
>
> Thierry
>
>
>
>> Hi,
>>
>> Having a system with a third of our users on POP3 (2300000 of them), no
>> trouble with dovecot (v2.1.5, on CentOS 5).
>> But one thing surprises me in your config, the INDEX=MEMORY in the
>> location parameter. That means that for each POP3 connection, dovecot has
>> to read each and every mails to create the index in memory. That might be
>> why the machine becomes unresponsive.
>> Unless you have a specific reason to use memory index (and I would be
>> curious to know about it), I would suggest to remove this and keep standard
>> file indexes, and your performances should improve a lot.
>>
>> Regards,
>>
>>        Thierry
>>
>>
>
>