Too many wait in auth process
itanguy at univ-brest.fr
itanguy at univ-brest.fr
Tue Feb 8 10:27:38 UTC 2022
Hello,
thank you for your advices and sorry to not have detailed infra
> ismael> I'm currently benchmarking new hardware aimed to serve around
> ismael> 70k users For now, our IMAP server have 13k users.
>
> This doesn't help us help you. Is this a new rasperry Pi 4? Is it a
> Dual CPU AMD Rzyzen with 128gb of memory and fast NVMe disks? What is
> your system setup?
Sorry, I have two servers to bench :
- first one (a model like our current IMAP servers) is 18To HDD, 256Go
RAM, 8c/16th
- second (new one aimed to serve many more customers) is 24 x 14 TO (HDD
SAS), 192GB DDR4 2,6Ghz, 12c/24t - 2.4GHz/3.5GHz
OS is FreeBSD 12.2
>
> ismael> To run imaptest, I've spwan some bench clients.
>
> Are these tests run from remote hosts? What kind of network are you
> using?
Yes, imaptest is running from kvm remote virtual machines in the same DC.
They are some networks hops between them, but few.
>
> ismael> Each bench client can run imaptest with 1000 clients.
> ismael> More than 1000 clients will load CPU of this bench client
>
> ismael> imaptest command (command are chosen from usage stat on our other IMAP servers):
>
> ismael> imaptest host=xxxxx port=xxx userfile=userfile mbox=/root/dovecot-crlf
> ismael> pass=sxxxx seed=123 clients=1000 select=194 uidfetch=94 noop=70
> ismael> status=82 append=49 fetch=276 list=12 store=19 expunge=22
> ismael> msubs=4 search=4 logout=1 delete=81 no_pipelining
>
> ismael> With one bench client, everything runs smoothly.
>
> ismael> # ps aux | grep dovecot | awk '{print $11,$12,$13,$14,$15,$16,$17,$18}' | sort | uniq -c
> ismael> 1 anvil: [221 connections] (anvil)
> ismael> 1 auth: [13 wait, 0 passdb, 0 userdb] (auth)
> ismael> 1 dovecot/config
> ismael> 1 dovecot/imap
> ismael> 84 dovecot/imap-login
> ismael> 1 dovecot/log
> ismael> 20 dovecot/pop3-login
> ismael> 1 grep dovecot
> ismael> 1 stats: [1307 connections] (stats)
>
> ismael> When a second instance bench instance start imaptest, clients
> ismael> of first and second instance begin to stall :
>
> ismael> 1400 stalled for 20 secs in command: 1 LOGIN"fakeuser644 at mailbench...." "password"
>
> So how is your dovecot authentication setup? Are you using a mysql
> backend? LDAP? Where is the server you're querying against? Are you
> running mysql on the same server you're running dovecot on?
In production, we use a remote galera cluster.
On benchmarking, for now, I use static for passdb and a file for userdb.
>
> Are you running multiple dovecot servers with dovecot director in
> front of them to help spread the load and to offer resilience if/when
> a backend server fails?
No. I'm directly benchmarking backend.
>
> ismael> And :
>
> ismael> # ps aux | grep dovecot | awk '{print $11,$12,$13,$14,$15,$16,$17,$18}' | sort | uniq -c
> ismael> 1 anvil: [221 connections] (anvil)
> ismael> 1 auth: [1227 wait, 0 passdb, 0 userdb] (auth)
> ismael> 1 dovecot/config
> ismael> 1 dovecot/imap
> ismael> 37 dovecot/imap-login
> ismael> 1 dovecot/log
> ismael> 20 dovecot/pop3-login
> ismael> 1 grep dovecot
> ismael> 1 stats: [680 connections] (stats)
>
> ismael> Every auth go in wait, number of connection decreases.
>
> ismael> Using mysql or a password file give same results.
>
> Where is mysql located?
Remote one, but I'll go, for now, with a passwd-file to exclude
potentials DB problems at the beginning of benchmarking.
>
> ismael> I have used different values for service_count with also no success.
>
> Post your configuration details.
#doveconf -n
auth_cache_negative_ttl = 0
auth_cache_size = 100 M
auth_cache_ttl = 2 mins
auth_failure_delay = 5 secs
auth_master_user_separator = *
auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@%+
auth_username_translation = %@
auth_verbose = yes
auth_worker_max_count = 500
base_dir = /var/run/dovecot/
default_client_limit = 100000
disable_plaintext_auth = no
imap_idle_notify_interval = 30 secs
listen = xxxxxxxxxxxx
login_greeting = xxxxxxxxxxxxxxxxxx
login_trusted_networks = xxxxxxxxxxxxxxxxxxx
mail_gid = xxxx
mail_uid = xxxx
mailbox_list_index = no
namespace {
inbox = yes
location =
prefix = INBOX.
separator = .
type = private
}
namespace {
hidden = yes
inbox = no
list = no
location =
prefix =
separator = .
type = private
}
passdb {
args = password=#hidden_use-P_to_show#
driver = static
}
plugin {
acl = vfile
quota = maildir:User quota
}
protocols = imap pop3
service anvil {
client_limit = 97000
unix_listener anvil-auth-penalty {
mode = 00
}
}
service auth-worker {
client_limit = 1
idle_kill = 0
process_limit = 6000000
process_min_avail = 0
service_count = 1
vsz_limit = 18446744073709551615 B
}
service auth {
client_limit = 0
idle_kill = 0
process_limit = 1
process_min_avail = 1
service_count = 0
vsz_limit = 1000 M
}
service imap-login {
client_limit = 26000
process_min_avail = 16
service_count = 0
vsz_limit = 1 G
}
service imap {
drop_priv_before_exec = yes
process_limit = 10000
}
service pop3-login {
service_count = 0
}
service pop3 {
drop_priv_before_exec = yes
process_limit = 10000
}
ssl = no
userdb {
driver = passwd-file
args = username_format=%Ln /usr/local/etc/dovecot/passwd-file
default_fields = uid=xxxx gid=xxxx
}
verbose_proctitle = yes
version_ignore = yes
protocol imap {
imap_max_line_length = 64 k
mail_max_userip_connections = 100000
mail_plugins = quota imap_quota acl
}
protocol pop3 {
mail_max_userip_connections = 100
mail_plugins = quota
pop3_logout_format = top=%t/%p, retr=%r/%b, del=%d/%m, size=%s
pop3_uidl_format = %f
}
ismael> I think my use of imaptest could be false.
It could be. Are you thinking that 2000 users will all be logging
into the system at the same time?
No, except when a backend is restarted, so I put delay on imaptest command :
imaptest host=xxx port=143 userfile=userfile mbox=/root/dovecot-crlf pass=password seed=123
clients=1000 delay=5 secs=3600 select=194 uidfetch=94 noop=70 status=82 append=49
fetch=276 list=12 store=19 expunge=22 msubs=4 search=4 logout=10 delete=81 no_pipelining
>
> ismael> My understanding of service auth is limited for now because
> ismael> I'm quite new to Dovecot (I have previously worked with
> ismael> Cyrus).
>
> Can't really give you any hints until you tell us more about your
> setup.
I understand, sorry again, hoping this new details are enough.
Thanks,
Ismaël
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dovecot.org/pipermail/dovecot/attachments/20220208/60bf34fb/attachment.htm>
More information about the dovecot
mailing list