<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hello,</p>
<p>thank you for your advices and sorry to not have detailed infra<br>
</p>
<blockquote type="cite"
cite="mid:25089.22896.393574.82026@quad.stoffel.home">
<pre class="moz-quote-pre" wrap="">
ismael> I'm currently benchmarking new hardware aimed to serve around
ismael> 70k users For now, our IMAP server have 13k users.
This doesn't help us help you. Is this a new rasperry Pi 4? Is it a
Dual CPU AMD Rzyzen with 128gb of memory and fast NVMe disks? What is
your system setup? </pre>
</blockquote>
<p>Sorry, I have two servers to bench :</p>
<p>- first one (a model like our current IMAP servers) is 18To HDD,
256Go RAM, 8c/16th</p>
<p>- second (new one aimed to serve many more customers) is 24 x 14
TO (HDD SAS), <span>192GB DDR4 2,6Ghz, </span><span><span>
12c/24t - 2.4GHz/3.5GHz</span></span></p>
<p>OS is FreeBSD 12.2<br>
</p>
<blockquote type="cite"
cite="mid:25089.22896.393574.82026@quad.stoffel.home">
<pre class="moz-quote-pre" wrap="">
ismael> To run imaptest, I've spwan some bench clients.
Are these tests run from remote hosts? What kind of network are you
using? </pre>
</blockquote>
<p>Yes, imaptest is running from kvm remote virtual machines in the
same DC.<br>
They are some networks hops between them, but few.<br>
</p>
<blockquote type="cite"
cite="mid:25089.22896.393574.82026@quad.stoffel.home">
<pre class="moz-quote-pre" wrap="">
ismael> Each bench client can run imaptest with 1000 clients.
ismael> More than 1000 clients will load CPU of this bench client
ismael> imaptest command (command are chosen from usage stat on our other IMAP servers):
ismael> imaptest host=xxxxx port=xxx userfile=userfile mbox=/root/dovecot-crlf
ismael> pass=sxxxx seed=123 clients=1000 select=194 uidfetch=94 noop=70
ismael> status=82 append=49 fetch=276 list=12 store=19 expunge=22
ismael> msubs=4 search=4 logout=1 delete=81 no_pipelining
ismael> With one bench client, everything runs smoothly.
ismael> # ps aux | grep dovecot | awk '{print $11,$12,$13,$14,$15,$16,$17,$18}' | sort | uniq -c
ismael> 1 anvil: [221 connections] (anvil)
ismael> 1 auth: [13 wait, 0 passdb, 0 userdb] (auth)
ismael> 1 dovecot/config
ismael> 1 dovecot/imap
ismael> 84 dovecot/imap-login
ismael> 1 dovecot/log
ismael> 20 dovecot/pop3-login
ismael> 1 grep dovecot
ismael> 1 stats: [1307 connections] (stats)
ismael> When a second instance bench instance start imaptest, clients
ismael> of first and second instance begin to stall :
ismael> 1400 stalled for 20 secs in command: 1 LOGIN <a class="moz-txt-link-rfc2396E" href="mailto:fakeuser644@mailbench....">"fakeuser644@mailbench...."</a> "password"
So how is your dovecot authentication setup? Are you using a mysql
backend? LDAP? Where is the server you're querying against? Are you
running mysql on the same server you're running dovecot on?</pre>
</blockquote>
<p>In production, we use a remote galera cluster.<br>
On benchmarking, for now, I use static for passdb and a file for
userdb.<br>
</p>
<p><br>
</p>
<blockquote type="cite"
cite="mid:25089.22896.393574.82026@quad.stoffel.home">
<pre class="moz-quote-pre" wrap="">
Are you running multiple dovecot servers with dovecot director in
front of them to help spread the load and to offer resilience if/when
a backend server fails? </pre>
</blockquote>
<p>No. I'm directly benchmarking backend.<br>
</p>
<p><br>
</p>
<blockquote type="cite"
cite="mid:25089.22896.393574.82026@quad.stoffel.home">
<pre class="moz-quote-pre" wrap="">
ismael> And :
ismael> # ps aux | grep dovecot | awk '{print $11,$12,$13,$14,$15,$16,$17,$18}' | sort | uniq -c
ismael> 1 anvil: [221 connections] (anvil)
ismael> 1 auth: [1227 wait, 0 passdb, 0 userdb] (auth)
ismael> 1 dovecot/config
ismael> 1 dovecot/imap
ismael> 37 dovecot/imap-login
ismael> 1 dovecot/log
ismael> 20 dovecot/pop3-login
ismael> 1 grep dovecot
ismael> 1 stats: [680 connections] (stats)
ismael> Every auth go in wait, number of connection decreases.
ismael> Using mysql or a password file give same results.
Where is mysql located? </pre>
</blockquote>
Remote one, but I'll go, for now, with a passwd-file to exclude
potentials DB problems at the beginning of benchmarking.<br>
<blockquote type="cite"
cite="mid:25089.22896.393574.82026@quad.stoffel.home">
<pre class="moz-quote-pre" wrap="">
ismael> I have used different values for service_count with also no success.
Post your configuration details.</pre>
</blockquote>
<pre>#doveconf -n</pre>
<pre>auth_cache_negative_ttl = 0
auth_cache_size = 100 M
auth_cache_ttl = 2 mins
auth_failure_delay = 5 secs
auth_master_user_separator = *
auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@%+
auth_username_translation = %@
auth_verbose = yes
auth_worker_max_count = 500
base_dir = /var/run/dovecot/
default_client_limit = 100000
disable_plaintext_auth = no
imap_idle_notify_interval = 30 secs
listen = xxxxxxxxxxxx
login_greeting = xxxxxxxxxxxxxxxxxx
login_trusted_networks = xxxxxxxxxxxxxxxxxxx
mail_gid = xxxx
mail_uid = xxxx
mailbox_list_index = no
namespace {
inbox = yes
location =
prefix = INBOX.
separator = .
type = private
}
namespace {
hidden = yes
inbox = no
list = no
location =
prefix =
separator = .
type = private
}
passdb {
args = password=#hidden_use-P_to_show#
driver = static
}
plugin {
acl = vfile
quota = maildir:User quota
}
protocols = imap pop3
</pre>
<pre>service anvil {
client_limit = 97000
unix_listener anvil-auth-penalty {
mode = 00
}
}
service auth-worker {
client_limit = 1
idle_kill = 0
process_limit = 6000000
process_min_avail = 0
service_count = 1
vsz_limit = 18446744073709551615 B
}
service auth {
client_limit = 0
idle_kill = 0
process_limit = 1
process_min_avail = 1
service_count = 0
vsz_limit = 1000 M
}
service imap-login {
client_limit = 26000
process_min_avail = 16
service_count = 0
vsz_limit = 1 G
}
service imap {
drop_priv_before_exec = yes
process_limit = 10000
}
service pop3-login {
service_count = 0
}
service pop3 {
drop_priv_before_exec = yes
process_limit = 10000
}
ssl = no</pre>
<pre>userdb {
driver = passwd-file
args = username_format=%Ln /usr/local/etc/dovecot/passwd-file
default_fields = uid=xxxx gid=xxxx
}
verbose_proctitle = yes
version_ignore = yes
protocol imap {
imap_max_line_length = 64 k
mail_max_userip_connections = 100000
mail_plugins = quota imap_quota acl
}
protocol pop3 {
mail_max_userip_connections = 100
mail_plugins = quota
pop3_logout_format = top=%t/%p, retr=%r/%b, del=%d/%m, size=%s
pop3_uidl_format = %f
}
ismael> I think my use of imaptest could be false.
It could be. Are you thinking that 2000 users will all be logging
into the system at the same time? </pre>
<blockquote type="cite"
cite="mid:25089.22896.393574.82026@quad.stoffel.home">
<pre class="moz-quote-pre" wrap=""></pre>
</blockquote>
<p>No, except when a backend is restarted, so I put delay on
imaptest command :</p>
<pre>imaptest host=xxx port=143 userfile=userfile mbox=/root/dovecot-crlf pass=password seed=123
clients=1000 delay=5 secs=3600 select=194 uidfetch=94 noop=70 status=82 append=49
fetch=276 list=12 store=19 expunge=22 msubs=4 search=4 logout=10 delete=81 no_pipelining
</pre>
<blockquote type="cite"
cite="mid:25089.22896.393574.82026@quad.stoffel.home">
<pre class="moz-quote-pre" wrap="">
ismael> My understanding of service auth is limited for now because
ismael> I'm quite new to Dovecot (I have previously worked with
ismael> Cyrus).
Can't really give you any hints until you tell us more about your
setup.</pre>
</blockquote>
I understand, sorry again, hoping this new details are enough.
<pre class="moz-quote-pre" wrap="">
Thanks,
Ismaël
</pre>
</body>
</html>