[Dovecot] POP3 Performance
Hi,
We actually have a mail hosting solutions with aprox. 100 thousand of email account, where about 90% of a customers use POP3 like email configuration. About a few mounths (we perfomed a lot of migration throught mbox email software to Maildir with dovecot) but i can see that the performance is very poor and receive complaint about delays of autentications of accounts.
The solution lives in Metrocluster Netapp storage, filesystem NFS, VMware as a virtualization (the mtas are a virtual machines lives in netapp too) about 4T of data mails and a 10G network connection (betwen mtas and nfs storage) All account information work in LDAP plataform (two servers in replicated mode, no high average or delays detected in this servers)
When the traffic have a peak of 1800 concurrent connections POP3, all of service suffer a high load average (about 8 - 20 load average in each dovecot) and authenticacion takes about 2 -10 seconds (in low traffic, autenticacion takes about 60 miliseconds)
Also, each dovecot instance lives un MTA server CentOS 5.8 x86_64 with 6G RAM (virtual machine) and share's hardware with a exim instance, like a MTA relay system (autenticated relay)
Usage of network have peaks of 80Mbits (all dmz network have 1Gbits of bandwith)
Attach of dovecot -n output:
# 2.0.18: /etc/dovecot/dovecot.conf # OS: Linux 2.6.18-308.1.1.el5 x86_64 CentOS release 5.8 (Final) auth_debug_passwords = yes auth_default_realm = portalplata.cl auth_realms = portalplata.cl auth_verbose = yes auth_verbose_passwords = plain auth_worker_max_count = 100 base_dir = /var/run/dovecot/ debug_log_path = /var/log/dovecot.log default_process_limit = 200 default_vsz_limit = 512 M disable_plaintext_auth = no first_valid_gid = 12 first_valid_uid = 8 lock_method = dotlock login_greeting = Dovecot mta10 mail_cache_min_mail_count = 5 mail_debug = yes mail_fsync = always mail_full_filesystem_access = yes mail_gid = 12 mail_location = maildir:%h/Maildir:INDEX=/data/cache/indexes/%2d/%1u/%2u/%u mail_nfs_storage = yes mail_plugins = " quota" mail_uid = 8 maildir_copy_with_hardlinks = no passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = box from subject quota = maildir } postmaster_address = mail@mail.com protocols = imap pop3 sendmail_path = /usr/lib/sendmail service auth { unix_listener auth-userdb { mode = 0600 user = exim } } service imap-login { service_count = 0 } service imap-postlogin { executable = script-login /usr/local/bin/postlogin.sh user = root } service imap { executable = imap imap-postlogin } service pop3-login { inet_listener pop3s { port = 995 ssl = yes } service_count = 0 } service pop3 { process_limit = 1024 } ssl_cert =
Output of account information in a LDAP:
# nettester, email.net, MAIL, USERS, cl dn: uid=nettester,dc=email.net,o=MAIL,o=USERS,c=cl dc: email.net mailMessageStore: /export/mdir/3/12/nettester@email.net/Maildir uid: nettester cn: nettester@email.net sn: nettester@email.net gidNumber: 12 homeDirectory: /export/mdir/3/12/nettester@email.net mail: nettester@email.net uidNumber: 8 objectClass: mailUser objectClass: posixAccount objectClass: mailSetting loginShell: /bin/false description: enable service: pop3 service: imap service2: webmail mailRate: 200 mailQuota: 1024M deliveryMode: none mailReplyText: .
Any suggestions? All ideas will be have a good receptions ;)
Pd: Sorry my english
-- Mauricio López Riffo Red Hat Certified Engineer 804006455319519 Administrador de Servicios Internet Área Ingeniería Gtd Internet S.A. http://www.grupogtd.com/ Moneda 920, Oficina 602 - Fono : +562 4139742
Am 16.03.2012 12:07, schrieb Mauricio López Riffo:
Hi,
We actually have a mail hosting solutions with aprox. 100 thousandof email account, where about 90% of a customers use POP3 like email configuration. About a few mounths (we perfomed a lot of migration throught mbox email software to Maildir with dovecot) but i can see that the performance is very poor and receive complaint about delays of autentications of accounts.
The solution lives in Metrocluster Netapp storage, filesystem NFS, VMware as a virtualization (the mtas are a virtual machines lives in netapp too) about 4T of data mails and a 10G network connection (betwen mtas and nfs storage) All account information work in LDAP plataform (two servers in replicated mode, no high average or delays detected in this servers)
When the traffic have a peak of 1800 concurrent connections POP3, all of service suffer a high load average (about 8 - 20 load average in each dovecot) and authenticacion takes about 2 -10 seconds (in low traffic, autenticacion takes about 60 miliseconds)
Also, each dovecot instance lives un MTA server CentOS 5.8 x86_64 with 6G RAM (virtual machine) and share's hardware with a exim instance, like a MTA relay system (autenticated relay)
Usage of network have peaks of 80Mbits (all dmz network have 1Gbits of bandwith)
Attach of dovecot -n output:
# 2.0.18: /etc/dovecot/dovecot.conf # OS: Linux 2.6.18-308.1.1.el5 x86_64 CentOS release 5.8 (Final) auth_debug_passwords = yes auth_default_realm = portalplata.cl auth_realms = portalplata.cl auth_verbose = yes auth_verbose_passwords = plain auth_worker_max_count = 100 base_dir = /var/run/dovecot/ debug_log_path = /var/log/dovecot.log default_process_limit = 200 default_vsz_limit = 512 M disable_plaintext_auth = no first_valid_gid = 12 first_valid_uid = 8 lock_method = dotlock login_greeting = Dovecot mta10 mail_cache_min_mail_count = 5 mail_debug = yes mail_fsync = always mail_full_filesystem_access = yes mail_gid = 12 mail_location = maildir:%h/Maildir:INDEX=/data/cache/indexes/%2d/%1u/%2u/%u mail_nfs_storage = yes mail_plugins = " quota" mail_uid = 8 maildir_copy_with_hardlinks = no passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = box from subject quota = maildir } postmaster_address = mail@mail.com protocols = imap pop3 sendmail_path = /usr/lib/sendmail service auth { unix_listener auth-userdb { mode = 0600 user = exim } } service imap-login { service_count = 0 } service imap-postlogin { executable = script-login /usr/local/bin/postlogin.sh user = root } service imap { executable = imap imap-postlogin } service pop3-login { inet_listener pop3s { port = 995 ssl = yes } service_count = 0 } service pop3 { process_limit = 1024 } ssl_cert =
Output of account information in a LDAP:
# nettester, email.net, MAIL, USERS, cl dn: uid=nettester,dc=email.net,o=MAIL,o=USERS,c=cl dc: email.net mailMessageStore: /export/mdir/3/12/nettester@email.net/Maildir uid: nettester cn: nettester@email.net sn: nettester@email.net gidNumber: 12 homeDirectory: /export/mdir/3/12/nettester@email.net mail: nettester@email.net uidNumber: 8 objectClass: mailUser objectClass: posixAccount objectClass: mailSetting loginShell: /bin/false description: enable service: pop3 service: imap service2: webmail mailRate: 200 mailQuota: 1024M deliveryMode: none mailReplyText: .
Any suggestions? All ideas will be have a good receptions ;)
Pd: Sorry my english
looks like you need to play with some config stuff and do more debug on your possible bottlenecks, what did you allready played with dovecot to high performance
guess Timo will help about config settings
after all for short to read
http://wiki.dovecot.org/Authentication/Caching http://wiki.dovecot.org/LoginProcess http://wiki2.dovecot.org/Services
any reasons for that
mail_full_filesystem_access = yes ?
general
nfs may not optimal, but that a long story also virtual machines have their pros and contras maildir is not so good in performance
Best Regards
MfG Robert Schetterer
Germany/Munich/Bavaria
One quick fix to try, if it's the login-time that's killing you, is to enable auth caching:
http://wiki2.dovecot.org/Authentication/Cachingthat should offload your backend LDAP-servers from doing bind() on ever login, had a huge login performance impact for us. We use "auth_cache_size = 100 M", which gives us 99% cache hits:
dovecot: auth: Authentication cache hits 3654591/3669119 (99%)
dovecot: auth: Authentication cache inserts: positive: 588030 80931909B, negative: 912 49888B-jf
On 3/16/2012 7:07 AM, Mauricio López Riffo wrote:
Hi,
We actually have a mail hosting solutions with aprox. 100 thousandof email account, where about 90% of a customers use POP3 like email configuration. About a few mounths (we perfomed a lot of migration throught mbox email software to Maildir with dovecot) but i can see that the performance is very poor and receive complaint about delays of autentications of accounts.
The solution lives in Metrocluster Netapp storage, filesystem NFS, VMware as a virtualization (the mtas are a virtual machines lives in netapp too) about 4T of data mails and a 10G network connection (betwen mtas and nfs storage) All account information work in LDAP plataform (two servers in replicated mode, no high average or delays detected in this servers)
When the traffic have a peak of 1800 concurrent connections POP3, all of service suffer a high load average (about 8 - 20 load average in each dovecot) and authenticacion takes about 2 -10 seconds (in low traffic, autenticacion takes about 60 miliseconds)
Also, each dovecot instance lives un MTA server CentOS 5.8 x86_64 with 6G RAM (virtual machine) and share's hardware with a exim instance, like a MTA relay system (autenticated relay)
Usage of network have peaks of 80Mbits (all dmz network have 1Gbits of bandwith)
Attach of dovecot -n output:
# 2.0.18: /etc/dovecot/dovecot.conf # OS: Linux 2.6.18-308.1.1.el5 x86_64 CentOS release 5.8 (Final) auth_debug_passwords = yes auth_default_realm = portalplata.cl auth_realms = portalplata.cl auth_verbose = yes auth_verbose_passwords = plain auth_worker_max_count = 100 base_dir = /var/run/dovecot/ debug_log_path = /var/log/dovecot.log default_process_limit = 200 default_vsz_limit = 512 M disable_plaintext_auth = no first_valid_gid = 12 first_valid_uid = 8 lock_method = dotlock login_greeting = Dovecot mta10 mail_cache_min_mail_count = 5 mail_debug = yes mail_fsync = always mail_full_filesystem_access = yes mail_gid = 12 mail_location = maildir:%h/Maildir:INDEX=/data/cache/indexes/%2d/%1u/%2u/%u mail_nfs_storage = yes mail_plugins = " quota" mail_uid = 8 maildir_copy_with_hardlinks = no passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = box from subject quota = maildir } postmaster_address = mail@mail.com protocols = imap pop3 sendmail_path = /usr/lib/sendmail service auth { unix_listener auth-userdb { mode = 0600 user = exim } } service imap-login { service_count = 0 } service imap-postlogin { executable = script-login /usr/local/bin/postlogin.sh user = root } service imap { executable = imap imap-postlogin } service pop3-login { inet_listener pop3s { port = 995 ssl = yes } service_count = 0 } service pop3 { process_limit = 1024 } ssl_cert =
Output of account information in a LDAP:
# nettester, email.net, MAIL, USERS, cl dn: uid=nettester,dc=email.net,o=MAIL,o=USERS,c=cl dc: email.net mailMessageStore: /export/mdir/3/12/nettester@email.net/Maildir uid: nettester cn: nettester@email.net sn: nettester@email.net gidNumber: 12 homeDirectory: /export/mdir/3/12/nettester@email.net mail: nettester@email.net uidNumber: 8 objectClass: mailUser objectClass: posixAccount objectClass: mailSetting loginShell: /bin/false description: enable service: pop3 service: imap service2: webmail mailRate: 200 mailQuota: 1024M deliveryMode: none mailReplyText: .
Any suggestions? All ideas will be have a good receptions ;)
Pd: Sorry my english
It doesn't seem to matter what type of hardware you might have, NFS can cause real bottlenecks, even to the point that your machine may report disk errors.
Unfortunately, it's an evil necessity in some shops, but any way to eliminate NFS when large throughput is occurring will definitely help. Make sure you're running the latest version of NFS on all machines since V3 and V4 don't always like each other.
I don't have a solution for it's replacement other than expensive hardware solutions.
steve
On 16.3.2012, at 13.07, Mauricio López Riffo wrote:
We actually have a mail hosting solutions with aprox. 100 thousand of email account, where about 90% of a customers use POP3 like email configuration. About a few mounths (we perfomed a lot of migration throught mbox email software to Maildir with dovecot) but i can see that the performance is very poor and receive complaint about delays of autentications of accounts.
The solution lives in Metrocluster Netapp storage, filesystem NFS, VMware as a virtualization (the mtas are a virtual machines lives in netapp too) about 4T of data mails and a 10G network connection (betwen mtas and nfs storage) All account information work in LDAP plataform (two servers in replicated mode, no high average or delays detected in this servers)
Maildir isn't very good for POP3, especially if the POP3 clients delete the mails after download. With Dovecot you could look into switching to multi-dbox format, which would have much better performance.
When the traffic have a peak of 1800 concurrent connections POP3, all of service suffer a high load average (about 8 - 20 load average in each dovecot) and authenticacion takes about 2 -10 seconds (in low traffic, autenticacion takes about 60 miliseconds)
What does the CPU usage and NFS IOPS usage look like during those times? Meaning is the problem related to disk usage or something else?
Note that for POP3 connections you don't get the "OK Logged in" reply until all of the message sizes have been read into memory.
Also, each dovecot instance lives un MTA server CentOS 5.8 x86_64 with 6G RAM (virtual machine) and share's hardware with a exim instance, like a MTA relay system (autenticated relay)
Are you randomly redirecting users to different Dovecot servers? Dovecot director would work better: http://wiki2.dovecot.org/Director
auth_worker_max_count = 100
Auth workers are irrelevant with LDAP.
lock_method = dotlock
fcntl would be faster, if your NFS setup can handle it.
mail_cache_min_mail_count = 5
I'm not really sure if it's a good idea to ever set this anything else than 0. Of course if you have detected that this actually decreases disk IO I'd be interested to see numbers.
mail_full_filesystem_access = yes
If your users are sharing the same UID, this means all the users can access each others' mails now! Even if they have different UIDs this is unlikely to be helpful.
mail_location = maildir:%h/Maildir:INDEX=/data/cache/indexes/%2d/%1u/%2u/%u mail_nfs_storage = yes
Is /data also on NFS? Or does each server have its own local indexes?
maildir_copy_with_hardlinks = no
This makes IMAP COPY command slower. Is there a reason why you've disabled it?
service imap-postlogin { executable = script-login /usr/local/bin/postlogin.sh user = root }
post-login script makes logins slower. What do you do in it? Why only for IMAP, not POP3?
plugin { autocreate = Sent Items autocreate2 = Spam autocreate3 = Drafts autocreate4 = Trash autosubscribe = Sent Items autosubscribe2 = Spam autosubscribe3 = Drafts autosubscribe4 = Trash }
These also slow logins down a little bit. v2.1 fixes that.
protocol pop3 { mail_plugins = " quota quota notify mail_log" pop3_reuse_xuidl = yes pop3_save_uidl = yes pop3_uidl_format = %u }
pop3_uidl_format=%u is a little bad, since it doesn't include %v. And you can improve pop3 performance with:
pop3_no_flag_changes=yes
And if the maildir filenames don't contain S=1234 sizes, this also makes a huge difference:
pop3_fast_size_lookups=yes
On 03/16/12 06:07, Timo Sirainen wrote:
Maildir isn't very good for POP3, especially if the POP3 clients delete the mails after download. With Dovecot you could look into switching to multi-dbox format, which would have much better performance.
Timo, can you explain why Maildir isn't a good for POP3 in this context?
Another thing our existing POP3 servers did was batch all of the deletes until after the +OK... was returned from quit. This doesn't reduce server load but has the impression of creating faster response times to the clients.
-K
On 16.3.2012, at 19.49, Kelsey Cummings wrote:
On 03/16/12 06:07, Timo Sirainen wrote:
Maildir isn't very good for POP3, especially if the POP3 clients delete the mails after download. With Dovecot you could look into switching to multi-dbox format, which would have much better performance.
Timo, can you explain why Maildir isn't a good for POP3 in this context?
Compared to mbox/mdbox: It needs to read and delete multiple separate files, which is typically much slower than reading and deleting a single file.
Another thing our existing POP3 servers did was batch all of the deletes until after the +OK... was returned from quit. This doesn't reduce server load but has the impression of creating faster response times to the clients.
You mean deleting the messages after +OK, instead of before? Does it really make a difference?.. Dovecot can reply with -ERR to QUIT if deletions failed for some reason.
On 03/16/12 10:54, Timo Sirainen wrote:
Another thing our existing POP3 servers did was batch all of the deletes until after the +OK... was returned from quit. This doesn't reduce server load but has the impression of creating faster response times to the clients.
You mean deleting the messages after +OK, instead of before? Does it really make a difference?
In the context of a clients "send and receive" phase taking a (small) fraction of a second less time, perhaps, but it is a small difference in any case. It was one of many small changes we made to try to improve interactive performance.
.. Dovecot can reply with -ERR to QUIT if deletions failed for some reason.
True, we decided that loosing that ability didn't really matter. (Like not counting newlines as two bytes in the message size.)
-K
On Fri, 16 Mar 2012 15:07:24 +0200, Timo Sirainen tss@iki.fi wrote:
On 16.3.2012, at 13.07, Mauricio López Riffo wrote:
pop3_no_flag_changes=yes
Is it the same as pop3_no_flag_updates=yes ?
M.
On 17.3.2012, at 16.14, Mark Alan wrote:
On Fri, 16 Mar 2012 15:07:24 +0200, Timo Sirainen tss@iki.fi wrote:
On 16.3.2012, at 13.07, Mauricio López Riffo wrote:
pop3_no_flag_changes=yes
Is it the same as pop3_no_flag_updates=yes ?
Yeah. I wrote it from my memory.
participants (7)
-
Jan-Frode Myklebust
-
Kelsey Cummings
-
Mark Alan
-
Mauricio López Riffo
-
Robert Schetterer
-
Steve Campbell
-
Timo Sirainen