[Dovecot] Dovecot optimisation
Hello all.
We start to use Dovecot at our email production but it runs not such fast as we expect.
We use Dell 2950 with 4GbRAM at FreeBSD-7.0-p9 - load avarage never upper a 0.5
We have about 500 clients most of it use Outlook 2007 via IMAP.
We run in this problems:
- Sync of imap folder is really slow(I think it is Outlook problem)
- Time after time we got Sync error from outlook
- Time after time our monitoring system says what IMAP port not answer in 10 sec. In real life it start to answer in 15-20 sec what is not really good - client thinks what server don`t work. If I restart dovecot
- it start to answer in 1-3 sec.
I think problem in my conf, may be I need tune it?
*dovecot -n don`t give full conf so i use this:* mail# grep -v '#' /usr/local/etc/dovecot.conf | egrep -v '^$' base_dir = /var/run/dovecot/ protocols = imap imaps pop3 pop3s managesieve listen = * disable_plaintext_auth = yes shutdown_clients = yes log_path = /var/log/dovecot/dovecot.log info_log_path = /var/log/dovecot/dovecot.log log_timestamp = "%b %d %H:%M:%S " syslog_facility = mail ssl_disable = no ssl_cert_file = /usr/local/etc/certs/fxclub_org.crt ssl_key_file = /usr/local/etc/certs/fxclub_org.key ssl_ca_file = /usr/local/etc/certs/GlobalSignCA.crt ssl_verify_client_cert = no ssl_parameters_regenerate = 168 ssl_cipher_list = ALL:!LOW:!SSLv2 verbose_ssl = no login_dir = /var/run/dovecot/login login_user = dovecot login_process_size = 64 login_process_per_connection = yes login_processes_count = 10 login_max_processes_count = 128 login_greeting = Dovecot ready. login_log_format_elements = user=<%u> method=%m rip=%r lip=%l %c login_log_format = %$: %s mail_location = maildir:/var/spool/dovecot/domains/%d/%n/ mail_uid = dmail mail_gid = dmail mail_privileged_group = mail mail_full_filesystem_access = no mail_debug = no mail_log_prefix = "%Us(%u): " mail_log_max_lines_per_sec = 10 mmap_disable = no dotlock_use_excl = yes fsync_disable = no mail_nfs_storage = no mail_nfs_index = no lock_method = fcntl mail_drop_priv_before_exec = no verbose_proctitle = yes first_valid_uid = 500 first_valid_gid = 0 max_mail_processes = 512 mail_process_size = 256 mail_max_keyword_length = 50 mail_cache_min_mail_count = 0 mailbox_idle_check_interval = 30 mail_save_crlf = no maildir_stat_dirs = no maildir_copy_with_hardlinks = yes maildir_copy_preserve_filename = no mbox_read_locks = fcntl mbox_write_locks = dotlock fcntl mbox_lock_timeout = 300 mbox_dotlock_change_timeout = 120 mbox_min_index_size = 0 protocol imap { login_executable = /usr/local/libexec/dovecot/imap-login
mail_executable = /usr/local/libexec/dovecot/imap imap_max_line_length = 65536 mail_max_userip_connections = 10 mail_plugins = quota imap_quota autocreate mail_plugin_dir = /usr/local/lib/dovecot/imap login_greeting_capability = yes imap_logout_format = bytes=%i/%o imap_client_workarounds = delay-newmail tb-extra-mailbox-sep }
protocol pop3 { login_executable = /usr/local/libexec/dovecot/pop3-login mail_executable = /usr/local/libexec/dovecot/pop3 pop3_enable_last = no pop3_lock_session = no pop3_uidl_format = %08Xu%08Xv pop3_logout_format = top=%t/%p, retr=%r/%b, del=%d/%m, size=%s mail_max_userip_connections = 3 mail_plugins = quota mail_plugin_dir = /usr/local/lib/dovecot/pop3 pop3_client_workarounds = outlook-no-nuls oe-ns-eoh } protocol lda { postmaster_address = postmaster@fxclub.org hostname = mail.fxclub.org mail_plugins = cmusieve quota mail_plugin_dir = /usr/local/lib/dovecot/lda quota_full_tempfail = yes deliver_log_format = msgid=%m: %$ log_path = /var/log/dovecot/deliver.log info_log_path = /var/log/dovecot/deliver.log sendmail_path = /usr/sbin/sendmail rejection_reason = Your message to <%t> was automatically rejected:%n%r auth_socket_path = /var/run/dovecot/auth-master sieve_global_path = /var/spool/dovecot/global.sieve } protocol managesieve { listen = *:2000 login_executable = /usr/local/libexec/dovecot/managesieve-login mail_executable = /usr/local/libexec/dovecot/managesieve managesieve_max_line_length = 65536 sieve_storage = % sieve = ~/.dovecot.sieve mail_location = managesieve_implementation_string = dovecot } auth_executable = /usr/local/libexec/dovecot/dovecot-auth auth_process_size = 256 auth_cache_size = 0 auth_cache_ttl = 3600 auth_cache_negative_ttl = 3600 auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@ auth_username_format = %Lu auth_verbose = yes auth_debug = no auth_debug_passwords = no auth_worker_max_count = 100 auth_failure_delay = 2 auth default { mechanisms = plain login passdb ldap { args = /usr/local/etc/dovecot-ldap.conf } userdb ldap { args = /usr/local/etc/dovecot-ldap.conf } user = dovecot-auth count = 1 ssl_require_client_cert = no socket listen { master { path = /var/run/dovecot/auth-master mode = 0600 user = dmail } client { path = /var/run/dovecot/auth-client mode = 0660 user = exim } } } dict { } plugin {
quota = maildir quota_rule = *:storage=1024M quota_rule2 = Trash:storage=1256M quota_rule3 = SPAM:ignore quota_warning = storage=95%% /usr/local/bin/quota-warning.sh 95 quota_warning2 = storage=80%% /usr/local/bin/quota-warning.sh 80 autocreate = Trash autocreate2 = Sent autocreate3 = Spam autosubscribe = Trash autosubscribe2 = Sent autosubscribe3 = Spam }
-- Best regards, Proskurin Kirill
On 1/20/2009 4:10 AM, Proskurin Kirill wrote:
We use Dell 2950 with 4GbRAM at FreeBSD-7.0-p9 - load avarage never upper a 0.5
You don't give dovecot version... don't you think that might be important?
We have about 500 clients most of it use Outlook 2007 via IMAP.
We run in this problems:
- Sync of imap folder is really slow(I think it is Outlook problem)
Yes, Outlook is a notoriously bad IMAP client, but 2007 is supposed to be a bit better behaved than previous versions...
*dovecot -n don`t give full conf so i use this:*
If dovecot -n doesn't give expected results, that may be a clue... doevecot -n gives the config that the running version of dovecot is actually USING - so if it contradcits what you expect, then you are most likely editing the wrong config file.
Please provide FULL dovecot -n output always, then copy/paste the stuff that isn't included (like the contents of dovecot-sql.conf if using it, etc)...
--
Best regards,
Charles
Charles Marcus wrote:
On 1/20/2009 4:10 AM, Proskurin Kirill wrote:
We use Dell 2950 with 4GbRAM at FreeBSD-7.0-p9 - load avarage never upper a 0.5
You don't give dovecot version... don't you think that might be important?
We have about 500 clients most of it use Outlook 2007 via IMAP.
We run in this problems:
- Sync of imap folder is really slow(I think it is Outlook problem)
Yes, Outlook is a notoriously bad IMAP client, but 2007 is supposed to be a bit better behaved than previous versions...
*dovecot -n don`t give full conf so i use this:*
If dovecot -n doesn't give expected results, that may be a clue... doevecot -n gives the config that the running version of dovecot is actually USING - so if it contradcits what you expect, then you are most likely editing the wrong config file.
Please provide FULL dovecot -n output always, then copy/paste the stuff that isn't included (like the contents of dovecot-sql.conf if using it, etc)...
Sorry - you a right. But dovecot -n don`t show things like a login_max_processes_count and other thing what I think must be a bootle neck.
mail# dovecot -n # 1.1.8: /usr/local/etc/dovecot.conf # OS: FreeBSD 7.0-RELEASE-p9 i386 ufs base_dir: /var/run/dovecot/ log_path: /var/log/dovecot/dovecot.log info_log_path: /var/log/dovecot/dovecot.log protocols: imap imaps pop3 pop3s managesieve listen(default): * listen(imap): * listen(pop3): * listen(managesieve): *:2000 ssl_ca_file: /usr/local/etc/certs/GlobalSignCA.crt ssl_cert_file: /usr/local/etc/certs/fxclub_org.crt ssl_key_file: /usr/local/etc/certs/fxclub_org.key ssl_cipher_list: ALL:!LOW:!SSLv2 login_dir: /var/run/dovecot/login login_executable(default): /usr/local/libexec/dovecot/imap-login login_executable(imap): /usr/local/libexec/dovecot/imap-login login_executable(pop3): /usr/local/libexec/dovecot/pop3-login login_executable(managesieve): /usr/local/libexec/dovecot/managesieve-login login_greeting_capability(default): yes login_greeting_capability(imap): yes login_greeting_capability(pop3): no login_greeting_capability(managesieve): no login_processes_count: 10 mail_max_userip_connections(default): 10 mail_max_userip_connections(imap): 10 mail_max_userip_connections(pop3): 3 mail_max_userip_connections(managesieve): 10 verbose_proctitle: yes first_valid_gid: 0 mail_privileged_group: mail mail_uid: dmail mail_gid: dmail mail_location(default): maildir:/var/spool/dovecot/domains/%d/%n/ mail_location(imap): maildir:/var/spool/dovecot/domains/%d/%n/ mail_location(pop3): maildir:/var/spool/dovecot/domains/%d/%n/ mail_location(managesieve): mail_executable(default): /usr/local/libexec/dovecot/imap mail_executable(imap): /usr/local/libexec/dovecot/imap mail_executable(pop3): /usr/local/libexec/dovecot/pop3 mail_executable(managesieve): /usr/local/libexec/dovecot/managesieve mail_plugins(default): quota imap_quota autocreate mail_plugins(imap): quota imap_quota autocreate mail_plugins(pop3): quota mail_plugins(managesieve): mail_plugin_dir(default): /usr/local/lib/dovecot/imap mail_plugin_dir(imap): /usr/local/lib/dovecot/imap mail_plugin_dir(pop3): /usr/local/lib/dovecot/pop3 mail_plugin_dir(managesieve): /usr/local/lib/dovecot/managesieve imap_client_workarounds(default): delay-newmail tb-extra-mailbox-sep imap_client_workarounds(imap): delay-newmail tb-extra-mailbox-sep imap_client_workarounds(pop3): imap_client_workarounds(managesieve): pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh pop3_client_workarounds(managesieve): sieve_storage(default): sieve_storage(imap): sieve_storage(pop3): sieve_storage(managesieve): % sieve(default): sieve(imap): sieve(pop3): sieve(managesieve): ~/.dovecot.sieve auth default: mechanisms: plain login user: dovecot-auth username_format: %Lu verbose: yes worker_max_count: 100 passdb: driver: ldap args: /usr/local/etc/dovecot-ldap.conf userdb: driver: ldap args: /usr/local/etc/dovecot-ldap.conf socket: type: listen client: path: /var/run/dovecot/auth-client mode: 432 user: exim master: path: /var/run/dovecot/auth-master mode: 384 user: dmail plugin: quota: maildir quota_rule: *:storage=1024M quota_rule2: Trash:storage=1256M quota_rule3: SPAM:ignore quota_warning: storage=95%% /usr/local/bin/quota-warning.sh 95 quota_warning2: storage=80%% /usr/local/bin/quota-warning.sh 80 autocreate: Trash autocreate2: Sent autocreate3: Spam autosubscribe: Trash autosubscribe2: Sent autosubscribe3: Spam
mail# grep -v '#' /usr/local/etc/dovecot-ldap.conf | egrep -v '^$' hosts = 127.0.0.1 dn = cn=root,dc=CAS dnpass = secret tls = no auth_bind = no ldap_version = 3 base = dc=CAS deref = never scope = subtree user_attrs = mailQuotaSize=quota_rule=*:storage=%$M ,=uid=1002,=gid=1002,=home=/var/spool/dovecot/domains/%d/%n user_filter = (&(objectClass=mailUser)(mail=%u)) pass_attrs = userPassword=password pass_filter = (&(objectClass=mailUser)(mail=%u)) default_pass_scheme = CRYPT
-- Best regards, Proskurin Kirill
On 1/20/2009, Proskurin Kirill (proskurin-kv@fxclub.org) wrote:
But dovecot -n don`t show things like a login_max_processes_count and other thing what I think must be a bootle neck.
I was aware it left out config info from other external files (like dovecot-sql configs) - but I thought thiall other settings should show... maybe this is a bug?
--
Best regards,
Charles
On Jan 20, 2009, at 6:05 AM, Proskurin Kirill wrote:
But dovecot -n don`t show things like a login_max_processes_count
and other thing what I think must be a bootle neck.
It doesn't show it, because you're using the default value for it.
Anyway if you think login processes is the bottleneck, you can try if
setting login_process_per_connection=no helps. http://wiki.dovecot.org/LoginProcess
Timo Sirainen wrote:
On Jan 20, 2009, at 6:05 AM, Proskurin Kirill wrote:
But dovecot -n don`t show things like a login_max_processes_count and other thing what I think must be a bootle neck.
It doesn't show it, because you're using the default value for it. Anyway if you think login processes is the bottleneck, you can try if setting login_process_per_connection=no helps. http://wiki.dovecot.org/LoginProcess
Problem what I dont know were is a bottle neck and I dont know how to
find it. I turn on mail_debug and all other but it not show anything
what I look.
I just increes all things what may be was a bottle neck and... seams to fix it. But I really want to know what it is was.
-- Best regards. Proskurin Kirill
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Tue, 20 Jan 2009, Proskurin Kirill wrote:
Problem what I don
t know were is a bottle neck and I dont know how to find it. I turn on mail_debug and all other but it not show anything what I look.
Does these problems occure all the time, e.g. if you restart the Dovecot demon and there are just a bunch of users on it.
Do you have a file descriptor limit for the demon process?
Do you have some "security" stuff running, a BSD-equivalent of SELinux or AppArmour?
Bye,
Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux)
iQEVAwUBSXXuLHWSIuGy1ktrAQL0BAf/TrQe4KWHHmSrlUoH2Kp4kj503cAJTBHt XwMbrek1K75BVseCfOZl36uHhkWzXULkhJOinN9cgsRG9yfYWfxj6bOguD+bsMp4 mhql4zkrl/w7yuUHqtGnekzNqFYiBDpkWxYWC3y1jOLEI7SLWcc9k/9AqM3l4cx1 KFEMoDJLJvl7BtSI2iMSgBNSbMyAgESxBQfg6MYqfY5kKSwQ7jkzZjmGLcZaCnV5 2VKzhyKIOC0E+h5xVWNX4GJ2jPJgZOX42FzrtiveIhGW96dF8G6Z8YARJuoaYrX6 KEyVgakMigbxixSIc4HWvbocuTqDtf6Xn99VMfte7NKwvak3a01SQA== =rSna -----END PGP SIGNATURE-----
Steffen Kaiser пишет:
Does these problems occure all the time, e.g. if you restart the Dovecot demon and there are just a bunch of users on it.
After restart - it work for some time, but it is just because no all connect to it again.
Do you have a file descriptor limit for the demon process?
System limit? 512mb. It not run on this.
Do you have some "security" stuff running, a BSD-equivalent of SELinux or AppArmour?
Nope.
As I sad before - i just increase *all* thing what can be a bottleneck twice. And seems to "fix" it. But it makes me think what it is real will be good if dovecot can say in debuging loging what he reach some limits from his config. It will make finding a bottleneck such easy.. heh. I just what to know - were was a problem, but seam to i don`t have any tool for this.
Any way - dovecot is a great and i glad what i migrate from a Cyrus. ;-)
-- Best regards, Proskurin Kirill
On Tue, 2009-01-20 at 22:52 +0300, Proskurin Kirill wrote:
But it makes me think what it is real will be good if dovecot can say in debuging loging what he reach some limits from his config.
Pretty much everything such thing does log an error/warning, except when login process max count is reached. I guess this should be fixed.
Proskurin Kirill wrote:
Steffen Kaiser пишет:
Does these problems occure all the time, e.g. if you restart the Dovecot demon and there are just a bunch of users on it.
After restart - it work for some time, but it is just because no all connect to it again.
Do you have a file descriptor limit for the demon process?
System limit? 512mb. It not run on this.
Do you have some "security" stuff running, a BSD-equivalent of SELinux or AppArmour?
Nope.
As I sad before - i just increase *all* thing what can be a bottleneck twice. And seems to "fix" it. But it makes me think what it is real will be good if dovecot can say in debuging loging what he reach some limits from his config. It will make finding a bottleneck such easy.. heh. I just what to know - were was a problem, but seam to i don`t have any tool for this.
ps ax | grep imap-login | wc -l
~Seth
Proskurin Kirill wrote:
Timo Sirainen wrote:
On Jan 20, 2009, at 6:05 AM, Proskurin Kirill wrote:
But dovecot -n don`t show things like a login_max_processes_count and other thing what I think must be a bootle neck.
It doesn't show it, because you're using the default value for it. Anyway if you think login processes is the bottleneck, you can try if setting login_process_per_connection=no helps. http://wiki.dovecot.org/LoginProcess
Problem what I don
t know were is a bottle neck and I dont know how to find it. I turn on mail_debug and all other but it not show anything what I look.I just increes all things what may be was a bottle neck and... seams to fix it. But I really want to know what it is was.
Setting "login_process_per_connection = no" is much more scalable for large environments. Obviously there is a slight security trade off. You can confirm if it's a login bottleneck by looking to see if you have the maximum number of "imap-login" processes (default is 128) rather than just arbitrarily increasing everything.
~Seth
On 1/20/2009, Timo Sirainen (tss@iki.fi) wrote:
It doesn't show it, because you're using the default value for it.
Ahh...
Which brings up a repeat request for alphabetical sorting of the output of dovecot -n and dovecot -a (makes it very easy to find settings and to make sure you aren't missing something), and for a new -d option to output only the default settings (as opposed to ALL), to make it easy to clean up redundant settings (setting something explicitly that has the value you are setting it to as the default), which makes for clean -n output.
--
Best regards,
Charles
On Tue, 2009-01-20 at 11:09 -0500, Charles Marcus wrote:
On 1/20/2009, Timo Sirainen (tss@iki.fi) wrote:
It doesn't show it, because you're using the default value for it.
Ahh...
Which brings up a repeat request for alphabetical sorting of the output of dovecot -n and dovecot -a (makes it very easy to find settings and to make sure you aren't missing something), and for a new -d option to output only the default settings (as opposed to ALL), to make it easy to clean up redundant settings (setting something explicitly that has the value you are setting it to as the default), which makes for clean -n output.
I'm now working on configuration handling rewrite for v1.3. Maybe for that. :)
On 1/20/2009 11:23 AM, Timo Sirainen wrote:
Which brings up a repeat request for alphabetical sorting of the output of dovecot -n and dovecot -a (makes it very easy to find settings and to make sure you aren't missing something), and for a new -d option to output only the default settings (as opposed to ALL), to make it easy to clean up redundant settings (setting something explicitly that has the value you are setting it to as the default), which makes for clean -n output.
I'm now working on configuration handling rewrite for v1.3. Maybe for that. :)
Somehow I'm not surprised... ;)
--
Best regards,
Charles
On 1/20/2009 11:23 AM, Timo Sirainen wrote:
Which brings up a repeat request for alphabetical sorting of the output of dovecot -n and dovecot -a (makes it very easy to find settings and to make sure you aren't missing something), and for a new -d option to output only the default settings (as opposed to ALL), to make it easy to clean up redundant settings (setting something explicitly that has the value you are setting it to as the default), which makes for clean -n output.
I'm now working on configuration handling rewrite for v1.3. Maybe for that. :)
While we're on the subject, and since you're already working on a (total?) rewrite of the config handling... ;)
To continue to borrow from postfix, how about adding some additional parameters - or at least coding to allow for their addition later if it is a lot of work now?
Suggestions:
Change command from 'dovecot -n[d][a]' to 'doveconf -n[d][a]' etc...
Allow config parameters to be added/edited via the commandline ala postfix, for example:
doveconf -e "mail_executable=/usr/libexec/dovecot/imap" would edit (-e) the config file and add this parameter setting
doveconf mail_executable would show the current setting for mail_executable
etc...
Obviously, I like the way postfix works... :)
--
Best regards,
Charles
Timo Sirainen пишет:
On Tue, 2009-01-20 at 11:09 -0500, Charles Marcus wrote:
On 1/20/2009, Timo Sirainen (tss@iki.fi) wrote:
It doesn't show it, because you're using the default value for it.
Ahh...
Which brings up a repeat request for alphabetical sorting of the output of dovecot -n and dovecot -a (makes it very easy to find settings and to make sure you aren't missing something), and for a new -d option to output only the default settings (as opposed to ALL), to make it easy to clean up redundant settings (setting something explicitly that has the value you are setting it to as the default), which makes for clean -n output.
I'm now working on configuration handling rewrite for v1.3. Maybe for that. :)
Heh... We so much wating for 2.0 :-)
-- Best regards, Proskurin Kirill
participants (6)
-
Charles Marcus
-
Proskurin Kirill
-
Proskurin Kirill
-
Seth Mattinen
-
Steffen Kaiser
-
Timo Sirainen