[Dovecot] Config review (2.0.5)
Since I have these performance problems after migration to 2.0.5 I'm publishing my config for review. Can anybody spot any obvious signs of trouble?
# 2.0.5: /usr/local/etc/dovecot.conf # OS: Linux 2.6.32-24-generic-pae i686 Debian squeeze/sid
auth_debug = yes auth_debug_passwords = yes auth_mechanisms = plain login disable_plaintext_auth = no auth_master_user_separator = *
mail_location = maildir:~/Maildir
# wegen webmail! mail_max_userip_connections = 1024
managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date
# Namespace für courier-compatibilitaet namespace { inbox = yes location = prefix = INBOX. separator = . type = private }
# fuer user*masteruser logins passdb { args = /usr/dovecot-2/etc/dovecot/dovecot.masteruser driver = passwd-file master = yes pass = yes }
# Authorisierung via PAM, /etc/pam.d/dovecot auth_cache_size = 64 M passdb { args = cache_key=%u driver = pam }
# User via passwd userdb { driver = passwd }
# plugin Konfiguration plugin {
# mailboxen anlegen und subscriben autocreate = Trash autocreate2 = spam autocreate3 = Sent autocreate4 = Drafts autosubscribe = Trash autosubscribe2 = spam autosubscribe3 = Sent autosubscribe4 = Drafts
# FullTextSearch
fts = squat
# Quota quota = maildir quota_rule = INBOX.Trash:storage=+2048M quota_warning = storage=99%% quota-warning 99 %u quota_warning2 = storage=95%% quota-warning 95 %u quota_warning3 = storage=90%% quota-warning 90 %u quota_warning4 = storage=85%% quota-warning 85 %u
# Sieve sieve = ~/.dovecot.sieve sieve_dir = ~/sieve
# Trash (wenn Mailbox voll, dann Trash und spam leeren)
trash = /usr/dovecot-2/etc/dovecot/dovecot-trash.conf
}
# Wir sprechen alles protocols = imap sieve pop3
# unix domain socket fuer Postfix service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } user = root }
# imap-login Prozess # high performance mode service imap-login { service_count = 0 }
# der eigentliche IMAPD service imap { drop_priv_before_exec = yes process_limit = 2048 }
# managesieve-login, wird nur genutzt von webmail aus service managesieve-login { service_count = 0 inet_listener sieve_deprecated { port = 2000 } }
# der eigentliche managesieve service managesieve { drop_priv_before_exec = yes process_limit = 128 }
# pop3-login Prozess # high performance mode service pop3-login { service_count = 0 # kein pop3, nur pop3s! inet_listener pop3 { port = 0 } }
# der eigentliche pop3 service pop3 { drop_priv_before_exec = yes process_limit = 512 }
# die ganzen SSL Zertifikate ssl_ca =
# schoene Ausgabe in ps auxwww verbose_proctitle = yes version_ignore = yes
mail_fsync = never maildir_very_dirty_syncs = yes
# globale settings, die anderen werden ja nach Protokoll individuell ergaenzt! mail_plugins = notify mail_log
# imap kann am meisten protocol imap { mail_plugins = $mail_plugins quota imap_quota trash fts fts_squat autocreate }
# pop3 hat workarounds protocol pop3 { pop3_client_workarounds = outlook-no-nuls oe-ns-eoh pop3_lock_session = yes pop3_uidl_format = %v-%u }
# LDA macht noch zusaetzlich sieve quota trash protocol lda { mail_plugins = $mail_plugins sieve quota trash postmaster_address = postmaster@charite.de quota_full_tempfail = yes syslog_facility = local4 }
# der schickt die Quota warnmails service quota-warning { executable = script /usr/local/scripts/quota-warning2 user = root unix_listener quota-warning { mode = 0666 user = vmail group = users } }
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | http://www.charite.de
On 10/7/2010 8:16 AM, Ralf Hildebrandt wrote:
Since I have these performance problems after migration to 2.0.5 I'm publishing my config for review. Can anybody spot any obvious signs of trouble? <snip> # plugin Konfiguration plugin {
# mailboxen anlegen und subscriben autocreate = Trash autocreate2 = spam autocreate3 = Sent autocreate4 = Drafts autosubscribe = Trash autosubscribe2 = spam autosubscribe3 = Sent autosubscribe4 = Drafts
# FullTextSearch fts = squat
I'm probably talking out of the wrong hole again - but have you tried removing squat from your plugin list to see if it makes a difference?
Daniel
- Daniel L. Miller dmiller@amfes.com:
# plugin Konfiguration plugin {
# mailboxen anlegen und subscriben autocreate = Trash autocreate2 = spam autocreate3 = Sent autocreate4 = Drafts autosubscribe = Trash autosubscribe2 = spam autosubscribe3 = Sent autosubscribe4 = Drafts
# FullTextSearch fts = squat
I'm probably talking out of the wrong hole again - but have you tried removing squat from your plugin list to see if it makes a difference?
I thought of that as well. I'll try that tonight :)
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | http://www.charite.de
- Daniel L. Miller dmiller@amfes.com:
# FullTextSearch fts = squat
I'm probably talking out of the wrong hole again - but have you tried removing squat from your plugin list to see if it makes a difference?
Oh my god, load is down this morning. WOW.
But what has changed? My old 1.2.x config ALSO had:
mail_plugins = quota imap_quota trash mail_log fts fts_squat zlib autocreate
and
plugin { fts = squat ...
Has the implementation of FTS changed in some way? Reading the two wikis doesn't yield any important difference. My clients/users surely have not changed!
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | http://www.charite.de
# FullTextSearch fts = squat
I'm probably talking out of the wrong hole again - but have you tried removing squat from your plugin list to see if it makes a difference?
Oh my god, load is down this morning. WOW.
But what has changed?
Perhaps: "The initial Squat index building for large mailboxes can be very CPU and memory hungry." -> http://wiki2.dovecot.org/Plugins/FTS/Squat
- ml@eulberg.name ml@eulberg.name:
Oh my god, load is down this morning. WOW.
But what has changed?
Perhaps: "The initial Squat index building for large mailboxes can be very CPU and memory hungry." -> http://wiki2.dovecot.org/Plugins/FTS/Squat
Should that info be logged? Also: since the LDA doesn't update the search index, repeated logins should cause repeated rebuilds of the index.
BTW, where's the index stored?
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | http://www.charite.de
- ml@eulberg.name ml@eulberg.name:
Oh my god, load is down this morning. WOW.
But what has changed?
Perhaps: "The initial Squat index building for large mailboxes can be very CPU and memory hungry." -> http://wiki2.dovecot.org/Plugins/FTS/Squat
Should that info be logged? Also: since the LDA doesn't update the search index, repeated logins should cause repeated rebuilds of the index.
BTW, where's the index stored?
I guess it will not be logged.
Wiki says: The Squat indexes are stored among Dovecot's other index files. They're called dovecot.index.search and dovecot.index.search.uids. It's safe to delete both of these files to trigger a rebuild.
- ml@eulberg.name ml@eulberg.name:
I guess it will not be logged.
It would probably happen too often.
Wiki says: The Squat indexes are stored among Dovecot's other index files. They're called dovecot.index.search and dovecot.index.search.uids. It's safe to delete both of these files to trigger a rebuild.
I wonder if the old files from 1.2.5 were still valid with 2.0.5...
But given identical clients and user behaviour (all I changed was the dovecot version), one would expect both 1.2.x and 2.0.x behave identical in terms of load.
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | http://www.charite.de
- Ralf Hildebrandt Ralf.Hildebrandt@charite.de:
I wonder if the old files from 1.2.5 were still valid with 2.0.5...
But given identical clients and user behaviour (all I changed was the dovecot version), one would expect both 1.2.x and 2.0.x behave identical in terms of load.
I put some more effort into the analysis of the performance problems: It's not the FTS plugin, since re-enabling doesn't change a thing.
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | http://www.charite.de
On 10/12/2010 4:25 AM, Ralf Hildebrandt wrote:
- Ralf HildebrandtRalf.Hildebrandt@charite.de:
I wonder if the old files from 1.2.5 were still valid with 2.0.5...
But given identical clients and user behaviour (all I changed was the dovecot version), one would expect both 1.2.x and 2.0.x behave identical in terms of load. I put some more effort into the analysis of the performance problems: It's not the FTS plugin, since re-enabling doesn't change a thing.
Now wait a minute! You said you found the problem and it was exactly what I suggested! I've already received my prize for most intelligent @ss in a discussion group - you can't take that away from me!
What changed? You turned off FTS, performance was better - and then it developed the problem again? You didn't specify.
Daniel
- Daniel L. Miller dmiller@amfes.com:
Now wait a minute! You said you found the problem and it was exactly what I suggested! I've already received my prize for most intelligent @ss in a discussion group - you can't take that away from me!
Yes I can. *it gone*
What changed? You turned off FTS, performance was better - and then it developed the problem again? You didn't specify.
Colleague returned from vacation today and gave some ideas. Today I turned off the "mail_log" and "notify" plugins and the load dropped considerably. Both plugins were used with default settings.
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | http://www.charite.de
- Ralf Hildebrandt Ralf.Hildebrandt@charite.de:
Colleague returned from vacation today and gave some ideas. Today I turned off the "mail_log" and "notify" plugins and the load dropped considerably. Both plugins were used with default settings.
I left all settings unchanged and examined the machine. Load stays low, so something the way how logging is done in 2.0.x must have changed.
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | http://www.charite.de
On Thu, 2010-10-14 at 13:42 +0200, Ralf Hildebrandt wrote:
- Ralf Hildebrandt Ralf.Hildebrandt@charite.de:
Colleague returned from vacation today and gave some ideas. Today I turned off the "mail_log" and "notify" plugins and the load dropped considerably. Both plugins were used with default settings.
I left all settings unchanged and examined the machine. Load stays low, so something the way how logging is done in 2.0.x must have changed.
Load as in I/O load or CPU load?.. If CPU, which process was eating it? I can't think of any good reason why mail_log would be doing anything that would cause such problems (other than maybe just logging too much stuff and causing problems because of that).
- Timo Sirainen tss@iki.fi:
On Thu, 2010-10-14 at 13:42 +0200, Ralf Hildebrandt wrote:
- Ralf Hildebrandt Ralf.Hildebrandt@charite.de:
Colleague returned from vacation today and gave some ideas. Today I turned off the "mail_log" and "notify" plugins and the load dropped considerably. Both plugins were used with default settings.
I left all settings unchanged and examined the machine. Load stays low, so something the way how logging is done in 2.0.x must have changed.
Load as in I/O load or CPU load?.. If CPU, which process was eating it?
CPU, specifically system (not user, not idle, not wait).
Process: No special process was peaking. How is the logging implemented? Each imap process is logging by itself?
I can't think of any good reason why mail_log would be doing anything that would cause such problems (other than maybe just logging too much stuff and causing problems because of that).
That kind of logging was on with 1.2.x, unless the defaults changed between 1.2.x and 2.0.x, it logged the same amount of data...
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | http://www.charite.de
On Thu, 2010-10-14 at 15:47 +0200, Ralf Hildebrandt wrote:
Colleague returned from vacation today and gave some ideas. Today I turned off the "mail_log" and "notify" plugins and the load dropped considerably. Both plugins were used with default settings. Load as in I/O load or CPU load?.. If CPU, which process was eating it?
CPU, specifically system (not user, not idle, not wait).
Process: No special process was peaking. How is the logging implemented? Each imap process is logging by itself?
dovecot-lda logs directly, with everything else it goes through "dovecot/log" process. Are you logging to syslog? Hmm. System CPU usage is about time spent in syscalls.. Is it doing too many tiny write()s or something? .. Maybe I should strace the processes and see if something's too slow there.
- Timo Sirainen tss@iki.fi:
Process: No special process was peaking. How is the logging implemented? Each imap process is logging by itself?
dovecot-lda logs directly,
OK!
with everything else it goes through "dovecot/log" process.
Interesting: 1400+ processes logging through one process... Has this changed from 1.2?
Are you logging to syslog?
yes!
Hmm. System CPU usage is about time spent in syscalls..
Exactly.
Is it doing too many tiny write()s or something?
I was thinking of this as well, because the total amount of disk IO didn't change (between 1.2 and 2.0) only the time spent in system...
.. Maybe I should strace the processes and see if something's too slow there.
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | http://www.charite.de
On Thu, 2010-10-14 at 16:47 +0200, Ralf Hildebrandt wrote:
with everything else it goes through "dovecot/log" process.
Interesting: 1400+ processes logging through one process... Has this changed from 1.2?
With v1.2 all logging also went through dovecot master process, each using their own pipe fd. With v2.0 each service has their own fd (e.g. all imap processes log via the same shared pipe fd, all pop3 processes log via another fd, etc). So it should be better in v2.0 than v1.2..
It would be possible to make all processes log directly though, by adding -L parameter to the service executables.
- Timo Sirainen tss@iki.fi:
With v1.2 all logging also went through dovecot master process,
OK. I never bothered, because I never had problems :)
I think dovecot could benefit from a diagram like the "Postfix big picture" where all daemons are listed and their relations/dependencies.
each using their own pipe fd. With v2.0 each service has their own fd (e.g. all imap processes log via the same shared pipe fd, all pop3 processes log via another fd, etc). So it should be better in v2.0 than v1.2..
Yes.
My idea is to turn back on logging and then trace the dovecot/log process using
strace -p PID -c
but not today, I'd rather do this on the weekend.
It would be possible to make all processes log directly though, by adding -L parameter to the service executables.
That would be interesting.
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | http://www.charite.de
On Fri, 2010-10-15 at 09:51 +0200, Ralf Hildebrandt wrote:
- Timo Sirainen tss@iki.fi:
With v1.2 all logging also went through dovecot master process,
OK. I never bothered, because I never had problems :)
I think dovecot could benefit from a diagram like the "Postfix big picture" where all daemons are listed and their relations/dependencies.
Yeah, I guess I should draw something some day. :)
My idea is to turn back on logging and then trace the dovecot/log process using
strace -p PID -c
-tt is also nice.
but not today, I'd rather do this on the weekend.
Each log line is written in a separate write() call. This would be a bit tricky to change to buffer the writes.. But I did get rid of some unnecessary time() + stat(/etc/localtime) calls with:
http://hg.dovecot.org/dovecot-2.0/rev/4933c3095ee2 http://hg.dovecot.org/dovecot-2.0/rev/e68366e88099 http://hg.dovecot.org/dovecot-2.0/rev/80097e5c38e9
- Timo Sirainen tss@iki.fi:
OK. I never bothered, because I never had problems :)
I think dovecot could benefit from a diagram like the "Postfix big picture" where all daemons are listed and their relations/dependencies.
Yeah, I guess I should draw something some day. :)
Postfix is a bad example. I had to redraw Wietse's diagram for a course on Postfix, since he kept adding new daemons :)
Each log line is written in a separate write() call. This would be a bit tricky to change to buffer the writes.. But I did get rid of some unnecessary time() + stat(/etc/localtime) calls with:
http://hg.dovecot.org/dovecot-2.0/rev/4933c3095ee2 http://hg.dovecot.org/dovecot-2.0/rev/e68366e88099 http://hg.dovecot.org/dovecot-2.0/rev/80097e5c38e9
I love you man :-) I'll try those.
Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | http://www.charite.de
- Timo Sirainen tss@iki.fi:
My idea is to turn back on logging and then trace the dovecot/log process using
strace -p PID -c
-tt is also nice.
I traced around a bit today, in dovecot-lda. A pattern I'm seeing quite often is this:
0.000853 open("/usr/dovecot-2/lib/dovecot/settings", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = 7
0.000463 fcntl64(7, F_GETFD) = 0x1 (flags FD_CLOEXEC)
0.000555 getdents64(7, /* 8 entries */, 32768) = 360
0.023219 getdents64(7, /* 0 entries */, 32768) = 0
0.001545 close(7) = 0two consecutive getdents64 calls on the same FD, one fast, one slow. WTF?
# ll /usr/dovecot-2/lib/dovecot/settings total 88 -rw-r--r-- 1 root root 24766 Okt 21 21:27 libmanagesieve_login_settings.a -rwxr-xr-x 1 root root 1107 Okt 21 21:27 libmanagesieve_login_settings.la -rwxr-xr-x 1 root root 23291 Okt 21 21:27 libmanagesieve_login_settings.so -rw-r--r-- 1 root root 11630 Okt 21 21:27 libmanagesieve_settings.a -rwxr-xr-x 1 root root 1065 Okt 21 21:27 libmanagesieve_settings.la -rwxr-xr-x 1 root root 12858 Okt 21 21:27 libmanagesieve_settings.so
And this is happening once more in the same trace:
0.002036 open("/usr/dovecot-2/lib/dovecot", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = 8
0.000468 fcntl64(8, F_GETFD) = 0x1 (flags FD_CLOEXEC)
0.003281 getdents64(8, /* 85 entries */, 32768) = 3736
0.150525 getdents64(8, /* 0 entries */, 32768) = 0
0.000490 close(8) = 0on fast call, one slow call.
Sorting all traces by execution time:
% cat dovecot-lda.* | sort -n -k1 |less 0.094459 fstat64(8, {st_mode=S_IFREG|0755, st_size=1870288, ...}) = 0 0.095051 brk(0x8d73000) = 0x8d73000 0.098191 getdents64(8, /* 0 entries */, 32768) = 0 0.098912 time(NULL) = 1288084930 0.103059 getdents64(8, /* 0 entries */, 32768) = 0 0.110251 getdents64(8, /* 0 entries */, 32768) = 0 0.115710 getdents64(8, /* 0 entries */, 32768) = 0 0.117688 time(NULL) = 1288084937 0.128360 getdents64(8, /* 0 entries */, 32768) = 0 0.130091 mprotect(0xb75b4000, 4096, PROT_READ) = 0 0.131681 rt_sigaction(SIGALRM, {0xb76cb490, [], SA_SIGINFO}, NULL, 8) = 0 0.133327 getdents64(8, /* 0 entries */, 32768) = 0 0.134047 stat64("/home/c/o/cousername/Maildir/dovecot-uidlist", {st_mode=S_IFREG|0600, st_size=1130, ...}) = 0 0.135077 rt_sigaction(SIGPIPE, {SIG_IGN, [], SA_RESTART}, NULL, 8) = 0 0.150525 getdents64(8, /* 0 entries */, 32768) = 0 0.153457 open("/home/f/k/fkusername/Maildir/dovecot-uidlist", O_RDWR|O_LARGEFILE) = 11 0.170190 getdents64(8, /* 0 entries */, 32768) = 0 0.188536 getdents64(8, /* 0 entries */, 32768) = 0 0.193665 umask(077) = 0177 0.270003 getdents64(8, /* 0 entries */, 32768) = 0
Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | http://www.charite.de
- Ralf Hildebrandt Ralf.Hildebrandt@charite.de:
I traced around a bit today, in dovecot-lda. A pattern I'm seeing quite often is this:
I also actived system accounting and found this:
# sa -l |egrep "(deliver|dovecot-lda)"
17365 277.05re 5.98u 159.69s 0avio 1489k dovecot-lda
17951 179.22re 2.04u 74.71s 0avio 1360k deliveron the morning I was running "dovecot-lda" for local delivery, but then switched to the old 1.2.x deliver for performance reasons.
Both are using sieve etc. (I converted the config when switching to 2.0.x). It seems the old deliver is using less real time, and only 50% system time.
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | http://www.charite.de
On Tue, 2010-10-26 at 15:46 +0200, Ralf Hildebrandt wrote:
on the morning I was running "dovecot-lda" for local delivery, but then switched to the old 1.2.x deliver for performance reasons.
If the slowness is caused by config parsing, that's actually easy to avoid:
service config { unix_listener config { mode = 0666 } }
(Of course assuming there isn't anything secret in the config.)
Now dovecot-lda and others will read the config from the socket rather than executing doveconf.
- Timo Sirainen tss@iki.fi:
On Tue, 2010-10-26 at 15:46 +0200, Ralf Hildebrandt wrote:
on the morning I was running "dovecot-lda" for local delivery, but then switched to the old 1.2.x deliver for performance reasons.
If the slowness is caused by config parsing, that's actually easy to avoid:
service config { unix_listener config { mode = 0666 } }
(Of course assuming there isn't anything secret in the config.)
Now dovecot-lda and others will read the config from the socket rather than executing doveconf.
I will test this immediately!!
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | http://www.charite.de
On 3.11.2010, at 7.51, Ralf Hildebrandt wrote:
service config { unix_listener config { mode = 0666 } }
Now dovecot-lda and others will read the config from the socket rather than executing doveconf.
I will test this immediately!!
Forgot to mention: This requires that base_dir hasn't been changed from the built-in default. But that can also be worked around with symlinks, e.g.:
ln -s /var/run/dovecot /usr/local/var/run/dovecot
- Timo Sirainen tss@iki.fi:
On 3.11.2010, at 7.51, Ralf Hildebrandt wrote:
service config { unix_listener config { mode = 0666 } }
Now dovecot-lda and others will read the config from the socket rather than executing doveconf.
I will test this immediately!!
Forgot to mention: This requires that base_dir hasn't been changed from the built-in default. But that can also be worked around with symlinks, e.g.:
ln -s /var/run/dovecot /usr/local/var/run/dovecot
Well, dovecot itself seems to be working. Would it log problems accessing base_dir?
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | http://www.charite.de
On 3.11.2010, at 10.32, Ralf Hildebrandt wrote:
Forgot to mention: This requires that base_dir hasn't been changed from the built-in default. But that can also be worked around with symlinks, e.g.:
ln -s /var/run/dovecot /usr/local/var/run/dovecot
Well, dovecot itself seems to be working. Would it log problems accessing base_dir?
No. If it can't read the config socket (which it can't by default), it silently fallbacks to executing doveconf.
- Timo Sirainen tss@iki.fi:
No. If it can't read the config socket (which it can't by default), it silently fallbacks to executing doveconf.
:)
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | http://www.charite.de
- Timo Sirainen tss@iki.fi:
On 3.11.2010, at 7.51, Ralf Hildebrandt wrote:
service config { unix_listener config { mode = 0666 } }
Now dovecot-lda and others will read the config from the socket rather than executing doveconf.
I will test this immediately!!
Forgot to mention: This requires that base_dir hasn't been changed from the built-in default. But that can also be worked around with symlinks, e.g.:
ln -s /var/run/dovecot /usr/local/var/run/dovecot
ln -s /usr/dovecot-2/var/run/dovecot /var/run/dovecot
(since /var/run/dovecot didn't exist here, I have everything below --prefix=/usr/dovecot-2)
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | http://www.charite.de
On 3.11.2010, at 10.34, Ralf Hildebrandt wrote:
ln -s /var/run/dovecot /usr/local/var/run/dovecot
ln -s /usr/dovecot-2/var/run/dovecot /var/run/dovecot
That symlink probably gets deleted at system bootup.
(since /var/run/dovecot didn't exist here, I have everything below --prefix=/usr/dovecot-2)
If you haven't changed base_dir, then the symlink isn't necessary. In the example above I was assuming base_dir had been changed to /var/run/dovecot from the built-in /usr/local/var/run/dovecot.
- Timo Sirainen tss@iki.fi:
On 3.11.2010, at 10.34, Ralf Hildebrandt wrote:
ln -s /var/run/dovecot /usr/local/var/run/dovecot
ln -s /usr/dovecot-2/var/run/dovecot /var/run/dovecot
That symlink probably gets deleted at system bootup.
(since /var/run/dovecot didn't exist here, I have everything below --prefix=/usr/dovecot-2)
If you haven't changed base_dir, then the symlink isn't necessary. In the example above I was assuming base_dir had been changed to /var/run/dovecot from the built-in /usr/local/var/run/dovecot.
I left base_dir at it's default, which is $prefix/var/run/dovecot So, no harm, no foul
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | http://www.charite.de
On Tue, 2010-10-26 at 11:44 +0200, Ralf Hildebrandt wrote:
I traced around a bit today, in dovecot-lda. A pattern I'm seeing quite often is this:
0.000853 open("/usr/dovecot-2/lib/dovecot/settings", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = 7 0.000463 fcntl64(7, F_GETFD) = 0x1 (flags FD_CLOEXEC) 0.000555 getdents64(7, /* 8 entries */, 32768) = 360 0.023219 getdents64(7, /* 0 entries */, 32768) = 0 0.001545 close(7) = 0two consecutive getdents64 calls on the same FD, one fast, one slow. WTF?
I don't know.. Is that directory on local filesystem? Are your mails on the same disk?
And this is happening once more in the same trace:
0.002036 open("/usr/dovecot-2/lib/dovecot", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = 8 0.000468 fcntl64(8, F_GETFD) = 0x1 (flags FD_CLOEXEC) 0.003281 getdents64(8, /* 85 entries */, 32768) = 3736 0.150525 getdents64(8, /* 0 entries */, 32768) = 0 0.000490 close(8) = 0on fast call, one slow call.
v1.x was also doing this. It's for loading plugins.
0.098912 time(NULL) = 1288084930
time(NULL) should always be a really fast operation. On my somewhat old machine it takes 0.000016 seconds. So if it's taking this long, it seems that the system in general is under heavy load and the individual system call times don't really tell anything.
0.193665 umask(077) = 0177
This should be about 0.000016 seconds too.
- Timo Sirainen tss@iki.fi:
On Tue, 2010-10-26 at 11:44 +0200, Ralf Hildebrandt wrote:
I traced around a bit today, in dovecot-lda. A pattern I'm seeing quite often is this:
0.000853 open("/usr/dovecot-2/lib/dovecot/settings", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = 7 0.000463 fcntl64(7, F_GETFD) = 0x1 (flags FD_CLOEXEC) 0.000555 getdents64(7, /* 8 entries */, 32768) = 360 0.023219 getdents64(7, /* 0 entries */, 32768) = 0 0.001545 close(7) = 0two consecutive getdents64 calls on the same FD, one fast, one slow. WTF?
I don't know.. Is that directory on local filesystem?
Yes.
Are your mails on the same disk?
No!
We have: /dev/sda1 on / type ext3 (rw,errors=panic) /dev/mapper/volg1-logv1 on /home type ext4 (rw,noexec,nodev,noatime,errors=panic,data=writeback)
time(NULL) should always be a really fast operation. On my somewhat old machine it takes 0.000016 seconds. So if it's taking this long, it seems that the system in general is under heavy load and the individual system call times don't really tell anything.
Yes, but where does the heavy load come from :)
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | http://www.charite.de
On 8.10.2010, at 12.55, Ralf Hildebrandt wrote:
Has the implementation of FTS changed in some way? Reading the two wikis doesn't yield any important difference. My clients/users surely have not changed!
No.. I haven't touched Squat for a long time. Then again, I haven't really tried it much either for a long time. I guess I'll have to try.
On 10/7/2010 8:16 AM, Ralf Hildebrandt wrote:
Since I have these performance problems after migration to 2.0.5 I'm publishing my config for review. Can anybody spot any obvious signs of trouble? <snip> # plugin Konfiguration plugin {
# mailboxen anlegen und subscriben autocreate = Trash autocreate2 = spam autocreate3 = Sent autocreate4 = Drafts autosubscribe = Trash autosubscribe2 = spam autosubscribe3 = Sent autosubscribe4 = Drafts
# FullTextSearch fts = squat
I'm probably talking out of the wrong hole again - but have you tried removing squat from your plugin list to see if it makes a difference? Daniel Miller,
Please do not get all defensive when someone asks why they're having performance issues. You act like he said "Because of dovecot, i'm having POP3 or IMAP performance problems". He only said he noticed a change since going to 2.0.5. You could have suggested to try disabling all the plugins and then enabling each one until he notices a huge performance decrease, using whatever he was using to benchmark, instead of ignorantly implying he should have already known to do so.
Thanks for contributing,
Jerrale G. SC Senior Admin
- Jerrale G jerralegayle@sheltoncomputers.com:
Please do not get all defensive when someone asks why they're having performance issues. You act like he said "Because of dovecot, i'm having POP3 or IMAP performance problems". He only said he noticed a change since going to 2.0.5.
Indeed.
You could have suggested to try disabling all the plugins and then enabling each one until he notices a huge performance decrease, using whatever he was using to benchmark, instead of ignorantly implying he should have already known to do so.
The FTS plugin was the culprit.
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | http://www.charite.de
On 10/10/2010 2:57 PM, Jerrale G wrote:
I'm probably talking out of the wrong hole again - but have you tried removing squat from your plugin list to see if it makes a difference?
Daniel Miller,
Please do not get all defensive when someone asks why they're having performance issues. You act like he said "Because of dovecot, i'm having POP3 or IMAP performance problems". He only said he noticed a change since going to 2.0.5. You could have suggested to try disabling all the plugins and then enabling each one until he notices a huge performance decrease, using whatever he was using to benchmark, instead of ignorantly implying he should have already known to do so.
I find this an all too frequent occurrence in e-mail correspondence - that humor doesn't translate well. My statement had NOTHING to do with the original author - and everything to do with the fact that *I* am a Linux novice in general, a noob e-mail admin in particular - and I was being *SELF*-deprecating.
I would go on - but have no wish to start a flame war. Suffice it to say if offense was taken, by anyone, then please accept my most sincere & humble apologies. No offense was intended. I will say, however, that I've found most people who feel slighted on discussion groups are quite capable of voicing their issues themselves - a third party "defender" is rarely necessary and usually creates more problems than were ever there originally. I'm quite certain Mr. Hildebrandt either took no offense because he recognized none was intended - and/or because he knows his own worth and disregards my opinion as insignficant - both of which are equally valid.
Daniel
- Daniel L. Miller dmiller@amfes.com:
I find this an all too frequent occurrence in e-mail correspondence
- that humor doesn't translate well. My statement had NOTHING to do with the original author - and everything to do with the fact that *I* am a Linux novice in general, a noob e-mail admin in particular - and I was being *SELF*-deprecating.
Hey, no need to. Your idea was good, but unlikely. Yet it worked, to my great amazement... :)
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | http://www.charite.de
On Sun, 2010-10-10 at 17:57 -0400, Jerrale G wrote:
On 10/7/2010 8:16 AM, Ralf Hildebrandt wrote:
fts = squat
I'm probably talking out of the wrong hole again - but have you tried removing squat from your plugin list to see if it makes a difference? Daniel Miller,
Please do not get all defensive when someone asks why they're having performance issues. You act like he said "Because of dovecot, i'm having POP3 or IMAP performance problems". He only said he noticed a change since going to 2.0.5. You could have suggested to try disabling all the plugins and then enabling each one until he notices a huge performance decrease, using whatever he was using to benchmark, instead of ignorantly implying he should have already known to do so.
I don't think he got all defensive, I don't believe he implied anything, and I don't think his reply was in any wrong. As I understand it, it was a gut feeling of his, and turned out to be the issue indeed. (As confirmed a while ago.)
Granted, I mostly lurk here only, and deleted most of this thread already, but from memory...
Jerrale, dude, you got all defensive. I don't know who is the author of the squat fts, and I am too lazy to look it up. But unless you are, I don't really understand your screaming ATTN attribution on-list in the first place.
Anyway, I don't feel like a flame war. I will try not to contribute to this thread any further. However, David, err, Daniel -- don't feel bad. I did not understand your comment offensive in any way, and I believe the guys originally involved did neither.
This was a strange (sub)thread to read indeed. Back to my recording.
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i
participants (6)
-
Daniel L. Miller
-
Jerrale G
-
Karsten Bräckelmann
-
ml＠eulberg.name
-
Ralf Hildebrandt
-
Timo Sirainen