September 2010 - dovecot

[Dovecot] Maybe wrong quota recalculation behaviour on overquota delivery try
by Luca Palazzo 03 Sep '10

03 Sep '10

Hi all, we are experencing some, may be strange, slowness in our dovecot/postfix mail system. We have three dovecot 2.0.1 server with LMTP, POP3 and IMAP enabled. Three shared OCFS2 filesystem are used for mail storage. We have too a load balancer based on dovecot director in front of these three server. Our SMTP cluster receives mail and send by LMTP to load balancer. Everything work flowless, but sometime, under heavy load, three server slow down a lot. Our mail gateway logs a lot of: status=deferred (host 10.194.1.20[10.194.1.20] said: 451 4.4.0 Remote server not answering (DATA output timeout) IP 10.194.1.20 is dovecot director. Looking at mbox servers log nothing seem to happen, but stracing lmtp process we can see stat system calls to all Maildir files, as if dovecot was recalculating quota and regenerating maildirsize file. It happens always when LMTP tries to deliver a mail to an overquota account. Is this possible and right? How can we look deeper in system to have more information? Best regards Luca PS: dovecot configuration for maildir servers is: # 2.0.1: /etc/dovecot/dovecot.conf # OS: Linux 2.6.26-2-686-bigmem i686 Debian 5.0.5 auth_master_user_separator = * auth_verbose = yes disable_plaintext_auth = no mail_location = maildir:~/Maildir mail_plugins = $mail_plugins notify mail_log quota managesieve_notify_capability = mailto managesieve_sieve_capability = comparator-i;octet comparator-i;ascii-casemap fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date spamtest spamtestplus virustest passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid box msgid from size quota = maildir:User quota quota_warning = storage=95%% quota-warning.sh 95 %u quota_warning2 = storage=80%% quota-warning.sh 80 %u sieve = ~/.dovecot.sieve sieve_dir = ~/sieve sieve_global_dir = /etc/dovecot/sieve/global/ sieve_global_path = /etc/dovecot/sieve/default.sieve } protocols = imap pop3 lmtp sieve service auth { unix_listener auth-userdb { group = dovecot mode = 0600 user = dovecot } } service director { inet_listener { port = 8100 } } service lmtp { inet_listener lmtp { address = 10.194.1.31 127.0.0.1 ::1 port = 24 } } service managesieve-login { inet_listener sieve { address = 10.194.1.31 127.0.0.1 ::1 port = 4190 } process_min_avail = 0 service_count = 1 } service quota-warning { executable = script /virtual/shared/quota-warning.sh unix_listener quota-warning { user = vmail } user = dovecot } ssl_cert = </etc/ssl/certs/dovecot.pem ssl_key = </etc/ssl/private/dovecot.pem userdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } verbose_proctitle = yes protocol lmtp { mail_plugins = $mail_plugins quota sieve } protocol lda { mail_plugins = $mail_plugins quota } protocol imap { mail_plugins = $mail_plugins quota imap_quota } protocol pop3 { mail_plugins = $mail_plugins quota }

2 3

[Dovecot] minor director issues
by Cor Bosman 03 Sep '10

03 Sep '10

I noticed 2 minor things with the director. 1) if I start 2 directors, but in 1 of them I dont add the other as a director_servers, they both start spamming the logfiles really really fast. It seems there is no backoff mechanism in the protocol between directors. 2) user mail server ip expire time cor 194.109.26.171 2010-09-02 14:32:13 <unknown> 194.109.26.172 2010-09-02 14:33:13 Why does it say 'unknown'? That session is active, im manually logged in with telnet, issued a login and a select inbox. The first one is a normal mail client login. Regards, Cor

2 6

[Dovecot] safe_memset
by Len7hir 03 Sep '10

03 Sep '10

samples % lib function 4 0.0525 libdovecot.so.0.0.0 sha1_result_libmysqlcli(..) 7 0.0919 libdovecot.so.0.0.0 io_loop_handler_run 36 0.4729 libdovecot.so.0.0.0 imap_parser_reset 227 2.9817 libdovecot.so.0.0.0 pool_alloconly_unref 7339 96.4009 libdovecot.so.0.0.0 pool_alloconly_clear 7710 3.2463 libdovecot.so.0.0.0 safe_memset 7710 100.000 libdovecot.so.0.0.0 safe_memset [self] ------- 1 0.0148 libdovecot.so.0.0.0 i_stream_limit_read 2 0.0296 libdovecot.so.0.0.0 i_stream_header_filt(..) 6 0.0888 libdovecot.so.0.0.0 imap_parser_destroy 8 0.1184 libdovecot.so.0.0.0 settings_parser_deinit 230 3.4039 libdovecot.so.0.0.0 imap_parser_reset 6510 96.3445 libdovecot.so.0.0.0 pool_alloconly_unref 598 0.2518 libdovecot.so.0.0.0 pool_alloconly_clear 7339 89.8286 libdovecot.so.0.0.0 safe_memset 598 7.3195 libdovecot.so.0.0.0 pool_alloconly_clear [self] 232 2.8397 libc-2.3.6.so (no symbols) 1 0.0122 libdovecot.so.0.0.0 body ------- 4 0.0525 libdovecot.so.0.0.0 sha1_result_libmysqlclient_craps_all_over 7 0.0919 libdovecot.so.0.0.0 io_loop_handler_run 36 0.4729 libdovecot.so.0.0.0 imap_parser_reset 227 2.9817 libdovecot.so.0.0.0 pool_alloconly_unref 7339 96.4009 libdovecot.so.0.0.0 pool_alloconly_clear 7710 3.2463 libdovecot.so.0.0.0 safe_memset 7710 100.000 libdovecot.so.0.0.0 safe_memset [self] Mostly safe_memset is called from pool_alloconly_clear. I attach result from oprofile - functions (with call tree) that have more than 2% of all oprofile samples in libdovecot.so. p_strdup is on the list, but it is lower than before last optimisations. As You see there is no other optimised functions :) -- Len7hir

2 1

[Dovecot] apply sieve filter to old mails
by Giovanni Mancuso 03 Sep '10

03 Sep '10

Hi, i install dovecot 2.0.1 with managesieve support in my test machine. If i create a sieve filter and i send the mail to my user thi filter is applied. Is there a way to apply this filter to old mails too? Thanks

1 0

[Dovecot] Suggested location for single-dbox
by William Blunn 03 Sep '10

03 Sep '10

I've updated http://wiki2.dovecot.org/MailLocation regarding a typical mail_location entry for dbox. Previously it said: mail_location = dbox:~/dbox but of course "dbox" is now the old name for "sdbox", so the example should be updated to say "sdbox" and there might/ought also to be an example for "mdbox". In http://wiki2.dovecot.org/MailboxFormat/dbox the example mail_location for mdbox is given as "mdbox:~/mdbox", so I have followed that for "MailLocation". So I have changed it to say # single-dbox mail_location = sdbox:~/dbox or: # multi-dbox mail_location = mdbox:~/mdbox I didn't know whether to change the directory for sdbox to "~/sdbox" or to leave it as it is. For the time being I have left it at "~/dbox" --- following the example given should allow new installations of Dovecot 2 to be able to read any existing single-dbox data which used the historically-suggested "~/dbox" location, whilst at the same time avoiding using the old mailbox format name of "dbox". Bill

2 1

Re: [Dovecot] Some doubts about quota
by Andre 03 Sep '10

03 Sep '10

Il giorno 03/set/2010, alle ore 10.37, Andre ha scritto: > Hi to all, > we are using quota to collect statistics about usage of mailboxes (thanks to the one who implemented :noenforcing) and we are using “maildir” backend, but we are thinking to migrate to *dbox (as soon as “per-user \Seen flag” would be implemented) and we should need a faster way to collect statistics, so I’ve tried to use Postgresql as a backend. I’ve follwed literally instructions from wiki2 (and part from wiki), but quota is not automatically updated as soon as a messages is delivered to user or deleted from mailbox, we need to perform a “doveadm quota recalc -A” to have it updated. Is it the normal behavior of quota plugin or do I missed something? > > TIA > Andrea Mistrali > am(a)am.cx Solved! I’ve had disabled quota plugin in LDA & LMTP. Now it works, sorry to have bothered you all and wasted your bandwidth Andrea Mistrali am(a)am.cx

1 0

[Dovecot] Some doubts about quota
by Andre 03 Sep '10

03 Sep '10

Hi to all, we are using quota to collect statistics about usage of mailboxes (thanks to the one who implemented :noenforcing) and we are using “maildir” backend, but we are thinking to migrate to *dbox (as soon as “per-user \Seen flag” would be implemented) and we should need a faster way to collect statistics, so I’ve tried to use Postgresql as a backend. I’ve follwed literally instructions from wiki2 (and part from wiki), but quota is not automatically updated as soon as a messages is delivered to user or deleted from mailbox, we need to perform a “doveadm quota recalc -A” to have it updated. Is it the normal behavior of quota plugin or do I missed something? TIA Andrea Mistrali am(a)am.cx

1 0

[Dovecot] bogus fts percentage
by Mike Abbott 03 Sep '10

03 Sep '10

Sometimes dovecot-2.0 sends this untagged response to clients: * OK Indexed -2147483648% of the mailbox, ETA 0:00 I imagine there's a bug in there somewhere.

3 3

[Dovecot] LMTP plugin with MySQL query
by Charly 03 Sep '10

03 Sep '10

Hello, I want to create a plugin to dovecot 2.0.1, to call a on the deliver action. My plugin work even if I include other libs like mysql or wathever I tried to use the "sql-api.h" (lib-sql), but at runtime, I have : "undefined symbol: sql_init", "Fatal: Couldn't load required plugins" Is it possible to use lib-sql api or mysql libs in a plugin ? I've tried with lda or lmtp plugin with the same result... Any idea ? Charly.

2 2

[Dovecot] My "%%" variables in the namespace settings do not appear to be expanding.
by Kenneth Cope 02 Sep '10

02 Sep '10

Hello, I've been using the dovecot 1.2.x branch server software for a while now. With the advent of the 2.0.x branch going 'stable' I decided to attempt a migration and make use of some of the features that I've not needed before. One of these is the shared folder/namespace feature. No luck. I've hit a wall and can not determine the problem. At the moment it looks like a bug to me, but it may well be that I'm missing some related configuration, or file permission. I can not get the %% variable expansion in a namespace to work. What I end up with, on an IMAP connection is a directory named, for instance, '%h'. The same goes for any of the %% variables used at this point. My current set-up is 'not' the same as the current setup and I'm using virtual users that are not 'live' on the current system. I would really appreciate it if someone could cast an eye over my config and let me know if I've missed something obvious. Thanks. Kenneth Cope kenneth(a)CopeOnThe.net ---- I'm using a dovecot compiled with the following options: ./configure --prefix=/usr/local/dovecot-2.0.1 \ --enable-dependency-tracking INSTALL_DATA="install -c -p -m644" \ --docdir=/usr/local/dovecot-2.0.1/docs/ --disable-static \ --disable-rpath --with-nss --with-zlib --with-bzlib --with-libcap \ --with-ssl=openssl --with-ssldir=/usr/local/dovecot-2.0.1/etc/ssl \ --with-docs \ ---- The 'test' configuration is as follows: # > /usr/local/dovecot-2.0.1/bin/doveconf -n # 2.0.1: /usr/local/dovecot-2.0.1/etc/dovecot/dovecot.conf # OS: Linux 2.6.9-89.0.26.ELsmp i686 Red Hat Enterprise Linux ES release 4 (Nahant Update 8) auth_debug = yes auth_debug_passwords = yes auth_mechanisms = plain login cram-md5 digest-md5 apop auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_ @& auth_username_format = %Lu auth_verbose = yes default_client_limit = 10000 default_internal_user = mail default_login_user = dovecot disable_plaintext_auth = no first_valid_gid = 12 first_valid_uid = 8 last_valid_gid = 12 last_valid_uid = 8 log_path = /usr/local/dovecot-2.0.1/dovecot.log login_log_format_elements = [%p:%e] %Us(%u) method=%m rip=%r:%b lip=%l:%a %c %k mail_debug = yes mail_location = maildir:%h/ mail_log_prefix = [%p] %Us(%u): mail_max_keyword_length = 30 mail_plugins = quota imap_quota zlib acl mailbox_idle_check_interval = 120 s namespace { inbox = yes location = prefix = separator = . type = private } namespace { hidden = yes inbox = no list = no location = prefix = INBOX separator = . type = private } namespace { list = children location = maildir:%%h:LAYOUT=maildir++:INDEX=~/shared.%Dn_%Dd prefix = shared/%n@%d. separator = . subscriptions = no type = shared } namespace { location = prefix = } passdb { args = scheme=CRYPT username_format=%u /usr/local/dovecot-2.0.1/auth/passwd.dovecot driver = passwd-file } plugin { acl = vfile:/usr/local/dovecot-2.0.1/etc/dovecot/global-acls:cache_secs=300 acl_shared_dict = file:/usr/local/dovecot-2.0.1/var/lib/dovecot/shared-mailboxes mail_log_events = all mail_log_fields = uid, box, msgid, from, subject, size, vsize, flags mail_log_group_events = yes quota = maildir:User quota quota_rule = ?:storage=400M quota_rule2 = Trash:storage=+10% zlib_save = gz zlib_save_level = 9 } service auth-worker { user = $default_internal_user } service auth { unix_listener auth-userdb { group = dovecot mode = 0664 user = mail } user = $default_internal_user } service imap-login { inet_listener imap { port = 9143 } inet_listener imaps { port = 9993 ssl = yes } process_min_avail = 10 service_count = 1 } service imap { process_limit = 1024 vsz_limit = 268435456 } service lmtp { unix_listener lmtp { mode = 0666 } } service pop3-login { inet_listener pop3 { port = 9110 } inet_listener pop3s { port = 9995 ssl = yes } } service pop3 { process_limit = 1024 } ssl_cert = </etc/ssl/mail.cert ssl_key = </etc/ssl/mail.cert userdb { args = username_format=%u /usr/local/dovecot-2.0.1/auth/passwd.dovecot driver = passwd-file } verbose_ssl = yes protocol imap { imap_client_workarounds = delay-newmail imap_id_log = * imap_idle_notify_interval = 120 s imap_logout_format = bytes=%i/%o imap_max_line_length = 65536 mail_max_userip_connections = 40 mail_plugins = $mail_plugins imap_quota zlib imap_zlib imap_acl } protocol pop3 { mail_max_userip_connections = 40 mail_plugins = $mail_plugins zlib pop3_client_workarounds = outlook-no-nuls oe-ns-eoh pop3_enable_last = no pop3_lock_session = no pop3_logout_format = top=%t/%p, retr=%r/%b, del=%d/%m, size=%s bytes=%i/%o pop3_reuse_xuidl = yes pop3_save_uidl = yes pop3_uidl_format = %Mf } ---- The contents of .../var/run/dovecot # > ls -lA total 12 srw------- 1 root root 0 Aug 31 11:25 anvil srw------- 1 root root 0 Aug 31 11:25 anvil-auth-penalty srw------- 1 root root 0 Aug 31 11:25 auth-client srw------- 1 mail root 0 Aug 31 11:25 auth-login srw------- 1 root root 0 Aug 31 11:25 auth-master srw-rw-r-- 1 mail dovecot 0 Aug 31 11:25 auth-userdb srw------- 1 mail root 0 Aug 31 11:25 auth-worker srw------- 1 root root 0 Aug 31 11:25 config srw------- 1 root root 0 Aug 31 11:25 dict srw------- 1 root root 0 Aug 31 11:25 director-admin srw------- 1 root root 0 Aug 31 11:25 director-userdb srw-rw-rw- 1 root root 0 Aug 31 11:25 dns-client srw------- 1 root root 0 Aug 31 11:25 doveadm-server lrwxrwxrwx 1 root root 49 Aug 31 11:25 dovecot.conf -> /usr/local/dovecot-2.0.1/etc/dovecot/dovecot.conf drwxr-xr-x 2 root root 4096 Aug 31 11:25 empty srw-rw-rw- 1 root root 0 Aug 31 11:25 lmtp drwxr-x--- 2 root dovecot 4096 Aug 31 11:25 login -rw------- 1 root root 6 Aug 31 11:25 master.pid

3 10