February 2016 - dovecot

Load increase and imap processes in state D
by Alessio Cecchi 09 Feb '16

09 Feb '16

Hi, in recent weeks I have notice a sudden increase in the load average of our Dovecot (2.2.19) IMAP/POP servers (and consequently in the NFS storage). After an investigation I found some dovecot/imap processes in D state and running UID FETCH or UID SORT from Webmail (Roundcube) without that user was actually more connected to webmail. Some minutes after the dovecot/imap processes stop to run FETCH/SORT but in "ps aux" I can still see it. If the user try to login in Webmail again the same process start again to run FETCH/SORT (and it consumes a lot of I/O resources) but the webmail does not load any message and return a "FETCH 1: UID INTERNALDATE BODY.PEEK ..." error. The only way to restore the webmail access is to kill the related dovecot/imap process (or with doveadm kick username). I noticed that these processes in D state had started many hours before. Example, now are 17:30 (important, see the time of the dovecot/imap processes) the user alessio@skye try to login in webmail without success and close the webmail, processes are still running: # doveadm who | grep alessio@skye username # proto (pids) (ips) alessio@skye 3 imap (10440 4624) (192.168.5.215) # ps aux | grep SORT vmail 4624 0.0 0.0 108792 16048 ? D 13:32 0:01 dovecot/imap [alessio@skye 192.168.5.215 UID SORT] vmail 10440 0.0 0.1 108520 17832 ? D 12:32 0:02 dovecot/imap [alessio@skye 192.168.5.215 UID SORT] # doveadm kick alessio@skye kicked connections from the following users: alessio@skye Processes are now died, and user can login in Webmail easily. Can be a "reusing processes" issue since I have set service_count=100 for imap and pop3 services or there may be other problems related to my configuration? Thanks my configuration: # 2.2.19.1 (16dc61d3dab3): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.9 (5d26f470efe6) # OS: Linux 2.6.32-573.12.1.el6.x86_64 x86_64 CentOS release 6.7 (Final) auth_cache_negative_ttl = 5 mins auth_cache_size = 10 M auth_cache_ttl = 20 mins auth_master_user_separator = * auth_worker_max_count = 50 deliver_log_format = msgid=%m, from=%f, subject="%s": %$ dict { acl = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext expire = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext sqlquota = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext } disable_plaintext_auth = no first_valid_gid = 89 first_valid_uid = 89 imap_client_workarounds = delay-newmail tb-extra-mailbox-sep tb-lsub-flags imap_idle_notify_interval = 29 mins imap_logout_format = in=%i out=%o session=<%{session}> last_valid_gid = 89 last_valid_uid = 89 lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes listen = 10.96.3.156 login_trusted_networks = 192.168.5.0/24 mail_fsync = always mail_location = maildir:~/Maildir mail_plugins = quota acl expire fts fts_solr zlib maildir_very_dirty_syncs = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate vnd.dovecot.duplicate mmap_disable = yes namespace { list = children location = maildir:%%h/Maildir:INDEX=~/Maildir/shared/%%u prefix = shared/%%n/ separator = / subscriptions = no type = shared } namespace inbox { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Spam { auto = subscribe special_use = \Junk } mailbox Trash { auto = subscribe special_use = \Trash } prefix = separator = / } passdb { args = /etc/dovecot/dovecot-deny-sql.conf.ext deny = yes driver = sql } passdb { args = /etc/dovecot/extra/master-users driver = passwd-file master = yes pass = yes } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { acl = vfile acl_shared_dict = proxy::acl antispam_backend = mailtrain antispam_mail_notspam = --ham antispam_mail_sendmail = /usr/bin/sa-learn antispam_mail_spam = --spam antispam_spam = Spam antispam_trash = Trash expire = Trash expire2 = Spam expire_dict = proxy::expire fts = solr fts_solr = url=http://10.0.0.5:8983/solr/ quota = maildir:UserQuota quota2 = dict:Quota Usage::noenforcing:proxy::sqlquota quota_grace = 10M quota_rule2 = Trash:storage=+100M quota_warning = storage=95%% quota-warning 95 %u quota_warning2 = storage=80%% quota-warning 80 %u sieve = ~/.dovecot.sieve sieve_before = /etc/dovecot/sieve/before.sieve sieve_dir = ~/sieve sieve_extensions = +vnd.dovecot.duplicate -vacation zlib_save = gz zlib_save_level = 6 } pop3_client_workarounds = outlook-no-nuls oe-ns-eoh pop3_logout_format = top=%t/%p, retr=%r/%b, del=%d/%m, size=%s, bytes=%i/%o, session=<%{session}> protocols = imap pop3 sieve sendmail_path = /bin/sendmail service auth { client_limit = 6524 unix_listener auth-userdb { group = vmail mode = 0660 user = vmail } } service dict { process_limit = 500 unix_listener dict { group = vmail mode = 0660 user = vmail } } service imap-login { process_min_avail = 4 service_count = 0 } service imap-postlogin { executable = script-login /etc/dovecot/imap-postlogin.sh unix_listener imap-postlogin { group = vmail mode = 0660 user = vmail } user = vmail } service imap { executable = imap imap-postlogin process_limit = 5000 service_count = 100 vsz_limit = 384 M } service managesieve-login { inet_listener sieve { port = 4190 } } service pop3-login { process_min_avail = 4 service_count = 0 } service pop3-postlogin { executable = script-login /etc/dovecot/pop3-postlogin.sh unix_listener pop3-postlogin { group = vmail mode = 0660 user = vmail } user = vmail } service pop3 { executable = pop3 pop3-postlogin process_limit = 1024 service_count = 100 } service quota-warning { executable = script /etc/dovecot/quota-warning.sh unix_listener quota-warning { user = vmail } user = vmail } service stats { fifo_listener stats-mail { group = vmail mode = 0660 user = vmail } } ssl_cert = </etc/dovecot/wildcard.pem ssl_key = </etc/dovecot/wildcard.key ssl_protocols = !SSLv2 !SSLv3 submission_host = 127.0.0.1 userdb { driver = prefetch } userdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } verbose_proctitle = yes protocol lda { mail_location = maildir:~/Maildir:INDEX=MEMORY mail_plugins = quota acl expire fts fts_solr zlib sieve } protocol imap { mail_max_userip_connections = 25 mail_plugins = quota acl expire fts fts_solr zlib imap_quota imap_acl imap_zlib } protocol sieve { mail_max_userip_connections = 2 } -- Alessio Cecchi Postmaster @ http://www.qboxmail.it https://www.linkedin.com/in/alessice

1 0

Bug: The "listescape" plugin does not work with the option "mail_shared_explicit_inbox = no". [with patch]
by Rudolf Körner 09 Feb '16

09 Feb '16

Hello, there seems to be a problem with the listescape plugin and shared INBOX: === Preconditions: - Dovecot 2.2.21 (or 2.2.13) - IMAP - Separator / - Namespace-prefixes "INBOX/" and "shared/%%u/" - Maildir(++) - option mail_plugins = acl listescape - option mail_shared_explicit_inbox = no - User "owner" shares the INBOX - ACLs set for another user "reader", so "reader" is able to access the shared INBOX of "owner" === Steps to reproduce by using IMAP commands at a telnet session: - a1 login reader password - a2 select shared/owner === Result: Selecting the shared INBOX fails with a response like: a2 NO Mailbox doesn't exist: shared/owner === Expected: The "select" command works. === Analysis: The function "mailbox_list_default_get_storage_name" (lib-storage/mailbox-list.c) does not work as expected when using the mail_plugin "listescape". It compares the mailbox name to the namespace prefix. Since the mailbox name is escaped a few lines before, the comparison yields "strings differ", which makes accessing the shared INBOX fail. For example: for a string vname (mailbox name) "shared/owner", it compares "shared.owner" (escaped string) to the list->ns->prefix which is "shared/owner". Asuming that the stored prefix should not be escaped, there should be a comparison of the unescaped name to the unescaped prefix (or the escaped name to the escaped prefix?). === Patch: Attached is a small patch which shows a possible solution. When applied, the problem is solved, i.e. the shared INBOX is accessible then. Important note: this patch is experimental, it works for the described special case but might not work with a different configuration or might even break things. Best Regards, Rudolf Körner

1 0

RFC: HTTP based storage API
by Nagy, Attila 09 Feb '16

09 Feb '16

Hi, Nearly every popular programming language has an LMTP/POP/IMAP implementation, most of them suck in many different ways. I don't know any server or library which provides a well-established, compatible protocol frontend with an open backend API, which could be used to easily make a custom storage backend for the LMTP/POP/IMAP frontend in any language, in any programming paradigm, irregardless of the frontend implementation. Dovecot is very close to this: it speaks IMAP to the backend, but you still have to make a nearly complete IMAP implementation, which is a PITA. Therefore I am curious about your opinion: what about an HTTP-based open backend (lib-storage) API to Dovecot? Some thoughts on this: 1. it should be a simple REST API 2. it should support efficient operations on both sides (client, server), for example listing a multi-million folder shouldn't be a single JSON response, but a JSON stream, which could be produced and parsed individually or in smaller batches efficiently 3. it should support capabilities, somewhat like the current imapc backend, so if the backend service suppports search, it should announce this capability and dovecot could offload these requests to it, but if it doesn't, dovecot could build its indices. 4. only the needed amount of POP/IMAP legacy should appear in the protocol 5. the operations should be asynchronous on Dovecot's side (to be scalable) 6. maybe even Dovecot's index-queries could be offloaded to the backend service (this way the local file system wouldn't be used at all and all caches could be stored in a distributed dictionary), this could also be a capability 7. it could support some kind of push (for IDLE-ing and the like) 8. it should be stateless If Dovecot could make this right, it could be the FUSE of the mail protocols, acting as a glue for experimental or production ready mail storage backends, without the need to write these backends in "Dovecot C" (I mean it's process and API structure). Of course HTTP here is not a strict requirement. It could be anything, which can be easy to implement and understand and is able to fulfill the needs. An example could be Dovecot's own dictionary protocol. What do you think about this?

4 10

autoexpunge - different settings for different users
by Tom Johnson 09 Feb '16

09 Feb '16

I have some users where I'd like to expunge their mail at different times. For example, I may have some users who I want to expunge everything from every folder at 30 days. Others where I'd like to expunge everything at 90 days. And still others where I only want to expunge mail from the Trash and Junk folders, and nowhere else. Is there a way to do some sort of lookup for this? Or is this beyond the ability of the autoexpunge feature? Should I just use a cron job and use "doveadm expunge" on a user-by-user basis? Thanks-

5 9

Dovecot with pam authentication and user@domain
by Christian Schneider 08 Feb '16

08 Feb '16

Hello all, I'm trying to setup dovecot for local users with pam authentication. The passdb and userdb entries are as follows: passdb { args = username_format=%n driver = pam } userdb { args = username_format=%n driver = passwd } Using "doveadm user chriss" returns the user record as expected, but "doveadm user chriss(a)testmail.ch-sc.de" gives an error: field valueuserdb lookup: user chriss(a)testmail.ch-sc.de doesn't exist As far as I understand, username_format=%n should drop the domain part and only search for the user in the userdb, but it doesn't. What am I missing? Greetings Christian

1 1

Permissions issue with Debian 8, dovecot 2.21, vpopmail
by Stephen 08 Feb '16

08 Feb '16

There is a permissions issue in dovecot 2.21 in the ~/run directory (mine is /var/run/dovecot) at least in Debian 8. Files created by dovecot imap: srw------- 1 root root 0 Feb 3 16:46 imap-hibernate srw------- 1 root root 0 Feb 3 16:46 imap-master changed to make work: srw-rw-rw- 1 root root 0 Feb 3 16:46 imap-hibernate srw-rw-rw- 1 root root 0 Feb 3 16:46 imap-master Errors were reported in /var/log/dovecot.log which we use mainly for fail2ban (it gets huge, but ...) vpopmail version 5.4.33 with one patch I don't think this is a Debian issue ... possibly could be a vpopmail issue, but it's probably caused by the fact that imap daemons are running as vpopmail: vpopmail 30182 9187 0 15:02 ? 00:00:00 dovecot/imap

2 1

Released Pigeonhole v0.4.12 for Dovecot v2.2.21
by Stephan Bosch 08 Feb '16

08 Feb '16

Hello Dovecot users, Here is the final v0.4.12 release of Pigeonhole for Dovecot v2.2.21. Nothing changed since the RC. Changelog v0.4.12: + Implemented the Sieve extracttext extension (RFC 5703; Section 7). It is now possible to extract body text from a message into a variable. * Increased ABI version due to changes in the Sieve interpreter's object definitions. - multiscript: Fixed bug in handling of (implicit) keep; final keep action was always executed as though there was a failure. This caused the keep action to revert back to the initial message, causing editheader actions to be ignored. - managesieve-login: Fixed proxy to allow SASL mechanisms other than PLAIN. Before, the proxy would fail if the server did not support the PLAIN mechanism. - ldap storage: Prevent segfault occurring when assigning certain (global) configuration options. The release is available as follows: http://pigeonhole.dovecot.org/releases/2.2/dovecot-2.2-pigeonhole-0.4.12.ta… http://pigeonhole.dovecot.org/releases/2.2/dovecot-2.2-pigeonhole-0.4.12.ta… Refer to http://pigeonhole.dovecot.org and the Dovecot v2.x wiki for more information. Have fun testing this release and don't hesitate to notify me when there are any problems. Regards, -- Stephan Bosch stephan(a)rename-it.nl

2 3

"mail_shared_explicit_inbox = no", but shared INBOX has a "\Noselect" tag?
by Rudolf Körner 08 Feb '16

08 Feb '16

Regarding the given information, it seems to me that the option "mail_shared_explicit_inbox=no" does not work as expected. Quote from NEWS-2.2: mail_shared_explicit_inbox setting to specify if a shared INBOX should be accessible as "shared/$user" or "shared/$user/INBOX" Using Dovecot 2.2.13 and Maildir++ Working on the commandline: # doveconf mail_shared_explicit_inbox mail_shared_explicit_inbox = no # service dovecot restart # ls -lda /mail/reader/Shared/owner(a)example.com/.* Shows no (hidden) folders. # telnet 127.0.0.1 143 a0 login reader(a)example.com *** a1 list "" "Shared/owner(a)example.com" * LIST (\Noselect \HasChildren) "/" Shared/owner(a)example.com a1 OK List completed. Is the retrieved "\Noselect" tag correct with the option "mail_shared_explicit_inbox = no"? I would expect the folder "Shared/owner(a)example.com" to be "selectable", because I thought the option "mail_shared_explicit_inbox = no" would make e-mails of a shared INBOX visible directly at the folder "Shared/owner(a)example.com" (which does not work in my case). Note: Shared *sub*folders of an INBOX work as expected. Just to show that the shared INBOX is accessible (note that it is not shown by a list "" "*" here, which is expected with that option): a2 list "" "Shared/owner(a)example.com/INBOX" a2 OK List completed. a3 select Shared/owner(a)example.com/INBOX * FLAGS (\Answered \Flagged \Deleted \Seen \Draft) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted. * 27 EXISTS Using "mail_shared_explicit_inbox = yes" does show the INBOX as a subfolder of "Shared/owner(a)example.com" and allows accessing the e-mails of the shared INBOX. However, it would be nice (and keep users happy at using their known shared folder layout) to see the e-mails of the shared INBOX at the Shared/owner(a)example.com folder. Best Regards, R.Koerner output of (removed irrelevant things like auth,pop,passdb,userdb,ssl,debug) # dovecot -n # 2.2.13: /etc/dovecot/dovecot.conf mail_location = maildir:~/Maildir mail_log_prefix = "%Us(%u): " mail_plugins = acl listescape namespace { hidden = no ignore_on_failure = no inbox = no list = children location = maildir:%%h/Maildir:INDEX=%h/Shared/%%u:CONTROL=%h/Shared/%%u:INDEXPVT=%h/Shared/%%u prefix = Shared/%%u/ separator = / subscriptions = yes type = shared } namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = INBOX/ separator = / subscriptions = yes type = private } plugin { acl = vfile acl_shared_dict = file:/var/lib/dovecot/db/shared-mailboxes.db mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename } protocols = imap lmtp protocol imap { mail_plugins = acl listescape imap_acl }

1 1

ACL user
by Peter Chiochetti 08 Feb '16

08 Feb '16

How would I go, If I wanted ACL processing to start with %{auth_user} instead of %{user} when determining rights? -- peter

2 2

Send Dovecot logs to rsyslog
by Nicolas Fo 08 Feb '16

08 Feb '16

Hello, I'm trying to send Dovecot logs to a Graylog server. To do this, I'd like to pass logs to rsyslog and rsyslog pass logs to remote Graylog server. I set in dovecot.conf : syslog_facility = local5.info I set in rsyslog.conf : local5.info @192.168.xxx.xxx:5555 Restarted services and it doesn't work. I use nmap to test if port 5555 is opened and this port is open. What missed I ? Thanks Nicolas

4 7