April 2016 - dovecot - dovecot.org

v2.2.24 released
by Timo Sirainen 26 Apr '16

26 Apr '16

http://dovecot.org/releases/2.2/dovecot-2.2.24.tar.gz http://dovecot.org/releases/2.2/dovecot-2.2.24.tar.gz.sig This should be a good release. :) * doveconf now warns if it sees a global setting being changed when the same setting was already set inside some filters. (A common mistake has been adding more plugins to a global mail_plugins setting after it was already set inside protocol { .. }, which caused the global setting to be ignored for that protocol.) * LMTP proxy: Increased default timeout 30s -> 125s. This makes it less likely to reach the timeout and cause duplicate deliveries. * LMTP and indexer now append ":suffix" to session IDs to make it unique for the specific user's delivery. (Fixes duplicate session ID warnings in stats process.) + Added dict-ldap for performing read-only LDAP dict lookups. + lazy-expunge: All mails can be saved to a single specified mailbox. + mailbox { autoexpunge } supports now wildcards in mailbox names. + doveadm HTTP API: Added support for proxy commands + imapc: Reconnect when getting disconnected in non-selected state. + imapc: Added imapc_features=modseq to access MODSEQs/HIGHESTMODSEQ. This is especially useful for incremental dsync. + doveadm auth/user: Auth lookup performs debug logging if -o auth_debug=yes is given to doveadm. + Added passdb/userdb { auth_verbose=yes|no } setting. + Cassandra: Added user, password, num_threads, connect_timeout and request_timeout settings. + doveadm user -e <value>: Print <value> with %variables expanded. - Huge header lines could have caused Dovecot to use too much memory (depending on config and used IMAP commands). (Typically this would result in only the single user's process dying with out of memory due to reaching service { vsz_limit } - not a global DoS). - dsync: Detect and handle invalid/stale -s state string better. - dsync: Fixed crash caused by specific mailbox renames - auth: Auth cache is now disabled passwd-file. It was unnecessary and it broke %variables in extra fields. - fts-tika: Don't crash if it returns 500 error - dict-redis: Fixed timeout handling - SEARCH INTHREAD was crashing - stats: Only a single fifo_listeners was supported, making it impossible to use both auth_stats=yes and mail stats plugin. - SSL errors were logged in separate "Stacked error" log lines instead of as part of the disconnection reason. - MIME body parser didn't handle properly when a child MIME part's --boundary had the same prefix as the parent.

2 2

Subject: Re: v2.2.24 released
by Joseph Tam 26 Apr '16

26 Apr '16

Timo Sirainen wrote: > - auth: Auth cache is now disabled passwd-file. It was unnecessary and > it broke %variables in extra fields. So settings like auth_cache_size = 65536 auth_cache_ttl = 86400 auth_cache_negative_ttl = 600 are ignored/useless, and passwd-file auth lookups always results in file read operation? Joseph Tam <jtam.home(a)gmail.com>

2 2

Solr with shared mailboxes
by Ricardo Branco 26 Apr '16

26 Apr '16

We have multiple logins assigned to single mailboxes, so we can give one login to different devices. Does this cause a problem with mailbox guids in solr or is the ID stored within the indexes and not related to username.

2 3

doveadm search when using solr problems
by Ricardo Branco 26 Apr '16

26 Apr '16

Is there a way when using doveadm to search a users mailbox to disable using solr plugin (without changing config file). We have found that solr does not work well with certain search types and reverting to non solr searching works much better.

2 3

Possible to use Doveadm for a time range deletion
by Kenneth Ulrich 26 Apr '16

26 Apr '16

Hello All, I was wondering if it s possible to remove emails between a time range. For example, End Users have emails dating back to Aug 2000. He would like all emails deleted between the time stamp of 07/2014 to 07/2015 yet keep any emails outside of this area. I've ran into some similar examples, but nothing definite.

2 1

Strange search behavior
by Cornelius Charlie 26 Apr '16

26 Apr '16

Hello, I experienced a strange behavior in the way dovecot is performing searches, I only experienced this exact issue with the flag "seen". If I perform this command: UID SORT (SIZE) UTF-8 FLAGGED SEEN Dovecot return every seen messages even if the message is not "flagged'. This is only happening when there is no unseen messages in the mailbox, if there is at least one, this is working properly. Any though ? Thanks.

3 4

push-notification plugin and imap-metadata permissions
by Giovanni S. Fois 26 Apr '16

26 Apr '16

Ultra short version: Why cant I set the following mailbox metadata key? /private/vendor/vendor.dovecot/pvt/server/vendor/vendor.dovecot/http-notify Let me explain the context: I'm using the Dovecot version 2.23.1, but the same happens for the 2.2.22 The push-notification plugin is supposed to send out a notification whenever a mailbox get a new email message. In order to tell if a mailbox is enabled to send out the notification, the plugin looks out for the following mailbox metadata key: /private/vendor/vendor.dovecot/pvt/server/vendor/vendor.dovecot/http-notify If the key is set then everything is OK and the notification is sent, otherwise the action is skipped. If I try to setup the metadata key by hand (telnet as the user over the imap port): setmetadata INBOX (/private/vendor/vendor.dovecot/pvt/server/vendor/vendor.dovecot/http-notify "user=myuser(a)mydomain.com") I get the error message: "Internal mailbox attributes cannot be accessed" Digging in the Dovecot 2.2.22 sources i found that: This string is from lib-imap-storage/imap-metadata.c - line 36 - Dovecot 2.2.22 The message is triggered by the following condition - same file - line 125 - Dovecot 2.2.22 if (strncmp(*key_r, MAILBOX_ATTRIBUTE_PREFIX_DOVECOT_PVT, strlen(MAILBOX_ATTRIBUTE_PREFIX_DOVECOT_PVT)) == 0) { So the path pvt/server appears to be forbidden. But, in the file lib-storage/mailbox-attribute.h we can read the following comment: /* User can get/set all non-pvt/ attributes and also pvt/server/ (but not pvt/server/pvt/) attributes. */ And, after said comment there is the definition of the macro MAILBOX_ATTRIBUTE_KEY_IS_USER_ACCESSIBLE(key) which has the same basic function of the condition in imap-metadata.c , but in this case the same imap key is seen as accessible. Now my questions: Can we use a negated version of MAILBOX_ATTRIBUTE_KEY_IS_USER_ACCESSIBLE(key) in imap-metadata? How can the push-notification plugin work out-of-the-box without changes and recompilation? Thank you for your valuable time and forgive me if I'm posing a dumb question. Best wishes, Giovanni S. Fois

4 6

Kernel panic in dovecot-ee-lmtp on Debian 8
by Tobi 26 Apr '16

26 Apr '16

Hi list I just realized that that I don not receive all mails in my mailbox (running dovecot-ee 2.2.23.1-1 on Debian 8). On my frontend servers (running postfix) the queue fills up with mails that cannot be delivered via lmtp to my backend servers. The error message on the frontend is "lost connection with backend while sending data" When I checked the logs on the backend server I found upon every delivery attempt kernel panics Apr 25 12:33:36 mbox1 dovecot: lmtp(REDACTED): Panic: epoll_ctl(del, 18) failed: Bad file descriptor Apr 25 12:33:36 mbox1 dovecot: lmtp(REDACTED): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x85dee) [0x7faa19225dee] -> /usr/lib/dovecot/libdovecot.so.0(+0x85edc) [0x7faa19225edc] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7faa191ca5be] -> /usr/lib/dovecot/libdovecot.so.0(+0x9afc1) [0x7faa1923afc1] -> /usr/lib/dovecot/libdovecot.so.0(+0x994af) [0x7faa192394af] -> /usr/lib/dovecot/libdovecot.so.0(+0xa5eda) [0x7faa19245eda] -> /usr/lib/dovecot/libdovecot.so.0(+0xa4566) [0x7faa19244566] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_destroy+0xe) [0x7faa192446ee] -> /usr/lib/dovecot/libdovecot-sieve.so.0(+0x7e44b) [0x7faa17d3a44b] -> /usr/lib/dovecot/libdovecot-sieve.so.0(+0x7ea14) [0x7faa17d3aa14] -> /usr/lib/dovecot/libdovecot.so.0(+0xa6d88) [0x7faa19246d88] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x4c) [0x7faa19239c6c] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x101) [0x7faa1923b0d1] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x25) [0x7faa19239cf5] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7faa19239e98] -> /usr/lib/dovecot/libdovecot-sieve.so.0(program_client_run+0xe8) [0x7faa17d3b308] -> /usr/lib/dovecot/modules/sieve/lib90_sieve_extprograms_plugin.so(+0x45b8) [0x7faa15d855b8] -> /usr/lib/dovecot/libdovecot-sieve.so.0(sieve_interpreter_continue+0x7c) [0x7faa17cfa2ac] -> /usr/lib/dovecot/libdovecot-sieve.so.0(sieve_interpreter_run+0x2b) [0x7faa17cfa4cb] -> /usr/lib/dovecot/libdovecot-sieve.so.0(+0x519e0) [0x7faa17d0d9e0] -> /usr/lib/dovecot/libdovecot-sieve.so.0(sieve_execute+0x47) [0x7faa17d0e567] -> /usr/lib/dovecot/modules/lib90_sieve_plugin.so(+0x3b6b) [0x7faa17f73b6b] -> /usr/lib/dovecot/libdovecot-lda.so.0(mail_deliver+0x49) [0x7faa197e5849] -> dovecot/lmtp() [0x4064b3] -> dovecot/lmtp() [0x406cb4] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x4c) [0x7faa19239c6c] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x101) [0x7faa1923b0d1] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x25) [0x7faa19239cf5] Apr 25 12:33:36 mbox1 dovecot: lmtp(REDACTED): Fatal: master: service(lmtp): child 32081 killed with signal 6 (core dumps disabled) the weird thing is that this is not concerning all incoming mails. From time to time the frontends can deliver messages to my backends. I just tested it with a mail from work. It arrived via a frontend that has other mails in queue that could not be delivered so far. Does anyone have an idea where I could look for the root cause of this panic messages? Thanks a lot tobi

4 5

doveadm-server segfaults when searching or expunging with the -A flag
by Burak Seydioglu 26 Apr '16

26 Apr '16

Hello, I have an openldap backend that contains a large number of users and serves passdb and userdb requests. I was trying to work around the pop3 deletion issue that I had reported earlier (http://www.dovecot.org/list/dovecot/2016-April/103973.html) with the pop3_deleted_flag setting so I updated my configuration and created some background jobs to expunge deleted emails. However, doveadm-server segfaults when running search or expunge commands with the -A flag: doveadm search mailbox INBOX KEYWORD '$Pop3Deleted' -A doveadm expunge mailbox INBOX KEYWORD '$Pop3Deleted' -A ###################################### gdb /usr/libexec/dovecot/doveadm-server /var/crash/core-doveadm-server-11-0-0-2589-1461628665 #0 o_stream_cork (stream=0x0) at ostream.c:141 #1 0x0000000000430193 in server_connection_output (conn=0xfe10e0) at server-connection.c:126 #2 0x00007fcde9fc80c7 in plain_flush_callback (sstream=0xfdacf0) at ostream-openssl.c:180 #3 0x00007fcdeb6f9929 in stream_send_io (fstream=0xfe65e0) at ostream-file.c:482 #4 0x00007fcdeb6ebceb in io_loop_call_io (io=0xfdb110) at ioloop.c:564 #5 0x00007fcdeb6ed2e7 in io_loop_handler_run_internal (ioloop=<value optimized out>) at ioloop-epoll.c:220 #6 0x00007fcdeb6ebd95 in io_loop_handler_run (ioloop=0xfdf4b0) at ioloop.c:612 #7 0x00007fcdeb6ebf38 in io_loop_run (ioloop=0xfdf4b0) at ioloop.c:588 #8 0x000000000041f542 in dsync_connect_tcp (ctx=0xfc7d70, mail_set=<value optimized out>, target=<value optimized out>, ssl=true, error_r=0x7ffde6127020) at doveadm-dsync.c:806 #9 0x000000000041ff95 in parse_location (_ctx=0xfc7d70, service_user=<value optimized out>, error_r=0x7ffde6127020) at doveadm-dsync.c:837 #10 cmd_dsync_prerun (_ctx=0xfc7d70, service_user=<value optimized out>, error_r=0x7ffde6127020) at doveadm-dsync.c:903 #11 0x00000000004205f2 in doveadm_mail_next_user (ctx=0xfc7d70, cctx=<value optimized out>, error_r=0x7ffde6127020) at doveadm-mail.c:383 #12 0x000000000043151f in doveadm_mail_cmd_server_run (conn=0xfc3950) at client-connection.c:183 #13 doveadm_cmd_handle (conn=0xfc3950) at client-connection.c:266 #14 client_handle_command (conn=0xfc3950) at client-connection.c:330 #15 client_connection_input (conn=0xfc3950) at client-connection.c:444 #16 0x00007fcdeb6ebceb in io_loop_call_io (io=0xfc34f0) at ioloop.c:564 #17 0x00007fcdeb6ed2e7 in io_loop_handler_run_internal (ioloop=<value optimized out>) at ioloop-epoll.c:220 #18 0x00007fcdeb6ebd95 in io_loop_handler_run (ioloop=0xf90780) at ioloop.c:612 #19 0x00007fcdeb6ebf38 in io_loop_run (ioloop=0xf90780) at ioloop.c:588 #20 0x00007fcdeb684753 in master_service_run (service=0xf90620, callback=<value optimized out>) at master-service.c:640 #21 0x0000000000433897 in main (argc=1, argv=0xf903a0) at main.c:126 ###################################### Dovecot version ###################################### dovecot --version 2.2.23 (806d709) ###################################### doveconf -n ###################################### # 2.2.23 (806d709): /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-573.22.1.el6.x86_64 x86_64 CentOS release 6.7 (Final) auth_debug = yes auth_mechanisms = plain login auth_verbose = yes base_dir = /var/run/dovecot/ disable_plaintext_auth = no doveadm_password = # hidden, use -P to show it first_valid_uid = 89 last_valid_uid = 89 login_greeting = Dovecot ready on mda01.dc1.domain.com. login_trusted_networks = 127.0.0.1 mail_debug = yes mail_gid = 89 mail_plugins = " notify replication" mail_uid = 89 mbox_write_locks = fcntl namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = /etc/dovecot/conf.d/dovecot-ldap-passdb.conf.ext driver = ldap } plugin { mail_replica = tcps:mdareplica:12345 } pop3_deleted_flag = $POP3Deleted protocols = pop3 lmtp replication_full_sync_interval = 3 hours replication_max_conns = 20 service aggregator { fifo_listener replication-notify-fifo { group = postfix mode = 0640 user = postfix } unix_listener replication-notify { group = postfix mode = 0640 user = postfix } } service auth { unix_listener auth { mode = 0660 } } service doveadm { inet_listener { port = 12345 ssl = yes } } service lmtp { process_min_avail = 10 unix_listener lmtp { group = postfix mode = 0640 user = postfix } } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { group = postfix mode = 0640 user = postfix } } ssl_cert = </etc/pki/tls/certs/dovecot.crt ssl_client_ca_file = /etc/pki/tls/CA/dovecot.bundle ssl_key = </etc/pki/tls/private/dovecot.key ssl_prefer_server_ciphers = yes userdb { driver = prefetch } userdb { args = /etc/dovecot/conf.d/dovecot-ldap-userdb.conf.ext driver = ldap } protocol lmtp { mail_plugins = " notify replication" } ###################################### LDAP configuration ###################################### cat /etc/dovecot/conf.d/dovecot-ldap-userdb.conf.ext hosts = localhost dn = cn=dovecot,dc=csi,dc=domain,dc=com dnpass = xxx ldap_version = 3 base = ou=instances,dc=csi,dc=domain,dc=com deref = never scope = subtree user_attrs = \ =home=/glide/mail/spool/%{ldap:uid}, \ =mail=maildir:/glide/mail/spool/%{ldap:maildrop} user_filter = (&(objectClass=mailUser)(uid=%n)) iterate_attrs = uid=user iterate_filter = (objectClass=mailUser) cat /etc/dovecot/conf.d/dovecot-ldap-passdb.conf.ext hosts = localhost dn = cn=dovecot,dc=csi,dc=domain,dc=com dnpass = xxx ldap_version = 3 base = ou=instances,dc=csi,dc=domain,dc=com deref = never scope = subtree pass_attrs = uid=user,userPassword=password, \ =userdb_home=/glide/mail/spool/%{ldap:uid}, \ =userdb_mail=maildir:/glide/mail/spool/%{ldap:maildrop} pass_filter = (&(objectClass=inetLocalMailRecipient)(uid=%n)) ###################################### Only interesting thing from the debug logs is a prefetch related entry: ###################################### Apr 25 17:31:22 mda01 dovecot: auth: Debug: master in: USER#01119118#011dev13654#011service=doveadm Apr 25 17:31:22 mda01 dovecot: auth: Debug: prefetch(dev13654): passdb didn't return userdb entries, trying the next userdb Apr 25 17:31:22 mda01 dovecot: auth: Debug: ldap(dev13654): user search: base=ou=instances,dc=csi,dc=domain,dc=com scope=subtree filter=(&(objectClass=mailUser)(uid=dev13654)) fields=uid,maildrop Apr 25 17:31:22 mda01 dovecot: auth: Debug: ldap(dev13654): result: uid=dev13654 maildrop=dev13654/maildir/; maildrop,uid unused Apr 25 17:31:22 mda01 dovecot: auth: Debug: ldap(dev13654): result: uid=dev13654 maildrop=dev13654/maildir/ Apr 25 17:31:22 mda01 dovecot: auth: Debug: userdb out: USER#01119118#011dev13654#011home=/glide/mail/spool/dev13654#011mail=maildir:/glide/mail/spool/dev13654/maildir/ ######################################

1 0

doveadm backup 2.2.23 ignores -u, but reads USER environment?
by Heiko Schlittermann 26 Apr '16

26 Apr '16

Hi, during a migration (cyrus -> dovecot) we ran into trouble¹ using 'doveadm backup' with 2.2.9. So I built 2.2.22 and 2.2.23, but they expose quite different behaviour. 2.2.22 seems to work at least partially, but 2.2.23 complains about a non existing user (root?). The command I'm running is: doveadm -o pop3_uidl_format=%v.%u \ -o imapc_master_user=cyrus -o imapc_password=* -o imapc_host=* \ -o mail_fsync=never \ -o imapc_ssl_verify=no -o imapc_ssl=imaps -o imapc_port=993 \ backup -f -u heiko -R imapc: Using 2.2.9 or 2.2.22 it works, using 2.2.23 "-u …" is ignored and I've to pass the user via the USER environment. But according to the manpage of doveadm-sync -u user/mask Run the command only for the given user. It's also possible to use '*' and '?' wildcards (e.g. -u *(a)example.org) When neither the -A option, nor the -F file option, nor the -u user was specified, the command will be executed with the environment of the currently logged in user. Interestingly enough -u 'heiko*' works fine, even with USER=root. So the environment doesn't seem to override -u … always. Maybe an important side note: the user 'heiko' doesn't exist as a local system user, he is known via LDAP only. And for e.g. 'doveadm quota -u heiko' it works as expected, even with 2.2.23. ¹) solved meanwhile by using -o mail_plugins= as the quota plugin seems to crash 'doveadm backup' Best regards from Dresden/Germany Viele Grüße aus Dresden Heiko Schlittermann -- SCHLITTERMANN.de ---------------------------- internet & unix support - Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} - gnupg encrypted messages are welcome --------------- key ID: F69376CE - ! key id 7CBF764A and 972EAC9F are revoked since 2015-01 ------------ -

1 0