October 2016 - dovecot

v2.2.26 release candidate released
by Timo Sirainen 21 Oct '16

21 Oct '16

http://dovecot.org/releases/2.2/rc/dovecot-2.2.26.rc1.tar.gz http://dovecot.org/releases/2.2/rc/dovecot-2.2.26.rc1.tar.gz.sig There are quite a lot of changes since v2.2.25. Please try out this RC so we can get a good and stable v2.2.26 out. * master: Removed hardcoded 511 backlog limit for listen(). The kernel should limit this as needed. * doveadm import: Source user is now initialized the same as target user. Added -U parameter to override the source user. * Mailbox names are no longer limited to 16 hierarchy levels. We'll check another way to make sure mailbox names can't grow larger than 4096 bytes. + Added a concept of "alternative usernames" by returning user_* extra field(s) in passdb. doveadm proxy list shows these alt usernames in "doveadm proxy list" output. "doveadm director&proxy kick" adds -f <passdb field> parameter. The alt usernames don't have to be unique, so this allows creation of user groups and kicking them in one command. + auth: passdb/userdb dict allows now %variables in key settings. + auth: If passdb returns noauthenticate=yes extra field, assume that it only set extra fields and authentication wasn't actually performed. + auth: passdb static now supports password={scheme} prefix. + imapc: Added imapc_max_line_length to limit maximum memory usage. + imap, pop3: Added rawlog_dir setting to store IMAP/POP3 traffic logs. This replaces at least partially the rawlog plugin. + dsync: Added dsync_features=empty-header-workaround setting. This makes incremental dsyncs work better for servers that randomly return empty headers for mails. When an empty header is seen for an existing mail, dsync assumes that it matches the local mail. + doveadm sync/backup: Added -I <max size> parameter to skip too large mails. + doveadm sync/backup: Fixed -t parameter and added -e for "end date". + doveadm mailbox metadata: Added -s parameter to allow accessing server metadata by using empty mailbox name. - master process's listener socket was leaked to all child processes. This might have allowed untrusted processes to capture and prevent "doveadm service stop" comands from working. - auth: userdb fields weren't passed to auth-workers, so %{userdb:*} from previous userdbs didn't work there. - auth: Each userdb lookup from cache reset its TTL. - auth: Fixed auth_bind=yes + sasl_bind=yes to work together - auth: Blocking userdb lookups reset extra fields set by previous userdbs. - auth: Cache keys didn't include %{passdb:*} and %{userdb:*} - auth-policy: Fixed crash due to using already-freed memory if policy lookup takes longer than auth request exists. - lib-auth: Unescape passdb/userdb extra fields. Mainly affected returning extra fields with LFs or TABs. - lmtp_user_concurrency_limit>0 setting was logging unnecessary anvil errors. - lmtp_user_concurrency_limit is now checked before quota check with lmtp_rcpt_check_quota=yes to avoid unnecessary quota work. - lmtp: %{userdb:*} variables didn't work in mail_log_prefix - autoexpunge settings for mailboxes with wildcards didn't work when namespace prefix was non-empty. - Fixed writing >2GB to iostream-temp files (used by fs-compress, fs-metawrap, doveadm-http) - director: Ignore duplicates in director_servers setting. - zlib, IMAP BINARY: Fixed internal caching when accessing multiple newly created mails. They all had UID=0 and the next mail could have wrongly used the previously cached mail. - doveadm stats reset wasn't reseting all the stats. - auth_stats=yes: Don't update num_logins, since it doubles them when using with mail stats. - quota count: Fixed deadlocks when updating vsize header. - dict-quota: Fixed crashes happening due to memory corruption. - dict proxy: Fixed various timeout-related bugs. - doveadm proxying: Fixed -A and -u wildcard handling. - doveadm proxying: Fixed hangs and bugs related to printing. - imap: Fixed wrongly triggering assert-crash in client_check_command_hangs. - imap proxy: Don't send ID command pipelined with nopipelining=yes - imap-hibernate: Don't execute quota_over_script or last_login after un-hibernation. - imap-hibernate: Don't un-hibernate if client sends DONE+IDLE in one IP packet. - imap-hibernate: Fixed various failures when un-hibernating. - fts: fts_autoindex=yes was broken in 2.2.25 unless fts_autoindex_exclude settings existed. - fts-solr: Fixed searching multiple mailboxes (patch by x16a0) - doveadm fetch body.snippet wasn't working in 2.2.25. Also fixed a crash with certain emails. - pop3-migration + dbox: Various fixes related to POP3 UIDL optimization in 2.2.25. - pop3-migration: Fixed "truncated email header" workaround.

10 17

Migrating users from a 2.0.19 to a 2.2.24 installation
by Gerben Wierda 21 Oct '16

21 Oct '16

Hello, I am currently still running an older dovecot (2.0.19apple1 on Mac OS X 10.8.5) and I want to migrate my users to a new server (macOS 10.12 with Server 5, which contains dovecot 2.2.24 (a82c823)). Basically, I want to create a new server installation on the new server so I don't bring any junk over (new user accounts, with the same uid/gid (still need to figure that one out), but after I have done that I need to move the data over from the old instalation to the new. Has anything changed in the formats between 2.0 and 2.2 that will stop me from doing this? Thanks, G

2 1

Extending IMAP daemon to comply with legacy Proxy
by Thorsten Hater 21 Oct '16

21 Oct '16

Hello, I am currently investigating if and how to migrate a legacy setup based on a modified Courier installation to Dovecot. So far I am really liking Dovecot. however, I will have to work with an existing IMAP proxy for the time being. In particular, the proxy authenticates the user and looks up the location of the mail directory and transfers the data to the actual server via an extension of the IMAP protocol. This extension will have to stay and the information transferred is not easily recoverable. Further database lookups on the real server would be possible to solve the issue, but I would prefer to keep the design intact at least for now. Is it possible to handle IMAP commands after LOGIN/AUTHENTICATE but before opening any mailboxes? Basically these commands aim to replace the userdb lookup. Any pointers on how to implement this, perhaps as a plugin? Best regards, Thorsten

1 0

sieve duplicate locking
by Alexander 'Leo' Bergolth 20 Oct '16

20 Oct '16

Hi! Does the duplicate sieve plugin do any locking to avoid duplicate parallel delivery of the same message? I sometimes experience duplicate mail delivery of messages with the same message-id, despite the use of a sieve duplicate filter. According to the log files, those messages are delivered in the same second by two parallel dovecot-lda processes. (Duplicate filtering works fine in all other cases.) RFC7352 states that the ID of a message may only be committed to the duplicate tracking list at the _end_ of a successful script execution, which may lead to race conditions. Maybe I am running into this? Is there an easy way to serialize mail delivery using some locking inside sieve? Or do I have to serialize per-user dovecot-lda delivery? Any experiences with that? I am using dovecot-2.2.25 and pidgeonhole-0.4.15. Mail is delivered using postfix-2.10 and dovecot-lda as mailbox_command. Mailbox format is maildir with LAYOUT=fs. Cheers, --leo -- e-mail ::: Leo.Bergolth (at) wu.ac.at fax ::: +43-1-31336-906050 location ::: IT-Services | Vienna University of Economics | Austria

2 2

Re: Massive LMTP Problems with dovecot
by Ralf Hildebrandt 20 Oct '16

20 Oct '16

> We'll need the `dovecot -n` output first. Here we go: > # 2.2.devel (933d16f): /etc/dovecot/dovecot.conf > # Pigeonhole version 0.4.devel (63f9b42) > # OS: Linux 3.13.0-98-generic x86_64 Ubuntu 14.04.5 LTS > default_vsz_limit = 2 G > lmtp_user_concurrency_limit = 10000 > mail_attachment_dir = /home/copymail/attachments > mail_location = mdbox:~/mdbox > mail_plugins = zlib fts fts_lucene > mdbox_rotate_size = 128 M > namespace inbox { > inbox = yes > location = > mailbox Drafts { > special_use = \Drafts > } > mailbox Junk { > special_use = \Junk > } > mailbox Sent { > special_use = \Sent > } > mailbox "Sent Messages" { > special_use = \Sent > } > mailbox Trash { > special_use = \Trash > } > prefix = > } > passdb { > args = username_format=%u /etc/dovecot/passwd > driver = passwd-file > } > plugin { > fts = lucene > fts_autoindex = yes > fts_languages = de,en > fts_lucene = whitespace_chars=@. > sieve = file:~/sieve;active=~/.dovecot.sieve > zlib_save = gz > zlib_save_level = 5 > } > protocols = " imap lmtp" > service imap-login { > inet_listener imap { > address = 127.0.0.1 > port = 143 > } > inet_listener imaps { > port = 993 > ssl = yes > } > } > service lmtp { > inet_listener lmtp { > address = 141.42.1.208 > port = 1025 > } > unix_listener /var/spool/postfix/private/dovecot-lmtp { > group = postfix > mode = 0660 > user = postfix > } > } > ssl_ca = /etc/ssl/certs/ca-certificates.crt > ssl_cert = </etc/ssl/certs/mail-cbf.pem > ssl_cipher_list = EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4 > ssl_key = # hidden, use -P to show it > ssl_prefer_server_ciphers = yes > ssl_protocols = !SSLv2 !SSLv3 > userdb { > args = username_format=%u /etc/dovecot/passwd > driver = passwd-file > } > protocol lmtp { > mail_plugins = zlib fts fts_lucene > } Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt(a)charite.de | http://www.charite.de

2 6

Lmtp Fatal Error
by Sven Roellig 19 Oct '16

19 Oct '16

Hi,dovecot is create an Fatal Panik Error. <AM5DMwaCBlgpDQAAWm89Cw>: Fatal: master: service(lmtp): child 3369 killed with signal 6 (core dumps disabled) <1jx3DhuCBlg1DQAAWm89Cw>: Panic: file lda-sieve-plugin.c: line 447 (lda_sieve_execute_scripts): assertion failed: (script != NULL) <1jx3DhuCBlg1DQAAWm89Cw>: Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x93fae) [0x7f547f6a3fae] -> /usr/lib/dovecot/libdovecot.so.0(+0x9409c) [0x7f547f6a409c] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f547f63d56e] -> /usr/lib/dovecot/modules/lib90_sieve_plugin.so(+0x3ae8) [0x7f547d765ae8] -> /usr/lib/dovecot/libdovecot-lda.so.0(mail_deliver+0x49) [0x7f547fc709a9] -> dovecot/lmtp(+0x7201) [0x7f54800a1201] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x5f) [0x7f547f6b88bf] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x10a) [0x7f547f6b9d8a] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x25) [0x7f547f6b8965] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x30) [0x7f547f6b8b00] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f547f643ac3] -> dovecot/lmtp(main+0x1c9) [0x7f548009f2c9] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f547f286b45] -> dovecot/lmtp(+0x53ae) [0x7f548009f3ae] Normal Mails are not deliverd but filterd mails are deliverd. System is Debian 8.6 Dovecot Core is 2%3a2.3.0-alpha0-1-auto+385 AMD64 can anyone help. Sven

3 4

Iteration base for LDAP
by Taso N. Devetzis 18 Oct '16

18 Oct '16

Greetings, The iteration machinery uses the LDAP search base set with the "base" directive (typically from dovecot-ldap.conf.ext); the same base used during nominal operations (e.g., passdb/userdb searches). Consider a directory: dc=ROOT |_ dc=foo,dc=com,dc=ROOT (foo.com subtree) |_ dc=bar,dc=net,dc=ROOT (bar.net subtree) A search base setting appropriate for mail operations might be: base = dc=%Dd,dc=ROOT # e.g. dc=foo,dc=com,dc=ROOT for user(a)foo.com This fails when iterating, as the variable substitution is meaningless in this context (and even a static subtree search base would only cover a portion of the overall directory during iterative searches). Setting the base to "dc=ROOT" obviously solves the issue at the expense of searching the entire directory for all operations. This is less than optimal. I could not find a way to override this setting at runtime via a doveadm option or similar. Ideally, a separate "iterate_base" setting would solve this issue. Any other solutions? Thanks, /taso

1 0

RE: First steps in Dovecot; IMAP not working
by Joseph Tam 18 Oct '16

18 Oct '16

> > doveadm user mailtest > field value > uid 1002 > gid 8 > home /home/mailtest > mail mbox:~/mail:INBOX=/var/mail/mailtest > system_groups_user mailtest Looks fine. > [log files] I'll try to locate them. Your best hope. Maybe this helps doveconf -n | grep log_ Or just force it somewhere by settings the *log_path configuration values. Also your mail server's log when trying to deliver. Joseph Tam <jtam.home(a)gmail.com>

1 0

Dict proxy client returning empty string instead of multiline string
by Pierre Jaury 18 Oct '16

18 Oct '16

Hello, I am using a dict proxy for my sieve extdata plugin to access some fields from an SQLite database (autoreply text and other database-configured items). All tests are performed against version 2.2.25. $ dovecot --version 2.2.25 (7be1766) My configuration looks like: dict { sieve = sqlite:/etc/dovecot/pigeonhole-sieve.dict } [...] sieve_extdata_dict_uri = proxy::sieve I am able to read pretty much any attribute without any issue, except when the value contains a special character like "\r" or "\n". By using the doveadm dict client, I narrowed it down to the dictionary management part (either server, protocol or client). I was suspecting escaping functions from "lib/strescape.c" (mostly str_tabescape and its counterpart, used by "lib-dict/client.c"), so I monitored socket communications. It seems that escaping is done properly on the server and the socket is not an issue either. The following strace dump results from running "doveadm dict get" against the dict socket: connect(8, {sa_family=AF_UNIX, sun_path="..."}, 110) = 0 fstat(8, {st_mode=S_IFSOCK|0777, st_size=0, ...}) = 0 [...] write(8, "H2\t0\t0\tadmin(a)domain.tld\tsieve\n", 30) = 30 [...] read(8, "Otest\1r\1ntest\n", 8192) = 14 Indeed "\1r" and "\1n" are the escape sequences used by "lib/strescape.c". I went deeped and debugged the call to "dict_lookup" performed by doveadm. Indeed the client gets the proper string from the socket and to my surprise, it is properly unescaped. Then, in "client_dict_lookup" ("lib-dict/dict-client.c"), the call to "p_strdup" returns an empty string (null byte set at the target address). Before the call to the dict "->lookup" attribute (client_dict_lookup): RAX: 0x7ffff73a37c0 (push r14) RBX: 0x6831b8 ("priv/reply_body") RCX: 0x7fffffffe240 --> 0x682a60 --> 0x6831b8 ("priv/reply_body") RDX: 0x6831b8 ("priv/reply_body") RSI: 0x683288 --> 0x7ffff7653120 --> 0x7ffff73ea620 ([...]) RDI: 0x690ad0 --> 0x7ffff7400713 --> 0x75250079786f7270 ('proxy') 0x7ffff73a1f10 <dict_lookup+32>: mov rcx,r11 (value_r) 0x7ffff73a1f13 <dict_lookup+35>: mov rdx,r8 (key) 0x7ffff73a1f16 <dict_lookup+38>: mov rsi,r10 (pool) 0x7ffff73a1f19 <dict_lookup+41>: mov rdi,r9 (dict) 0x7ffff73a1f1c <dict_lookup+44>: add rsp,0x8 => 0x7ffff73a1f20 <dict_lookup+48>: jmp rax Before the call to p_strdup in "client_dict_lookup": RSI: 0x6832d8 ("test\r\ntest") (lookup.result.value) RDI: 0x683288 --> 0x7ffff7653120 --> [...] (pool) RAX: 0x0 (result) 0x7ffff73a384f: nop 0x7ffff73a3850: mov rsi,QWORD PTR [rsp+0x8] 0x7ffff73a3855: mov rdi,r14 => 0x7ffff73a3858: call 0x7ffff736d3c0 <p_strdup@plt> 0x7ffff73a385d: mov QWORD PTR [r13+0x0],rax 0x7ffff73a3861: mov rsi,QWORD PTR [rsp+0x18] 0x7ffff73a3866: xor rsi,QWORD PTR fs:0x28 0x7ffff73a386f: mov eax,ebx After the call: 0x7ffff73a3850: mov rsi,QWORD PTR [rsp+0x8] 0x7ffff73a3855: mov rdi,r14 0x7ffff73a3858: call 0x7ffff736d3c0 <p_strdup@plt> => 0x7ffff73a385d: mov QWORD PTR [r13+0x0],rax 0x7ffff73a3861: mov rsi,QWORD PTR [rsp+0x18] 0x7ffff73a3866: xor rsi,QWORD PTR fs:0x28 0x7ffff73a386f: mov eax,ebx 0x7ffff73a3871: jne 0x7ffff73a38da RSI: 0x0 RDI: 0x6832d8 --> 0x0 RAX: 0x6832d8 --> 0x0 (result) It is worth noting that I can reproduce the exact same execution flow with a non-multiline result string (lookup.result.value) that is properly copied by "p_strdup" and returned in RAX, then displayed by doveadm. I am not familiar with the pooling mechanism hidden behind the call to p_strdump and not quite sure why this behaviour is emerging. Maybe I am even miles away from an understanding of the issue here, but it sounds to me like something is wrong in the way "p_strdup" performs the copy. Hope this helps, kaiyou.

2 11

Re: First steps in Dovecot; IMAP not working
by Joseph Tam 18 Oct '16

18 Oct '16

Marnaud <anic297(a)mac.com> writes: > "mailtest", the new user, is in group mail(8). In addition, I've added > write permission for "others" to /var/mail. Now, I'm trying to send a > message to "mailtest" from another, working, e-mail account and nothing > happens. This time, "doveadm log errors" is empty. > > In short, I don't get any error but no mail either. Two things to check: # What does dovecot think user "mailtest" has? doveadm user mailtest # The sticky bit should be set on /var/mail (you didn't mention # setting it. It probably doesn't have bearing on this problem, # but it will make it a little more secure. chmod 1777 /var/mail > Since "doveadm log errors" returns an empty result, where should I look > for the problem? I usually don't use this command, I look at the log file which seems to have more details. Try looking there for more diagnostics. Also, look at your MTA's logs as well. Joseph Tam <jtam.home(a)gmail.com>

4 4