v2.3.0 release candidate released

[Mark Moseley]

On Mon, Dec 18, 2017 at 7:23 AM, Timo Sirainen <tss at iki.fi> wrote:

> https://dovecot.org/releases/2.3/rc/dovecot-2.3.0.rc1.tar.gz
> https://dovecot.org/releases/2.3/rc/dovecot-2.3.0.rc1.tar.gz.sig
>
> It's finally time for v2.3 release branch! There are several new and
> exciting features in it. I'm especially happy about the new logging and
> statistics code, which will allow us to generate statistics for just about
> everything. We didn't have time to implement everything we wanted for them
> yet, and there especially aren't all that many logging events yet that can
> be used for statistics. We'll implement those to v2.3.1, which might also
> mean that some of the APIs might still change in v2.3.1 if that's required.
>
> We also have new lib-smtp server code, which was used to implement SMTP
> submission server and do a partial rewrite for LMTP server. Please test
> these before v2.3.0 to make sure we don't have any bad bugs left!
>
> BTW. The v2.3.0 will most likely be signed with a new PGP key ED409DA1.
>
> Some of the larger changes:
>
>  * Various setting changes, see https://wiki2.dovecot.org/Upgrading/2.3
>  * Logging rewrite started: Logging is now based on hierarchical events.
>    This makes it possible to do various things, like: 1) giving
>    consistent log prefixes, 2) enabling debug logging with finer
>    granularity, 3) provide logs in more machine readable formats
>    (e.g. json). Everything isn't finished yet, especially a lot of the
>    old logging code still needs to be translated to the new way.
>  * Statistics rewrite started: Stats are now based on (log) events.
>    It's possible to gather statistics about any event that is logged.
>    See http://wiki2.dovecot.org/Statistics for details
>  * ssl_dh setting replaces the old generated ssl-parameters.dat
>  * IMAP: When BINARY FETCH finds a broken mails, send [PARSE] error
>    instead of [UNKNOWNCTE]
>  * Linux: core dumping via PR_SET_DUMPABLE is no longer enabled by
>    default due to potential security reasons (found by cPanel Security
>    Team).
>
>  + Added support for SMTP submission proxy server, which includes
>    support for BURL and CHUNKING extension.
>  + LMTP rewrite. Supports now CHUNKING extension and mixing of
>    local/proxy recipients.
>  + auth: Support libsodium to add support for ARGON2I and ARGON2ID
>    password schemes.
>  + auth: Support BLF-CRYPT password scheme in all platforms
>  + auth: Added LUA scripting support for passdb/userdb.
>    See https://wiki2.dovecot.org/AuthDatabase/Lua
>  - Input streams are more reliable now when there are errors or when
>    the maximum buffer size is reached. Previously in some situations
>    this could have caused Dovecot to try to read already freed memory.
>  - Output streams weren't previously handling failures when writing a
>    trailer at the end of the stream. This mainly affected encrypt and
>    zlib compress ostreams, which could have silently written truncated
>    files if the last write happened to fail (which shouldn't normally
>    have ever happened).
>  - virtual plugin: Fixed panic when fetching mails from virtual
>    mailboxes with IMAP BINARY extension.
>  - Many other smaller fixes
>
>

No issue compilng (and very very excited about this release, esp the Lua
code, which is already super useful).

I did have this one issue so far with the RC. I was previously using a git
checkout of ecfca41e9d998a0f21ce7a4bce1dc78c58c3e015 with some of the Lua
patches attached. That was working just fine (except for one thing I'll
mention below). I rolled the RC and got this (and I was actually testing
for the issue I had with ecfca41e9d998a0f21ce7a4bce1dc78c58c3e015):

# doveadm -D acl set -u test1-shared at test.com INBOX user=test1 at test.com
read  list
Debug: Loading modules from directory: /usr/lib/dovecot/modules
Debug: Module loaded: /usr/lib/dovecot/modules/lib01_acl_plugin.so
Debug: Module loaded: /usr/lib/dovecot/modules/lib02_lazy_expunge_plugin.so
Debug: Module loaded: /usr/lib/dovecot/modules/lib10_quota_plugin.so
Debug: Module loaded: /usr/lib/dovecot/modules/lib20_fts_plugin.so
Debug: Module loaded: /usr/lib/dovecot/modules/lib20_virtual_plugin.so
Debug: Module loaded: /usr/lib/dovecot/modules/lib20_zlib_plugin.so
Debug: Module loaded: /usr/lib/dovecot/modules/lib21_fts_lucene_plugin.so
Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm
Debug: Module loaded:
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so
Debug: Skipping module doveadm_expire_plugin, because dlopen() failed:
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined
symbol: expire_set_deinit (this is usually intentional, so just ignore this
message)
Debug: Module loaded:
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so
Debug: Module loaded:
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so
Debug: Module loaded:
/usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so
Debug: Module loaded:
/usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so
Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed:
/usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so: undefined
symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so
just ignore this message)
doveadm(test1-shared at test.com)<37433><>: Debug: auth USER input:
test1-shared at test.com uid=200 home=/m/mail//7/1/fe/test.com/test1-shared
gid=200 quota=maildir:User quota mail_location=maildir:/m/mail//7/1/fe/
test.com/test1-shared/maildir
doveadm(test1-shared at test.com)<37433><>: Debug: Added userdb setting:
mail_location=maildir:/m/mail//7/1/fe/test.com/test1-shared/maildir
doveadm(test1-shared at test.com)<37433><>: Debug: Added userdb setting:
plugin/quota=maildir:User quota
doveadm(test1-shared at test.com): Debug: Effective uid=200, gid=200,
home=/m/mail//7/1/fe/test.com/test1-shared
doveadm(test1-shared at test.com): Panic: file buffer.c: line 97
(buffer_check_limits): assertion failed: (buf->used <= buf->alloc)
doveadm(test1-shared at test.com): Error: Raw backtrace:
/usr/lib/dovecot/libdovecot.so.0(+0xc5d7a) [0x6ba69ea77d7a] ->
/usr/lib/dovecot/libdovecot.so.0(+0xc5de9) [0x6ba69ea77de9] ->
/usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x6ba69e9ea521] ->
/usr/lib/dovecot/libdovecot.so.0(buffer_write+0x160) [0x6ba69ea72210] ->
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so(+0x43fe)
[0x6ba6997c33fe] ->
/usr/lib/dovecot/libdovecot-storage.so.0(hook_mail_user_created+0x20a)
[0x6ba69ed72a2a] ->
/usr/lib/dovecot/libdovecot-storage.so.0(mail_user_init+0x220)
[0x6ba69ed78870] ->
/usr/lib/dovecot/libdovecot-storage.so.0(mail_storage_service_next_with_session_suffix+0x494)
[0x6ba69ed76314] -> doveadm(+0x296ff) [0x5bde3fb56ff] -> doveadm(+0x2a3ab)
[0x5bde3fb63ab] -> doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x21d)
[0x5bde3fb71bd] -> doveadm(doveadm_cmd_run_ver2+0x575) [0x5bde3fc6c15] ->
doveadm(doveadm_cmd_try_run_ver2+0x37) [0x5bde3fc6c67] ->
doveadm(main+0x1d2) [0x5bde3fa5f52] ->
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x6ba69e609830] ->
doveadm(_start+0x29) [0x5bde3fa6329]
Aborted


This same command was working with ecfca41e9d998a0f21ce7a4bce1dc78c58c3e015
earlier today -- this is run from command line history. The issue I had
noticed with ecfca41e9d998a0f21ce7a4bce1dc78c58c3e015 (and was going to ask
about it today anyway) is that when I would use doveadm acl to add an ACL
with *just* 'read', dovecot would create the ACL file correctly, but using
a file-based acl_shared_dict, it would create a 0 byte acl_shared_dict
file. However using anything else for the ACL, e.g. 'read list', it would
create the acl_shared_dict file with what you'd expect. I don't know if
that's expected behavior or not (but couldn't find anything on google to
say so). So just to sum up: With just 'read', it'd create an empty
acl_shared_dict file; with 'read list' or anything else, it'd create a
working, non-empty acl_shared_dict file.

I was going to see if it was different in the RC but instead, there's just
the above error.

This is just a test rig, so I can try out whatever would be useful to
debugging. Happy to post doveconf -n, if helpful. The only config changes
from ecfca41e9d998a0f21ce7a4bce1dc78c58c3e015 to RC1 that I made were just
the things that needed to be done to let dovecot start up (removing some of
the stats stuff, getting rid of ssl_protocols, etc).