v2.3.0 release candidate released

Tue Dec 19 00:31:58 EET 2017

On Mon, Dec 18, 2017 at 1:16 PM, Mark Moseley <moseleymark at gmail.com> wrote:

> On Mon, Dec 18, 2017 at 7:23 AM, Timo Sirainen <tss at iki.fi> wrote:
>
>> https://dovecot.org/releases/2.3/rc/dovecot-2.3.0.rc1.tar.gz
>> https://dovecot.org/releases/2.3/rc/dovecot-2.3.0.rc1.tar.gz.sig
>>
>> It's finally time for v2.3 release branch! There are several new and
>> exciting features in it. I'm especially happy about the new logging and
>> statistics code, which will allow us to generate statistics for just about
>> everything. We didn't have time to implement everything we wanted for them
>> yet, and there especially aren't all that many logging events yet that can
>> be used for statistics. We'll implement those to v2.3.1, which might also
>> mean that some of the APIs might still change in v2.3.1 if that's required.
>>
>> We also have new lib-smtp server code, which was used to implement SMTP
>> submission server and do a partial rewrite for LMTP server. Please test
>> these before v2.3.0 to make sure we don't have any bad bugs left!
>>
>> BTW. The v2.3.0 will most likely be signed with a new PGP key ED409DA1.
>>
>> Some of the larger changes:
>>
>>  * Various setting changes, see https://wiki2.dovecot.org/Upgrading/2.3
>>  * Logging rewrite started: Logging is now based on hierarchical events.
>>    This makes it possible to do various things, like: 1) giving
>>    consistent log prefixes, 2) enabling debug logging with finer
>>    granularity, 3) provide logs in more machine readable formats
>>    (e.g. json). Everything isn't finished yet, especially a lot of the
>>    old logging code still needs to be translated to the new way.
>>  * Statistics rewrite started: Stats are now based on (log) events.
>>    It's possible to gather statistics about any event that is logged.
>>    See http://wiki2.dovecot.org/Statistics for details
>>  * ssl_dh setting replaces the old generated ssl-parameters.dat
>>  * IMAP: When BINARY FETCH finds a broken mails, send [PARSE] error
>>    instead of [UNKNOWNCTE]
>>  * Linux: core dumping via PR_SET_DUMPABLE is no longer enabled by
>>    default due to potential security reasons (found by cPanel Security
>>    Team).
>>
>>  + Added support for SMTP submission proxy server, which includes
>>    support for BURL and CHUNKING extension.
>>  + LMTP rewrite. Supports now CHUNKING extension and mixing of
>>    local/proxy recipients.
>>  + auth: Support libsodium to add support for ARGON2I and ARGON2ID
>>    password schemes.
>>  + auth: Support BLF-CRYPT password scheme in all platforms
>>  + auth: Added LUA scripting support for passdb/userdb.
>>    See https://wiki2.dovecot.org/AuthDatabase/Lua
>>  - Input streams are more reliable now when there are errors or when
>>    the maximum buffer size is reached. Previously in some situations
>>    this could have caused Dovecot to try to read already freed memory.
>>  - Output streams weren't previously handling failures when writing a
>>    trailer at the end of the stream. This mainly affected encrypt and
>>    zlib compress ostreams, which could have silently written truncated
>>    files if the last write happened to fail (which shouldn't normally
>>    have ever happened).
>>  - virtual plugin: Fixed panic when fetching mails from virtual
>>    mailboxes with IMAP BINARY extension.
>>  - Many other smaller fixes
>>
>>
>
> No issue compilng (and very very excited about this release, esp the Lua
> code, which is already super useful).
>
> I did have this one issue so far with the RC. I was previously using a git
> checkout of ecfca41e9d998a0f21ce7a4bce1dc78c58c3e015 with some of the Lua
> patches attached. That was working just fine (except for one thing I'll
> mention below). I rolled the RC and got this (and I was actually testing
> for the issue I had with ecfca41e9d998a0f21ce7a4bce1dc78c58c3e015):
>
> # doveadm -D acl set -u test1-shared at test.com INBOX user=test1 at test.com
> read  list
> Debug: Loading modules from directory: /usr/lib/dovecot/modules
> Debug: Module loaded: /usr/lib/dovecot/modules/lib01_acl_plugin.so
> Debug: Module loaded: /usr/lib/dovecot/modules/
> lib02_lazy_expunge_plugin.so
> Debug: Module loaded: /usr/lib/dovecot/modules/lib10_quota_plugin.so
> Debug: Module loaded: /usr/lib/dovecot/modules/lib20_fts_plugin.so
> Debug: Module loaded: /usr/lib/dovecot/modules/lib20_virtual_plugin.so
> Debug: Module loaded: /usr/lib/dovecot/modules/lib20_zlib_plugin.so
> Debug: Module loaded: /usr/lib/dovecot/modules/lib21_fts_lucene_plugin.so
> Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm
> Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_
> plugin.so
> Debug: Skipping module doveadm_expire_plugin, because dlopen() failed:
> /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so:
> undefined symbol: expire_set_deinit (this is usually intentional, so just
> ignore this message)
> Debug: Module loaded: /usr/lib/dovecot/modules/
> doveadm/lib10_doveadm_quota_plugin.so
> Debug: Module loaded: /usr/lib/dovecot/modules/
> doveadm/lib10_doveadm_sieve_plugin.so
> Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_
> lucene_plugin.so
> Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_
> plugin.so
> Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed:
> /usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so:
> undefined symbol: mail_crypt_box_get_pvt_digests (this is usually
> intentional, so just ignore this message)
> doveadm(test1-shared at test.com)<37433><>: Debug: auth USER input:
> test1-shared at test.com uid=200 home=/m/mail//7/1/fe/test.com/test1-shared
> gid=200 quota=maildir:User quota mail_location=maildir:/m/mail//7/1/fe/
> test.com/test1-shared/maildir
> doveadm(test1-shared at test.com)<37433><>: Debug: Added userdb setting:
> mail_location=maildir:/m/mail//7/1/fe/test.com/test1-shared/maildir
> doveadm(test1-shared at test.com)<37433><>: Debug: Added userdb setting:
> plugin/quota=maildir:User quota
> doveadm(test1-shared at test.com): Debug: Effective uid=200, gid=200,
> home=/m/mail//7/1/fe/test.com/test1-shared
> doveadm(test1-shared at test.com): Panic: file buffer.c: line 97
> (buffer_check_limits): assertion failed: (buf->used <= buf->alloc)
> doveadm(test1-shared at test.com): Error: Raw backtrace:
> /usr/lib/dovecot/libdovecot.so.0(+0xc5d7a) [0x6ba69ea77d7a] ->
> /usr/lib/dovecot/libdovecot.so.0(+0xc5de9) [0x6ba69ea77de9] ->
> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x6ba69e9ea521] ->
> /usr/lib/dovecot/libdovecot.so.0(buffer_write+0x160) [0x6ba69ea72210] ->
> /usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so(+0x43fe)
> [0x6ba6997c33fe] -> /usr/lib/dovecot/libdovecot-
> storage.so.0(hook_mail_user_created+0x20a) [0x6ba69ed72a2a] ->
> /usr/lib/dovecot/libdovecot-storage.so.0(mail_user_init+0x220)
> [0x6ba69ed78870] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_storage_
> service_next_with_session_suffix+0x494) [0x6ba69ed76314] ->
> doveadm(+0x296ff) [0x5bde3fb56ff] -> doveadm(+0x2a3ab) [0x5bde3fb63ab] ->
> doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x21d) [0x5bde3fb71bd] ->
> doveadm(doveadm_cmd_run_ver2+0x575) [0x5bde3fc6c15] ->
> doveadm(doveadm_cmd_try_run_ver2+0x37) [0x5bde3fc6c67] ->
> doveadm(main+0x1d2) [0x5bde3fa5f52] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)
> [0x6ba69e609830] -> doveadm(_start+0x29) [0x5bde3fa6329]
> Aborted
>
>
> This same command was working with ecfca41e9d998a0f21ce7a4bce1dc78c58c3e015
> earlier today -- this is run from command line history. The issue I had
> noticed with ecfca41e9d998a0f21ce7a4bce1dc78c58c3e015 (and was going to
> ask about it today anyway) is that when I would use doveadm acl to add an
> ACL with *just* 'read', dovecot would create the ACL file correctly, but
> using a file-based acl_shared_dict, it would create a 0
> byte acl_shared_dict file. However using anything else for the ACL, e.g.
> 'read list', it would create the acl_shared_dict file with what you'd
> expect. I don't know if that's expected behavior or not (but couldn't find
> anything on google to say so). So just to sum up: With just 'read', it'd
> create an empty acl_shared_dict file; with 'read list' or anything else,
> it'd create a working, non-empty acl_shared_dict file.
>
> I was going to see if it was different in the RC but instead, there's just
> the above error.
>
> This is just a test rig, so I can try out whatever would be useful to
> debugging. Happy to post doveconf -n, if helpful. The only config changes
> from ecfca41e9d998a0f21ce7a4bce1dc78c58c3e015 to RC1 that I made were
> just the things that needed to be done to let dovecot start up (removing
> some of the stats stuff, getting rid of ssl_protocols, etc).
>

Sorry, the backtrace went away and the command completes successfully after
installing the 0.5.0rc1 pigeonhole.

The issue with the zero byte acl_shared_dict files from an acl set of just
'read' still persists. And the acl_shared_dict file is still created and
populated correctly with anything beyond just read. E.g. it's created and
populated if the only ACL is 'lookup'. I only started playing with shared
mailboxes in 2.3 so I don't have spot where I can test on 2.2. It's not
super critical since I'd always be using 'read write-seen' at least, but I
figured I'd bring it up.