v2.3.16 released

Timo Sirainen

6 Aug 2021 6 Aug '21

1:42 p.m.

Hi,

One interesting thing in this release is the support for configuring OAUTH2 openid-configuration element. It would be nice if IMAP clients started supporting this feature to enable OAUTH2 for all IMAP servers, not just Gmail and a few others. This would allow all kinds of new authentication methods for IMAP and improve the authentication security in general.

https://dovecot.org/releases/2.3/dovecot-2.3.16.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.16.tar.gz.sig

Binary packages in https://repo.dovecot.org/ Docker images in https://hub.docker.com/r/dovecot/dovecot https://hub.docker.com/r/dovecot/dovecot

Any unexpected exit() will now result in a core dump. This can especially help notice problems when a Lua script causes exit(0).
auth-worker process is now restarted when the number of auth requests reaches service auth-worker { service_count }. The default is still unlimited.

Event improvements: Added data_stack_grow event and http-client category. See https://doc.dovecot.org/admin_manual/list_of_events/
oauth2: Support RFC 7628 openid-configuration element. This allows clients to support OAUTH2 for any server, not just a few hardcoded servers like they do now. See openid_configuration_url setting in dovecot-oauth2.conf.ext.
mysql: Single statements are no longer enclosed with BEGIN/COMMIT.
dovecot-sysreport --core supports multiple core files now and does not require specifying the binary path.
imapc: When imap_acl plugin is loaded and imapc_features=acl is used, IMAP ACL commands are proxied to the remote server. See https://doc.dovecot.org/configuration_manual/mail_location/imapc/
dict-sql now supports the "UPSERT" syntax for SQLite and PostgreSQL.
imap: If IMAP client disconnects during a COPY command, the copying is aborted, and changes are reverted. This may help to avoid many email duplicates if client disconnects during COPY and retries it after reconnecting.

master process was using 100% CPU if service attempted to create more processes due to process_min_avail, but process_limit was already reached. v2.3.15 regression.
Using attachment detection flags wrongly logged unnecessary "Failed to add attachment keywords" errors. v2.3.13 regression.
IMAP QRESYNC: Expunging UID 1 mail resulted in broken VANISHED response, which could have confused IMAP clients. v2.3.13 regression.
imap: STORE didn't send untagged replies for \Seen changes for (shared) mailboxes using INDEXPVT. v2.3.10 regression.
rawlog_dir setting would not log input that was pipelined after authentication command.
Fixed potential infinite looping with autoexpunging.
Log event exporter: Truncate long fields to 1000 bytes
LAYOUT=index: ACL inheritance didn't work when creating mailboxes
Event filters: Unquoted '?' wildcard caused a crash at startup
fs-metawrap: Fix to handling zero sized files
imap-hibernate: Fixed potential crash at deinit.
acl: dovecot-acl-list files were written for acl_ignore_namespaces
program-client (used by Sieve extprograms, director_flush_socket) may have missed status response from UNIX and network sockets, resulting in unexpected failures.

Attachments:

attachment.html (text/html — 4.9 KB)

Show replies by date

Joan Moreau

6 Aug 6 Aug

2:43 p.m.

New subject: [Dovecot-news] v2.3.16 released

Thank you Timo

However, this leads to

kernel: imap[228122]: segfault at 50 ip 00007f7015ee332b sp 00007fffa7178740 error 4 in lib20_fts_plugin.so[7f7015ee1000+11000]

Returning to 2.3.15 resolves the problem

On 2021-08-06 12:42, Timo Sirainen wrote:

...

Hi,

One interesting thing in this release is the support for configuring OAUTH2 openid-configuration element. It would be nice if IMAP clients started supporting this feature to enable OAUTH2 for all IMAP servers, not just Gmail and a few others. This would allow all kinds of new authentication methods for IMAP and improve the authentication security in general. https://dovecot.org/releases/2.3/dovecot-2.3.16.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.16.tar.gz.sig

Binary packages in https://repo.dovecot.org/ Docker images in https://hub.docker.com/r/dovecot/dovecot

Any unexpected exit() will now result in a core dump. This can especially help notice problems when a Lua script causes exit(0).

auth-worker process is now restarted when the number of auth requests reaches service auth-worker { service_count }. The default is still unlimited.

Event improvements: Added data_stack_grow event and http-client category. See https://doc.dovecot.org/admin_manual/list_of_events/

oauth2: Support RFC 7628 openid-configuration element. This allows clients to support OAUTH2 for any server, not just a few hardcoded servers like they do now. See openid_configuration_url setting in dovecot-oauth2.conf.ext.

mysql: Single statements are no longer enclosed with BEGIN/COMMIT.

dovecot-sysreport --core supports multiple core files now and does not require specifying the binary path.

imapc: When imap_acl plugin is loaded and imapc_features=acl is used, IMAP ACL commands are proxied to the remote server. See https://doc.dovecot.org/configuration_manual/mail_location/imapc/

dict-sql now supports the "UPSERT" syntax for SQLite and PostgreSQL.

imap: If IMAP client disconnects during a COPY command, the copying is aborted, and changes are reverted. This may help to avoid many email duplicates if client disconnects during COPY and retries it after reconnecting.

master process was using 100% CPU if service attempted to create more processes due to process_min_avail, but process_limit was already reached. v2.3.15 regression.

Using attachment detection flags wrongly logged unnecessary "Failed to add attachment keywords" errors. v2.3.13 regression.

IMAP QRESYNC: Expunging UID 1 mail resulted in broken VANISHED response, which could have confused IMAP clients. v2.3.13 regression.

imap: STORE didn't send untagged replies for \Seen changes for (shared) mailboxes using INDEXPVT. v2.3.10 regression.

rawlog_dir setting would not log input that was pipelined after authentication command.

Fixed potential infinite looping with autoexpunging.

Log event exporter: Truncate long fields to 1000 bytes

LAYOUT=index: ACL inheritance didn't work when creating mailboxes

Event filters: Unquoted '?' wildcard caused a crash at startup

fs-metawrap: Fix to handling zero sized files

imap-hibernate: Fixed potential crash at deinit.

acl: dovecot-acl-list files were written for acl_ignore_namespaces

program-client (used by Sieve extprograms, director_flush_socket) may have missed status response from UNIX and network sockets, resulting in unexpected failures.

Dovecot-news mailing list Dovecot-news@dovecot.org https://dovecot.org/mailman/listinfo/dovecot-news

Aki Tuomi

2:49 p.m.

New subject: [Dovecot-news] v2.3.16 released

...

On 06/08/2021 15:43 Joan Moreau jom@grosjo.net wrote:

Thank you Timo However, this leads to kernel: imap[228122]: segfault at 50 ip 00007f7015ee332b sp 00007fffa7178740 error 4 in lib20_fts_plugin.so[7f7015ee1000+11000] Returning to 2.3.15 resolves the problem

Can you provide gdb bt full output for the crash?

Aki

Joan Moreau

3:08 p.m.

New subject: [Dovecot-news] v2.3.16 released

Below

(gdb) bt full #0 fts_user_autoindex_exclude (box=<optimized out>, box@entry=0x55e0bc7e0fe8) at fts-user.c:347 fuser = <optimized out> #1 0x00007f42e8e9b4a6 in fts_mailbox_allocated (box=0x55e0bc7e0fe8) at fts-storage.c:806 flist = <optimized out> v = 0x55e0bc7e1010 fbox = 0x55e0bc7e1608 #2 0x00007f42e952652c in hook_mailbox_allocated (box=box@entry=0x55e0bc7e0fe8) at mail-storage-hooks.c:256 _data_stack_cur_id = 5 _foreach_end = 0x55e0bc7d28a0 _foreach_ptr = 0x55e0bc7d2890 hooks = 0x7f42e8ec9ba0 ctx = 0x55e0bc7e2818 #3 0x00007f42e95219c1 in mailbox_alloc (list=0x55e0bc7d97b8, vname=0x55e0bc78f608 "INBOX", flags=flags@entry=MAILBOX_FLAG_DROP_RECENT) at mail-storage.c:860 _data_stack_cur_id = 4 new_list = 0x55e0bc7d97b8 storage = 0x55e0bc7d9fc8 box = 0x55e0bc7e0fe8 open_error = MAIL_ERROR_NONE errstr = 0x0 __func__ = "mailbox_alloc" #4 0x000055e0bbd0a5c2 in select_open (readonly=false, mailbox=<optimized out>, ctx=0x55e0bc7d6fa0) at cmd-select.c:285 client = 0x55e0bc7d6298 status = {messages = 32, recent = 48, unseen = 814554448, uidvalidity = 32766, uidnext = 814554256, first_unseen_seq = 32766, first_recent_uid = 1633369088, last_cached_seq = 3805518085, highest_modseq = 0, highest_pvt_modseq = 139925357787644, keywords = 0x55e0bc78f398, permanent_flags = 0, flags = 0, permanent_keywords = false, allow_new_keywords = false, nonpermanent_modseqs = false, no_modseq_tracking = false, have_guids = false, have_save_guids = true, have_only_guid128 = false} flags = MAILBOX_FLAG_DROP_RECENT ret = 0 client = <optimized out> status = {messages = <optimized out>, recent = <optimized out>, unseen = <optimized out>, uidvalidity = <optimized out>, uidnext = <optimized out>, first_unseen_seq = <optimized out>, first_recent_uid = <optimized out>, last_cached_seq = <optimized out>, highest_modseq = <optimized out>, highest_pvt_modseq = <optimized out>, keywords = <optimized out>, permanent_flags = <optimized out>, flags = <optimized out>, permanent_keywords = <optimized out>, allow_new_keywords = <optimized out>, nonpermanent_modseqs = <optimized out>, no_modseq_tracking = <optimized out>, have_guids = <optimized out>, have_save_guids = <optimized out>, have_only_guid128 = <optimized out>} flags = <optimized out> ret = <optimized out> #5 cmd_select_full (cmd=<optimized out>, readonly=<optimized out>) at cmd-select.c:416 client = 0x55e0bc7d6298 ctx = 0x55e0bc7d6fa0 args = 0x55e0bc7a58d8 list_args = 0x7ffe308d1c74 mailbox = 0x55e0bc78f608 "INBOX" client_error = 0x1 ret = <optimized out> __func__ = "cmd_select_full" #6 0x000055e0bbd12484 in command_exec (cmd=cmd@entry=0x55e0bc7d6e08) at imap-commands.c:201 hook = 0x55e0bc79b5d0 finished = <optimized out> __func__ = "command_exec" #7 0x000055e0bbd104b2 in client_command_input (cmd=<optimized out>) at imap-client.c:1230 client = 0x55e0bc7d6298 command = <optimized out> tag = 0x7f42e942d8fa "]A\\A]\303\061\300\303ff.\017\037\204" name = 0x55e0bbd26e50 "SELECT" ret = <optimized out>

On 2021-08-06 13:49, Aki Tuomi wrote:

...

...
On 06/08/2021 15:43 Joan Moreau jom@grosjo.net wrote:

Thank you Timo However, this leads to kernel: imap[228122]: segfault at 50 ip 00007f7015ee332b sp 00007fffa7178740 error 4 in lib20_fts_plugin.so[7f7015ee1000+11000] Returning to 2.3.15 resolves the problem

Can you provide gdb bt full output for the crash?

Aki

Timo Sirainen

4:07 p.m.

New subject: [Dovecot-news] v2.3.16 released

On 6. Aug 2021, at 15.08, Joan Moreau jom@grosjo.net wrote:

...

Below

(gdb) bt full #0 fts_user_autoindex_exclude (box=<optimized out>, box@entry=0x55e0bc7e0fe8) at fts-user.c:347

There is no such function in 2.3.16 release. That's only in the current git master. What did you install and from where?

Joan Moreau

4:28 p.m.

New subject: [Dovecot-news] v2.3.16 released

git clone -b release-2.3.16

On 2021-08-06 15:07, Timo Sirainen wrote:

...

On 6. Aug 2021, at 15.08, Joan Moreau jom@grosjo.net wrote:

...
Below

(gdb) bt full #0 fts_user_autoindex_exclude (box=<optimized out>, box@entry=0x55e0bc7e0fe8) at fts-user.c:347

There is no such function in 2.3.16 release. That's only in the current git master. What did you install and from where?

Timo Sirainen

5:30 p.m.

New subject: [Dovecot-news] v2.3.16 released

I don't see anything wrong in git branch either:

% git clone -b release-2.3.16 https://github.com/dovecot/core.git https://github.com/dovecot/core.git % git grep fts_user_autoindex_exclude %

% git show commit 7e2e900c1a420006371e8a8cf9b4ddb36e69a986 Author: Timo Sirainen timo.sirainen@open-xchange.com Date: Thu Aug 5 18:25:31 2021 +0300

NEWS: Add 100% master process CPU item

I think somehow you built it from master instead of release-2.3.16.

...

On 6. Aug 2021, at 16.28, Joan Moreau jom@grosjo.net wrote:

git clone -b release-2.3.16

On 2021-08-06 15:07, Timo Sirainen wrote:

...
On 6. Aug 2021, at 15.08, Joan Moreau mailto:jom@grosjo.net> wrote:

...
Below

(gdb) bt full #0 fts_user_autoindex_exclude (box=<optimized out>, box@entry=0x55e0bc7e0fe8) at fts-user.c:347

There is no such function in 2.3.16 release. That's only in the current git master. What did you install and from where?

Joan Moreau

9 Aug 9 Aug

11:03 a.m.

New subject: [Dovecot-news] v2.3.16 released

Well, I do not think I am mistaken.

I also get the following error for "indexer" process

#0 0x00007f2370f7fe3d in o_stream_nsendv (stream=0x0, iov=iov@entry=0x7ffeb9dabd70, iov_count=iov_count@entry=1) at ostream.c:333 333 if (unlikely(stream->closed || stream->stream_errno != 0 || (gdb) bt full #0 0x00007f2370f7fe3d in o_stream_nsendv (stream=0x0, iov=iov@entry=0x7ffeb9dabd70, iov_count=iov_count@entry=1) at ostream.c:333 overflow = false #1 0x00007f2370f7feca in o_stream_nsend (stream=<optimized out>, data=<optimized out>, size=<optimized out>) at ostream.c:325 iov = {iov_base = 0x55b8af41d470, iov_len = 5} #2 0x00007f2370f7ff1a in o_stream_nsend_str (stream=<optimized out>, str=<optimized out>) at ostream.c:344 No locals. #3 0x000055b8af391f84 in indexer_client_status_callback (percentage=56, context=0x55b8af434b70) at indexer-client.c:146 _data_stack_cur_id = 4 ctx = 0x55b8af434b70 #4 0x000055b8af3921a0 in indexer_queue_request_status_int (queue=0x55b8af4299a0, request=0x55b8af434b90, percentage=56) at indexer-queue.c:182 context = <optimized out> i = 0 #5 0x000055b8af3919a2 in worker_status_callback (percentage=56, context=0x55b8af434cb0) at indexer.c:104 conn = 0x55b8af434cb0 request = 0x55b8af434b90 #6 0x000055b8af392ac4 in worker_connection_call_callback (percentage=<optimized out>, worker=0x55b8af434cb0) at worker-connection.c:42 No locals. #7 worker_connection_input_args (conn=0x55b8af434cb0, args=0x55b8af41d348) at worker-connection.c:109 worker = 0x55b8af434cb0 percentage = 56 ret = <optimized out> _tmp_event = <optimized out> #8 0x00007f2370f53853 in connection_input_default (conn=0x55b8af434cb0) at connection.c:95 _data_stack_cur_id = 3 line = 0x55b8af438625 "56" input = 0x55b8af436210 output = 0x55b8af436430 ret = 1 #9 0x00007f2370f71919 in io_loop_call_io (io=0x55b8af436550) at ioloop.c:727 ioloop = 0x55b8af425ec0 t_id = 2 __func__ = "io_loop_call_io" #10 0x00007f2370f72fc2 in io_loop_handler_run_internal (ioloop=ioloop@entry=0x55b8af425ec0) at ioloop-epoll.c:222

On 2021-08-06 13:49, Aki Tuomi wrote:

...

...
On 06/08/2021 15:43 Joan Moreau jom@grosjo.net wrote:

Thank you Timo However, this leads to kernel: imap[228122]: segfault at 50 ip 00007f7015ee332b sp 00007fffa7178740 error 4 in lib20_fts_plugin.so[7f7015ee1000+11000] Returning to 2.3.15 resolves the problem

Can you provide gdb bt full output for the crash?

Aki

Timo Sirainen

11:24 a.m.

New subject: [Dovecot-news] v2.3.16 released

On 9. Aug 2021, at 11.03, Joan Moreau jom@grosjo.net wrote:

...

#0 0x00007f2370f7fe3d in o_stream_nsendv (stream=0x0, iov=iov@entry=0x7ffeb9dabd70, iov_count=iov_count@entry=1) at ostream.c:333

overflow = false #1 0x00007f2370f7feca in o_stream_nsend (stream=<optimized out>, data=<optimized out>, size=<optimized out>) at ostream.c:325 iov = {iov_base = 0x55b8af41d470, iov_len = 5} #2 0x00007f2370f7ff1a in o_stream_nsend_str (stream=<optimized out>, str=<optimized out>) at ostream.c:344 No locals. #3 0x000055b8af391f84 in indexer_client_status_callback (percentage=56, context=0x55b8af434b70) at indexer-client.c:146 _data_stack_cur_id = 4 ctx = 0x55b8af434b70 #4 0x000055b8af3921a0 in indexer_queue_request_status_int (queue=0x55b8af4299a0, request=0x55b8af434b90, percentage=56) at indexer-queue.c:182 context = <optimized out>

Looks like v2.3.15 already broke this. Happens when indexer-client disconnects early. Hopefully doesn't happen very often.

Timo Sirainen

11:27 a.m.

New subject: [Dovecot-news] v2.3.16 released

On 9. Aug 2021, at 11.24, Timo Sirainen timo@sirainen.com wrote:

...

On 9. Aug 2021, at 11.03, Joan Moreau mailto:jom@grosjo.net> wrote:

...
#0 0x00007f2370f7fe3d in o_stream_nsendv (stream=0x0, iov=iov@entry=0x7ffeb9dabd70, iov_count=iov_count@entry=1) at ostream.c:333

overflow = false #1 0x00007f2370f7feca in o_stream_nsend (stream=<optimized out>, data=<optimized out>, size=<optimized out>) at ostream.c:325 iov = {iov_base = 0x55b8af41d470, iov_len = 5} #2 0x00007f2370f7ff1a in o_stream_nsend_str (stream=<optimized out>, str=<optimized out>) at ostream.c:344 No locals. #3 0x000055b8af391f84 in indexer_client_status_callback (percentage=56, context=0x55b8af434b70) at indexer-client.c:146 _data_stack_cur_id = 4 ctx = 0x55b8af434b70 #4 0x000055b8af3921a0 in indexer_queue_request_status_int (queue=0x55b8af4299a0, request=0x55b8af434b90, percentage=56) at indexer-queue.c:182 context = <optimized out>

Looks like v2.3.15 already broke this. Happens when indexer-client disconnects early. Hopefully doesn't happen very often.

Oh, actually v2.3.15.1, but looks like it wasn't even released to community.

Joan Moreau

11:32 a.m.

New subject: [Dovecot-news] v2.3.16 released

Well, I don't really understand your note.

Bottom-line : 2.3.16 crashes every now and then.

Maybe is there a quick fix for production servers ?

On 2021-08-09 10:27, Timo Sirainen wrote:

...

On 9. Aug 2021, at 11.24, Timo Sirainen timo@sirainen.com wrote:

On 9. Aug 2021, at 11.03, Joan Moreau jom@grosjo.net wrote:

#0 0x00007f2370f7fe3d in o_stream_nsendv (stream=0x0, iov=iov@entry=0x7ffeb9dabd70, iov_count=iov_count@entry=1) at ostream.c:333

overflow = false #1 0x00007f2370f7feca in o_stream_nsend (stream=<optimized out>, data=<optimized out>, size=<optimized out>) at ostream.c:325 iov = {iov_base = 0x55b8af41d470, iov_len = 5} #2 0x00007f2370f7ff1a in o_stream_nsend_str (stream=<optimized out>, str=<optimized out>) at ostream.c:344 No locals. #3 0x000055b8af391f84 in indexer_client_status_callback (percentage=56, context=0x55b8af434b70) at indexer-client.c:146 _data_stack_cur_id = 4 ctx = 0x55b8af434b70 #4 0x000055b8af3921a0 in indexer_queue_request_status_int (queue=0x55b8af4299a0, request=0x55b8af434b90, percentage=56) at indexer-queue.c:182 context = <optimized out>

Looks like v2.3.15 already broke this. Happens when indexer-client disconnects early. Hopefully doesn't happen very often.

Oh, actually v2.3.15.1, but looks like it wasn't even released to community.

Michael Ströder

11:58 a.m.

New subject: [Dovecot-news] v2.3.16 released

On 8/9/21 11:32 AM, Joan Moreau wrote:

...

Bottom-line : 2.3.16 crashes every now and then.

I see occasional failures in the test suite on openSUSE build system (OBS):

https://build.opensuse.org/package/show/home:stroeder:network/dovecot23

But I'm not sure whether that's a timing-related red hering specific for running the tests on OBS.

Ciao, Michael.

Timo Sirainen

12:19 p.m.

New subject: [Dovecot-news] v2.3.16 released

On 9. Aug 2021, at 11.58, Michael Ströder michael@stroeder.com wrote:

...

On 8/9/21 11:32 AM, Joan Moreau wrote:

...
Bottom-line : 2.3.16 crashes every now and then.

I see occasional failures in the test suite on openSUSE build system (OBS):

https://build.opensuse.org/package/show/home:stroeder:network/dovecot23

But I'm not sure whether that's a timing-related red hering specific for running the tests on OBS.

The test-cpu-limit failures are a bit annoying. Probably has something to do with overloaded VMs. Maybe we need to just disable those failing asserts entirely since they seem to pop up once in a while.

These probably have something to do with big endian CPUs. Would need to debug further:

[ 762s] test-mail-cache-fields.c:50: Assert failed: cache_field.last_used == priv->field.last_used && cache_field.decision == priv->field.decision [ 762s] test-mail-cache-fields.c:65: Assert failed: cache_field.last_used == priv->field.last_used && cache_field.decision == priv->field.decision [ 762s] test-mail-cache-fields.c:94: Assert failed: cache_field.last_used == priv->field.last_used && cache_field.decision == priv->field.decision

This seems to happen because setrlimit(RLIMIT_AS) is somehow not working as expected in the RISCV target - not sure what could be done about that other than disabling the test entirely:

[ 9496s] test-file-cache.c:266: Assert failed: file_cache_set_size(cache, 1024) == -1 [ 9496s] Panic: file test-common.c: line 211 (test_expect_error_string_n_times): assertion failed: (expected_errors == 0)

Timo Sirainen

12:10 p.m.

New subject: [Dovecot-news] v2.3.16 released

On 9. Aug 2021, at 11.32, Joan Moreau jom@grosjo.net wrote:

...

Bottom-line : 2.3.16 crashes every now and then. Maybe is there a quick fix for production servers ?

Attached a fix.

Daniel J. Luke

6 Aug 6 Aug

8:03 p.m.

New subject: [Dovecot-news] v2.3.16 released

On Aug 6, 2021, at 7:42 AM, Timo Sirainen timo@sirainen.com wrote:

...

https://dovecot.org/releases/2.3/dovecot-2.3.16.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.16.tar.gz.sig

The patch from https://dovecot.org/pipermail/dovecot/2021-June/122375.html is still necessary to build on newer Mac OS X (for the same reason). Tested on Mac OS version 11.5.1.

-- Daniel J. Luke

dovecot＠ptld.com

8 Aug 8 Aug

2:54 a.m.

Was the update issue sorted? Is it safe to update or was/is there a glitch? Or did the person with the issue update from the wrong branch?

And just to confirm, based on the change log in this email it doesn't look like we need to change anything in our config for this update? (Coming from 2.3.15)

Christian Kivalo

8:53 a.m.

On August 8, 2021 2:54:30 AM GMT+02:00, dovecot@ptld.com wrote:

...

Was the update issue sorted? Is it safe to update or was/is there a glitch? Had no problem upgrading here.

-- Christian Kivalo

1200

Age (days ago)

1203

Last active (days ago)

List overview

16 comments

7 participants

participants (7)

Aki Tuomi
Christian Kivalo
Daniel J. Luke
dovecot＠ptld.com
Joan Moreau
Michael Ströder
Timo Sirainen

v2.3.16 released

tags

participants (7)