Hi,
One interesting thing in this release is the support for configuring OAUTH2 openid-configuration element. It would be nice if IMAP clients started supporting this feature to enable OAUTH2 for all IMAP servers, not just Gmail and a few others. This would allow all kinds of new authentication methods for IMAP and improve the authentication security in general.
https://dovecot.org/releases/2.3/dovecot-2.3.16.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.16.tar.gz.sig
Binary packages in https://repo.dovecot.org/ Docker images in https://hub.docker.com/r/dovecot/dovecot <https://hub.docker.com/r/dovecot/dovecot>
- Any unexpected exit() will now result in a core dump. This can especially help notice problems when a Lua script causes exit(0).
- auth-worker process is now restarted when the number of auth requests reaches service auth-worker { service_count }. The default is still unlimited.
- Event improvements: Added data_stack_grow event and http-client category. See https://doc.dovecot.org/admin_manual/list_of_events/
- oauth2: Support RFC 7628 openid-configuration element. This allows clients to support OAUTH2 for any server, not just a few hardcoded servers like they do now. See openid_configuration_url setting in dovecot-oauth2.conf.ext.
- mysql: Single statements are no longer enclosed with BEGIN/COMMIT.
- dovecot-sysreport --core supports multiple core files now and does not require specifying the binary path.
- imapc: When imap_acl plugin is loaded and imapc_features=acl is used, IMAP ACL commands are proxied to the remote server. See https://doc.dovecot.org/configuration_manual/mail_location/imapc/
- dict-sql now supports the "UPSERT" syntax for SQLite and PostgreSQL.
- imap: If IMAP client disconnects during a COPY command, the copying is aborted, and changes are reverted. This may help to avoid many email duplicates if client disconnects during COPY and retries it after reconnecting.
- master process was using 100% CPU if service attempted to create more processes due to process_min_avail, but process_limit was already reached. v2.3.15 regression.
- Using attachment detection flags wrongly logged unnecessary "Failed to add attachment keywords" errors. v2.3.13 regression.
- IMAP QRESYNC: Expunging UID 1 mail resulted in broken VANISHED response, which could have confused IMAP clients. v2.3.13 regression.
- imap: STORE didn't send untagged replies for \Seen changes for (shared) mailboxes using INDEXPVT. v2.3.10 regression.
- rawlog_dir setting would not log input that was pipelined after authentication command.
- Fixed potential infinite looping with autoexpunging.
- Log event exporter: Truncate long fields to 1000 bytes
- LAYOUT=index: ACL inheritance didn't work when creating mailboxes
- Event filters: Unquoted '?' wildcard caused a crash at startup
- fs-metawrap: Fix to handling zero sized files
- imap-hibernate: Fixed potential crash at deinit.
- acl: dovecot-acl-list files were written for acl_ignore_namespaces
- program-client (used by Sieve extprograms, director_flush_socket) may have missed status response from UNIX and network sockets, resulting in unexpected failures.
Thank you Timo
However, this leads to
kernel: imap[228122]: segfault at 50 ip 00007f7015ee332b sp 00007fffa7178740 error 4 in lib20_fts_plugin.so[7f7015ee1000+11000]
Returning to 2.3.15 resolves the problem
On 2021-08-06 12:42, Timo Sirainen wrote:
Hi,
One interesting thing in this release is the support for configuring OAUTH2 openid-configuration element. It would be nice if IMAP clients started supporting this feature to enable OAUTH2 for all IMAP servers, not just Gmail and a few others. This would allow all kinds of new authentication methods for IMAP and improve the authentication security in general. https://dovecot.org/releases/2.3/dovecot-2.3.16.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.16.tar.gz.sig
Binary packages in https://repo.dovecot.org/ Docker images in https://hub.docker.com/r/dovecot/dovecot
- Any unexpected exit() will now result in a core dump. This can especially help notice problems when a Lua script causes exit(0).
- auth-worker process is now restarted when the number of auth requests reaches service auth-worker { service_count }. The default is still unlimited.
- Event improvements: Added data_stack_grow event and http-client category. See https://doc.dovecot.org/admin_manual/list_of_events/
- oauth2: Support RFC 7628 openid-configuration element. This allows clients to support OAUTH2 for any server, not just a few hardcoded servers like they do now. See openid_configuration_url setting in dovecot-oauth2.conf.ext.
- mysql: Single statements are no longer enclosed with BEGIN/COMMIT.
- dovecot-sysreport --core supports multiple core files now and does not require specifying the binary path.
- imapc: When imap_acl plugin is loaded and imapc_features=acl is used, IMAP ACL commands are proxied to the remote server. See https://doc.dovecot.org/configuration_manual/mail_location/imapc/
- dict-sql now supports the "UPSERT" syntax for SQLite and PostgreSQL.
- imap: If IMAP client disconnects during a COPY command, the copying is aborted, and changes are reverted. This may help to avoid many email duplicates if client disconnects during COPY and retries it after reconnecting.
- master process was using 100% CPU if service attempted to create more processes due to process_min_avail, but process_limit was already reached. v2.3.15 regression.
- Using attachment detection flags wrongly logged unnecessary "Failed to add attachment keywords" errors. v2.3.13 regression.
- IMAP QRESYNC: Expunging UID 1 mail resulted in broken VANISHED response, which could have confused IMAP clients. v2.3.13 regression.
- imap: STORE didn't send untagged replies for \Seen changes for (shared) mailboxes using INDEXPVT. v2.3.10 regression.
- rawlog_dir setting would not log input that was pipelined after authentication command.
- Fixed potential infinite looping with autoexpunging.
- Log event exporter: Truncate long fields to 1000 bytes
- LAYOUT=index: ACL inheritance didn't work when creating mailboxes
- Event filters: Unquoted '?' wildcard caused a crash at startup
- fs-metawrap: Fix to handling zero sized files
- imap-hibernate: Fixed potential crash at deinit.
- acl: dovecot-acl-list files were written for acl_ignore_namespaces
- program-client (used by Sieve extprograms, director_flush_socket) may have missed status response from UNIX and network sockets, resulting in unexpected failures.
Dovecot-news mailing list Dovecot-news@dovecot.org https://dovecot.org/mailman/listinfo/dovecot-news
On 06/08/2021 15:43 Joan Moreau <jom@grosjo.net> wrote:
Thank you Timo However, this leads to kernel: imap[228122]: segfault at 50 ip 00007f7015ee332b sp 00007fffa7178740 error 4 in lib20_fts_plugin.so[7f7015ee1000+11000] Returning to 2.3.15 resolves the problem
Can you provide gdb bt full output for the crash?
Aki
Below
(gdb) bt full #0 fts_user_autoindex_exclude (box=<optimized out>, box@entry=0x55e0bc7e0fe8) at fts-user.c:347 fuser = <optimized out> #1 0x00007f42e8e9b4a6 in fts_mailbox_allocated (box=0x55e0bc7e0fe8) at fts-storage.c:806 flist = <optimized out> v = 0x55e0bc7e1010 fbox = 0x55e0bc7e1608 #2 0x00007f42e952652c in hook_mailbox_allocated (box=box@entry=0x55e0bc7e0fe8) at mail-storage-hooks.c:256 _data_stack_cur_id = 5 _foreach_end = 0x55e0bc7d28a0 _foreach_ptr = 0x55e0bc7d2890 hooks = 0x7f42e8ec9ba0 <fts_mail_storage_hooks> ctx = 0x55e0bc7e2818 #3 0x00007f42e95219c1 in mailbox_alloc (list=0x55e0bc7d97b8, vname=0x55e0bc78f608 "INBOX", flags=flags@entry=MAILBOX_FLAG_DROP_RECENT) at mail-storage.c:860 _data_stack_cur_id = 4 new_list = 0x55e0bc7d97b8 storage = 0x55e0bc7d9fc8 box = 0x55e0bc7e0fe8 open_error = MAIL_ERROR_NONE errstr = 0x0 __func__ = "mailbox_alloc" #4 0x000055e0bbd0a5c2 in select_open (readonly=false, mailbox=<optimized out>, ctx=0x55e0bc7d6fa0) at cmd-select.c:285 client = 0x55e0bc7d6298 status = {messages = 32, recent = 48, unseen = 814554448, uidvalidity = 32766, uidnext = 814554256, first_unseen_seq = 32766, first_recent_uid = 1633369088, last_cached_seq = 3805518085, highest_modseq = 0, highest_pvt_modseq = 139925357787644, keywords = 0x55e0bc78f398, permanent_flags = 0, flags = 0, permanent_keywords = false, allow_new_keywords = false, nonpermanent_modseqs = false, no_modseq_tracking = false, have_guids = false, have_save_guids = true, have_only_guid128 = false} flags = MAILBOX_FLAG_DROP_RECENT ret = 0 client = <optimized out> status = {messages = <optimized out>, recent = <optimized out>, unseen = <optimized out>, uidvalidity = <optimized out>, uidnext = <optimized out>, first_unseen_seq = <optimized out>, first_recent_uid = <optimized out>, last_cached_seq = <optimized out>, highest_modseq = <optimized out>, highest_pvt_modseq = <optimized out>, keywords = <optimized out>, permanent_flags = <optimized out>, flags = <optimized out>, permanent_keywords = <optimized out>, allow_new_keywords = <optimized out>, nonpermanent_modseqs = <optimized out>, no_modseq_tracking = <optimized out>, have_guids = <optimized out>, have_save_guids = <optimized out>, have_only_guid128 = <optimized out>} flags = <optimized out> ret = <optimized out> #5 cmd_select_full (cmd=<optimized out>, readonly=<optimized out>) at cmd-select.c:416 client = 0x55e0bc7d6298 ctx = 0x55e0bc7d6fa0 args = 0x55e0bc7a58d8 list_args = 0x7ffe308d1c74 mailbox = 0x55e0bc78f608 "INBOX" client_error = 0x1 <error: Cannot access memory at address 0x1> ret = <optimized out> __func__ = "cmd_select_full" #6 0x000055e0bbd12484 in command_exec (cmd=cmd@entry=0x55e0bc7d6e08) at imap-commands.c:201 hook = 0x55e0bc79b5d0 finished = <optimized out> __func__ = "command_exec" #7 0x000055e0bbd104b2 in client_command_input (cmd=<optimized out>) at imap-client.c:1230 client = 0x55e0bc7d6298 command = <optimized out> tag = 0x7f42e942d8fa <p_strdup+74> "]A\\A]\303\061\300\303ff.\017\037\204" name = 0x55e0bbd26e50 "SELECT" ret = <optimized out>
On 2021-08-06 13:49, Aki Tuomi wrote:
On 06/08/2021 15:43 Joan Moreau <jom@grosjo.net> wrote:
Thank you Timo However, this leads to kernel: imap[228122]: segfault at 50 ip 00007f7015ee332b sp 00007fffa7178740 error 4 in lib20_fts_plugin.so[7f7015ee1000+11000] Returning to 2.3.15 resolves the problem
Can you provide
gdb bt fulloutput for the crash?Aki
On 6. Aug 2021, at 15.08, Joan Moreau <jom@grosjo.net> wrote:
Below
(gdb) bt full #0 fts_user_autoindex_exclude (box=<optimized out>, box@entry=0x55e0bc7e0fe8) at fts-user.c:347
There is no such function in 2.3.16 release. That's only in the current git master. What did you install and from where?
git clone -b release-2.3.16
On 2021-08-06 15:07, Timo Sirainen wrote:
On 6. Aug 2021, at 15.08, Joan Moreau <jom@grosjo.net> wrote:
Below
(gdb) bt full #0 fts_user_autoindex_exclude (box=<optimized out>, box@entry=0x55e0bc7e0fe8) at fts-user.c:347
There is no such function in 2.3.16 release. That's only in the current git master. What did you install and from where?
I don't see anything wrong in git branch either:
% git clone -b release-2.3.16 https://github.com/dovecot/core.git <https://github.com/dovecot/core.git> % git grep fts_user_autoindex_exclude %
% git show commit 7e2e900c1a420006371e8a8cf9b4ddb36e69a986 Author: Timo Sirainen <timo.sirainen@open-xchange.com> Date: Thu Aug 5 18:25:31 2021 +0300
NEWS: Add 100% master process CPU itemI think somehow you built it from master instead of release-2.3.16.
On 6. Aug 2021, at 16.28, Joan Moreau <jom@grosjo.net> wrote:
git clone -b release-2.3.16
On 2021-08-06 15:07, Timo Sirainen wrote:
On 6. Aug 2021, at 15.08, Joan Moreau <jom@grosjo.net <mailto:jom@grosjo.net>> wrote:
Below
(gdb) bt full #0 fts_user_autoindex_exclude (box=<optimized out>, box@entry=0x55e0bc7e0fe8) at fts-user.c:347
There is no such function in 2.3.16 release. That's only in the current git master. What did you install and from where?
Well, I do not think I am mistaken.
I also get the following error for "indexer" process
#0 0x00007f2370f7fe3d in o_stream_nsendv (stream=0x0, iov=iov@entry=0x7ffeb9dabd70, iov_count=iov_count@entry=1) at ostream.c:333 333 if (unlikely(stream->closed || stream->stream_errno != 0 || (gdb) bt full #0 0x00007f2370f7fe3d in o_stream_nsendv (stream=0x0, iov=iov@entry=0x7ffeb9dabd70, iov_count=iov_count@entry=1) at ostream.c:333 overflow = false #1 0x00007f2370f7feca in o_stream_nsend (stream=<optimized out>, data=<optimized out>, size=<optimized out>) at ostream.c:325 iov = {iov_base = 0x55b8af41d470, iov_len = 5} #2 0x00007f2370f7ff1a in o_stream_nsend_str (stream=<optimized out>, str=<optimized out>) at ostream.c:344 No locals. #3 0x000055b8af391f84 in indexer_client_status_callback (percentage=56, context=0x55b8af434b70) at indexer-client.c:146 _data_stack_cur_id = 4 ctx = 0x55b8af434b70 #4 0x000055b8af3921a0 in indexer_queue_request_status_int (queue=0x55b8af4299a0, request=0x55b8af434b90, percentage=56) at indexer-queue.c:182 context = <optimized out> i = 0 #5 0x000055b8af3919a2 in worker_status_callback (percentage=56, context=0x55b8af434cb0) at indexer.c:104 conn = 0x55b8af434cb0 request = 0x55b8af434b90 #6 0x000055b8af392ac4 in worker_connection_call_callback (percentage=<optimized out>, worker=0x55b8af434cb0) at worker-connection.c:42 No locals. #7 worker_connection_input_args (conn=0x55b8af434cb0, args=0x55b8af41d348) at worker-connection.c:109 worker = 0x55b8af434cb0 percentage = 56 ret = <optimized out> _tmp_event = <optimized out> #8 0x00007f2370f53853 in connection_input_default (conn=0x55b8af434cb0) at connection.c:95 _data_stack_cur_id = 3 line = 0x55b8af438625 "56" input = 0x55b8af436210 output = 0x55b8af436430 ret = 1 #9 0x00007f2370f71919 in io_loop_call_io (io=0x55b8af436550) at ioloop.c:727 ioloop = 0x55b8af425ec0 t_id = 2 __func__ = "io_loop_call_io" #10 0x00007f2370f72fc2 in io_loop_handler_run_internal (ioloop=ioloop@entry=0x55b8af425ec0) at ioloop-epoll.c:222
On 2021-08-06 13:49, Aki Tuomi wrote:
On 06/08/2021 15:43 Joan Moreau <jom@grosjo.net> wrote:
Thank you Timo However, this leads to kernel: imap[228122]: segfault at 50 ip 00007f7015ee332b sp 00007fffa7178740 error 4 in lib20_fts_plugin.so[7f7015ee1000+11000] Returning to 2.3.15 resolves the problem
Can you provide
gdb bt fulloutput for the crash?Aki
On 9. Aug 2021, at 11.03, Joan Moreau <jom@grosjo.net> wrote:
#0 0x00007f2370f7fe3d in o_stream_nsendv (stream=0x0, iov=iov@entry=0x7ffeb9dabd70, iov_count=iov_count@entry=1) at ostream.c:333
overflow = false #1 0x00007f2370f7feca in o_stream_nsend (stream=<optimized out>, data=<optimized out>, size=<optimized out>) at ostream.c:325 iov = {iov_base = 0x55b8af41d470, iov_len = 5} #2 0x00007f2370f7ff1a in o_stream_nsend_str (stream=<optimized out>, str=<optimized out>) at ostream.c:344 No locals. #3 0x000055b8af391f84 in indexer_client_status_callback (percentage=56, context=0x55b8af434b70) at indexer-client.c:146 _data_stack_cur_id = 4 ctx = 0x55b8af434b70 #4 0x000055b8af3921a0 in indexer_queue_request_status_int (queue=0x55b8af4299a0, request=0x55b8af434b90, percentage=56) at indexer-queue.c:182 context = <optimized out>
Looks like v2.3.15 already broke this. Happens when indexer-client disconnects early. Hopefully doesn't happen very often.
On 9. Aug 2021, at 11.24, Timo Sirainen <timo@sirainen.com> wrote:
On 9. Aug 2021, at 11.03, Joan Moreau <jom@grosjo.net <mailto:jom@grosjo.net>> wrote:
#0 0x00007f2370f7fe3d in o_stream_nsendv (stream=0x0, iov=iov@entry=0x7ffeb9dabd70, iov_count=iov_count@entry=1) at ostream.c:333
overflow = false #1 0x00007f2370f7feca in o_stream_nsend (stream=<optimized out>, data=<optimized out>, size=<optimized out>) at ostream.c:325 iov = {iov_base = 0x55b8af41d470, iov_len = 5} #2 0x00007f2370f7ff1a in o_stream_nsend_str (stream=<optimized out>, str=<optimized out>) at ostream.c:344 No locals. #3 0x000055b8af391f84 in indexer_client_status_callback (percentage=56, context=0x55b8af434b70) at indexer-client.c:146 _data_stack_cur_id = 4 ctx = 0x55b8af434b70 #4 0x000055b8af3921a0 in indexer_queue_request_status_int (queue=0x55b8af4299a0, request=0x55b8af434b90, percentage=56) at indexer-queue.c:182 context = <optimized out>
Looks like v2.3.15 already broke this. Happens when indexer-client disconnects early. Hopefully doesn't happen very often.
Oh, actually v2.3.15.1, but looks like it wasn't even released to community.
Well, I don't really understand your note.
Bottom-line : 2.3.16 crashes every now and then.
Maybe is there a quick fix for production servers ?
On 2021-08-09 10:27, Timo Sirainen wrote:
On 9. Aug 2021, at 11.24, Timo Sirainen <timo@sirainen.com> wrote:
On 9. Aug 2021, at 11.03, Joan Moreau <jom@grosjo.net> wrote:
#0 0x00007f2370f7fe3d in o_stream_nsendv (stream=0x0, iov=iov@entry=0x7ffeb9dabd70, iov_count=iov_count@entry=1) at ostream.c:333
overflow = false #1 0x00007f2370f7feca in o_stream_nsend (stream=<optimized out>, data=<optimized out>, size=<optimized out>) at ostream.c:325 iov = {iov_base = 0x55b8af41d470, iov_len = 5} #2 0x00007f2370f7ff1a in o_stream_nsend_str (stream=<optimized out>, str=<optimized out>) at ostream.c:344 No locals. #3 0x000055b8af391f84 in indexer_client_status_callback (percentage=56, context=0x55b8af434b70) at indexer-client.c:146 _data_stack_cur_id = 4 ctx = 0x55b8af434b70 #4 0x000055b8af3921a0 in indexer_queue_request_status_int (queue=0x55b8af4299a0, request=0x55b8af434b90, percentage=56) at indexer-queue.c:182 context = <optimized out>
Looks like v2.3.15 already broke this. Happens when indexer-client disconnects early. Hopefully doesn't happen very often.
Oh, actually v2.3.15.1, but looks like it wasn't even released to community.
On 8/9/21 11:32 AM, Joan Moreau wrote:
Bottom-line : 2.3.16 crashes every now and then.
I see occasional failures in the test suite on openSUSE build system (OBS):
https://build.opensuse.org/package/show/home:stroeder:network/dovecot23
But I'm not sure whether that's a timing-related red hering specific for running the tests on OBS.
Ciao, Michael.
On 9. Aug 2021, at 11.58, Michael Ströder <michael@stroeder.com> wrote:
On 8/9/21 11:32 AM, Joan Moreau wrote:
Bottom-line : 2.3.16 crashes every now and then.
I see occasional failures in the test suite on openSUSE build system (OBS):
https://build.opensuse.org/package/show/home:stroeder:network/dovecot23
But I'm not sure whether that's a timing-related red hering specific for running the tests on OBS.
The test-cpu-limit failures are a bit annoying. Probably has something to do with overloaded VMs. Maybe we need to just disable those failing asserts entirely since they seem to pop up once in a while.
These probably have something to do with big endian CPUs. Would need to debug further:
[ 762s] test-mail-cache-fields.c:50: Assert failed: cache_field.last_used == priv->field.last_used && cache_field.decision == priv->field.decision [ 762s] test-mail-cache-fields.c:65: Assert failed: cache_field.last_used == priv->field.last_used && cache_field.decision == priv->field.decision [ 762s] test-mail-cache-fields.c:94: Assert failed: cache_field.last_used == priv->field.last_used && cache_field.decision == priv->field.decision
This seems to happen because setrlimit(RLIMIT_AS) is somehow not working as expected in the RISCV target - not sure what could be done about that other than disabling the test entirely:
[ 9496s] test-file-cache.c:266: Assert failed: file_cache_set_size(cache, 1024) == -1 [ 9496s] Panic: file test-common.c: line 211 (test_expect_error_string_n_times): assertion failed: (expected_errors == 0)
On Aug 6, 2021, at 7:42 AM, Timo Sirainen <timo@sirainen.com> wrote:
https://dovecot.org/releases/2.3/dovecot-2.3.16.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.16.tar.gz.sig
The patch from https://dovecot.org/pipermail/dovecot/2021-June/122375.html is still necessary to build on newer Mac OS X (for the same reason). Tested on Mac OS version 11.5.1.
-- Daniel J. Luke
Was the update issue sorted? Is it safe to update or was/is there a glitch? Or did the person with the issue update from the wrong branch?
And just to confirm, based on the change log in this email it doesn't look like we need to change anything in our config for this update? (Coming from 2.3.15)
participants (7)
-
Aki Tuomi
-
Christian Kivalo
-
Daniel J. Luke
-
dovecot＠ptld.com
-
Joan Moreau
-
Michael Ströder
-
Timo Sirainen