Hi,
One interesting thing in this release is the support for configuring OAUTH2 openid-configuration element. It would be nice if IMAP clients started supporting this feature to enable OAUTH2 for all IMAP servers, not just Gmail and a few others. This would allow all kinds of new authentication methods for IMAP and improve the authentication security in general.
https://dovecot.org/releases/2.3/dovecot-2.3.16.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.16.tar.gz.sig
Binary packages in https://repo.dovecot.org/ Docker images in https://hub.docker.com/r/dovecot/dovecot https://hub.docker.com/r/dovecot/dovecot
- Any unexpected exit() will now result in a core dump. This can especially help notice problems when a Lua script causes exit(0).
- auth-worker process is now restarted when the number of auth requests reaches service auth-worker { service_count }. The default is still unlimited.
- Event improvements: Added data_stack_grow event and http-client category. See https://doc.dovecot.org/admin_manual/list_of_events/
- oauth2: Support RFC 7628 openid-configuration element. This allows clients to support OAUTH2 for any server, not just a few hardcoded servers like they do now. See openid_configuration_url setting in dovecot-oauth2.conf.ext.
- mysql: Single statements are no longer enclosed with BEGIN/COMMIT.
- dovecot-sysreport --core supports multiple core files now and does not require specifying the binary path.
- imapc: When imap_acl plugin is loaded and imapc_features=acl is used, IMAP ACL commands are proxied to the remote server. See https://doc.dovecot.org/configuration_manual/mail_location/imapc/
- dict-sql now supports the "UPSERT" syntax for SQLite and PostgreSQL.
- imap: If IMAP client disconnects during a COPY command, the copying is aborted, and changes are reverted. This may help to avoid many email duplicates if client disconnects during COPY and retries it after reconnecting.
- master process was using 100% CPU if service attempted to create more processes due to process_min_avail, but process_limit was already reached. v2.3.15 regression.
- Using attachment detection flags wrongly logged unnecessary "Failed to add attachment keywords" errors. v2.3.13 regression.
- IMAP QRESYNC: Expunging UID 1 mail resulted in broken VANISHED response, which could have confused IMAP clients. v2.3.13 regression.
- imap: STORE didn't send untagged replies for \Seen changes for (shared) mailboxes using INDEXPVT. v2.3.10 regression.
- rawlog_dir setting would not log input that was pipelined after authentication command.
- Fixed potential infinite looping with autoexpunging.
- Log event exporter: Truncate long fields to 1000 bytes
- LAYOUT=index: ACL inheritance didn't work when creating mailboxes
- Event filters: Unquoted '?' wildcard caused a crash at startup
- fs-metawrap: Fix to handling zero sized files
- imap-hibernate: Fixed potential crash at deinit.
- acl: dovecot-acl-list files were written for acl_ignore_namespaces
- program-client (used by Sieve extprograms, director_flush_socket) may have missed status response from UNIX and network sockets, resulting in unexpected failures.
Thank you Timo
However, this leads to
kernel: imap[228122]: segfault at 50 ip 00007f7015ee332b sp 00007fffa7178740 error 4 in lib20_fts_plugin.so[7f7015ee1000+11000]
Returning to 2.3.15 resolves the problem
On 2021-08-06 12:42, Timo Sirainen wrote:
Hi,
One interesting thing in this release is the support for configuring OAUTH2 openid-configuration element. It would be nice if IMAP clients started supporting this feature to enable OAUTH2 for all IMAP servers, not just Gmail and a few others. This would allow all kinds of new authentication methods for IMAP and improve the authentication security in general. https://dovecot.org/releases/2.3/dovecot-2.3.16.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.16.tar.gz.sig
Binary packages in https://repo.dovecot.org/ Docker images in https://hub.docker.com/r/dovecot/dovecot
- Any unexpected exit() will now result in a core dump. This can especially help notice problems when a Lua script causes exit(0).
- auth-worker process is now restarted when the number of auth requests reaches service auth-worker { service_count }. The default is still unlimited.
- Event improvements: Added data_stack_grow event and http-client category. See https://doc.dovecot.org/admin_manual/list_of_events/
- oauth2: Support RFC 7628 openid-configuration element. This allows clients to support OAUTH2 for any server, not just a few hardcoded servers like they do now. See openid_configuration_url setting in dovecot-oauth2.conf.ext.
- mysql: Single statements are no longer enclosed with BEGIN/COMMIT.
- dovecot-sysreport --core supports multiple core files now and does not require specifying the binary path.
- imapc: When imap_acl plugin is loaded and imapc_features=acl is used, IMAP ACL commands are proxied to the remote server. See https://doc.dovecot.org/configuration_manual/mail_location/imapc/
- dict-sql now supports the "UPSERT" syntax for SQLite and PostgreSQL.
- imap: If IMAP client disconnects during a COPY command, the copying is aborted, and changes are reverted. This may help to avoid many email duplicates if client disconnects during COPY and retries it after reconnecting.
- master process was using 100% CPU if service attempted to create more processes due to process_min_avail, but process_limit was already reached. v2.3.15 regression.
- Using attachment detection flags wrongly logged unnecessary "Failed to add attachment keywords" errors. v2.3.13 regression.
- IMAP QRESYNC: Expunging UID 1 mail resulted in broken VANISHED response, which could have confused IMAP clients. v2.3.13 regression.
- imap: STORE didn't send untagged replies for \Seen changes for (shared) mailboxes using INDEXPVT. v2.3.10 regression.
- rawlog_dir setting would not log input that was pipelined after authentication command.
- Fixed potential infinite looping with autoexpunging.
- Log event exporter: Truncate long fields to 1000 bytes
- LAYOUT=index: ACL inheritance didn't work when creating mailboxes
- Event filters: Unquoted '?' wildcard caused a crash at startup
- fs-metawrap: Fix to handling zero sized files
- imap-hibernate: Fixed potential crash at deinit.
- acl: dovecot-acl-list files were written for acl_ignore_namespaces
- program-client (used by Sieve extprograms, director_flush_socket) may have missed status response from UNIX and network sockets, resulting in unexpected failures.
Dovecot-news mailing list Dovecot-news@dovecot.org https://dovecot.org/mailman/listinfo/dovecot-news
On 06/08/2021 15:43 Joan Moreau jom@grosjo.net wrote:
Thank you Timo However, this leads to kernel: imap[228122]: segfault at 50 ip 00007f7015ee332b sp 00007fffa7178740 error 4 in lib20_fts_plugin.so[7f7015ee1000+11000] Returning to 2.3.15 resolves the problem
Can you provide gdb bt full
output for the crash?
Aki
Below
(gdb) bt full
#0 fts_user_autoindex_exclude (box=<optimized out>,
box@entry=0x55e0bc7e0fe8) at fts-user.c:347
fuser = <optimized out>
#1 0x00007f42e8e9b4a6 in fts_mailbox_allocated (box=0x55e0bc7e0fe8) at
fts-storage.c:806
flist = <optimized out>
v = 0x55e0bc7e1010
fbox = 0x55e0bc7e1608
#2 0x00007f42e952652c in hook_mailbox_allocated
(box=box@entry=0x55e0bc7e0fe8) at mail-storage-hooks.c:256
_data_stack_cur_id = 5
_foreach_end = 0x55e0bc7d28a0
_foreach_ptr = 0x55e0bc7d2890
hooks = 0x7f42e8ec9ba0
On 2021-08-06 13:49, Aki Tuomi wrote:
On 06/08/2021 15:43 Joan Moreau jom@grosjo.net wrote:
Thank you Timo However, this leads to kernel: imap[228122]: segfault at 50 ip 00007f7015ee332b sp 00007fffa7178740 error 4 in lib20_fts_plugin.so[7f7015ee1000+11000] Returning to 2.3.15 resolves the problem
Can you provide
gdb bt full
output for the crash?Aki
On 6. Aug 2021, at 15.08, Joan Moreau jom@grosjo.net wrote:
Below
(gdb) bt full #0 fts_user_autoindex_exclude (box=<optimized out>, box@entry=0x55e0bc7e0fe8) at fts-user.c:347
There is no such function in 2.3.16 release. That's only in the current git master. What did you install and from where?
git clone -b release-2.3.16
On 2021-08-06 15:07, Timo Sirainen wrote:
On 6. Aug 2021, at 15.08, Joan Moreau jom@grosjo.net wrote:
Below
(gdb) bt full #0 fts_user_autoindex_exclude (box=<optimized out>, box@entry=0x55e0bc7e0fe8) at fts-user.c:347
There is no such function in 2.3.16 release. That's only in the current git master. What did you install and from where?
I don't see anything wrong in git branch either:
% git clone -b release-2.3.16 https://github.com/dovecot/core.git https://github.com/dovecot/core.git % git grep fts_user_autoindex_exclude %
% git show commit 7e2e900c1a420006371e8a8cf9b4ddb36e69a986 Author: Timo Sirainen timo.sirainen@open-xchange.com Date: Thu Aug 5 18:25:31 2021 +0300
NEWS: Add 100% master process CPU item
I think somehow you built it from master instead of release-2.3.16.
On 6. Aug 2021, at 16.28, Joan Moreau jom@grosjo.net wrote:
git clone -b release-2.3.16
On 2021-08-06 15:07, Timo Sirainen wrote:
On 6. Aug 2021, at 15.08, Joan Moreau
mailto:jom@grosjo.net> wrote: Below
(gdb) bt full #0 fts_user_autoindex_exclude (box=<optimized out>, box@entry=0x55e0bc7e0fe8) at fts-user.c:347
There is no such function in 2.3.16 release. That's only in the current git master. What did you install and from where?
Well, I do not think I am mistaken.
I also get the following error for "indexer" process
#0 0x00007f2370f7fe3d in o_stream_nsendv (stream=0x0, iov=iov@entry=0x7ffeb9dabd70, iov_count=iov_count@entry=1) at ostream.c:333 333 if (unlikely(stream->closed || stream->stream_errno != 0 || (gdb) bt full #0 0x00007f2370f7fe3d in o_stream_nsendv (stream=0x0, iov=iov@entry=0x7ffeb9dabd70, iov_count=iov_count@entry=1) at ostream.c:333 overflow = false #1 0x00007f2370f7feca in o_stream_nsend (stream=<optimized out>, data=<optimized out>, size=<optimized out>) at ostream.c:325 iov = {iov_base = 0x55b8af41d470, iov_len = 5} #2 0x00007f2370f7ff1a in o_stream_nsend_str (stream=<optimized out>, str=<optimized out>) at ostream.c:344 No locals. #3 0x000055b8af391f84 in indexer_client_status_callback (percentage=56, context=0x55b8af434b70) at indexer-client.c:146 _data_stack_cur_id = 4 ctx = 0x55b8af434b70 #4 0x000055b8af3921a0 in indexer_queue_request_status_int (queue=0x55b8af4299a0, request=0x55b8af434b90, percentage=56) at indexer-queue.c:182 context = <optimized out> i = 0 #5 0x000055b8af3919a2 in worker_status_callback (percentage=56, context=0x55b8af434cb0) at indexer.c:104 conn = 0x55b8af434cb0 request = 0x55b8af434b90 #6 0x000055b8af392ac4 in worker_connection_call_callback (percentage=<optimized out>, worker=0x55b8af434cb0) at worker-connection.c:42 No locals. #7 worker_connection_input_args (conn=0x55b8af434cb0, args=0x55b8af41d348) at worker-connection.c:109 worker = 0x55b8af434cb0 percentage = 56 ret = <optimized out> _tmp_event = <optimized out> #8 0x00007f2370f53853 in connection_input_default (conn=0x55b8af434cb0) at connection.c:95 _data_stack_cur_id = 3 line = 0x55b8af438625 "56" input = 0x55b8af436210 output = 0x55b8af436430 ret = 1 #9 0x00007f2370f71919 in io_loop_call_io (io=0x55b8af436550) at ioloop.c:727 ioloop = 0x55b8af425ec0 t_id = 2 __func__ = "io_loop_call_io" #10 0x00007f2370f72fc2 in io_loop_handler_run_internal (ioloop=ioloop@entry=0x55b8af425ec0) at ioloop-epoll.c:222
On 2021-08-06 13:49, Aki Tuomi wrote:
On 06/08/2021 15:43 Joan Moreau jom@grosjo.net wrote:
Thank you Timo However, this leads to kernel: imap[228122]: segfault at 50 ip 00007f7015ee332b sp 00007fffa7178740 error 4 in lib20_fts_plugin.so[7f7015ee1000+11000] Returning to 2.3.15 resolves the problem
Can you provide
gdb bt full
output for the crash?Aki
On 9. Aug 2021, at 11.03, Joan Moreau jom@grosjo.net wrote:
#0 0x00007f2370f7fe3d in o_stream_nsendv (stream=0x0, iov=iov@entry=0x7ffeb9dabd70, iov_count=iov_count@entry=1) at ostream.c:333
overflow = false #1 0x00007f2370f7feca in o_stream_nsend (stream=<optimized out>, data=<optimized out>, size=<optimized out>) at ostream.c:325 iov = {iov_base = 0x55b8af41d470, iov_len = 5} #2 0x00007f2370f7ff1a in o_stream_nsend_str (stream=<optimized out>, str=<optimized out>) at ostream.c:344 No locals. #3 0x000055b8af391f84 in indexer_client_status_callback (percentage=56, context=0x55b8af434b70) at indexer-client.c:146 _data_stack_cur_id = 4 ctx = 0x55b8af434b70 #4 0x000055b8af3921a0 in indexer_queue_request_status_int (queue=0x55b8af4299a0, request=0x55b8af434b90, percentage=56) at indexer-queue.c:182 context = <optimized out>
Looks like v2.3.15 already broke this. Happens when indexer-client disconnects early. Hopefully doesn't happen very often.
On 9. Aug 2021, at 11.24, Timo Sirainen timo@sirainen.com wrote:
On 9. Aug 2021, at 11.03, Joan Moreau
mailto:jom@grosjo.net> wrote: #0 0x00007f2370f7fe3d in o_stream_nsendv (stream=0x0, iov=iov@entry=0x7ffeb9dabd70, iov_count=iov_count@entry=1) at ostream.c:333
overflow = false #1 0x00007f2370f7feca in o_stream_nsend (stream=<optimized out>, data=<optimized out>, size=<optimized out>) at ostream.c:325 iov = {iov_base = 0x55b8af41d470, iov_len = 5} #2 0x00007f2370f7ff1a in o_stream_nsend_str (stream=<optimized out>, str=<optimized out>) at ostream.c:344 No locals. #3 0x000055b8af391f84 in indexer_client_status_callback (percentage=56, context=0x55b8af434b70) at indexer-client.c:146 _data_stack_cur_id = 4 ctx = 0x55b8af434b70 #4 0x000055b8af3921a0 in indexer_queue_request_status_int (queue=0x55b8af4299a0, request=0x55b8af434b90, percentage=56) at indexer-queue.c:182 context = <optimized out>
Looks like v2.3.15 already broke this. Happens when indexer-client disconnects early. Hopefully doesn't happen very often.
Oh, actually v2.3.15.1, but looks like it wasn't even released to community.
Well, I don't really understand your note.
Bottom-line : 2.3.16 crashes every now and then.
Maybe is there a quick fix for production servers ?
On 2021-08-09 10:27, Timo Sirainen wrote:
On 9. Aug 2021, at 11.24, Timo Sirainen timo@sirainen.com wrote:
On 9. Aug 2021, at 11.03, Joan Moreau jom@grosjo.net wrote:
#0 0x00007f2370f7fe3d in o_stream_nsendv (stream=0x0, iov=iov@entry=0x7ffeb9dabd70, iov_count=iov_count@entry=1) at ostream.c:333
overflow = false #1 0x00007f2370f7feca in o_stream_nsend (stream=<optimized out>, data=<optimized out>, size=<optimized out>) at ostream.c:325 iov = {iov_base = 0x55b8af41d470, iov_len = 5} #2 0x00007f2370f7ff1a in o_stream_nsend_str (stream=<optimized out>, str=<optimized out>) at ostream.c:344 No locals. #3 0x000055b8af391f84 in indexer_client_status_callback (percentage=56, context=0x55b8af434b70) at indexer-client.c:146 _data_stack_cur_id = 4 ctx = 0x55b8af434b70 #4 0x000055b8af3921a0 in indexer_queue_request_status_int (queue=0x55b8af4299a0, request=0x55b8af434b90, percentage=56) at indexer-queue.c:182 context = <optimized out>
Looks like v2.3.15 already broke this. Happens when indexer-client disconnects early. Hopefully doesn't happen very often.
Oh, actually v2.3.15.1, but looks like it wasn't even released to community.
On 8/9/21 11:32 AM, Joan Moreau wrote:
Bottom-line : 2.3.16 crashes every now and then.
I see occasional failures in the test suite on openSUSE build system (OBS):
https://build.opensuse.org/package/show/home:stroeder:network/dovecot23
But I'm not sure whether that's a timing-related red hering specific for running the tests on OBS.
Ciao, Michael.
On 9. Aug 2021, at 11.58, Michael Ströder michael@stroeder.com wrote:
On 8/9/21 11:32 AM, Joan Moreau wrote:
Bottom-line : 2.3.16 crashes every now and then.
I see occasional failures in the test suite on openSUSE build system (OBS):
https://build.opensuse.org/package/show/home:stroeder:network/dovecot23
But I'm not sure whether that's a timing-related red hering specific for running the tests on OBS.
The test-cpu-limit failures are a bit annoying. Probably has something to do with overloaded VMs. Maybe we need to just disable those failing asserts entirely since they seem to pop up once in a while.
These probably have something to do with big endian CPUs. Would need to debug further:
[ 762s] test-mail-cache-fields.c:50: Assert failed: cache_field.last_used == priv->field.last_used && cache_field.decision == priv->field.decision [ 762s] test-mail-cache-fields.c:65: Assert failed: cache_field.last_used == priv->field.last_used && cache_field.decision == priv->field.decision [ 762s] test-mail-cache-fields.c:94: Assert failed: cache_field.last_used == priv->field.last_used && cache_field.decision == priv->field.decision
This seems to happen because setrlimit(RLIMIT_AS) is somehow not working as expected in the RISCV target - not sure what could be done about that other than disabling the test entirely:
[ 9496s] test-file-cache.c:266: Assert failed: file_cache_set_size(cache, 1024) == -1 [ 9496s] Panic: file test-common.c: line 211 (test_expect_error_string_n_times): assertion failed: (expected_errors == 0)
On 9. Aug 2021, at 11.32, Joan Moreau jom@grosjo.net wrote:
Bottom-line : 2.3.16 crashes every now and then. Maybe is there a quick fix for production servers ?
Attached a fix.
On Aug 6, 2021, at 7:42 AM, Timo Sirainen timo@sirainen.com wrote:
https://dovecot.org/releases/2.3/dovecot-2.3.16.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.16.tar.gz.sig
The patch from https://dovecot.org/pipermail/dovecot/2021-June/122375.html is still necessary to build on newer Mac OS X (for the same reason). Tested on Mac OS version 11.5.1.
-- Daniel J. Luke
Was the update issue sorted? Is it safe to update or was/is there a glitch? Or did the person with the issue update from the wrong branch?
And just to confirm, based on the change log in this email it doesn't look like we need to change anything in our config for this update? (Coming from 2.3.15)
participants (7)
-
Aki Tuomi
-
Christian Kivalo
-
Daniel J. Luke
-
dovecot@ptld.com
-
Joan Moreau
-
Michael Ströder
-
Timo Sirainen