Dovecot release v2.3.7
Hi!
We are pleased to release Dovecot release v2.3.7.
Tarball is available at
https://dovecot.org/releases/2.3/dovecot-2.3.7.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.7.tar.gz.sig
Binary packages are available at https://repo.dovecot.org/
Changes
- fts-solr: Removed break-imap-search parameter
- Added more events for the new statistics, see https://doc.dovecot.org/admin_manual/list_of_events/
- mail-lua: Add IMAP metadata accessors, see https://doc.dovecot.org/admin_manual/lua/
- Add event exporters that allow exporting raw events to log files and external systems, see https://doc.dovecot.org/configuration_manual/event_export/
- SNIPPET is now PREVIEW and size has been increased to 200 characters.
- Add body option to fts_enforced. This triggers building FTS index only on body search, and an error using FTS index fails the search rather than reads through all the mails.
- Submission/LMTP: Fixed crash when domain argument is invalid in a second EHLO/LHLO command.
- Copying/moving mails using Maildir format loses IMAP keywords in the destination if the mail also has no system flags.
- mail_attachment_detection_options=add-flags-on-save caused email body to be unnecessarily opened when FETCHing mail headers that were already cached.
- mail attachment detection keywords not saved with maildir.
- dovecot.index.cache may have grown excessively large in some situations. This happened especially when using autoexpunging with lazy_expunge folders. Also with mdbox format in general the cache file wasn't recreated as often as it should have.
- Autoexpunged mails weren't immediately deleted from the disk. Instead, the deletion from disk happened the next time the folder was opened. This could have caused unnecessary delays if the opening was done by an interactive IMAP session.
- Dovecot's TCP connections sometimes add extra 40ms latency due to not enabling TCP_NODELAY. HTTP and SMTP/LMTP connections weren't affected, but everything else was. This delay wasn't always visible - only in some situations with some message/packet sizes.
- imapc: Fix various crash conditions
- Dovecot builds were not always reproducible.
- login-proxy: With shutdown_clients=no after config reload the existing connections could no longer be listed or kicked with doveadm.
- "doveadm proxy kick" with -f parameter caused a crash in some situations.
- Auth policy can cause segmentation fault crash during auth process shutdown if all auth requests have not been finished.
- Fix various minor bugs leading into incorrect behaviour in mailbox list index handling. These rarely caused noticeable problems.
- LDAP auth: Iteration accesses freed memory, possibly crashing auth-worker
- local_name { .. } filter in dovecot.conf does not correctly support multiple names and wildcards were matched incorrectly.
- replicator: dsync assert-crashes if it can't connect to remote TCP server.
- config: Memory leak in config process when ssl_dh setting wasn't set and there was no ssl-parameters.dat file. This caused config process to die once in a while with "out of memory".
Aki Tuomi Open-Xchange oy
Aki Tuomi via dovecot <dovecot@dovecot.org> wrote:
We are pleased to release Dovecot release v2.3.7.
My upgrade from 2.3.6 to 2.3.7 broke replication (FreeBSD 12.0-STABLE r349799):
| dovecot[76032]: master: Dovecot v2.3.7 (494d20bdc) starting up for imap, lmtp, sieve | dovecot[76035]: replicator: Error: file_ostream.net_set_tcp_nodelay((conn:unix:/var/run/dovecot/stats-writer), TRUE) failed: Invalid argument | dovecot[76035]: replicator: Error: file_ostream.net_set_tcp_nodelay((conn:unix:auth-userdb), TRUE) failed: Invalid argument | dovecot[76035]: auth: Error: file_ostream.net_set_tcp_nodelay((conn:unix:/var/run/dovecot/stats-writer), TRUE) failed: Invalid argument
Back to 2.3.6 fixed it.
FYI: I do not have any … metric name {} … in my config.
Regards, Michael
On 12 Jul 2019, at 21.05, Michael Grimm via dovecot <dovecot@dovecot.org> wrote:
I don't think these cause any actual breakage? Of course, there are tons of errors logged.. This likely fixes it anyway: diff --git a/src/lib/ostream-file.c b/src/lib/ostream-file.c index e7e6f62d1..766841f2f 100644 --- a/src/lib/ostream-file.c +++ b/src/lib/ostream-file.c @@ -334,7 +334,7 @@ static void o_stream_tcp_flush_via_nodelay(struct file_ostream *fstream) { if (net_set_tcp_nodelay(fstream->fd, TRUE) < 0) { if (errno != ENOTSUP && errno != ENOTSOCK && - errno != ENOPROTOOPT) { + errno != ENOPROTOOPT && errno != EINVAL) { i_error("file_ostream.net_set_tcp_nodelay(%s, TRUE) failed: %m", o_stream_get_name(&fstream->ostream.ostream)); }
Timo Sirainen via dovecot <dovecot@dovecot.org> wrote:
On 12 Jul 2019, at 21.05, Michael Grimm via dovecot <dovecot@dovecot.org> wrote:
My upgrade from 2.3.6 to 2.3.7 broke replication (FreeBSD 12.0-STABLE r349799):
I don't think these cause any actual breakage?
It doesn't break completely, but it leaves replication shaken: 1) replication is pretty much delayed in some cases (monitored on both instances) 2) replication doesn't become completed after 15 minutes in some cases Returning to 2.3.6 completes these hanging replications instantaneously.
Of course, there are tons of errors logged..
Yes.
I do not compile from source, I'm using FreeBSD's ports. Thus I tried to put this patch at the desired location to become used whilst compiling dovecot. But this patch fails to become applied, most probably due to incorrect whitespaces. As I do not have access to the dovecot's source tree, I do not have a clue who the ASCII version of the patch is. Thus, could you mail it to my as an attachment? Regards, Michael
Michael Grimm via dovecot <dovecot@dovecot.org> wrote:
Timo Sirainen via dovecot <dovecot@dovecot.org> wrote:
This likely fixes it anyway:
diff --git a/src/lib/ostream-file.c b/src/lib/ostream-file.c
I do not compile from source, I'm using FreeBSD's ports. Thus I tried to put this patch at the desired location to become used whilst compiling dovecot. But this patch fails to become applied, most probably due to incorrect whitespaces. As I do not have access to the dovecot's source tree, I do not have a clue who the ASCII version of the patch is. Thus, could you mail it to my as an attachment?
Never mind, Larry has applied your patch to FreeBSD's port system in the meantime, and yes, it does silence those error messages.
But, replication still doesn't works as known from previous versions. I will give it a try over night, but I am seeing …
| Queued 'full resync' requests 1
… after 1 minute, already.
Regards, Michael
Michael Grimm wrote:
The following modifications regarding my setup made replication work again (at least for the last 8 hours):
#) Re-compiling dovecot with the patch from https://dovecot.org/pipermail/dovecot/2019-July/116479.html #) Activating mailbox attributes as mentioned in https://dovecot.org/pipermail/dovecot/2019-July/116492.html
Now replication is working from my point of view, besides occational error messages like:
| imap-login: Error: file_ostream.net_set_tcp_nodelay(, TRUE) failed: Connection reset by peer
(This has been reported before: https://dovecot.org/pipermail/dovecot/2019-July/116491.html)
Regards, Michael
On 13 Jul 2019, at 18.39, Michael Grimm via dovecot <dovecot@dovecot.org> wrote:
Now replication is working from my point of view, besides occational error messages like:
| imap-login: Error: file_ostream.net_set_tcp_nodelay(, TRUE) failed: Connection reset by peer
Here's the final fix for it:
https://github.com/dovecot/core/commit/25028730cd1b76e373ff989625132d526eea2... <https://github.com/dovecot/core/commit/25028730cd1b76e373ff989625132d526eea2504>
Timo Sirainen via dovecot <dovecot@dovecot.org> wrote:
On 13 Jul 2019, at 18.39, Michael Grimm via dovecot <dovecot@dovecot.org> wrote:
No wonder, after applying your patch those error messages are history :-)
Thank you very much and regards, Michael
@Larry: from my point of view you may replace this patch with the previous one.
LMTP is broken on director:
Jul 13 13:42:41 lmtp(34824): Panic: file smtp-params.c: line 685 (smtp_params_mail_add_body_to_event): assertion failed: ((caps & SMTP_CAPABILITY_8BITMIME) != 0) Jul 13 13:42:41 lmtp(34824): Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0xdf06a) [0x7f561203806a] -> /usr/lib64/dovecot/libdovecot.so.0(+0xdf0b1) [0x7f56120380b1] -> /usr/lib64/dovecot/libdovecot.so.0(+0x40161) [0x7f5611f99161] -> /usr/lib64/dovecot/libdovecot.so.0(+0x44fc4) [0x7f5611f9dfc4] -> /usr/lib64/dovecot/libdovecot.so.0(smtp_client_transaction_start+0x78) [0x7f5611fa7a58] -> dovecot/lmtp 172.17.165.6 MAIL FROM [0x558eaf181d67] -> dovecot/lmtp 172.17.165.6 MAIL FROM [0x558eaf17eebe] -> /usr/lib64/dovecot/libdovecot.so.0(smtp_server_cmd_rcpt+0x222) [0x7f5611fafd12] -> /usr/lib64/dovecot/libdovecot.so.0(smtp_server_command_new+0x150) [0x7f5611fb58a0] -> /usr/lib64/dovecot/libdovecot.so.0(+0x606c8) [0x7f5611fb96c8] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x55) [0x7f561204f275] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x95) [0x7f561204f3a5] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f561204f588] -> /usr/lib64/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f5611fc68f3] -> dovecot/lmtp 172.17.165.6 MAIL FROM [0x558eaf17dc19] -> /lib64/libc.so.6(__libc_start_main+0x100) [0x7f5611be3d20] -> dovecot/lmtp 172.17.165.6 MAIL FROM [0x558eaf17d849]
Tom
On 2019-07-12 14:29, Aki Tuomi via dovecot wrote:
On 13 Jul 2019, at 14.44, Tom Sommer via dovecot <dovecot@dovecot.org> wrote:
LMTP is broken on director:
Jul 13 13:42:41 lmtp(34824): Panic: file smtp-params.c: line 685 (smtp_params_mail_add_body_to_event): assertion failed: ((caps & SMTP_CAPABILITY_8BITMIME) != 0)
Thanks, fixed: https://github.com/dovecot/core/commit/c4de81077c11d09eddf6a5c93676ee8235034... <https://github.com/dovecot/core/commit/c4de81077c11d09eddf6a5c93676ee82350343a6>
Version 2.3.7 appears to break director ring sync. After upgrading, our logs are full of these kinds of messages:
Jul 14 04:09:20 yyy dovecot: director: Error: director: User xxx host lookup failed: Timeout because ring not synced - queued for 30 secs (Ring not synced for 1410 secs, hash=271838221) Jul 14 04:09:20 yyy dovecot: director: Error: director: User xxx host lookup failed: Timeout because ring not synced - queued for 30 secs (Ring not synced for 1410 secs, hash=3482757093)
As well as:
Jul 14 04:09:21 yyy dovecot: director: Error: Ring SYNC seq=53 appears to have got lost, resending
These are absent from the logs prior to July 12.
I have downgraded to 2.3.6 in hopes of resolving the issue.
Regards, Eirik Rye
On 12/07/2019 14:29, Aki Tuomi via dovecot wrote:
Please disregard this. My apologies.
The issue appears to have been caused by an unrelated network issue with one of the directors, that coincidentally occurred at almost the exact same time.
A potential improvement would be if the "Ring SYNC seq=XX appears to have got lost" error message would indicate which director/server it pertains to.
Best regards, Eirik Rye
On 16/07/2019 11:17, Eirik Rye via dovecot wrote:
participants (5)
-
Aki Tuomi
-
Eirik Rye
-
Michael Grimm
-
Timo Sirainen
-
Tom Sommer