v2.2.32 release candidate released
https://dovecot.org/releases/2.2/rc/dovecot-2.2.32.rc1.tar.gz https://dovecot.org/releases/2.2/rc/dovecot-2.2.32.rc1.tar.gz.sig
There are various changes in this release that can be used to significantly reduce disk IO with:
- NFS storage especially, but I guess also other remote filesystems and even some with local disks
- When mail storage and INDEX storage are separated
- imapc: Info-level line is logged every time when successfully connected to the remote server. This includes local/remote IP/port, which can be useful for matching against external logs.
- config: Log a warning if plugin { key=no } is used explicitly. v2.3 will support "no" properly in plugin settings, but for now any value at all for a boolean plugin setting is treated as "yes", even if it's written as explicit "no". This change will now warn that it most likely won't work as intended.
- Various optimizations to avoid accessing files/directories when it's not necessary. Especially avoid accessing mail root directories when INDEX directories point to a different filesystem.
- mail_location can now include ITERINDEX parameter. This tells Dovecot to perform mailbox listing from the INDEX path instead of from the mail root path. It's mainly useful when the INDEX storage is on a faster storage.
- mail_location can now include VOLATILEDIR=<path> parameter. This is used for creating lock files and in future potentially other files that don't need to exist permanently. The path could point to tmpfs for example. This is especially useful to avoid creating lock files to NFS or other remote filesystems. For example: mail_location=sdbox:~/sdbox:VOLATILEDIR=/tmp/volatile/%2.256Nu/%u
- mail_location's LISTINDEX=<path> can now contain a full path. This allows storing mailbox list index to a different storage than the rest of the indexes, for example to tmpfs.
- mail_location can now include NO-NOSELECT parameter. This automatically deletes any \NoSelect mailboxes that have no children. These mailboxes are sometimes confusing to users.
- mail_location can now include BROKENCHAR=<char> parameter. This can be useful with imapc to access mailbox names that aren't valid mUTF-7 charset from remote servers.
- If mailbox_list_index_very_dirty_syncs=yes, the list index is no longer refreshed against filesystem when listing mailboxes. This allows the mailbox listing to be done entirely by only reading the mailbox list index.
- Added mailbox_list_index_include_inbox setting to control whether INBOX's STATUS information should be cached in the mailbox list index. The default is "no", but it may be useful to change it to "yes", especially if LISTINDEX points to tmpfs.
- userdb can return chdir=<path>, which override mail_home for the chdir location. This can be useful to avoid accessing home directory on login.
- userdb can return postlogin=<socket> to specify per-user imap/pop3 postlogin socket path.
- cassandra: Add support for result paging by adding page_size=<n> parameter to the connect setting.
- dsync/imapc, pop3-migration plugin: Strip also trailing tabs from headers when matching mails. This helps with migrations from Zimbra.
- imap_logout_format supports now %{appended} and %{autoexpunged}
- virtual plugin: Optimize IDLE to use mailbox list index for finding out when something has changed.
- virtual plugin: A lot of fixes. In many cases it was also working very inefficiently or even incorrectly.
- imap: NOTIFY parameter parsing was incorrectly "fixed" in v2.2.31. It was actually (mostly) working in previous versions, but broken in v2.2.31.
- Modseq tracking didn't always work correctly. This could have caused imap unhibernation to fail or IMAP QRESYNC/CONDSTORE extensions to not work perfectly.
- mdbox: "Inconsistency in map index" wasn't fixed automatically
- dict-ldap: %variable values used in the LDAP filter weren't escaped.
- quota=count: quota_warning = -storage=.. was never executed (try #2). v2.2.31 fixed it for -messages, but not for -storage.
- imapc: >= 32 kB mail bodies were supposed to be cached for subsequent FETCHes, but weren't.
- quota-status service didn't support recipient_delimiter
- acl: Don't access dovecot-acl-list files with acl_globals_only=yes
- mail_location: If INDEX dir is set, mailbox deletion deletes its childrens' indexes. For example if "box" is deleted, "box/child" index directory was deleted as well (but mails were preserved).
Hello Timo,
I am quite eager to test it, but I don't know if these changes can be applicable for my configuration (it is shared storage one, any mdbox on NFS can be accessed by multiple servers, I made it so it's mostly accessed by one, but at certain conditions multiple servers still access mailbox simultaneously).
So if possible I want to ask few dumb questions:
I assume VOLATILEDIR can't be used with configurations where multiple servers access the same mdbox because lockfiles need to be shared as well? Or does this option touch only some (lock)files that do not need to be shared between servers accessing same mailbox?
Would LISTINDEX indexes be regenerated properly for mdbox if lost/corrupted? Properly, meaning without losing flags, message numbering or some other vital data. If yes, then I assume they can be stored on local storage per server, and not on NFS. If LISTINDEX can be regenerated, will LISTINDEX be updated if it's detected to be obsolete compared to accessed mdbox contents?
Thanks.
On 15.08.2017 23:49, Timo Sirainen wrote:
https://dovecot.org/releases/2.2/rc/dovecot-2.2.32.rc1.tar.gz https://dovecot.org/releases/2.2/rc/dovecot-2.2.32.rc1.tar.gz.sig
There are various changes in this release that can be used to significantly reduce disk IO with:
- NFS storage especially, but I guess also other remote filesystems and even some with local disks
- When mail storage and INDEX storage are separated
On 16 Aug 2017, at 8.37, Alexey Asemov (Alex/AT) lists@alex-at.ru wrote:
Hello Timo,
I am quite eager to test it, but I don't know if these changes can be applicable for my configuration (it is shared storage one, any mdbox on NFS can be accessed by multiple servers, I made it so it's mostly accessed by one, but at certain conditions multiple servers still access mailbox simultaneously).
So if possible I want to ask few dumb questions:
- I assume VOLATILEDIR can't be used with configurations where multiple servers access the same mdbox because lockfiles need to be shared as well? Or does this option touch only some (lock)files that do not need to be shared between servers accessing same mailbox?
If you're using e.g. Dovecot director, it should be very unlikely that multiple servers access the same user simultaneously. Currently only two lock files use VOLATILEDIR:
- .vsize.lock: Used to update folder's size in dovecot.index. So the worst that can happen is that the vsize becomes wrong. If you're using quota=count, it could become wrong. But this gets fixed automatically anyway later on.
- autoexpunge.lock : Only used if you have autoexpunge settings.
- Would LISTINDEX indexes be regenerated properly for mdbox if lost/corrupted? Properly, meaning without losing flags, message numbering or some other vital data. If yes, then I assume they can be stored on local storage per server, and not on NFS. If LISTINDEX can be regenerated, will LISTINDEX be updated if it's detected to be obsolete compared to accessed mdbox contents?
dovecot.list.index* don't have any especially vital information, like message flags. So it can be fully regenerated. You have 3 options actually:
a) Use mailbox_list_index_very_dirty_syncs=yes and keep LISTINDEX in NFS: This optimizes LIST b) Use mailbox_list_index_very_dirty_syncs=no and put LISTINDEX to local: This automatically updates the list index as necessary, but it does some extra stat()s on the folders to make sure they're up-to-date in the list index. c) Use mailbox_list_index_very_dirty_syncs=yes and put LISTINDEX to local: This fully trusts the locally cached list indexes. It works only if you can guarantee that the local list indexes aren't obsolete. So it requires some scripting. For example you can store in NFS or in Redis or elsewhere the user's last backend hostname. Then create a post-login script that deletes the list indexes if the hostname doesn't match the current server where user is logging in. This can also be optimized to run the script only when the hostname changes by using the new userdb postlogin socket and %{if...} that is coming to v2.2.33. Although there are also other ways.
Hello Timo,
Many thanks for the explanation, it really clarified things. I'll go for option (b) "Use mailbox_list_index_very_dirty_syncs=no and put LISTINDEX to local" then and write later on if there are any issues with it.
Am 15.08.2017 um 22:49 schrieb Timo Sirainen:
https://dovecot.org/releases/2.2/rc/dovecot-2.2.32.rc1.tar.gz
my buildsystem complain on spelling errors patches attached.
- in binaries -> src/director/director-connection.c: reseting -> resetting
- in manpages -> doc/man/doveadm-exec.1.in: wich -> which + No newline at end of file
one other point I notice (not new in this versions) are "warnings" from Debian lintian:
I: dv-dovecot: hardening-no-fortify-functions usr/lib/dovecot/auth
N:
N: This package provides an ELF binary that lacks the use of fortified libc
N: functions. Either there are no potentially unfortified functions called
N: by any routines, all unfortified calls have already been fully validated
N: at compile-time, or the package was not built with the default Debian
N: compiler flags defined by dpkg-buildflags. If built using
N: dpkg-buildflags directly, be sure to import CPPFLAGS.
N:
N: NB: Due to false-positives, Lintian ignores some unprotected functions
N: (e.g. memcpy).
N:
N: Refer to https://wiki.debian.org/Hardening and
N: http://bugs.debian.org/673112 for details.
N:
N: Severity: normal, Certainty: wild-guess
N:
N: Check: binaries, Type: binary, udeb
N:
I: dv-dovecot: hardening-no-fortify-functions usr/lib/dovecot/config
I: dv-dovecot: hardening-no-fortify-functions usr/lib/dovecot/director
I: dv-dovecot: hardening-no-fortify-functions usr/lib/dovecot/gdbhelper
I: dv-dovecot: hardening-no-fortify-functions usr/lib/dovecot/imap
I: dv-dovecot: hardening-no-fortify-functions usr/lib/dovecot/libdovecot-login.so.0.0.0
I: dv-dovecot: hardening-no-fortify-functions usr/lib/dovecot/modules/lib10_quota_plugin.so
I: dv-dovecot: hardening-no-fortify-functions usr/lib/dovecot/modules/lib20_fts_plugin.so
I: dv-dovecot: hardening-no-fortify-functions usr/lib/dovecot/modules/lib20_replication_plugin.so
I: dv-dovecot: hardening-no-fortify-functions usr/lib/dovecot/modules/lib99_welcome_plugin.so
I: dv-dovecot: hardening-no-fortify-functions usr/lib/dovecot/quota-status
I: dv-dovecot: hardening-no-fortify-functions usr/lib/dovecot/script
I: dv-dovecot: hardening-no-fortify-functions usr/lib/dovecot/script-login
I: dv-dovecot: hardening-no-fortify-functions usr/lib/dovecot/xml2text
the text mention "CPPFLAGS" which occour exact two times in my Buildlog:
- as parameter to configure
- in that warning
I read this as "configure get an evironment variable "CPPFLAGS" and completely ignore them. full buildlog (for Debian Jessie, using aclocal-1.14, btw...) at https://andreasschulze.de/tmp/dv-dovecot_2.2.32~rc1-2017081601_amd64.build.t...
Andreas
On Tue, Aug 15, 2017 at 11:49:29PM +0300, Timo Sirainen wrote:
https://dovecot.org/releases/2.2/rc/dovecot-2.2.32.rc1.tar.gz https://dovecot.org/releases/2.2/rc/dovecot-2.2.32.rc1.tar.gz.sig
In case you are interested (and have the spare cycles), I get the following mostly new warnings with gcc-7
-- Eray
/var/tmp/portage/app-arch/lz4-1.7.5-r1/work/lz4-1.7.5/lib/lz4frame.c:1082:29: warning: this statement may fall through [-Wimplicit-fallthrough=] dctxPtr->dStage = dstage_storeHeader; ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~ /var/tmp/portage/app-arch/lz4-1.7.5-r1/work/lz4-1.7.5/lib/lz4frame.c:1085:9: note: here case dstage_storeHeader: ^~~~ /var/tmp/portage/app-arch/lz4-1.7.5-r1/work/lz4-1.7.5/lib/lz4frame.c:1194:33: warning: this statement may fall through [-Wimplicit-fallthrough=] dctxPtr->dStage = dstage_decodeCBlock; ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~ /var/tmp/portage/app-arch/lz4-1.7.5-r1/work/lz4-1.7.5/lib/lz4frame.c:1198:9: note: here case dstage_decodeCBlock: ^~~~ /var/tmp/portage/app-arch/lz4-1.7.5-r1/work/lz4-1.7.5/programs/lz4cli.c:453:36: warning: this statement may fall through [-Wimplicit-fallthrough=] case 'r': recursive=1; /* without break */ ~~~~~~~~~^~ /var/tmp/portage/app-arch/lz4-1.7.5-r1/work/lz4-1.7.5/programs/lz4cli.c:456:17: note: here case 'm': multiple_inputs=1; ^~~~ /var/tmp/portage/app-arch/lz4-1.7.5-r1/work/lz4-1.7.5/programs/lz4cli.c:453:36: warning: this statement may fall through [-Wimplicit-fallthrough=] case 'r': recursive=1; /* without break */ ~~~~~~~~~^~ /var/tmp/portage/app-arch/lz4-1.7.5-r1/work/lz4-1.7.5/programs/lz4cli.c:456:17: note: here case 'm': multiple_inputs=1; ^~~~ json-parser.c:537:6: warning: this statement may fall through [-Wimplicit-fallthrough=] if (parser->skipping && parser->nested_skip_count == 0) { ^ json-parser.c:541:2: note: here case JSON_STATE_ARRAY_NEXT_SKIP: ^~~~ istream-attachment-extractor.c:328:3: warning: this statement may fall through [-Wimplicit-fallthrough=] buffer_set_used_size(part_buf, 0); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ istream-attachment-extractor.c:330:2: note: here case MAIL_ATTACHMENT_STATE_YES: ^~~~ message-header-encode.c:21:4: warning: this statement may fall through [-Wimplicit-fallthrough=] i++; ~^~ message-header-encode.c:23:2: note: here case '\n': ^~~~ ldap-connection.c:484:16: warning: this statement may fall through [-Wimplicit-fallthrough=] conn->state = LDAP_STATE_AUTH; ~~~~~~~~~~~~^~~~~~~~~~~~~~~~~ ldap-connection.c:487:2: note: here case LDAP_STATE_AUTH: ^~~~ fts-filter-contractions.c:46:6: warning: this statement may fall through [-Wimplicit-fallthrough=] if (token[pos] == '\0' || token[pos] != 'u') ^ fts-filter-contractions.c:49:2: note: here case 'c': ^~~~ mail-transaction-log-file.c:1064:6: warning: this statement may fall through [-Wimplicit-fallthrough=] if ((hdr->type & MAIL_TRANSACTION_EXTERNAL) == 0) { ^ mail-transaction-log-file.c:1068:2: note: here case MAIL_TRANSACTION_APPEND: ^~~~ test-mail-transaction-log-file.c:412:18: warning: passing argument 1 of ‘test_run’ discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers] return test_run(test_functions); ^~~~~~~~~~~~~~ In file included from test-mail-transaction-log-file.c:5:0: ../../src/lib-test/test-common.h:55:5: note: expected ‘void (**)(void)’ but argument is of type ‘void (* const*)(void)’ int test_run(void (*test_functions[])(void)); ^~~~~~~~ maildir-save.c:277:30: warning: ‘*’ in boolean context, suggest ‘&&’ instead [-Wint-in-bool-context] mf->keywords_count * sizeof(unsigned int)); ../../../../src/lib/macros.h:161:26: note: in definition of macro ‘COMPILE_ERROR_IF_TRUE’ (sizeof(char[1 - 2 * !!(condition)]) - 1) ^~~~~~~~~ maildir-save.c:276:2: note: in expansion of macro ‘buffer_create_from_const_data’ buffer_create_from_const_data(&ctx->keywords_buffer, mf + 1, ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ imapc-search.c:99:6: warning: this statement may fall through [-Wimplicit-fallthrough=] if ((mbox->capabilities & IMAPC_CAPABILITY_WITHIN) == 0) { ^ imapc-search.c:110:2: note: here case SEARCH_ALL: ^~~~ mail-search.c:78:7: warning: this statement may fall through [-Wimplicit-fallthrough=] if (arg->value.str == NULL) ^ mail-search.c:81:3: note: here case SEARCH_KEYWORDS: ^~~~ password-scheme.c:186:6: warning: this statement may fall through [-Wimplicit-fallthrough=] if (!guessed_encoding) { ^ password-scheme.c:193:2: note: here case PW_ENCODING_BASE64: ^~~~ client-common-auth.c:58:43: warning: ?: using integer constants in boolean context [-Wint-in-bool-context] AUTH_WAITING_WARNING_TIMEOUT_MSECS : ../../src/lib/macros.h:161:26: note: in definition of macro ‘COMPILE_ERROR_IF_TRUE’ (sizeof(char[1 - 2 * !!(condition)]) - 1) ^~~~~~~~~ client-common-auth.c:57:3: note: in expansion of macro ‘timeout_add’ timeout_add(!client->notified_auth_ready ? ^~~~~~~~~~~ fts-search-args.c:206:7: warning: this statement may fall through [-Wimplicit-fallthrough=] if ((*argp)->value.str[0] == '\0') { ^ fts-search-args.c:211:3: note: here case SEARCH_BODY: ^~~~ doveadm-mail-crypt.c:861:14: warning: this statement may fall through [-Wimplicit-fallthrough=] ctx->force = TRUE; doveadm-mail-crypt.c:862:2: note: here default: ^~~~~~~
participants (4)
-
A. Schulze
-
Alexey Asemov (Alex/AT)
-
Eray Aslan
-
Timo Sirainen