[dovecot/core] d0dc63: lib-storage: Fix truncating header values at NUL w...

Mon Apr 23 14:30:18 EEST 2018

  Branch: refs/heads/master
  Home:   https://github.com/dovecot/core
  Commit: d0dc63004b67a111570d6db4b0fe596c9a56d55c
      https://github.com/dovecot/core/commit/d0dc63004b67a111570d6db4b0fe596c9a56d55c
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2018-04-21 (Sat, 21 Apr 2018)

  Changed paths:
    M src/lib-storage/index/index-mail-headers.c

  Log Message:
  -----------
  lib-storage: Fix truncating header values at NUL when adding to cache

For example FETCH BODY.PEEK[HEADER.FIELDS (Subject)] could have returned
the NUL on the first fetch when the mail wasn't in dovecot.index.cache,
but the second FETCH would have returned it truncated at the NUL.

  Commit: 9abd4cb55a90fe4ff5fd574da932e11c08189839
      https://github.com/dovecot/core/commit/9abd4cb55a90fe4ff5fd574da932e11c08189839
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2018-04-21 (Sat, 21 Apr 2018)

  Changed paths:
    M src/lib-mail/message-snippet.c
    M src/lib-mail/test-message-snippet.c

  Log Message:
  -----------
  lib-mail: message_snippet_generate() - Ignore NULs without shrinking snippet size

Previously the NULs also weren't in the snippet content, but they were
included in the snippet size.

  Commit: 980dc2b45ea7981c8eef628a14580a238a36cc44
      https://github.com/dovecot/core/commit/980dc2b45ea7981c8eef628a14580a238a36cc44
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2018-04-21 (Sat, 21 Apr 2018)

  Changed paths:
    M src/lib-mail/rfc822-parser.c

  Log Message:
  -----------
  lib-mail: rfc822-parser: Don't truncate comment/quoted-string/domain-literal at NUL

The other tokens couldn't have contained NULs in the first place.

  Commit: 6f456ea32afed46e62b06ef69c1965743fd73057
      https://github.com/dovecot/core/commit/6f456ea32afed46e62b06ef69c1965743fd73057
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2018-04-23 (Mon, 23 Apr 2018)

  Changed paths:
    M src/doveadm/doveadm-dump-index.c

  Log Message:
  -----------
  doveadm dump: Don't truncate dovecot.index.cache values at NULs

  Commit: 1bb4e53ea674fe84dc0e1633f1cf1cb5f2fd2e49
      https://github.com/dovecot/core/commit/1bb4e53ea674fe84dc0e1633f1cf1cb5f2fd2e49
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2018-04-23 (Mon, 23 Apr 2018)

  Changed paths:
    M src/doveadm/doveadm-print-table.c

  Log Message:
  -----------
  doveadm: Fix table printer to not truncate output at NULs

  Commit: a60deaa8cc47f8ecb0155b52ca4b849ee1fcfe5b
      https://github.com/dovecot/core/commit/a60deaa8cc47f8ecb0155b52ca4b849ee1fcfe5b
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2018-04-23 (Mon, 23 Apr 2018)

  Changed paths:
    M src/auth/db-checkpassword.c

  Log Message:
  -----------
  auth: checkpassword - Fail if input from script contains NULs

Previously the input was just silently truncated at NULs.

  Commit: 302708ee9a2bcc5876cd1c54e7832da4f3943786
      https://github.com/dovecot/core/commit/302708ee9a2bcc5876cd1c54e7832da4f3943786
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2018-04-23 (Mon, 23 Apr 2018)

  Changed paths:
    M src/lib-dict-extra/dict-fs.c
    M src/lib-dict/dict-memcached-ascii.c
    M src/lib-dict/dict-redis.c

  Log Message:
  -----------
  dict-fs/memcached-ascii/redis: Consistently truncate value at NULs

The dict API currently doesn't support NULs in values, so they have to be
truncated. However, previously they were truncated at NULs, but more data
may have still been appended to the value. This could have produced
somewhat random results.

  Commit: 3821df49f69db6fab8402a14bb8d7646dea1e835
      https://github.com/dovecot/core/commit/3821df49f69db6fab8402a14bb8d7646dea1e835
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2018-04-23 (Mon, 23 Apr 2018)

  Changed paths:
    M src/config/config-parser.c

  Log Message:
  -----------
  config: Consistently truncate value at the first NUL in value input file

When value contained "<file", the file was added as a value. The setting
values don't currently support NULs, so the value gets truncated at the
first NUL. However, the previous code could have appended more data to
the value. This could have produced somewhat confusing results.

  Commit: 0f9bbf22016c50de7f3cc30f4aaf5d877069e3dd
      https://github.com/dovecot/core/commit/0f9bbf22016c50de7f3cc30f4aaf5d877069e3dd
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2018-04-23 (Mon, 23 Apr 2018)

  Changed paths:
    M src/lib-master/master-login.c
    M src/util/script-login.c

  Log Message:
  -----------
  lib-master, util: Consistently truncate tabescaped input at the first NUL

The NULs are supposed to come tab-escaped, so there's no need to support
actual NUL characters. The previous code truncated at NULs, but could
have kept appending more data to the value. This could have produced
somewhat random results.

  Commit: 46aa1c4ce13f637161dd2a9c135f108555a6dbe9
      https://github.com/dovecot/core/commit/46aa1c4ce13f637161dd2a9c135f108555a6dbe9
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2018-04-23 (Mon, 23 Apr 2018)

  Changed paths:
    M src/login-common/client-common-auth.c

  Log Message:
  -----------
  login-common: Consistently truncate client SASL input at first NUL

The input is supposed to be base64-encoded, so there's no need to support
actual NUL characters. The previous code truncated at NULs, but could have
kept appending more data to the value. This could have produced somewhat
random results.

  Commit: fc92bc3d1126fdb15a7c6ef58a0fa61ad87dadb7
      https://github.com/dovecot/core/commit/fc92bc3d1126fdb15a7c6ef58a0fa61ad87dadb7
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2018-04-23 (Mon, 23 Apr 2018)

  Changed paths:
    M src/plugins/fts-squat/fts-backend-squat.c

  Log Message:
  -----------
  fts-squat: Fix truncating headers at NULs while indexing

  Commit: b60044d481498ca4bfd2a47cfd11ecabb0471a5d
      https://github.com/dovecot/core/commit/b60044d481498ca4bfd2a47cfd11ecabb0471a5d
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2018-04-23 (Mon, 23 Apr 2018)

  Changed paths:
    M src/lib-mail/rfc2231-parser.c
    M src/lib-mail/rfc822-parser.c
    M src/lib-mail/rfc822-parser.h
    M src/lib-mail/test-rfc822-parser.c

  Log Message:
  -----------
  lib-mail: Change rfc822_parse_content_param() API to allow NULs in value

This was the only function in rfc822-parser.h that wasn't NUL-safe.
This won't fix anything, but it makes the rfc822-parser.h API fully
consistent with the NUL handling.

Normally rfc2231_parse() should be called instead of calling
rfc822_parse_content_param() directly, so this shouldn't break any plugins.

  Commit: ee120303949217b7f9ac33e036cf38aea39822c3
      https://github.com/dovecot/core/commit/ee120303949217b7f9ac33e036cf38aea39822c3
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2018-04-23 (Mon, 23 Apr 2018)

  Changed paths:
    M src/lib-mail/rfc822-parser.c
    M src/lib-mail/test-rfc822-parser.c

  Log Message:
  -----------
  lib-mail: rfc822-parser: Don't allow preserving escaped [CR]LF

It's not valid to have "\<CR>" or "\<LF>", so the old behavior isn't really
wrong either. However, rfc822_parse_quoted_string() callers are more likely
to expect that the output won't contain any [CR]LF so this new behavior is
a bit better.

  Commit: 39f9e4d3db603f312a43fddce21f566a960ff0b6
      https://github.com/dovecot/core/commit/39f9e4d3db603f312a43fddce21f566a960ff0b6
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2018-04-23 (Mon, 23 Apr 2018)

  Changed paths:
    M src/lib-mail/rfc822-parser.c
    M src/lib-mail/test-rfc822-parser.c

  Log Message:
  -----------
  lib-mail: rfc822-parser: Improve domain-literal parsing

Strip away folding whitespace. Treat any extra '[' as an error.

  Commit: 8bdf274e70a562a611950de3765472337e8825e8
      https://github.com/dovecot/core/commit/8bdf274e70a562a611950de3765472337e8825e8
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2018-04-23 (Mon, 23 Apr 2018)

  Changed paths:
    M src/lib-mail/rfc822-parser.c
    M src/lib-mail/test-rfc822-parser.c

  Log Message:
  -----------
  lib-mail: rfc822-parser: Strip away folding whitespace from comments

  Commit: 8d2ad317fce454e0aca3bd34ebf5fcf967393e3c
      https://github.com/dovecot/core/commit/8d2ad317fce454e0aca3bd34ebf5fcf967393e3c
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2018-04-23 (Mon, 23 Apr 2018)

  Changed paths:
    M src/lib-mail/rfc822-parser.c
    M src/lib-mail/rfc822-parser.h
    M src/lib-mail/test-rfc822-parser.c

  Log Message:
  -----------
  lib-mail: rfc822-parser: Add nul_replacement_char

  Commit: 39323a4666ad1be728c7f13c05f2fe4e15d43aa0
      https://github.com/dovecot/core/commit/39323a4666ad1be728c7f13c05f2fe4e15d43aa0
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2018-04-23 (Mon, 23 Apr 2018)

  Changed paths:
    M src/lib-mail/message-address.c
    M src/lib-mail/test-message-address.c

  Log Message:
  -----------
  lib-mail: message_address_parse*() - Replace NULs with 0x80

Instead of truncating the strings at NULs.

  Commit: 621cd6da69cb9592d6324d70245863179a39c338
      https://github.com/dovecot/core/commit/621cd6da69cb9592d6324d70245863179a39c338
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2018-04-23 (Mon, 23 Apr 2018)

  Changed paths:
    M src/lib-mail/rfc2231-parser.c
    M src/lib-mail/rfc2231-parser.h
    M src/lib-mail/test-rfc2231-parser.c

  Log Message:
  -----------
  lib-mail: rfc2231_parse() - Replace NULs with 0x80

Instead of truncating the strings at NULs.

  Commit: 95fba2c295bb84639dc64ca5c9f2ccb9edc00ba7
      https://github.com/dovecot/core/commit/95fba2c295bb84639dc64ca5c9f2ccb9edc00ba7
  Author: Sergey Kitov <sergey.kitov at open-xchange.com>
  Date:   2018-04-23 (Mon, 23 Apr 2018)

  Changed paths:
    M src/lib-mail/message-header-parser.c
    M src/lib-mail/message-header-parser.h

  Log Message:
  -----------
  lib-mail: Add MESSAGE_HEADER_REPLACE_NULS_WITH_0x80 flag

The flag signals that input stream for message_parse_header() should replace
0x0 symbols with 0x80.

  Commit: ab32237ffd78259013b4b2186292b0c80c34e955
      https://github.com/dovecot/core/commit/ab32237ffd78259013b4b2186292b0c80c34e955
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2018-04-23 (Mon, 23 Apr 2018)

  Changed paths:
    M src/lib/str.h

  Log Message:
  -----------
  lib: Improve str_append_n() comment

  Commit: 495c324e4d361d52ac869306cc8c060b60fb317b
      https://github.com/dovecot/core/commit/495c324e4d361d52ac869306cc8c060b60fb317b
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2018-04-23 (Mon, 23 Apr 2018)

  Changed paths:
    M src/lib/str.c
    M src/lib/str.h
    M src/lib/test-str.c

  Log Message:
  -----------
  lib: Add str_append_max()

It's otherwise the same as str_append_n(), except it takes const char* as
input instead of const void*. This, as well as its name, should make it
less likely to be used wrong when the input may legitimately have NULs.

The unit test is changed from str_append_n() to str_append_max(), because
str_append_n() will be deprecated.

  Commit: 07cbb7b116e0be888b43b4d27510f69e25b6e41d
      https://github.com/dovecot/core/commit/07cbb7b116e0be888b43b4d27510f69e25b6e41d
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2018-04-23 (Mon, 23 Apr 2018)

  Changed paths:
    M src/auth/auth-cache.c
    M src/auth/db-dict-cache-key.c
    M src/auth/db-ldap.c
    M src/auth/password-scheme-md5crypt.c
    M src/config/config-parser.c
    M src/config/doveconf.c
    M src/lib-dict-backend/dict-ldap.c
    M src/lib-dict/dict.c
    M src/lib-fs/fs-posix.c
    M src/lib-fs/fs-sis.c
    M src/lib-fts/fts-tokenizer-address.c
    M src/lib-http/http-auth.c
    M src/lib-http/http-header-parser.c
    M src/lib-http/http-parser.c
    M src/lib-http/http-url.c
    M src/lib-imap/imap-quote.c
    M src/lib-imap/imap-url.c
    M src/lib-imap/imap-utf7.c
    M src/lib-mail/message-header-parser.c
    M src/lib-mail/message-snippet.c
    M src/lib-mail/rfc822-parser.c
    M src/lib-mail/test-istream-dot.c
    M src/lib-mail/test-istream-header-filter.c
    M src/lib-mail/test-istream-qp-decoder.c
    M src/lib-mail/test-istream-qp-encoder.c
    M src/lib-mail/test-message-header-parser.c
    M src/lib-mail/test-message-parser.c
    M src/lib-settings/settings-parser.c
    M src/lib-settings/settings.c
    M src/lib-smtp/smtp-address.c
    M src/lib-smtp/smtp-parser.c
    M src/lib-smtp/smtp-reply-parser.c
    M src/lib-smtp/smtp-server-reply.c
    M src/lib-smtp/smtp-syntax.c
    M src/lib-storage/index/index-mail-headers.c
    M src/lib-storage/index/maildir/maildir-uidlist.c
    M src/lib-storage/index/mbox/mbox-save.c
    M src/lib-storage/index/mbox/mbox-sync-parse.c
    M src/lib-storage/mailbox-attribute.c
    M src/lib-storage/mailbox-list.c
    M src/lib-storage/mailbox-tree.c
    M src/lib/failures.c
    M src/lib/file-dotlock.c
    M src/lib/istream.c
    M src/lib/str-sanitize.c
    M src/lib/strescape.c
    M src/lib/test-path-util.c
    M src/lib/uri-util.c
    M src/plugins/fts-solr/fts-backend-solr-old.c
    M src/plugins/fts-solr/fts-backend-solr.c

  Log Message:
  -----------
  global: Replace str_append_n() with str_append_data() wherever possible

It shouldn't be possible for input to have NULs in any of these places.
This makes the extra NUL-check in str_append_n() unnecessary.

  Commit: b8b0e5660b871c40024063308d40c9607e3e3aad
      https://github.com/dovecot/core/commit/b8b0e5660b871c40024063308d40c9607e3e3aad
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2018-04-23 (Mon, 23 Apr 2018)

  Changed paths:
    M src/lib/strescape.c

  Log Message:
  -----------
  lib: str_append_[tab]unescaped(): Don't truncate input at NULs

For str_append_tabunescaped() the input is supposed to have NULs already
escaped though, so it was truncating only with invalid input.

  Commit: c68782f4b1e3086cbadf533e6aae647c5e60fd00
      https://github.com/dovecot/core/commit/c68782f4b1e3086cbadf533e6aae647c5e60fd00
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2018-04-23 (Mon, 23 Apr 2018)

  Changed paths:
    M src/lib-fts/fts-tokenizer-address.c

  Log Message:
  -----------
  lib-fts: fts-tokenizer-address - Cleanup NUL handling in parent_data

The end result is anyway that the token will be truncated at the first NUL.
This change just makes the code a bit more understandable on where the
truncation will happen.

  Commit: b50b90941fc1f024b0a4e4fb4fed981fa6565726
      https://github.com/dovecot/core/commit/b50b90941fc1f024b0a4e4fb4fed981fa6565726
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2018-04-23 (Mon, 23 Apr 2018)

  Changed paths:
    M src/config/doveconf.c
    M src/doveadm/dsync/dsync-mailbox-tree-sync.c
    M src/lib/failures.c
    M src/lib/var-expand.c

  Log Message:
  -----------
  global: Replace str_append_n() with str_append_max()

  Commit: dd477f872229fb42715b0f9591b04df08a725775
      https://github.com/dovecot/core/commit/dd477f872229fb42715b0f9591b04df08a725775
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2018-04-23 (Mon, 23 Apr 2018)

  Changed paths:
    M src/lib/compat.h
    M src/lib/macros.h

  Log Message:
  -----------
  lib: Add ATTR_DEPRECATED macro

  Commit: d8449ed1b2327db1d02bf14b0f122479a26bf01c
      https://github.com/dovecot/core/commit/d8449ed1b2327db1d02bf14b0f122479a26bf01c
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2018-04-23 (Mon, 23 Apr 2018)

  Changed paths:
    M src/lib/str.h

  Log Message:
  -----------
  lib: Mark str_append_n() deprecated

It was almost always used wrongly. Most of the time str_append_data()
should have been used instead. str_append_max() has the old functionality.

  Commit: 955384f916aafb6912b03154bb4ca7743c88daf0
      https://github.com/dovecot/core/commit/955384f916aafb6912b03154bb4ca7743c88daf0
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2018-04-23 (Mon, 23 Apr 2018)

  Changed paths:
    M src/lib-storage/index/dbox-common/dbox-mail.c
    M src/lib-storage/index/imapc/imapc-mail.c
    M src/lib-storage/index/imapc/imapc-mailbox.c
    M src/lib-storage/index/index-mail.c
    M src/lib-storage/index/pop3c/pop3c-sync.c

  Log Message:
  -----------
  lib-storage: Don't add trailing NUL to strings in dovecot.index.cache

They are unnecessary and just unnecessarily consume disk space. The
existing reading code works even if they don't exist, so this change won't
break backwards compatibility.

Compare: https://github.com/dovecot/core/compare/5c62b6eebed4...955384f916aa