[dovecot/core] d506fc: lib: Add assert to io_stream_ref()

Wed Nov 1 14:00:12 EET 2017

  Branch: refs/heads/master
  Home:   https://github.com/dovecot/core
  Commit: d506fc38cc533607b75008a369b40c85d0c09b0a
      https://github.com/dovecot/core/commit/d506fc38cc533607b75008a369b40c85d0c09b0a
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2017-11-01 (Wed, 01 Nov 2017)

  Changed paths:
    M src/lib/iostream.c

  Log Message:
  -----------
  lib: Add assert to io_stream_ref()

This makes an assert-crash likely to happen if iostream was already freed.

  Commit: a6657bc038805b573bd1fac568b8eb0fae673847
      https://github.com/dovecot/core/commit/a6657bc038805b573bd1fac568b8eb0fae673847
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2017-11-01 (Wed, 01 Nov 2017)

  Changed paths:
    M src/imap/cmd-append.c

  Log Message:
  -----------
  imap: Set max buffer size for CATENATE istream-chain

This fixes a crash with mbox when nonexistent CATENATE URL is attempted to
be used. The istream doesn't get any size, and istream-header-filter panics
due to having reached the maximum buffer size (0), even though the stream is
empty and there's nothing to read.

  Commit: fdf15b2c4fd47078108f556a58c6b9f365449f9e
      https://github.com/dovecot/core/commit/fdf15b2c4fd47078108f556a58c6b9f365449f9e
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2017-11-01 (Wed, 01 Nov 2017)

  Changed paths:
    M src/lib-mail/istream-header-filter.c

  Log Message:
  -----------
  lib-mail: istream-header-filter - Fix header read() when max_buffer_size=0

If the buffer was too small to read anything into it, read_header() returned
-2, but then it continued reading the header as if it was the body already.
This should instead be an error situation - max buffer size just shouldn't be
0 at this point.

This was originally changed in a72355fef49c512c13b7d8c3e07f4f76ebd57a10 as
some kind of a workaround, but since then we implemented support for reading
a mixed header/body in c2afdbf963d0564a542d71ca784deb0c2f7776d0.

  Commit: 59a63791d4ec70a134cb0dcbad1255d952075efe
      https://github.com/dovecot/core/commit/59a63791d4ec70a134cb0dcbad1255d952075efe
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2017-11-01 (Wed, 01 Nov 2017)

  Changed paths:
    M src/lib/istream-chain.c
    M src/lib/istream-concat.c

  Log Message:
  -----------
  lib: istream-chain/concat cleanup - return early if no new data is read

There's no need to change the buffer or other variables. This simplifies
the following changes.

This cleanup is identical for istream-concat and istream-chain.

  Commit: 4c7d704408fc77467143a945dc2d0a856f72e674
      https://github.com/dovecot/core/commit/4c7d704408fc77467143a945dc2d0a856f72e674
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2017-11-01 (Wed, 01 Nov 2017)

  Changed paths:
    M src/lib/Makefile.am
    A src/lib/memarea.c
    A src/lib/memarea.h
    M src/lib/test-lib.inc
    A src/lib/test-memarea.c

  Log Message:
  -----------
  lib: Add memarea API

This can be used to create reference counted memory areas where a callback
is called once the refcount drops to zero.

  Commit: 785ee8becdb11e41abaaf64c28eb3923215d1f27
      https://github.com/dovecot/core/commit/785ee8becdb11e41abaaf64c28eb3923215d1f27
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2017-11-01 (Wed, 01 Nov 2017)

  Changed paths:
    M src/lib/istream-private.h
    M src/lib/istream.c

  Log Message:
  -----------
  lib: Add i_stream_free_buffer() to free i_stream_*alloc()ed memory

  Commit: 54bd0fec0be357266e299466a582f3c9269884e9
      https://github.com/dovecot/core/commit/54bd0fec0be357266e299466a582f3c9269884e9
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2017-11-01 (Wed, 01 Nov 2017)

  Changed paths:
    M src/lib-http/http-transfer-chunked.c
    M src/lib-ssl-iostream/istream-openssl.c
    M src/lib/istream-chain.c
    M src/lib/istream-concat.c
    M src/lib/istream-multiplex.c
    M src/lib/istream.c

  Log Message:
  -----------
  global: Use i_stream_free_buffer() instead of freeing w_buffer directly

  Commit: 5ca745ae8e9903ef5265360e4c882e9314595020
      https://github.com/dovecot/core/commit/5ca745ae8e9903ef5265360e4c882e9314595020
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2017-11-01 (Wed, 01 Nov 2017)

  Changed paths:
    M src/lib/istream-private.h
    M src/lib/istream.c

  Log Message:
  -----------
  lib: Use memarea for default istream memory allocations.

  Commit: 5181e190e968c25b19daa4489b278730326eec62
      https://github.com/dovecot/core/commit/5181e190e968c25b19daa4489b278730326eec62
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2017-11-01 (Wed, 01 Nov 2017)

  Changed paths:
    M src/lib-test/test-istream.c

  Log Message:
  -----------
  lib-test: istream-test - use memarea API

  Commit: 82ea464c113f43aaa2b2e23de334cf3081c332be
      https://github.com/dovecot/core/commit/82ea464c113f43aaa2b2e23de334cf3081c332be
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2017-11-01 (Wed, 01 Nov 2017)

  Changed paths:
    M src/lib/istream-private.h
    M src/lib/istream.c

  Log Message:
  -----------
  lib: Add istream.snapshot() method for referencing the current memarea

It will be used by following commits.

  Commit: 3fe5eaeb32a8bbc31ce0673793b1c37f72d00d47
      https://github.com/dovecot/core/commit/3fe5eaeb32a8bbc31ce0673793b1c37f72d00d47
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2017-11-01 (Wed, 01 Nov 2017)

  Changed paths:
    M src/lib/istream-seekable.c

  Log Message:
  -----------
  lib: istream-seekable - Fix fd leak if initial temp file couldn't be read back

That should normally never happen.

  Commit: 3e55775bc7a37ebc05e06c04cafb32eee9888e87
      https://github.com/dovecot/core/commit/3e55775bc7a37ebc05e06c04cafb32eee9888e87
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2017-11-01 (Wed, 01 Nov 2017)

  Changed paths:
    M src/lib/istream-seekable.c

  Log Message:
  -----------
  lib: istream-seekable - Change to use i_stream_*alloc()

This simplifies implementing the snapshotting.

  Commit: 448ab3576e0def2135c6f4681bd412dd59425d44
      https://github.com/dovecot/core/commit/448ab3576e0def2135c6f4681bd412dd59425d44
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2017-11-01 (Wed, 01 Nov 2017)

  Changed paths:
    M src/lib/istream-seekable.c

  Log Message:
  -----------
  lib: istream-seekable - Implement snapshot()

  Commit: d74c9f4bf7ee37f3f58a895032ea013d07294865
      https://github.com/dovecot/core/commit/d74c9f4bf7ee37f3f58a895032ea013d07294865
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2017-11-01 (Wed, 01 Nov 2017)

  Changed paths:
    M src/lib/istream-mmap.c

  Log Message:
  -----------
  lib: istream-mmap - Use memarea API

  Commit: f5d8e05492a3d2552b54129344e37ca01d241832
      https://github.com/dovecot/core/commit/f5d8e05492a3d2552b54129344e37ca01d241832
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2017-11-01 (Wed, 01 Nov 2017)

  Changed paths:
    M src/lib-mail/istream-attachment-extractor.c

  Log Message:
  -----------
  lib-mail: istream-attachment-extractor - Don't modify buffer during failure

If i_stream_read() returns -1 because the attachment file couldn't be
created, don't add the trailing data to the main istream. It's not useful,
and the following extra checks will cause it to assert-crash.

  Commit: e05039894bb8fa68d6e5bc2c36fd870463bcd09f
      https://github.com/dovecot/core/commit/e05039894bb8fa68d6e5bc2c36fd870463bcd09f
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2017-11-01 (Wed, 01 Nov 2017)

  Changed paths:
    M src/lib-compression/istream-lz4.c

  Log Message:
  -----------
  lib-compression: istream-lz4 - Use i_stream_alloc()

  Commit: 51d379cbc50242a13462d0fded50e013eb00cc07
      https://github.com/dovecot/core/commit/51d379cbc50242a13462d0fded50e013eb00cc07
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2017-11-01 (Wed, 01 Nov 2017)

  Changed paths:
    M src/lib/istream-private.h
    M src/lib/istream.c

  Log Message:
  -----------
  lib: Add i_stream_try_alloc_avoid_compress()

  Commit: ef4661aec4031e24944175ab914dc60942a2b68a
      https://github.com/dovecot/core/commit/ef4661aec4031e24944175ab914dc60942a2b68a
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2017-11-01 (Wed, 01 Nov 2017)

  Changed paths:
    M src/lib-compression/istream-bzlib.c
    M src/lib-compression/istream-lzma.c
    M src/lib-compression/istream-zlib.c

  Log Message:
  -----------
  lib-compression: Use i_stream_try_alloc*()

  Commit: 58e397e90c4b8f7f042879e029dfa6c093b3143a
      https://github.com/dovecot/core/commit/58e397e90c4b8f7f042879e029dfa6c093b3143a
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2017-11-01 (Wed, 01 Nov 2017)

  Changed paths:
    M src/lib-ssl-iostream/istream-openssl.c

  Log Message:
  -----------
  lib-ssl-iostream: Simplify istream-openssl

Partially reverts 739125f23e3312045e620014812fe2249a309cc4. This is no
longer needed because of the io_set_pending() changes.

  Commit: 2974dca6be5120e49279f06c8aa952e5fac56048
      https://github.com/dovecot/core/commit/2974dca6be5120e49279f06c8aa952e5fac56048
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2017-11-01 (Wed, 01 Nov 2017)

  Changed paths:
    M src/lib-compression/istream-bzlib.c
    M src/lib-compression/istream-lz4.c
    M src/lib-compression/istream-lzma.c
    M src/lib-compression/istream-zlib.c
    M src/lib-dcrypt/istream-decrypt.c
    M src/lib-fs/istream-fs-file.c
    M src/lib-fs/istream-fs-stats.c
    M src/lib-fs/istream-metawrap.c
    M src/lib-http/http-server-request.c
    M src/lib-http/http-transfer-chunked.c
    M src/lib-mail/istream-attachment-extractor.c
    M src/lib-mail/istream-binary-converter.c
    M src/lib-mail/istream-dot.c
    M src/lib-mail/istream-header-filter.c
    M src/lib-mail/istream-nonuls.c
    M src/lib-mail/istream-qp-decoder.c
    M src/lib-mail/istream-qp-encoder.c
    M src/lib-program-client/program-client-remote.c
    M src/lib-ssl-iostream/istream-openssl.c
    M src/lib-storage/index/istream-mail.c
    M src/lib-storage/index/mbox/istream-raw-mbox.c
    M src/lib-test/test-istream.c
    M src/lib/istream-base64-decoder.c
    M src/lib/istream-base64-encoder.c
    M src/lib/istream-callback.c
    M src/lib/istream-chain.c
    M src/lib/istream-concat.c
    M src/lib/istream-crlf.c
    M src/lib/istream-data.c
    M src/lib/istream-failure-at.c
    M src/lib/istream-file.c
    M src/lib/istream-hash.c
    M src/lib/istream-jsonstr.c
    M src/lib/istream-limit.c
    M src/lib/istream-mmap.c
    M src/lib/istream-multiplex.c
    M src/lib/istream-private.h
    M src/lib/istream-rawlog.c
    M src/lib/istream-seekable.c
    M src/lib/istream-sized.c
    M src/lib/istream-tee.c
    M src/lib/istream-timeout.c
    M src/lib/istream-try.c
    M src/lib/istream.c
    M src/plugins/mail-filter/istream-ext-filter.c

  Log Message:
  -----------
  global: Add flags parameter to i_stream_create() and set it to 0

This just changes the API - the following changes add the new flag.

  Commit: 89a89730a1dd98edb5c16a5b65f693389eb81124
      https://github.com/dovecot/core/commit/89a89730a1dd98edb5c16a5b65f693389eb81124
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2017-11-01 (Wed, 01 Nov 2017)

  Changed paths:
    M src/lib/istream-private.h
    M src/lib/istream.c

  Log Message:
  -----------
  lib: Add ISTREAM_CREATE_FLAG_NOOP_SNAPSHOT flag

  Commit: 25c833d3cff7047f5c383892c379f59d116ff218
      https://github.com/dovecot/core/commit/25c833d3cff7047f5c383892c379f59d116ff218
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2017-11-01 (Wed, 01 Nov 2017)

  Changed paths:
    M src/lib/istream-chain.c
    M src/lib/istream-concat.c
    M src/lib/istream-data.c
    M src/lib/istream-tee.c

  Log Message:
  -----------
  lib: Use ISTREAM_CREATE_FLAG_NOOP_SNAPSHOT where useful

  Commit: e0fab14602b73ff590b2a9c5d9e67e2dfb5d1f9e
      https://github.com/dovecot/core/commit/e0fab14602b73ff590b2a9c5d9e67e2dfb5d1f9e
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2017-11-01 (Wed, 01 Nov 2017)

  Changed paths:
    M src/lib/istream-private.h
    M src/lib/istream.c
    M src/lib/istream.h

  Log Message:
  -----------
  lib: i_stream_read() - Use snapshots

This guarantees that data returned by i_stream_get_data() won't be freed
until the next i_stream_read() returns >0. This is mainly important,
because often the error handling for <=0 doesn't update the buffer
pointers correctly and it leads to complicated bugs where already freed
memory is attempted to be read (but never written).

  Commit: 66967e15d8ed35f30493dd9694a0aabe1a266f9d
      https://github.com/dovecot/core/commit/66967e15d8ed35f30493dd9694a0aabe1a266f9d
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2017-11-01 (Wed, 01 Nov 2017)

  Changed paths:
    M src/lib/istream.c

  Log Message:
  -----------
  lib: i_stream_read() - Add buffer validity checks if DEBUG is enabled

  Commit: fa39c427573ada77265b833db687ca7e854794a1
      https://github.com/dovecot/core/commit/fa39c427573ada77265b833db687ca7e854794a1
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2017-11-01 (Wed, 01 Nov 2017)

  Changed paths:
    M src/lib/istream.c

  Log Message:
  -----------
  lib: i_stream_compress() - assert-crash if trying to compress shared memarea

The caller is always expected to check the refcount before calling this.

  Commit: bcd286622779a93f809b11993db0550f8c7cc9b5
      https://github.com/dovecot/core/commit/bcd286622779a93f809b11993db0550f8c7cc9b5
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2017-11-01 (Wed, 01 Nov 2017)

  Changed paths:
    M src/lib-dcrypt/istream-decrypt.c
    M src/lib-fs/istream-metawrap.c
    M src/lib-http/http-transfer-chunked.c
    M src/lib-mail/istream-dot.c
    M src/lib-mail/istream-header-filter.c
    M src/lib-mail/istream-nonuls.c
    M src/lib-mail/istream-qp-decoder.c
    M src/lib-mail/istream-qp-encoder.c
    M src/lib-program-client/program-client-remote.c
    M src/lib-storage/index/mbox/istream-raw-mbox.c
    M src/lib/istream-base64-decoder.c
    M src/lib/istream-base64-encoder.c
    M src/lib/istream-chain.c
    M src/lib/istream-concat.c
    M src/lib/istream-crlf.c
    M src/lib/istream-jsonstr.c
    M src/lib/istream-limit.c
    M src/lib/istream-multiplex.c
    M src/lib/istream-rawlog.c
    M src/lib/istream-seekable.c
    M src/lib/istream-sized.c
    M src/lib/istream.c
    M src/plugins/mail-filter/istream-ext-filter.c

  Log Message:
  -----------
  global: Use i_stream_read_memarea() wherever possible

Compare: https://github.com/dovecot/core/compare/97b6a92d5756...bcd286622779