zlib errors after upgrading to 2.3.11.3

[Robert Nowotny]

 On 9. Sep 2020, at 11.14, Robert Nowotny <rnowotny1966 at gmail.com> wrote:


Sep  3 08:33:25 lxc-imap dovecot: imap(mpaul)<48684><2/9E5mKuAezAqKjk>:
Error: Mailbox Sent: UID=2171:
read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed:
read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe
(FETCH BODY[])


Also this way you can see if the broken mail is actually xz or zstd or
zlib. It would be nice to know if there are any zstd or zlib compressed
mails that have problems. We did a lot of stress testing with zstd and also
with xz, but haven't been able to reproduce any problems. It's also strange
that it says taht the error is "Broken pipe" - that doesn't indicate that
the mail is corrupted but that there is something more strange going on. So
perhaps you don't actually have any mails written as corrupted, but Dovecot
is just somehow having trouble reading the mails.


I managed to reproduce this. The files aren't corrupted, it's just that
reading is failing. The attached patch should fix the xz code and should
make your files readable again.


Yes, it works ! I switched now to "zstd" compression and currently
re-compressing all user imap folders (one by one)


Actually, with further testing it looks like the mails were written
truncated. My patch simply hides the truncation when reading the mail. But
it's anyway best that can be done about them.

zstd also has a bug in writing compressed output, but it should always
crash instead of writing broken output.

We'll try to get fixes to these merged to git soon.


currently I am re-compressing the mailboxes and so far i can see only one
(1) Error, after about 120GB of recomresses emails :

Sep 10 11:32:24 lxc-imap dovecot: imap(ebay)<37070><1aDvPvKus+/AqKjk>:
Panic: file ostream.c: line 287 (o_stream_sendv_int): assertion failed:
(!stream->blocking)
Sep 10 11:32:24 lxc-imap dovecot: imap(ebay)<37070><1aDvPvKus+/AqKjk>:
Error: Raw backtrace:
/usr/local/lib/dovecot/libdovecot.so.0(backtrace_append+0x41)
[0x7f5b00de6da1] ->
/usr/local/lib/dovecot/libdovecot.so.0(backtrace_get+0x22) [0x7f5b00de6ec2]
-> /usr/local/lib/dovecot/libdovecot.so.0(+0xfe5eb) [0x7f5b00df05eb] ->
/usr/local/lib/dovecot/libdovecot.so.0(+0xfe687) [0x7f5b00df0687] ->
/usr/local/lib/dovecot/libdovecot.so.0(+0x598e1) [0x7f5b00d4b8e1] ->
/usr/local/lib/dovecot/libdovecot.so.0(+0x5e2c1) [0x7f5b00d502c1] ->
/usr/local/lib/dovecot/libdovecot.so.0(o_stream_sendv+0x32)
[0x7f5b00e14a92] ->
/usr/local/lib/dovecot/libdovecot.so.0(io_stream_copy+0x5a)
[0x7f5b00e1535a] ->
/usr/local/lib/dovecot/libdovecot.so.0(o_stream_send_istream+0x50)
[0x7f5b00e15060] ->
/usr/local/lib/dovecot/libdovecot-storage.so.0(index_storage_save_continue+0x2d)
[0x7f5b00f9659d] ->
/usr/local/lib/dovecot/libdovecot-storage.so.0(dbox_save_continue+0x30)
[0x7f5b00f3d3b0] ->
/usr/local/lib/dovecot/libdovecot-storage.so.0(mailbox_save_continue+0x3e)
[0x7f5b00f1a34e] -> dovecot/imap [ebay 192.168.168.228 APPEND](+0x13801)
[0x55d6905bf801] -> dovecot/imap [ebay 192.168.168.228
APPEND](command_exec+0x6c) [0x55d6905ce3ec] -> dovecot/imap [ebay
192.168.168.228 APPEND](+0x12fdb) [0x55d6905befdb] ->
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x6d)
[0x7f5b00e06ced] ->
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x145)
[0x7f5b00e08305] ->
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x54)
[0x7f5b00e06d94] ->
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x48) [0x7f5b00e06f08]
-> /usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x17)
[0x7f5b00d7f8d7] -> dovecot/imap [ebay 192.168.168.228 APPEND](main+0x459)
[0x55d6905bec99] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)
[0x7f5b00b0e0b3] -> dovecot/imap [ebay 192.168.168.228 APPEND](_start+0x2e)
[0x55d6905bed2e]
Sep 10 11:32:24 lxc-imap dovecot: imap-login: Panic: epoll_ctl(del, 21)
failed: Bad file descriptor
Sep 10 11:32:24 lxc-imap dovecot: imap(ebay)<37070><1aDvPvKus+/AqKjk>:
Fatal: master: service(imap): child 37070 killed with signal 6 (core dumps
disabled - https://dovecot.org/bugreport.html#coredumps)


until now I only have one broken mailbox, I need to repair (I will try that
later, just finish all the other users before):

Sep 10 12:13:13 lxc-imap dovecot: imap(mtrenner)<41872><KVTf0PKu7+7AqKi0>:
Error: lzma.read(/home/vmail/virtualmailboxes/mtrenner/storage/m.349):
corrupted data at 3763912
Sep 10 12:13:13 lxc-imap dovecot: imap(mtrenner)<41872><KVTf0PKu7+7AqKi0>:
Error: Mailbox Sent: UID=1480:
read(zlib(/home/vmail/virtualmailboxes/mtrenner/storage/m.349)) failed:
read(/home/vmail/virtualmailboxes/mtrenner/storage/m.349) failed:
lzma.read(/home/vmail/virtualmailboxes/mtrenner/storage/m.349): corrupted
data at 3763912 (FETCH BODY[])

> zstd also has a bug in writing compressed output, but it should always
crash instead of writing broken output.
> We'll try to get fixes to these merged to git soon.

Uh-Oh, that would be good ... pls notify !
how such thing can slip through the tests ?
I just can not remember, the last error with compression was like 10 years
ago (dont know if it was me who reported that)

However, thank You for Your fast response and taking care !!!

Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dovecot.org/pipermail/dovecot/attachments/20200910/b4f25fd9/attachment.html>