i_stream_read_memarea: assertion failed: (!stream->blocking)

Continuation of https://dovecot.org/pipermail/dovecot/2021-January/121235.html

• I didn't get a reply to my last mail, so ended up deleting the various messages (or in some cases mailboxes) and stopped running into problems for a while, but am now seeing it again.

I get "i_stream_read_memarea: assertion failed: (!stream->blocking)" when accessing some messages. Backtrace and doveconf -n below.

$ doveadm -D search from abcdefghij [...] Aug 13 17:30:02 doveadm(sthen): Debug: Mailbox trash: UID 1177: Looked up field guid from mail cache Aug 13 17:30:02 doveadm(sthen): Debug: Mailbox trash: UID 1177: Opened mail because: 1/1 headers not cached (first=from) (Mail has other cached fields, reset_id=1470764070) Aug 13 17:30:02 doveadm(sthen): Debug: Mailbox trash: UID 1178: Looked up field guid from mail cache Aug 13 17:30:02 doveadm(sthen): Debug: Mailbox trash: UID 1178: Looked up field guid from mail cache Aug 13 17:30:02 doveadm(sthen): Debug: Mailbox trash: UID 1178: Looked up field guid from mail cache Aug 13 17:30:02 doveadm(sthen): Debug: Mailbox trash: UID 1178: Opened mail because: 1/1 headers not cached (first=from) (Mail has other cached fields, reset_id=1470764070) Aug 13 17:30:02 doveadm(sthen): Panic: file istream.c: line 345 (i_stream_read_memarea): assertion failed: (!stream->blocking) Abort trap (core dumped)

Some other searches hitting the same message do succeed though e.g. "doveadm search mailbox trash".

Somehow after various other searches trying to figure out what triggered the crash and what didn't (e.g. "doveadm -D search mailbox trash uid 1170:12000 from abc" crashed but "uid 1170:1190" didn't) it has stopped failing with that particular message, but I am still hitting it in at least one other mailbox.

The messages where I'm seeing this so far seem to all be stored with lz4 (a couple of years old mostly).

Any ideas?

GNU gdb (GDB) 7.12.1 Copyright (C) 2017 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-openbsd6.9". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from doveadm...Reading symbols from /usr/local/bin/.debug/doveadm.dbg...done. done. [New process 130792] Core was generated by `doveadm'. Program terminated with signal SIGABRT, Aborted. #0 thrkill () at /tmp/-:3 3 /tmp/-: No such file or directory. (gdb) bt #0 thrkill () at /tmp/-:3 #1 0x00000c3b2a1ee58e in _libc_abort () at /usr/src/lib/libc/stdlib/abort.c:51 #2 0x00000c3b2780f876 in default_fatal_finish (type=LOG_TYPE_PANIC, status=0) at failures.c:459 #3 0x00000c3b2780dc94 in fatal_handler_real (ctx=0x7f7ffffcb040, format=<error reading variable: Unhandled dwarf expression opcode 0xa3>, args=<error reading variable: Unhandled dwarf expression opcode 0xa3>) at failures.c:471 #4 0x00000c3b2780dc51 in default_fatal_handler (ctx=0x0, format=0x6 <error: Cannot access memory at address 0x6>, args=0x0) at failures.c:479 #5 0x00000c3b2780ded2 in i_panic (format=0x6 <error: Cannot access memory at address 0x6>) at failures.c:524 #6 0x00000c3b2781db26 in i_stream_read_memarea (stream=0xc3b41c94080) at istream.c:345 #7 0x00000c3b27827c2f in read_more (sstream=0xc3b41c80c00) at istream-seekable.c:152 #8 0x00000c3b27827a70 in read_from_buffer (sstream=0xc3b41c80c00, ret_r=0x7f7ffffcb188) at istream-seekable.c:207 #9 0x00000c3b27826f67 in i_stream_seekable_read (stream=0xc3b41c80c00) at istream-seekable.c:269 #10 0x00000c3b2781da74 in i_stream_read_memarea (stream=0xc3b41c80c80) at istream.c:326 #11 0x00000c3b2782568c in i_stream_limit_read (stream=0xc3b1097d000) at istream-limit.c:51 #12 0x00000c3b2781da74 in i_stream_read_memarea (stream=0xc3b1097d080) at istream.c:326 #13 0x00000c3b2781e16c in i_stream_read_copy_from_parent (istream=<optimized out>) at istream.c:400 #14 0x00000c3bc821b24a in i_stream_mail_read (stream=0xc3b41c83200) at istream-mail.c:115 #15 0x00000c3b2781da74 in i_stream_read_memarea (stream=0xc3b41c83280) at istream.c:326 #16 0x00000c3b2781ef0a in i_stream_read (stream=0xc3b41c83280) at istream.c:283 #17 i_stream_read_data (stream=0xc3b41c83280, data_r=0x7f7ffffcb3b8, size_r=0x7f7ffffcb3b0, threshold=1) at istream.c:760 #18 0x00000c3b277e4016 in i_stream_read_bytes (stream=0x0, data_r=<optimized out>, size_r=<optimized out>, wanted=<optimized out>) at ../../src/lib/istream.h:220 #19 message_parse_header_next (ctx=0xc3b109a4300, hdr_r=0x7f7ffffcb470) at message-header-parser.c:85 #20 0x00000c3b277dc5f2 in read_header (mstream=0xc3b41cb0800) at istream-header-filter.c:195 #21 i_stream_header_filter_read (stream=0xc3b41cb0800) at istream-header-filter.c:450 #22 0x00000c3b2781da74 in i_stream_read_memarea (stream=0xc3b41cb0880) at istream.c:326 #23 0x00000c3b2781ef0a in i_stream_read (stream=0xc3b41cb0880) at istream.c:283 #24 i_stream_read_data (stream=0xc3b41cb0880, data_r=0x7f7ffffcb568, size_r=0x7f7ffffcb560, threshold=1) at istream.c:760 #25 0x00000c3b277e4016 in i_stream_read_bytes (stream=0x0, data_r=<optimized out>, size_r=<optimized out>, wanted=<optimized out>) at ../../src/lib/istream.h:220 #26 message_parse_header_next (ctx=0xc3b1097a800, hdr_r=0x7f7ffffcb5b8) at message-header-parser.c:85 #27 0x00000c3b277e4879 in message_parse_header (input=0xc3b41cb0880, hdr_size=0x0, flags=<optimized out>, callback=0xc3bc822a3a0 <search_header>, context=0x7f7ffffcb608) at message-header-parser.c:392 #28 0x00000c3bc8229c62 in search_arg_match_text (args=0xc3b10987110, ctx=0xc3b41c89400) at index-search.c:769 #29 search_match_once (ctx=0xc3b41c89400) at index-search.c:1408 #30 0x00000c3bc822883e in search_match_next (ctx=0xc3b41c89400) at index-search.c:1516 #31 search_more_with_mail (ctx=0xc3b41c89400, mail=<optimized out>) at index-search.c:1632 #32 search_more_with_prefetching (ctx=0xc3b41c89400, mail_r=0x7f7ffffcb770) at index-search.c:1714 #33 search_more (ctx=0xc3b41c89400, mail_r=0x7f7ffffcb770) at index-search.c:1788 #34 0x00000c3bc82282cf in index_storage_search_next_nonblock (_ctx=0xc3b41c89400, mail_r=0x7f7ffffcb848, tryagain_r=0x7f7ffffcb7bf) at index-search.c:1816 #35 0x00000c3bc8282636 in mailbox_search_next_nonblock (ctx=0xc3b41c89400, mail_r=0x7f7ffffcb848, tryagain_r=<optimized out>) at mail-storage.c:2290 #36 mailbox_search_next (ctx=0xc3b41c89400, mail_r=0x7f7ffffcb848) at mail-storage.c:2275 #37 0x00000c38e91eeb31 in cmd_search_box (ctx=<optimized out>, info=<optimized out>) at doveadm-mail-search.c:36 #38 cmd_search_run (ctx=<optimized out>, user=<error reading variable: Unhandled dwarf expression opcode 0xa3>) at doveadm-mail-search.c:61 #39 0x00000c38e91e3ec3 in doveadm_mail_next_user (ctx=0xc3b10980048, error_r=0x7f7ffffcb9b0) at doveadm-mail.c:462 #40 0x00000c38e91e565a in doveadm_mail_cmd_exec (ctx=0xc3b10980048, wildcard_user=0x0) at doveadm-mail.c:653 #41 0x00000c38e91e51e3 in doveadm_cmd_ver2_to_mail_cmd_wrapper (cctx=<optimized out>) at doveadm-mail.c:1153 #42 0x00000c38e91f7f38 in doveadm_cmd_run_ver2 (argc=3, argv=0x7f7ffffcbc98, cctx=0x7f7ffffcbb70) at doveadm-cmd.c:575 #43 0x00000c38e91f78e5 in doveadm_cmd_try_run_ver2 ( cmd_name=<error reading variable: Unhandled dwarf expression opcode 0xa3>, argc=0, argv=0x6, cctx=0x7f7ffffcbb70) at doveadm-cmd.c:497 #44 0x00000c38e91fb636 in main (argc=6, argv=0x0) at doveadm.c:366 (gdb) frame 6 #6 0x00000c3b2781db26 in i_stream_read_memarea (stream=0xc3b41c94080) at istream.c:345 345 i_assert(!stream->blocking); (gdb) p *stream $1 = { v_offset = 0, stream_errno = 0, mmaped = false, blocking = true, closed = false, readable_fd = false, seekable = false, eof = true, real_stream = 0xc3b41c94000

$ doveconf -n # some parts replaced with <snip> # 2.3.16 (7e2e900c1a): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.16 (09c29328) # OS: OpenBSD 6.9 amd64 # Hostname: naiad.spacehopper.org dsync_remote_cmd = /usr/bin/ssh <snip> first_valid_uid = 1000 lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c session=<%{session}> %k login_trusted_networks = 127.0.0.1 <snip> mail_location = mdbox:~/mdbox mail_plugins = " zlib notify replication" managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext vacation-seconds spamtest spamtestplus imapsieve vnd.dovecot.imapsieve mbox_write_locks = fcntl mdbox_rotate_interval = 4 days mdbox_rotate_size = 15 M mmap_disable = yes namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = separator = / } passdb { args = scheme=CRYPT username_format=%u /etc/dovecot/users driver = passwd-file } plugin { imapsieve_mailbox1_before = file:/etc/dovecot/sieve/report-spam.sieve imapsieve_mailbox1_causes = COPY imapsieve_mailbox1_name = zSpam imapsieve_mailbox2_before = file:/etc/dovecot/sieve/report-ham.sieve imapsieve_mailbox2_causes = COPY imapsieve_mailbox2_from = zSpam imapsieve_mailbox2_name = * imapsieve_mailbox3_before = file:/etc/dovecot/sieve/report-ham.sieve imapsieve_mailbox3_causes = COPY imapsieve_mailbox3_name = zHam mail_replica = <snip> sieve = file:~/sieve;active=~/.dovecot.sieve sieve_before = /etc/dovecot/sieve/before.d/ sieve_extensions = +spamtest +spamtestplus +vacation-seconds sieve_global_extensions = +vnd.dovecot.pipe sieve_pipe_bin_dir = /etc/dovecot/sieve/bin sieve_plugins = sieve_imapsieve sieve_extprograms sieve_spamtest_max_header = X-Spamd-Result: default: [[:alnum:]]+ \[-?[[:digit:]]+\.[[:digit:]]+ / (-?[[:digit:]]+\.[[:digit:]]+)\] sieve_spamtest_status_header = X-Spamd-Result: default: [[:alnum:]]+ \[(-?[[:digit:]]+\.[[:digit:]]+) / -?[[:digit:]]+\.[[:digit:]]+\] sieve_spamtest_status_type = score sieve_vacation_default_period = 6h sieve_vacation_max_period = 0 sieve_vacation_min_period = 1m } protocols = imap pop3 lmtp sieve replication_max_conns = 4 service aggregator { fifo_listener replication-notify-fifo { mode = 0666 } unix_listener replication-notify { mode = 0666 } } service auth { unix_listener /var/spool/postfix/private/auth { group = _postfix mode = 0660 user = _postfix } } service imap-login { inet_listener imap { port = 143 } inet_listener imaps { port = 993 ssl = yes } } service imap { vsz_limit = 512 M } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = _postfix mode = 0660 user = _postfix } vsz_limit = 512 M } service managesieve-login { inet_listener sieve { port = 4190 } inet_listener sieve_deprecated { port = 2000 } service_count = 1 } service pop3-login { inet_listener pop3s { port = 995 ssl = yes } } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0600 } } ssl_alt_cert = </etc/letsencrypt/live/junkpile.org/fullchain.pem ssl_cert = </etc/letsencrypt/live/junkpile.org/fullchain.ecpem ssl_cipher_list = TLSv1.2+AEAD+ECDHE:TLSv1.2+AEAD+DHE:ECDHE-RSA-AES128-SHA ssl_curve_list = X25519:P-521:P-384 ssl_dh = # hidden, use -P to show it ssl_min_protocol = TLSv1 ssl_prefer_server_ciphers = yes userdb { args = username_format=%u /etc/dovecot/users driver = passwd-file } verbose_proctitle = yes protocol lmtp { mail_plugins = " zlib notify replication sieve" } protocol lda { mail_plugins = " zlib notify replication sieve" postmaster_address = postmaster@spacehopper.org } protocol imap { mail_max_userip_connections = 21 mail_plugins = " zlib notify replication imap_sieve imap_zlib" }