[Dovecot] Antwort: Antwort: Re: WG: on High Load using IMAPSYNC : Panic: file ostream-lzma.c: line 147: unreached. Dovecot 2.2.12 with zlib/XZ compression --> solved !

[Robert Nowotny]

when using xz compression, on high load (syncing thousands of emails) 
dovecot panics with : Panic: file ostream-lzma.c: line 147: unreached

after some investigation I found out that the call 

ret = lzma_code(zs, LZMA_FINISH);

returns LZMA_OK (what is enumerated integer 0) sometimes.

again - this error will only show up under heavy load, for instance when 
syncing a lot of emails to a new mailbox with imapsync or dsync.

After patching the file ostream-lzma.c as follows, it seems to work now 
correctly.

Someone with more knowledge about the lzma library and that codepart 
should probably look over it, before merging it into the code.

I did extensive testing with huge mailboxes successfully-
just insert the lines marked with '++' and recompile.

-- code snipped of ostream-lzma.c, around line 124  --

        do {
                len = sizeof(zstream->outbuf) - zs->avail_out;
                if (len != 0) {
                        zs->next_out = zstream->outbuf;
                        zs->avail_out = sizeof(zstream->outbuf);

                        zstream->outbuf_used = len;
                        if ((ret = o_stream_zlib_send_outbuf(zstream)) <= 
0)
                                return ret;
                        if (done)
                                break;
                }

                ret = lzma_code(zs, LZMA_FINISH);
                switch (ret) {
                case LZMA_STREAM_END:
                        done = TRUE;
                        break;
++              /* LZMA_FINISH should be called until LZMA_STREAM_END */
++              /* got LZMA_OK on high server load when syncing through 
imap */
++              case LZMA_OK:
++                      break;
                case LZMA_MEM_ERROR:
                        i_fatal_status(FATAL_OUTOFMEM,
                                       "lzma.write(%s): Out of memory",
 o_stream_get_name(&zstream->ostream.ostream));
                default:
++          /* log the unexpected error code if any */
++                      i_fatal("lzma returncode when flushing : %d", 
ret);
                        i_unreached();
                }
        } while (zs->avail_out != sizeof(zstream->outbuf));

-- /code snipped --


yours sincerely

Ing. Robert Nowotny
Rotek GmbH
Vienna, Austria