[Dovecot] Antwort: Re: WG: on High Load using IMAPSYNC : Panic: file ostream-lzma.c: line 147: unreached. Dovecot 2.2.12 with zlib/XZ compression

[Robert Nowotny]

On 05/06/2014 04:39 PM, Robert Nowotny wrote:
> I propose it would be good practice to write some logging in the default 

> switch cases, to see the unexpected returnvalues in the errorlog.
> 
> something like : 
> 
> ostream-lzma.c : 
> -- Line 147 :   i_unreached();
> ++ Line 147 :   i_fatal("unexpected lzma errorcode when flushing : %s", 
> ret);
> 
> after that modification the errorlog shows : 
> 
> May  6 14:03:05 vm-imap dovecot: imap(test): Fatal: lzma errorcode when 
> flushing : (null)

> The code change should be i_fatal("unexpected lzma errorcode when 
flushing : %d", ret);
> since ret is an integer.

sure - You are Right.

After changing the code, the Error Log shows : 
lzma errorcode when flushing : 0

0 means LZMA_OK

from the description in the LMZA base.h : 

LZMA_FINISH = 3
                                 /**<
                                  * \brief       Finish the coding 
operation
                                  *
                                  * All the input data must have been 
given to the encoder
                                  * (the last bytes can still be pending 
in next_in).
                                  * Call lzma_code() with LZMA_FINISH 
until it returns
                                  * LZMA_STREAM_END. Once LZMA_FINISH has 
been used,
                                  * the amount of input must no longer be 
changed by
                                  * the application.
                                  *
                                  * When decoding, using LZMA_FINISH is 
optional unless the
                                  * LZMA_CONCATENATED flag was used when 
the decoder was
                                  * initialized. When LZMA_CONCATENATED 
was not used, the only
                                  * effect of LZMA_FINISH is that the 
amount of input must not
                                  * be changed just like in the encoder.
                                  */

> A logging change like that is a good idea and we have been meaning to do 
something similar. It would help debugging, if you
> could continue running it in your setup, where the error seems to occur 
quite frequently. Also tell me if you figure out what
> data or action triggers it.

The error only occurs if I sync a lot (thousands) of emails through 
IMAPSYNC to a new mailbox.
It seems calling lzma_code(zs, LZMA_FINISH); returns LZMA_OK - but is 
still not ready.
I guess the call gives back LZMA_OK (meaning - ok I will do that) but 
needs some more time to LZMA_FINISH

On low load it seems to be able to finish in time - so that error never 
occurs. 
But on high (write to SSD Drive) load, it returns LZMA_OK instead of 
LZMA_FINISH.

I changed the code now as follows and it seems to work now.
I will convert some other huge boxes and lat You know if we will have some 
otrher negative effects on that ... 


This seems to work: 

ostream-lzma.c : 


Line 137 fff : 

                ret = lzma_code(zs, LZMA_FINISH);
                switch (ret) {
                case LZMA_STREAM_END:
                        done = TRUE;
                        break;
+               /* LZMA_FINISH should be called until LZMA_STREAM_END */
+               case LZMA_OK:
+                       break;
                case LZMA_MEM_ERROR:
                        i_fatal_status(FATAL_OUTOFMEM,
                                       "lzma.write(%s): Out of memory",
 o_stream_get_name(&zstream->ostream.ostream));
                default:
+                       /* got error on high load syncing through imap */
+                       i_fatal("lzma errorcode when flushing : %d", ret);
                        i_unreached();
                }
        } while (zs->avail_out != sizeof(zstream->outbuf));


since I am not into lmza programming - is the while clause correct, or do 
we need another loop until 
lzma_code(zs, LZMA_FINISH) returns LZMA_STREAM_END ???


br

Ing. Robert Nowotny
Vienna