On 05/06/2014 04:39 PM, Robert Nowotny wrote:
I propose it would be good practice to write some logging in the default
switch cases, to see the unexpected returnvalues in the errorlog.
something like :
ostream-lzma.c : -- Line 147 : i_unreached(); ++ Line 147 : i_fatal("unexpected lzma errorcode when flushing : %s", ret);
after that modification the errorlog shows :
May 6 14:03:05 vm-imap dovecot: imap(test): Fatal: lzma errorcode when flushing : (null)
The code change should be i_fatal("unexpected lzma errorcode when flushing : %d", ret); since ret is an integer.
sure - You are Right.
After changing the code, the Error Log shows : lzma errorcode when flushing : 0
0 means LZMA_OK
from the description in the LMZA base.h :
LZMA_FINISH = 3 /**< * \brief Finish the coding operation * * All the input data must have been given to the encoder * (the last bytes can still be pending in next_in). * Call lzma_code() with LZMA_FINISH until it returns * LZMA_STREAM_END. Once LZMA_FINISH has been used, * the amount of input must no longer be changed by * the application. * * When decoding, using LZMA_FINISH is optional unless the * LZMA_CONCATENATED flag was used when the decoder was * initialized. When LZMA_CONCATENATED was not used, the only * effect of LZMA_FINISH is that the amount of input must not * be changed just like in the encoder. */
A logging change like that is a good idea and we have been meaning to do something similar. It would help debugging, if you could continue running it in your setup, where the error seems to occur quite frequently. Also tell me if you figure out what data or action triggers it.
The error only occurs if I sync a lot (thousands) of emails through IMAPSYNC to a new mailbox. It seems calling lzma_code(zs, LZMA_FINISH); returns LZMA_OK - but is still not ready. I guess the call gives back LZMA_OK (meaning - ok I will do that) but needs some more time to LZMA_FINISH
On low load it seems to be able to finish in time - so that error never occurs. But on high (write to SSD Drive) load, it returns LZMA_OK instead of LZMA_FINISH.
I changed the code now as follows and it seems to work now. I will convert some other huge boxes and lat You know if we will have some otrher negative effects on that ...
This seems to work:
ostream-lzma.c :
Line 137 fff :
ret = lzma_code(zs, LZMA_FINISH);
switch (ret) {
case LZMA_STREAM_END:
done = TRUE;
break;/* LZMA_FINISH should be called until LZMA_STREAM_END */case LZMA_OK:
o_stream_get_name(&zstream->ostream.ostream)); default:break; case LZMA_MEM_ERROR: i_fatal_status(FATAL_OUTOFMEM, "lzma.write(%s): Out of memory",/* got error on high load syncing through imap */i_fatal("lzma errorcode when flushing : %d", ret); i_unreached(); } } while (zs->avail_out != sizeof(zstream->outbuf));
since I am not into lmza programming - is the while clause correct, or do we need another loop until lzma_code(zs, LZMA_FINISH) returns LZMA_STREAM_END ???
br
Ing. Robert Nowotny Vienna