On Oct 15, 2013, at 4:33 PM, Reinaldo Matukuma wrote:

Hello. Probably only Timo can help-me with this.

I have a self-made plugin based on the zlib plugin that i use to cryptograph the messages at inbox.

As a side-effect of the cryptography, my plugin changes the size of the message, but until 2.0.19 this works well with dovecot index and the W/S flags.

But now, i'm going to upgrade to 2.1.17 and now i have these messages on log at my test ambiance:

Oct 15 20:19:25 test dovecot: imap(reinaldo@exemplo.com.br): Error: Cached message size smaller than expected (367 < 529) Oct 15 20:19:25 test dovecot: imap(reinaldo@exemplo.com.br): Error: Maildir filename has wrong S value, renamed the file from /storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/cur/1381879158.M634385P5208.test,S=367,W=378:2,S to /storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/cur/1381879158.M634385P5208.test,S=529:2,S Oct 15 20:19:25 test dovecot: imap(reinaldo@exemplo.com.br): Error: Corrupted index cache file /storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/dovecot.index.cache: Broken physical size for mail UID 6

I think there's something else up, this looks quite similar to what I reported the other day:

http://dovecot.org/list/dovecot/2013-October/092917.html

Do you get further messages after the "broken physical size" indicating that dovecot is then trying to open the file based on the original filename?

Regardless, setting this should turn off the file size/name corrections, perhaps that will help:

maildir_broken_filename_sizes=yes

Charles

I understood with these messages that the dovecot is arguing to get the real size of the file now, overriding the return of size from my crypto plugin. But i don't understood if this is a consequence from the changes to correct the index issue (the dovecot.index.cache issue).

So... I want know if the correct way to fix this would be change my plugin to return the real size of the file (that will be larger than the effective message that the plugin returne after the de-cryptography) or if i need use some new function to the plugin replace the expected size based on the real size of the message, not of the file.

Thanks

Reinaldo

Humm... Looking deeper on my plugin i saw that have a call to the i_stream_create_header_filter:

full_input[0] = i_stream_create_header_filter(input, HEADER_FILTER_EXCLUDE | HEADER_FILTER_NO_CR, exclude_headers, 3, filter_callback, &zuser->hash_value);

I need this because i really have 3 lines on the header that i need to hide from the client...

And i have a call to i_stream_create_concat too:

imail->data.stream = i_stream_create_concat(full_input);

Because i have a header and body separation too just for the fact that the header could be searched via dovecot cache once it remains un-cryptographed.

So... The size of S flag before dovecot change it is exactly the header size minus 3 lines of the header plus the un-cryptographed body content.

I notice too two things:

• that dovecot does the rename of the file and can open the message (including the un-cryptographed body part) but removes the W flag on the filename. Isn't the W flag used anymore?

• that the size used by dovecot when it renames the file is only the size of the header part, excluding the body part... I did a test with a larger message and got this at the log:

Oct 17 17:24:27 test dovecot: imap(reinaldo@exemplo.com.br): Error: Cached message size larger than expected (3472 > 1010) Oct 17 17:24:27 test dovecot: imap(reinaldo@exemplo.com.br): Error: Maildir filename has wrong S value, renamed the file from /storage/test/messages/exemplo.com.br/reinaldo/Maildir/cur/1382041442.M605987P2439V2051I475262.test,S=3472,W=3538:2, to /storage/test/messages/exemplo.com.br/reinaldo/Maildir/cur/1382041442.M605987P2439V2051I475262.test,S=1010:2, Oct 17 17:24:27 test dovecot: imap(reinaldo@exemplo.com.br): Error: Corrupted index cache file /storage/test/messages/exemplo.com.br/reinaldo/Maildir/dovecot.index.cache: Broken physical size for mail UID 18 Oct 17 17:24:27 test dovecot: imap(reinaldo@exemplo.com.br): Error: read() failed: Input/output error (FETCH for mailbox INBOX UID 18) Oct 17 17:24:27 test dovecot: imap(reinaldo@exemplo.com.br): Disconnected: Internal error occurred. Refer to server log for more information. [2013-10-17 17:24:27] in=339 out=2915

So... I'm assuming that on dovecot 2.0.19 the index.cache was indexing all the header and un-cryptographed body, really using the information on S and W flags.

Does the dovecot 2.1.17 index work in another way, needing to index only the header part of the message?

From: ramatukuma@hotmail.com To: dovecot@dovecot.org Date: Tue, 15 Oct 2013 17:33:44 -0300 Subject: [Dovecot] Plugin issue with update from 2.0.19 to 2.1.17

Hello. Probably only Timo can help-me with this.

I have a self-made plugin based on the zlib plugin that i use to cryptograph the messages at inbox.

As a side-effect of the cryptography, my plugin changes the size of the message, but until 2.0.19 this works well with dovecot index and the W/S flags.

But now, i'm going to upgrade to 2.1.17 and now i have these messages on log at my test ambiance:

Oct 15 20:19:25 test dovecot: imap(reinaldo@exemplo.com.br): Error: Cached message size smaller than expected (367 < 529) Oct 15 20:19:25 test dovecot: imap(reinaldo@exemplo.com.br): Error: Maildir filename has wrong S value, renamed the file from /storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/cur/1381879158.M634385P5208.test,S=367,W=378:2,S to /storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/cur/1381879158.M634385P5208.test,S=529:2,S Oct 15 20:19:25 test dovecot: imap(reinaldo@exemplo.com.br): Error: Corrupted index cache file /storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/dovecot.index.cache: Broken physical size for mail UID 6

I understood with these messages that the dovecot is arguing to get the real size of the file now, overriding the return of size from my crypto plugin. But i don't understood if this is a consequence from the changes to correct the index issue (the dovecot.index.cache issue).

So... I want know if the correct way to fix this would be change my plugin to return the real size of the file (that will be larger than the effective message that the plugin returne after the de-cryptography) or if i need use some new function to the plugin replace the expected size based on the real size of the message, not of the file.

Thanks

Reinaldo