[Dovecot] Plugin issue with update from 2.0.19 to 2.1.17
Hello. Probably only Timo can help-me with this.
I have a self-made plugin based on the zlib plugin that i use to cryptograph the messages at inbox.
As a side-effect of the cryptography, my plugin changes the size of the message, but until 2.0.19 this works well with dovecot index and the W/S flags.
But now, i'm going to upgrade to 2.1.17 and now i have these messages on log at my test ambiance:
Oct 15 20:19:25 test dovecot: imap(reinaldo@exemplo.com.br): Error: Cached message size smaller than expected (367 < 529) Oct 15 20:19:25 test dovecot: imap(reinaldo@exemplo.com.br): Error: Maildir filename has wrong S value, renamed the file from /storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/cur/1381879158.M634385P5208.test,S=367,W=378:2,S to /storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/cur/1381879158.M634385P5208.test,S=529:2,S Oct 15 20:19:25 test dovecot: imap(reinaldo@exemplo.com.br): Error: Corrupted index cache file /storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/dovecot.index.cache: Broken physical size for mail UID 6
I understood with these messages that the dovecot is arguing to get the real size of the file now, overriding the return of size from my crypto plugin. But i don't understood if this is a consequence from the changes to correct the index issue (the dovecot.index.cache issue).
So... I want know if the correct way to fix this would be change my plugin to return the real size of the file (that will be larger than the effective message that the plugin returne after the de-cryptography) or if i need use some new function to the plugin replace the expected size based on the real size of the message, not of the file.
Thanks
Reinaldo
On Oct 15, 2013, at 4:33 PM, Reinaldo Matukuma wrote:
Hello. Probably only Timo can help-me with this.
I have a self-made plugin based on the zlib plugin that i use to cryptograph the messages at inbox.
As a side-effect of the cryptography, my plugin changes the size of the message, but until 2.0.19 this works well with dovecot index and the W/S flags.
But now, i'm going to upgrade to 2.1.17 and now i have these messages on log at my test ambiance:
Oct 15 20:19:25 test dovecot: imap(reinaldo@exemplo.com.br): Error: Cached message size smaller than expected (367 < 529) Oct 15 20:19:25 test dovecot: imap(reinaldo@exemplo.com.br): Error: Maildir filename has wrong S value, renamed the file from /storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/cur/1381879158.M634385P5208.test,S=367,W=378:2,S to /storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/cur/1381879158.M634385P5208.test,S=529:2,S Oct 15 20:19:25 test dovecot: imap(reinaldo@exemplo.com.br): Error: Corrupted index cache file /storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/dovecot.index.cache: Broken physical size for mail UID 6
I think there's something else up, this looks quite similar to what I reported the other day:
http://dovecot.org/list/dovecot/2013-October/092917.html
Do you get further messages after the "broken physical size" indicating that dovecot is then trying to open the file based on the original filename?
Regardless, setting this should turn off the file size/name corrections, perhaps that will help:
maildir_broken_filename_sizes=yes
Charles
I understood with these messages that the dovecot is arguing to get the real size of the file now, overriding the return of size from my crypto plugin. But i don't understood if this is a consequence from the changes to correct the index issue (the dovecot.index.cache issue).
So... I want know if the correct way to fix this would be change my plugin to return the real size of the file (that will be larger than the effective message that the plugin returne after the de-cryptography) or if i need use some new function to the plugin replace the expected size based on the real size of the message, not of the file.
Thanks
Reinaldo
Hi Charles.
I don't think that is the same issue...
I tried to set the maildir_broken_filename_sizes=yes and have tryed the Timo fix patch (attached here: http://www.dovecot.org/list/dovecot/2012-March/064731.html) but the index continues to recalculate the size.
This is after the Timo fix patch:
Oct 16 20:50:13 teste dovecot: imap(reinaldo@exemplo.com.br): Error: Cached message size smaller than expected (367 < 529) Oct 16 20:50:13 teste dovecot: imap(reinaldo@exemplo.com.br): Error: Maildir filename has wrong S value, renamed the file from /storage/teste/messages/exemplo.com.br/reinaldo/Maildir/.Sent/cur/1381967409.M54518P30563.teste,S=367,W=378:2,S to /storage/teste/messages/exemplo.com.br/reinaldo/Maildir/.Sent/cur/1381967409.M54518P30563.teste,S=529:2,S Oct 16 20:50:13 teste dovecot: imap(reinaldo@exemplo.com.br): Error: Corrupted index cache file /storage/teste/messages/exemplo.com.br/reinaldo/Maildir/.Sent/dovecot.index.cache: Broken physical size for mail UID 8
And this is after the maildir_broken_filename_sizes=yes conf:
Oct 17 12:21:04 teste dovecot: imap(reinaldo@exemplo.com.br): Error: Cached message size smaller than expected (368 < 530) Oct 17 12:21:04 teste dovecot: imap(reinaldo@exemplo.com.br): Error: Maildir filename has wrong S value, renamed the file from /storage/teste/messages/exemplo.com.br/reinaldo/Maildir/.Sent/cur/1381969983.M452882P30597.teste,S=530:2,S to /storage/teste/messages/exemplo.com.br/reinaldo/Maildir/.Sent/cur/1381969983.M452882P30597.teste,S=530:2,S Oct 17 12:21:04 teste dovecot: imap(reinaldo@exemplo.com.br): Error: Corrupted index cache file /storage/teste/messages/exemplo.com.br/reinaldo/Maildir/.Sent/dovecot.index.cache: Broken physical size for mail UID 11
So... I noticed that after the maildir_broken_filename_sizes=yes i still got the error but there no changes on the S flag of the filename.
Is the dovecot.index.cache used for text searching too? Because if it is i think that i really need some way to make my plugin informs the correct size to dovecot, as i think that works on dovecot 2.0.19.
PS: I had to change some calls on my plugin in order to comply with some changes on dovecot 2.1.17. That is the list of the calls that had been changed... Could be that I had make another choice for values of the functions?
Before (2.0.19):
After (2.1.17):return zmail->super.get_stream(_mail, hdr_size, body_size, stream_r);
return zmail->super.get_stream(_mail, 1, hdr_size, body_size, stream_r);
Before (2.0.19):
- if (zmail->super.get_stream(_mail, NULL, NULL, &input) < 0){ After (2.1.17):
- if (zmail->super.get_stream(_mail, 1, NULL, NULL, &input) < 0){
Before (2.0.19):
After (2.1.17):full_input[1] = i_stream_create_fd(fd1, 0, TRUE);
full_input[1] = i_stream_create_fd(fd1, (size_t)-1, TRUE);
Before (2.0.19):
After (2.1.17):if (zmail->super.get_stream(ctx->dest_mail, NULL, NULL, &input) >= 0){
if (zmail->super.get_stream(ctx->dest_mail, 1, NULL, NULL, &input) >= 0){
Before (2.0.19):
- return o_stream_create(&emexis_stream->ostream); After (2.1.17):
- return o_stream_create(&emexis_stream->ostream, NULL);
And after define: mail->v.get_stream = myplugin_permail_get_stream;
i have this function declaration on 2.0.19: static int myplugin_permail_get_stream(struct mail *_mail, struct message_size *hdr_size, struct message_size *body_size, struct istream **stream_r) {
and this on 2.1.17: static int emexis_permail_get_stream(struct mail *_mail, bool get_body ATTR_UNUSED, struct message_size *hdr_size, struct message_size *body_size, struct istream **stream_r) {
CC: dovecot@dovecot.org From: spork@bway.net Subject: Re: [Dovecot] Plugin issue with update from 2.0.19 to 2.1.17 Date: Tue, 15 Oct 2013 17:15:05 -0400 To: ramatukuma@hotmail.com
On Oct 15, 2013, at 4:33 PM, Reinaldo Matukuma wrote:
Hello. Probably only Timo can help-me with this.
I have a self-made plugin based on the zlib plugin that i use to cryptograph the messages at inbox.
As a side-effect of the cryptography, my plugin changes the size of the message, but until 2.0.19 this works well with dovecot index and the W/S flags.
But now, i'm going to upgrade to 2.1.17 and now i have these messages on log at my test ambiance:
Oct 15 20:19:25 test dovecot: imap(reinaldo@exemplo.com.br): Error: Cached message size smaller than expected (367 < 529) Oct 15 20:19:25 test dovecot: imap(reinaldo@exemplo.com.br): Error: Maildir filename has wrong S value, renamed the file from /storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/cur/1381879158.M634385P5208.test,S=367,W=378:2,S to /storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/cur/1381879158.M634385P5208.test,S=529:2,S Oct 15 20:19:25 test dovecot: imap(reinaldo@exemplo.com.br): Error: Corrupted index cache file /storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/dovecot.index.cache: Broken physical size for mail UID 6
I think there's something else up, this looks quite similar to what I reported the other day:
http://dovecot.org/list/dovecot/2013-October/092917.html
Do you get further messages after the "broken physical size" indicating that dovecot is then trying to open the file based on the original filename?
Regardless, setting this should turn off the file size/name corrections, perhaps that will help:
maildir_broken_filename_sizes=yes
Charles
I understood with these messages that the dovecot is arguing to get the real size of the file now, overriding the return of size from my crypto plugin. But i don't understood if this is a consequence from the changes to correct the index issue (the dovecot.index.cache issue).
So... I want know if the correct way to fix this would be change my plugin to return the real size of the file (that will be larger than the effective message that the plugin returne after the de-cryptography) or if i need use some new function to the plugin replace the expected size based on the real size of the message, not of the file.
Thanks
Reinaldo
Humm... Looking deeper on my plugin i saw that have a call to the i_stream_create_header_filter:
full_input[0] = i_stream_create_header_filter(input, HEADER_FILTER_EXCLUDE | HEADER_FILTER_NO_CR, exclude_headers, 3, filter_callback, &zuser->hash_value);
I need this because i really have 3 lines on the header that i need to hide from the client...
And i have a call to i_stream_create_concat too:
imail->data.stream = i_stream_create_concat(full_input);
Because i have a header and body separation too just for the fact that the header could be searched via dovecot cache once it remains un-cryptographed.
So... The size of S flag before dovecot change it is exactly the header size minus 3 lines of the header plus the un-cryptographed body content.
I notice too two things:
that dovecot does the rename of the file and can open the message (including the un-cryptographed body part) but removes the W flag on the filename. Isn't the W flag used anymore?
that the size used by dovecot when it renames the file is only the size of the header part, excluding the body part... I did a test with a larger message and got this at the log:
Oct 17 17:24:27 test dovecot: imap(reinaldo@exemplo.com.br): Error: Cached message size larger than expected (3472 > 1010) Oct 17 17:24:27 test dovecot: imap(reinaldo@exemplo.com.br): Error: Maildir filename has wrong S value, renamed the file from /storage/test/messages/exemplo.com.br/reinaldo/Maildir/cur/1382041442.M605987P2439V2051I475262.test,S=3472,W=3538:2, to /storage/test/messages/exemplo.com.br/reinaldo/Maildir/cur/1382041442.M605987P2439V2051I475262.test,S=1010:2, Oct 17 17:24:27 test dovecot: imap(reinaldo@exemplo.com.br): Error: Corrupted index cache file /storage/test/messages/exemplo.com.br/reinaldo/Maildir/dovecot.index.cache: Broken physical size for mail UID 18 Oct 17 17:24:27 test dovecot: imap(reinaldo@exemplo.com.br): Error: read() failed: Input/output error (FETCH for mailbox INBOX UID 18) Oct 17 17:24:27 test dovecot: imap(reinaldo@exemplo.com.br): Disconnected: Internal error occurred. Refer to server log for more information. [2013-10-17 17:24:27] in=339 out=2915
So... I'm assuming that on dovecot 2.0.19 the index.cache was indexing all the header and un-cryptographed body, really using the information on S and W flags.
Does the dovecot 2.1.17 index work in another way, needing to index only the header part of the message?
From: ramatukuma@hotmail.com To: dovecot@dovecot.org Date: Tue, 15 Oct 2013 17:33:44 -0300 Subject: [Dovecot] Plugin issue with update from 2.0.19 to 2.1.17
Hello. Probably only Timo can help-me with this.
I have a self-made plugin based on the zlib plugin that i use to cryptograph the messages at inbox.
As a side-effect of the cryptography, my plugin changes the size of the message, but until 2.0.19 this works well with dovecot index and the W/S flags.
But now, i'm going to upgrade to 2.1.17 and now i have these messages on log at my test ambiance:
Oct 15 20:19:25 test dovecot: imap(reinaldo@exemplo.com.br): Error: Cached message size smaller than expected (367 < 529) Oct 15 20:19:25 test dovecot: imap(reinaldo@exemplo.com.br): Error: Maildir filename has wrong S value, renamed the file from /storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/cur/1381879158.M634385P5208.test,S=367,W=378:2,S to /storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/cur/1381879158.M634385P5208.test,S=529:2,S Oct 15 20:19:25 test dovecot: imap(reinaldo@exemplo.com.br): Error: Corrupted index cache file /storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/dovecot.index.cache: Broken physical size for mail UID 6
I understood with these messages that the dovecot is arguing to get the real size of the file now, overriding the return of size from my crypto plugin. But i don't understood if this is a consequence from the changes to correct the index issue (the dovecot.index.cache issue).
So... I want know if the correct way to fix this would be change my plugin to return the real size of the file (that will be larger than the effective message that the plugin returne after the de-cryptography) or if i need use some new function to the plugin replace the expected size based on the real size of the message, not of the file.
Thanks
Reinaldo
On 15.10.2013, at 23.33, Reinaldo Matukuma <ramatukuma@hotmail.com> wrote:
Hello. Probably only Timo can help-me with this.
I have a self-made plugin based on the zlib plugin that i use to cryptograph the messages at inbox.
As a side-effect of the cryptography, my plugin changes the size of the message, but until 2.0.19 this works well with dovecot index and the W/S flags.
But now, i'm going to upgrade to 2.1.17 and now i have these messages on log at my test ambiance:
Oct 15 20:19:25 test dovecot: imap(reinaldo@exemplo.com.br): Error: Cached message size smaller than expected (367 < 529) Oct 15 20:19:25 test dovecot: imap(reinaldo@exemplo.com.br): Error: Maildir filename has wrong S value, renamed the file from /storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/cur/1381879158.M634385P5208.test,S=367,W=378:2,S to /storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/cur/1381879158.M634385P5208.test,S=529:2,S Oct 15 20:19:25 test dovecot: imap(reinaldo@exemplo.com.br): Error: Corrupted index cache file /storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/dovecot.index.cache: Broken physical size for mail UID 6
Is it intended that when decrypting messages you’ll get exactly the original message back? Or are you also modifying the message? Assuming there is no modification, you could take a look at how http://dovecot.org/patches/2.2/mail-filter.tar.gz does that without problems. Message modifications should also be possible with that code with small modifications. I’m not sure if that code works for v2.1.
participants (3)
-
Charles Sprickman
-
Reinaldo Matukuma
-
Timo Sirainen