[Dovecot] zlib plugin maildir broken ?/ Cached message size smaller than expected
Hi Timo, by having errors like
Cached message size smaller than expected and Maildir filename has wrong S value and Corrupted index cache file
in dove 2.1.8
i can reproduce the problem with tb 14 with 2 nat machines connected to the same server
the problem relates with create new folder then copy some uncompressed and compressed mail to it, the i.e open it by other client
sometime this happens at once , sometime after a few copy actions
however thunderbird ,then errors above apear, sometimes no mail is shown, sometimes only uncompressed or only compressed
so funktion read both formats gets lost somehow any idea, you need a better debug ?
Best Regards MfG Robert Schetterer
On 1.8.2012, at 0.29, Robert Schetterer wrote:
Cached message size smaller than expected and Maildir filename has wrong S value and Corrupted index cache file
in dove 2.1.8
i can reproduce the problem with tb 14 with 2 nat machines connected to the same server
the problem relates with create new folder then copy some uncompressed and compressed mail to it, the i.e open it by other client
sometime this happens at once , sometime after a few copy actions
I can't reproduce this. Are you sure the problem isn't that the old mails already have a wrong S value and the errors happen when Dovecot tries to read them? It would help to know how to reproduce this on an entirely empty mail account.
Am 31.07.2012 23:44, schrieb Timo Sirainen:
On 1.8.2012, at 0.29, Robert Schetterer wrote:
Cached message size smaller than expected and Maildir filename has wrong S value and Corrupted index cache file
in dove 2.1.8
i can reproduce the problem with tb 14 with 2 nat machines connected to the same server
the problem relates with create new folder then copy some uncompressed and compressed mail to it, the i.e open it by other client
sometime this happens at once , sometime after a few copy actions
I can't reproduce this. Are you sure the problem isn't that the old mails already have a wrong S value and the errors happen when Dovecot tries to read them?
at my recent onwledge mixing compressed new mails from my sent folder, and older uncompressed ones ( which are shown nice both in tb ), with some copy action in a new folder from another client leads to wrong S and tb does not display them any longer sometime, perhaps a double zip happens?
It would help to know how to reproduce this on an entirely empty mail account.
i will try to test, but first i have to deal with now broken maildirs i try disable zlip plugin now ,as workaround hopefully this does not lead to more problems on old maildirs , and fix the problem with new mail
-- Best Regards MfG Robert Schetterer
Am 01.08.2012 07:36, schrieb Robert Schetterer:
Am 31.07.2012 23:44, schrieb Timo Sirainen:
On 1.8.2012, at 0.29, Robert Schetterer wrote:
Cached message size smaller than expected and Maildir filename has wrong S value and Corrupted index cache file
in dove 2.1.8
i can reproduce the problem with tb 14 with 2 nat machines connected to the same server
the problem relates with create new folder then copy some uncompressed and compressed mail to it, the i.e open it by other client
sometime this happens at once , sometime after a few copy actions
I can't reproduce this. Are you sure the problem isn't that the old mails already have a wrong S value and the errors happen when Dovecot tries to read them?
at my recent onwledge mixing compressed new mails from my sent folder, and older uncompressed ones ( which are shown nice both in tb ), with some copy action in a new folder from another client leads to wrong S and tb does not display them any longer sometime, perhaps a double zip happens?
It would help to know how to reproduce this on an entirely empty mail account.
i will try to test, but first i have to deal with now broken maildirs i try disable zlip plugin now ,as workaround hopefully this does not lead to more problems on old maildirs , and fix the problem with new mail
hi Timo, rolled back all stuff you recommended offlist so situation is a bit better now, however why, i had to handle cause more mailldirs get broken
i guess iam dealing with equal Problems described here
http://www.mail-archive.com/dovecot@dovecot.org/msg43986.html
conversation with ralf, looks like your patch wasnt enough
i have a few mailboxes and folders now i cant fix compressed files stay unreadable after move copy
by Maildir filename has wrong S value
i need some kind of mass repair actions etc , or some new patch
Best Regards MfG Robert Schetterer
See what you get by running the maildir through: http://dovecot.org/tools/maildir-size-check.sh
Am 01.08.2012 14:06, schrieb Timo Sirainen:
See what you get by running the maildir through: http://dovecot.org/tools/maildir-size-check.sh
tons of i.e
Freunde/cur/1321795242.M359794P29783.mail02,S=2647728:2,S: Wrong S=2647728 value, should be S=3838835
-- Best Regards MfG Robert Schetterer
On 1.8.2012, at 15.12, Robert Schetterer wrote:
Am 01.08.2012 14:06, schrieb Timo Sirainen:
See what you get by running the maildir through: http://dovecot.org/tools/maildir-size-check.sh
tons of i.e
Freunde/cur/1321795242.M359794P29783.mail02,S=2647728:2,S: Wrong S=2647728 value, should be S=3838835
If you don't mind the messages' IMAP UIDs changing, you could give the -f parameter to fix those.
(There is also http://dovecot.org/tools/maildir-size-fix.pl which I of course noticed only after writing the whole script. That perl script should preferably be modified to support compressed mails and do those other checks, since it updates the uidlists and doesn't cause UIDs to change.)
Am 01.08.2012 14:17, schrieb Timo Sirainen:
On 1.8.2012, at 15.12, Robert Schetterer wrote:
Am 01.08.2012 14:06, schrieb Timo Sirainen:
See what you get by running the maildir through: http://dovecot.org/tools/maildir-size-check.sh
tons of i.e
Freunde/cur/1321795242.M359794P29783.mail02,S=2647728:2,S: Wrong S=2647728 value, should be S=3838835
If you don't mind the messages' IMAP UIDs changing, you could give the -f parameter to fix those.
i will try things couldn not get more worst
(There is also http://dovecot.org/tools/maildir-size-fix.pl which I of course noticed only after writing the whole script. That perl script should preferably be modified to support compressed mails
jep compressed is a must have
and do those other checks, since it updates the uidlists and doesn't cause UIDs to change.)
i will report next using script
-- Best Regards MfG Robert Schetterer
Am 01.08.2012 14:25, schrieb Robert Schetterer:
Am 01.08.2012 14:17, schrieb Timo Sirainen:
On 1.8.2012, at 15.12, Robert Schetterer wrote:
Am 01.08.2012 14:06, schrieb Timo Sirainen:
See what you get by running the maildir through: http://dovecot.org/tools/maildir-size-check.sh
tons of i.e
Freunde/cur/1321795242.M359794P29783.mail02,S=2647728:2,S: Wrong S=2647728 value, should be S=3838835
If you don't mind the messages' IMAP UIDs changing, you could give the -f parameter to fix those.
i will try things couldn not get more worst
(There is also http://dovecot.org/tools/maildir-size-fix.pl which I of course noticed only after writing the whole script. That perl script should preferably be modified to support compressed mails
jep compressed is a must have
and do those other checks, since it updates the uidlists and doesn't cause UIDs to change.)
i will report next using script
./maildir-size-check.sh -f maildir is an unknown option...?
-- Best Regards MfG Robert Schetterer
On 1.8.2012, at 15.43, Robert Schetterer wrote:
If you don't mind the messages' IMAP UIDs changing, you could give the -f parameter to fix those.
i will try things couldn not get more worst
./maildir-size-check.sh -f maildir is an unknown option...?
Oh, looks like I forgot to re-upload it after adding that option. It's there now.
Am 01.08.2012 14:58, schrieb Timo Sirainen:
On 1.8.2012, at 15.43, Robert Schetterer wrote:
If you don't mind the messages' IMAP UIDs changing, you could give the -f parameter to fix those.
i will try things couldn not get more worst
./maildir-size-check.sh -f maildir is an unknown option...?
Oh, looks like I forgot to re-upload it after adding that option. It's there now.
looks like working fine, but it has some problems with pathnames with special char and spaces
-- Best Regards MfG Robert Schetterer
Am 01.08.2012 14:58, schrieb Timo Sirainen:
On 1.8.2012, at 15.43, Robert Schetterer wrote:
If you don't mind the messages' IMAP UIDs changing, you could give the -f parameter to fix those.
i will try things couldn not get more worst
./maildir-size-check.sh -f maildir is an unknown option...?
Oh, looks like I forgot to re-upload it after adding that option. It's there now.
Hi Timo, most of the Problems are fixed now many thx for your script , but how to avoid that in future ( Maildir filename has wrong S value ) with Ralf you released a patch diff -r 3599790da3d7 src/lib-storage/index/istream-mail.c --- a/src/lib-storage/index/istream-mail.c Wed Mar 21 14:25:12 2012 +0200 +++ b/src/lib-storage/index/istream-mail.c Wed Mar 21 17:58:00 2012 +0200 @@ -20,6 +20,7 @@ struct mail *mail = mstream->mail; enum mail_lookup_abort orig_lookup_abort; + return FALSE; if (mstream->expected_size != (uoff_t)-1) return TRUE; would it be a good idea to include it in 2.1.9 for me ? as it looks like its not in the public code ? -- Best Regards MfG Robert Schetterer
On 2.8.2012, at 10.09, Robert Schetterer wrote:
Hi Timo, most of the Problems are fixed now many thx for your script
, but how to avoid that in future ( Maildir filename has wrong S value )
You'd need to find out why it started happening in the first place. What software wrote those filenames? Are new filenames still creating with wrong S value? Does your dovecot.conf use zlib plugin globally (instead of only in some protocol sections, which might cause this kind of trouble)?
you released a patch
diff -r 3599790da3d7 src/lib-storage/index/istream-mail.c --- a/src/lib-storage/index/istream-mail.c Wed Mar 21 14:25:12 2012 +0200 +++ b/src/lib-storage/index/istream-mail.c Wed Mar 21 17:58:00 2012 +0200 @@ -20,6 +20,7 @@ struct mail *mail = mstream->mail; enum mail_lookup_abort orig_lookup_abort;
+ return FALSE; if (mstream->expected_size != (uoff_t)-1) return TRUE;
would it be a good idea to include it in 2.1.9 for me ? as it looks like its not in the public code ?
It's an ugly workaround for the wrong S=size problem when zlib is used. Preferably you wouldn't ever need to use it.
Am 03.08.2012 15:56, schrieb Timo Sirainen:
On 2.8.2012, at 10.09, Robert Schetterer wrote:
Hi Timo, most of the Problems are fixed now many thx for your script
, but how to avoid that in future ( Maildir filename has wrong S value )
You'd need to find out why it started happening in the first place.
the problem is/was getting maildirs fixed, no time to debug on production system in deed i have no idea why this happend, i found out some broken were old maildirs, so for them i bet coruption by migration 2.0 -2.1 other cuted out attachments imap online with serveral clients on = user fault in my meaning left the cases which i reproduced myself, or some brand new mailboxes it was about compressed mail and nocompressed with wrong message size in one folder What software wrote those filenames? Are new filenames still creating with wrong S value? Does your dovecot.conf use zlib plugin globally (instead of only in some protocol sections, which might cause this kind of trouble)? 10-mail.conf:mail_plugins = $mail_plugins zlib expire virtual fts fts_lucene mail_log autocreate notify acl listescape 20-imap.conf: mail_plugins = imap_acl acl fts fts_lucene mail_log autocreate notify expire virtual quota imap_quota zlib imap_zlib listescape 90-plugin.conf:zlib_save_level = 6 90-plugin.conf:zlib_save = gz should be ok ?
you released a patch
diff -r 3599790da3d7 src/lib-storage/index/istream-mail.c --- a/src/lib-storage/index/istream-mail.c Wed Mar 21 14:25:12 2012 +0200 +++ b/src/lib-storage/index/istream-mail.c Wed Mar 21 17:58:00 2012 +0200 @@ -20,6 +20,7 @@ struct mail *mail = mstream->mail; enum mail_lookup_abort orig_lookup_abort;
+ return FALSE; if (mstream->expected_size != (uoff_t)-1) return TRUE;
would it be a good idea to include it in 2.1.9 for me ? as it looks like its not in the public code ?
It's an ugly workaround for the wrong S=size problem when zlib is used. Preferably you wouldn't ever need to use it.
ok , i fixed all mailboxes with your script and one from ralf with compressed maildirs, updated to 2.1.9, using lucene now, disabled stats ( for kernel bugs ), and for now all problems are gone if i cant find any better reason, i go for shit happens, and learned how to fix it, however it were only 6 maildirs from 200 which got broken going trough lot of updates in years , so this is ok in my mind good time for users learn to build some mail archive from time to time however no mail got lost
-- Best Regards MfG Robert Schetterer
participants (2)
-
Robert Schetterer
-
Timo Sirainen