[Dovecot] Fwd: Weird problem opening mbox in beta1/2
Hmm, this is peculiar!
This mailbox has been unchanged on my test server since September. I've been using it to test each new version of Dovecot just after I've compiled it.
When beta1 was released, I opened this mbox with no problems. However, when I upgraded to beta2, I got "file size unexpectedly shrinked in mbox file /export/mail/a/24/vis99003/INBOX (16895967 vs 16899267)".
Reverting to beta1, it also failed with the same error! I've deleted indexes, restored the mbox from a backup made in October (no changes) and I still get the error. It *does* open OK in 1.0-stable.
I've put it through mboxcrypt.pl, deleted the UW-IMAP pseudo-message (which wasn't converted properly by mboxcrypt) and it still fails to open with a similar error, so I've sent it to Timo (gzipped, of course!)
I've e-mailed this to the list minus the attachment as sending unsolicited 1MB attachments is unfriendly! However, if anybody wants to look at it, please let me know.
Best Wishes, Chris
P.S. The past participle of "shrink" is "shrunk" not "shrinked" - it's yet another irregular English verb :)
-- --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- Christopher Wakelin, c.d.wakelin@reading.ac.uk IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439 Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094
-- --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- Christopher Wakelin, c.d.wakelin@reading.ac.uk IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439 Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094
Chris Wakelin wrote:
Hmm, this is peculiar!
This mailbox has been unchanged on my test server since September. I've been using it to test each new version of Dovecot just after I've compiled it.
When beta1 was released, I opened this mbox with no problems. However, when I upgraded to beta2, I got "file size unexpectedly shrinked in mbox file /export/mail/a/24/vis99003/INBOX (16895967 vs 16899267)".
Reverting to beta1, it also failed with the same error! I've deleted indexes, restored the mbox from a backup made in October (no changes) and I still get the error. It *does* open OK in 1.0-stable.
I've put it through mboxcrypt.pl, deleted the UW-IMAP pseudo-message (which wasn't converted properly by mboxcrypt) and it still fails to open with a similar error, so I've sent it to Timo (gzipped, of course!)
I've e-mailed this to the list minus the attachment as sending unsolicited 1MB attachments is unfriendly! However, if anybody wants to look at it, please let me know.
Best Wishes, Chris
P.S. The past participle of "shrink" is "shrunk" not "shrinked" - it's yet another irregular English verb :)
Right! Some more testing seems to give :-
1.0-alpha5 opens it fine dovecot-20060110 opens it fine dovecot-20060114 (plus the minor fixes for compile errors) fails 1.0-beta1 and 1.0-beta2 fail
dovecot-20060110 plus the patches from the CVS commits
2006-01-11 20:45 Timo Sirainen <tss@iki.fi>
* src/lib-storage/index/mbox/: istream-raw-mbox.c, mbox-mail.c:
Handle unexpectedly breaking mboxes better without crashing.2006-01-14 13:59 Timo Sirainen <tss@iki.fi>
* src/lib-storage/index/mbox/mbox-sync-update.c: Use longer line
wrapping with X-IMAP, X-IMAPbase and X-Keywords headers since
UW-IMAP doesn't like the wrapping.works (but I've just realised that the latter was after dovecot-20060114 anyway).
So it looks like it might be something to do with the int -> bool changes.
The really strange thing, is I'm pretty sure I checked this mbox with all the above versions when they came out, and it was fine! I've also got another mbox giving the same sort of error in beta1/2, and I've tried a third one with beta2 and it was fine.
I've tried using UW-IMAP's mailutil to copy the mbox to a new folder (which sometimes fixes mbox problems), and that new folder still fails.
I've got a few users (including me, of course!) using beta1 on the live server without problems, but I'm not keen on deploying beta1 or beta2 everywhere without getting to the bottom of this :(
Best Wishes, (Puzzled) Chris
-- --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- Christopher Wakelin, c.d.wakelin@reading.ac.uk IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439 Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094
Chris Wakelin wrote:
Chris Wakelin wrote:
Hmm, this is peculiar!
This mailbox has been unchanged on my test server since September. I've been using it to test each new version of Dovecot just after I've compiled it.
When beta1 was released, I opened this mbox with no problems. However, when I upgraded to beta2, I got "file size unexpectedly shrinked in mbox file /export/mail/a/24/vis99003/INBOX (16895967 vs 16899267)".
Reverting to beta1, it also failed with the same error! I've deleted indexes, restored the mbox from a backup made in October (no changes) and I still get the error. It *does* open OK in 1.0-stable.
I've put it through mboxcrypt.pl, deleted the UW-IMAP pseudo-message (which wasn't converted properly by mboxcrypt) and it still fails to open with a similar error, so I've sent it to Timo (gzipped, of course!)
I've e-mailed this to the list minus the attachment as sending unsolicited 1MB attachments is unfriendly! However, if anybody wants to look at it, please let me know.
Best Wishes, Chris
P.S. The past participle of "shrink" is "shrunk" not "shrinked" - it's yet another irregular English verb :)
Right! Some more testing seems to give :-
1.0-alpha5 opens it fine dovecot-20060110 opens it fine dovecot-20060114 (plus the minor fixes for compile errors) fails 1.0-beta1 and 1.0-beta2 fail
dovecot-20060110 plus the patches from the CVS commits
2006-01-11 20:45 Timo Sirainen <tss@iki.fi>
- src/lib-storage/index/mbox/: istream-raw-mbox.c, mbox-mail.c: Handle unexpectedly breaking mboxes better without crashing.
2006-01-14 13:59 Timo Sirainen <tss@iki.fi>
- src/lib-storage/index/mbox/mbox-sync-update.c: Use longer line wrapping with X-IMAP, X-IMAPbase and X-Keywords headers since UW-IMAP doesn't like the wrapping.
works (but I've just realised that the latter was after dovecot-20060114 anyway).
So it looks like it might be something to do with the int -> bool changes.
The really strange thing, is I'm pretty sure I checked this mbox with all the above versions when they came out, and it was fine! I've also got another mbox giving the same sort of error in beta1/2, and I've tried a third one with beta2 and it was fine.
I've tried using UW-IMAP's mailutil to copy the mbox to a new folder (which sometimes fixes mbox problems), and that new folder still fails.
I've got a few users (including me, of course!) using beta1 on the live server without problems, but I'm not keen on deploying beta1 or beta2 everywhere without getting to the bottom of this :(
Best Wishes, (Puzzled) Chris
I'm getting somewhere at last. I tried chopping one of the mboxes into smaller and smaller bits until I got the error with only one message!
It seems I've got a couple of test messages at the end of the suspect mboxes with what looks like incorrect "Content-Length" headers.
But that leaves the question of what changed? The only thing in CVS I can think of is the int -> bool change to parse_content_length() in src/lib-storage/index/mbox/mbox-sync-parse.c; perhaps it never successfully parsed them before and now does (we're still returning "TRUE" or "FALSE" rather than -1 or 0)?
That still doesn't explain how come I opened them OK, even with beta1 until this week? Maybe I'm going bonkers and actually didn't open them successfully after all!
Chris
-- --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- Christopher Wakelin, c.d.wakelin@reading.ac.uk IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439 Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094
On Wed, 2006-01-25 at 18:33 +0000, Chris Wakelin wrote:
So it looks like it might be something to do with the int -> bool changes.
That's correct. I had mixed integers and booleans in one function, and after doing int -> bool change, the returned -1 got translated to same as TRUE. Would be nice if gcc complained about those mistakes.. Fixed in CVS, and attached patch also fixes it.
Thanks, that's fixed it! I'm still puzzled about why I *thought* I'd opened it successfully before. Maybe I used a different compilation environment? I'd been experimenting with gcc 3.3.2 as well as 2.95. Best Wishes, Chris Timo Sirainen wrote:
On Wed, 2006-01-25 at 18:33 +0000, Chris Wakelin wrote:
So it looks like it might be something to do with the int -> bool changes.
That's correct. I had mixed integers and booleans in one function, and after doing int -> bool change, the returned -1 got translated to same as TRUE. Would be nice if gcc complained about those mistakes.. Fixed in CVS, and attached patch also fixes it.
------------------------------------------------------------------------
? src/lib-storage/index/mbox/diff ? src/lib-storage/index/mbox/mbox-lock.c2 ? src/lib-storage/index/mbox/mbox-min-index-size.diff Index: src/lib-storage/index/mbox/istream-raw-mbox.c =================================================================== RCS file: /var/lib/cvs/dovecot/src/lib-storage/index/mbox/istream-raw-mbox.c,v retrieving revision 1.39 diff -u -r1.39 istream-raw-mbox.c --- src/lib-storage/index/mbox/istream-raw-mbox.c 14 Jan 2006 18:47:52 -0000 1.39 +++ src/lib-storage/index/mbox/istream-raw-mbox.c 26 Jan 2006 20:41:45 -0000 @@ -340,7 +340,7 @@ input->real_stream->abs_start_offset); }
-static bool istream_raw_mbox_is_valid_from(struct raw_mbox_istream *rstream) +static int istream_raw_mbox_is_valid_from(struct raw_mbox_istream *rstream) { const unsigned char *data; size_t size; @@ -354,7 +354,7 @@ if ((size == 1 && data[0] == '\n') || (size == 2 && data[0] == '\r' && data[1] == '\n')) { /* EOF */ - return TRUE; + return 1; }
if (size > 31 && memcmp(data, "\nFrom ", 6) == 0) { @@ -364,7 +364,7 @@ data += 7; size -= 7; } else { - return FALSE; + return 0; }
while (memchr(data, '\n', size) == NULL) { @@ -373,12 +373,12 @@ }
if (mbox_from_parse(data, size, &received_time, &sender) < 0) - return FALSE; + return 0;
rstream->next_received_time = received_time; i_free(rstream->next_sender); rstream->next_sender = sender; - return TRUE; + return 1; }
uoff_t istream_raw_mbox_get_start_offset(struct istream *stream)
-- --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- Christopher Wakelin, c.d.wakelin@reading.ac.uk IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439 Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094
participants (2)
-
Chris Wakelin
-
Timo Sirainen