On Tue, Feb 14, 2006 at 10:59:49AM +0200, Timo Sirainen wrote:
Attached is an example of one such last message saved from Thunderbird after new mail arrived that was not seen. The filler characters are bytes of 0x80, and then the next message from "n-Path: " is present.
They're actually 0x00 inside the mbox file, but Dovecot just translates them to 0x80 since IMAP doesn't allow sending 0x00 characters.
Oh, I see...
BTW, why 0x80, of all characters?
Now, these 0x00 characters are added to mbox file when it's rewriting the headers and it figures out that it needs more space. So it first grows the file by writing 0x00 characters to the end of the file. Then it starts moving message data towards the end of file.
I'll look for the code that does that, and put some debugging prints there to see if I can verify it.
So you mean that this corruption is only seen by Thunderbird, and it won't stay in the mbox file itself?
Indeed. The corruption is as seen by Thunderbird through IMAP. Accessing the mbox file directly, there is no corruption in it.
In that case it means that mbox locking isn't working properly, ie. a Dovecot process is reading the mbox while another Dovecot process is modifying it.
May very well be the case. Especially considering that Thunderbird has multiple connections to the server open. I'm not sure what causes Dovecot to rewrite headers at each such point (given no client initiated write or even read), but I can think of some...
I don't remember having tested mbox_read_locks=dotlock, but I'd have thought it would have worked..
I'll try to investigate more soon. If you can provide some ideas on how to check the locking it could help (given that Dovecot locks for a short period of time only, I can't easily fabricate concurrent locking from outside, and it appears just holding a read lock all the time will not do much).
Perhaps some of the other details of the problem I have described can shed some more light (i.e. the tracing of mbox_sync, the fact that opening a new IMAP connection corrects the problem also for the old one, etc.)?
I will look some more and probably follow up to you with some further details.
Cheers, -- Tom
-- Tom Alsberg - hacker (being the best description fitting this space) Web page: http://www.cs.huji.ac.il/~alsbergt/ DISCLAIMER: The above message does not even necessarily represent what my fingers have typed on the keyboard, save anything further.