[Dovecot] mbox empty messages in Sent folder
I'm talking about mbox Sent folder, where some mailers append through imap server a copy of each message they send.
Sometimes, just three header lines got appended instead of the whole mail message, such as:
From xxxxxxxx@xxxxxx.xxxxxx.xxxxx.xx.xx Fri May 23 12:30:14 2008 X-UID: 2852 Status: RO
This happened in the past (dovecot-1.1-beta/rc with Evolution and with Thunderbird), and happened last week with dovecot-1.1-rc5 (+ last week hg patches).
Anyone had a similar issue?
Regards, Diego.
On Mon, 2008-05-26 at 15:08 +0200, Diego Liziero wrote:
I'm talking about mbox Sent folder, where some mailers append through imap server a copy of each message they send.
Sometimes, just three header lines got appended instead of the whole mail message, such as:
From xxxxxxxx@xxxxxx.xxxxxx.xxxxx.xx.xx Fri May 23 12:30:14 2008 X-UID: 2852 Status: RO
Are there other messages around it with the same From-line? Could this From-line be part of the previous message's body?
On Mon, May 26, 2008 at 3:34 PM, Timo Sirainen tss@iki.fi wrote:
On Mon, 2008-05-26 at 15:08 +0200, Diego Liziero wrote:
I'm talking about mbox Sent folder, where some mailers append through imap server a copy of each message they send.
Sometimes, just three header lines got appended instead of the whole mail message, such as:
From xxxxxxxx@xxxxxx.xxxxxx.xxxxx.xx.xx Fri May 23 12:30:14 2008 X-UID: 2852 Status: RO
Are there other messages around it with the same From-line? Could this From-line be part of the previous message's body?
The empty mail header is part of a sent email that is missing in the Sent mbox file. The previous From header and the following one are correctly the ones of the previous and following sent emails. Here is the mbox lines starting fom the previous mail, and ending to the header of the following one (in this case it seems that the actual previous mail has been deleted as UID 2851 is missing, but I have other examples where no UID is missing):
From xxxxxxxx@xxxxxx.xxxxxx.xxxxx.xx.xx Fri May 23 12:15:17 2008 Message-ID: xxxxxxxx.xxxxx@xxxxxx.xxxxx.xx.xx Date: Fri, 23 May 2008 12:15:06 +0200 From: xxxx xxxxxxx xxxxxxx xxxxxxxx xxxx@xxxxxx.xxxxx.xx.xx User-Agent: xxxxxxxxxxx x.x.x.xx (xxxxxxx/xxxxxxxx) MIME-Version: x.x To: xxxxxxx xxxxxxx@xxxxxx.xxxxx.xx.xx, xxxxxxx xxxx xxxxxxxxx@xxxxxx.xxxxx.xx.xx Subject: xxxxxxxx xxxxxxxx xxxxx x xxxxxxx xxx xxx xxxxxxxxx xx xxxxxxxx... Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit X-UID: 2850 Status: RO X-Keywords: Content-Length: 336
xxxxxxx,
xx xxxxxxxx xxx xxxxxx xxx xxxxxxx xxxxxxxxxxxxx.xx - xxxxxxxxxxxx xxxxxxx xx xxxxx xx xxxxxxx xxxx'xxxxxx xxxxxxxx.
-- xxxxxx xx xxxxx - xxxxxx xxxxxxxxxxxx xxxx xxxxxxxxxx xxxxxxx xxxxxxxxxxx xxx xxx xxxxxxxx, x - xxxxx xxxxx (xx) xxx.: xxx.xxxxxx xxx: xxx xxx.xxx xxxx.: xxx.xxxxxxx xxxxx: xxxx@xxxxxx.xxxxx.xx.xx
From xxxxxxxx@xxxxxx.xxxxxx.xxxxx.xx.xx Fri May 23 12:30:14 2008 X-UID: 2852 Status: RO
From xxxxxxxx@xxxxxx.xxxxxx.xxxxx.xx.xx Sat May 24 08:24:09 2008 Message-ID: xxxxxxxx.xxxxxxx@xxxxxx.xxxxx.xx.xx Date: Sat, 24 May 2008 08:24:02 +0200 From: xxxx xxxxxxx xxxxxxx xxxxxxxx xxxx@xxxxxx.xxxxx.xx.xx User-Agent: xxxxxxxxxxx x.x.x.xx (xxxxxxx/xxxxxxxx) MIME-Version: x.x To: xxxxxxxx xxxxxxxx xxxxxxxx.xxxxxxxx@xxxxxx.xxxxx.xx.xx Subject: xxxxxxx xxx xxxxxxxxx xxxxxxxx,xxxxxxxxxx xxxx (x xxxxxxx xxxx) xxxxxxxx xxx xxxxxxx xxxxxxxx Content-Type: multipart/alternative; xxxxxxxx="------------xxxxxxxxxxxxxxxxxxxxxxxx" X-UID: 2853 Status: R X-Keywords: Content-Length: 4591
Regards, Diego
On Mon, 2008-05-26 at 19:11 +0200, Diego Liziero wrote:
On Mon, May 26, 2008 at 3:34 PM, Timo Sirainen tss@iki.fi wrote:
On Mon, 2008-05-26 at 15:08 +0200, Diego Liziero wrote:
I'm talking about mbox Sent folder, where some mailers append through imap server a copy of each message they send.
Sometimes, just three header lines got appended instead of the whole mail message, such as:
From xxxxxxxx@xxxxxx.xxxxxx.xxxxx.xx.xx Fri May 23 12:30:14 2008 X-UID: 2852 Status: RO
Are there other messages around it with the same From-line? Could this From-line be part of the previous message's body?
The empty mail header is part of a sent email that is missing in the Sent mbox file.
Could it be that the connection died client was trying to save it, so it never got there? Although it still shouldn't have left the From-line there. Could Dovecot have crashed there?
Maybe this helps? http://hg.dovecot.org/dovecot-1.1/rev/dd9d344ba140
On Mon, May 26, 2008 at 11:17 PM, Timo Sirainen tss@iki.fi wrote:
Could it be that the connection died client was trying to save it, so it never got there? Although it still shouldn't have left the From-line there.
The mailer didn't crashed. When it happened to me I thought about a bug of Evolution. When the helpdesk told me that is happening also to other users (with thunderbird), I wrote to this list to ask if someone had a similar issue.
Could Dovecot have crashed there?
I don't know, however nothing unusual is written to the log.
Maybe this helps? http://hg.dovecot.org/dovecot-1.1/rev/dd9d344ba140
Thanks Timo, I'll let you know if it happens again with this patch.
Diego.
On Tue, 2008-05-27 at 15:48 +0200, Diego Liziero wrote:
On Mon, May 26, 2008 at 11:17 PM, Timo Sirainen tss@iki.fi wrote:
Could it be that the connection died client was trying to save it, so it never got there? Although it still shouldn't have left the From-line there.
The mailer didn't crashed. When it happened to me I thought about a bug of Evolution. When the helpdesk told me that is happening also to other users (with thunderbird), I wrote to this list to ask if someone had a similar issue.
I've been using mboxes for years with Evolution and haven't seen that problem. I just grepped my Sent Messages and there wasn't a single empty message.
Could Dovecot have crashed there?
I don't know, however nothing unusual is written to the log.
There should have then been logged "killed with signal".
Maybe this helps? http://hg.dovecot.org/dovecot-1.1/rev/dd9d344ba140
Thanks Timo, I'll let you know if it happens again with this patch.
That only causes the empty message not to be written at all, it doesn't help about losing saved messages.
On Tue, May 27, 2008 at 5:43 PM, Timo Sirainen tss@iki.fi wrote:
Maybe this helps? http://hg.dovecot.org/dovecot-1.1/rev/dd9d344ba140
Thanks Timo, I'll let you know if it happens again with this patch.
That only causes the empty message not to be written at all, it doesn't help about losing saved messages.
Amoung the various emails with header corruption I got with 1.1.rc8, one of them is similar to the one of this bug report. Just the first 5 lines of header without body, and without the remaining lines of the header, either.
But this time there are two header lines more: X-Keywords and Content-Length.
From xxxxxxxx@xxxx.xxxxxx.xxxxx.xx.xx Mon Jun 02 23:53:29 2008 X-UID: 7863 Status: R X-Keywords: Content-Length: 0
Actually this is the Content-Length after having selected the mail with evolution, and having this error in dovecot.log:
Error: IMAP(username): FETCH for mailbox sent-mail UID 7863 got too little data: 2 vs 615 Error: IMAP(username): Corrupted index cache file /mailhome/username/.imap/sent-mail/dovecot.index.cache: Broken virtual size for mail UID 7863
I don't know if the original email content length was correctly written in the header, or in the index.
Diego.
On Tue, May 27, 2008 at 3:48 PM, Diego Liziero diegoliz@gmail.com wrote:
On Mon, May 26, 2008 at 11:17 PM, Timo Sirainen tss@iki.fi wrote:
Could it be that the connection died client was trying to save it, so it never got there? Although it still shouldn't have left the From-line there.
The mailer didn't crashed. When it happened to me I thought about a bug of Evolution. When the helpdesk told me that is happening also to other users (with thunderbird), I wrote to this list to ask if someone had a similar issue.
Could Dovecot have crashed there?
I don't know, however nothing unusual is written to the log.
Maybe this helps? http://hg.dovecot.org/dovecot-1.1/rev/dd9d344ba140
Thanks Timo, I'll let you know if it happens again with this patch.
Unfortunately it happened again with dovecot-1.1.rc13 (=1.1.0)
This time using horde/imp
Diego.
On Mon, Jun 23, 2008 at 11:03 AM, Diego Liziero diegoliz@gmail.com wrote:
On Tue, May 27, 2008 at 3:48 PM, Diego Liziero diegoliz@gmail.com wrote:
On Mon, May 26, 2008 at 11:17 PM, Timo Sirainen tss@iki.fi wrote:
Maybe this helps? http://hg.dovecot.org/dovecot-1.1/rev/dd9d344ba140
Thanks Timo, I'll let you know if it happens again with this patch.
Unfortunately it happened again with dovecot-1.1.rc13 (=1.1.0)
And many users complains about this also with latest dovecot 1.1.2
In the last 2 days it happened 7 times to my user. I've the rawlog and valgrind logs. Nothing wrong according to valgrind, and nothing strange in the rawlog.
Is there any further debug I can use?
On Aug 1, 2008, at 1:49 AM, Diego Liziero wrote:
On Mon, Jun 23, 2008 at 11:03 AM, Diego Liziero diegoliz@gmail.com
wrote:On Tue, May 27, 2008 at 3:48 PM, Diego Liziero diegoliz@gmail.com
wrote:On Mon, May 26, 2008 at 11:17 PM, Timo Sirainen tss@iki.fi wrote:
Maybe this helps? http://hg.dovecot.org/dovecot-1.1/rev/ dd9d344ba140
Thanks Timo, I'll let you know if it happens again with this patch.
Unfortunately it happened again with dovecot-1.1.rc13 (=1.1.0)
And many users complains about this also with latest dovecot 1.1.2
In the last 2 days it happened 7 times to my user. I've the rawlog and valgrind logs. Nothing wrong according to valgrind, and nothing strange in the
rawlog.Is there any further debug I can use?
You mean the messages show up as APPENDed in the rawlog with an OK
reply?
On Fri, Aug 1, 2008 at 1:09 AM, Timo Sirainen tss@iki.fi wrote:
On Aug 1, 2008, at 1:49 AM, Diego Liziero wrote:
On Mon, Jun 23, 2008 at 11:03 AM, Diego Liziero diegoliz@gmail.com
wrote:
On Tue, May 27, 2008 at 3:48 PM, Diego Liziero diegoliz@gmail.com wrote:
On Mon, May 26, 2008 at 11:17 PM, Timo Sirainen tss@iki.fi wrote:
Maybe this helps? http://hg.dovecot.org/dovecot-1.1/rev/dd9d344ba140
Thanks Timo, I'll let you know if it happens again with this patch.
Unfortunately it happened again with dovecot-1.1.rc13 (=1.1.0)
And many users complains about this also with latest dovecot 1.1.2
In the last 2 days it happened 7 times to my user. I've the rawlog and valgrind logs. Nothing wrong according to valgrind, and nothing strange in the rawlog.
Is there any further debug I can use?
You mean the messages show up as APPENDed in the rawlog with an OK reply?
It seems so, I sent you privately the rawlog and the last part of a sent-mail mailbox.
Anyone besides me is seeing this? (random empty mails in Sent folder)
In the last two weeks users are complainig almost every day about lost sent mails.
Regards, Diego.
On Aug 3, 2008, at 10:31 PM, Diego Liziero wrote:
It seems so, I sent you privately the rawlog and the last part of a sent-mail
mailbox.
The interesting thing about that rawlog was that it shows the APPEND
returning it saved the message with UID x, but in the mbox file
there's no UID x, but there is the empty message with UID x+1.
Anyone besides me is seeing this? (random empty mails in Sent folder)
I'm using mboxes all the time, never seen this..
Perhaps if you put all processes through strace (-s 1000000) and when
it again happens for some user send me the strace? Although I'd guess
it shows that the message was properly written to the mbox file. The
real question is then what truncates the mbox file..
On Mon, Aug 4, 2008 at 12:07 AM, Timo Sirainen tss@iki.fi wrote:
On Aug 3, 2008, at 10:31 PM, Diego Liziero wrote:
It seems so, I sent you privately the rawlog and the last part of a sent-mail mailbox.
The interesting thing about that rawlog was that it shows the APPEND returning it saved the message with UID x, but in the mbox file there's no UID x, but there is the empty message with UID x+1.
Anyone besides me is seeing this? (random empty mails in Sent folder)
I'm using mboxes all the time, never seen this..
Perhaps if you put all processes through strace (-s 1000000) and when it again happens for some user send me the strace? Although I'd guess it shows that the message was properly written to the mbox file. The real question is then what truncates the mbox file..
Just sent to Timo the strace of the same mail sent twice, the fist time appeared empty, the second time correctly written to sent-mail folder.
On Aug 4, 2008, at 5:21 AM, Diego Liziero wrote:
On Mon, Aug 4, 2008 at 12:07 AM, Timo Sirainen tss@iki.fi wrote:
On Aug 3, 2008, at 10:31 PM, Diego Liziero wrote:
It seems so, I sent you privately the rawlog and the last part of a sent-mail
mailbox.The interesting thing about that rawlog was that it shows the APPEND returning it saved the message with UID x, but in the mbox file
there's no UID x, but there is the empty message with UID x+1.Anyone besides me is seeing this? (random empty mails in Sent
folder)I'm using mboxes all the time, never seen this..
Perhaps if you put all processes through strace (-s 1000000) and
when it again happens for some user send me the strace? Although I'd guess
it shows that the message was properly written to the mbox file. The real
question is then what truncates the mbox file..Just sent to Timo the strace of the same mail sent twice, the fist time appeared empty, the second time correctly written to sent-mail folder.
Maybe this helps? http://hg.dovecot.org/dovecot-1.1/rev/8ab845d3c96d
On Mon, Aug 4, 2008 at 4:17 PM, Timo Sirainen tss@iki.fi wrote:
Maybe this helps? http://hg.dovecot.org/dovecot-1.1/rev/8ab845d3c96d
It seems so, thanks Timo.
With this patch, by now, all sent mails are correctly written in "Sent" folder, I'let you know if I've just been lucky :)
BTW I didn't succeed in reproducing this issue with imaptest, what was the trick to trigger it?
On Aug 6, 2008, at 6:11 AM, Diego Liziero wrote:
On Mon, Aug 4, 2008 at 4:17 PM, Timo Sirainen tss@iki.fi wrote:
Maybe this helps? http://hg.dovecot.org/dovecot-1.1/rev/8ab845d3c96d
It seems so, thanks Timo.
With this patch, by now, all sent mails are correctly written in "Sent" folder, I'let you know if I've just been lucky :)
BTW I didn't succeed in reproducing this issue with imaptest, what was the trick to trigger it?
I'm not sure if there's an easy way to reproduce it. You'd have to
cause the first read to return EAGAIN but the second read that comes
only microseconds later to return the entire message. Perhaps if
imaptest sent first the APPEND command, then did a small pause and
after that sent the message.
On Wed, Aug 6, 2008 at 4:26 PM, Timo Sirainen tss@iki.fi wrote:
On Aug 6, 2008, at 6:11 AM, Diego Liziero wrote:
On Mon, Aug 4, 2008 at 4:17 PM, Timo Sirainen tss@iki.fi wrote:
Maybe this helps? http://hg.dovecot.org/dovecot-1.1/rev/8ab845d3c96d
It seems so, thanks Timo.
With this patch, by now, all sent mails are correctly written in "Sent" folder, I'let you know if I've just been lucky :)
Definitely solved. I asked the most complainig users to test if it's fixed and they say "yes". The most affected client was horde/imp webmail.
Thanks again Timo.
BTW I didn't succeed in reproducing this issue with imaptest, what was the trick to trigger it?
I'm not sure if there's an easy way to reproduce it. You'd have to cause the first read to return EAGAIN but the second read that comes only microseconds later to return the entire message. Perhaps if imaptest sent first the APPEND command, then did a small pause and after that sent the message.
Mmm.. I tried to comment out the "cork" part and added a 10% random sleep after sending the command if (!(rand()%9)) usleep(rand()%5000000);
and I started getting the famous "Error: IMAP(testdove): FETCH for mailbox INBOX UID xxx got too little data: yyy vs zzz" instead.
Regards, Diego.
On Aug 13, 2008, at 6:19 PM, Diego Liziero wrote:
Mmm.. I tried to comment out the "cork" part and added a 10% random sleep after sending the command if (!(rand()%9)) usleep(rand()%5000000);
and I started getting the famous "Error: IMAP(testdove): FETCH for mailbox INBOX UID xxx got too little data: yyy vs zzz" instead.
With the latest code? Sounds interesting. Could you send me all the
changes as a patch? I probably won't have time to check it until next
week though.
participants (2)
-
Diego Liziero
-
Timo Sirainen