Bug#776094: dovecot-imapd: corrupts mailbox after trying to retrieve it (fwd)
Hello.
I wrote about this three weeks ago but got no answer. I'm going to officially "forward" the Debian bug this time, with all the details.
The test case is just 840 bytes long. Please give it a try.
---------- Forwarded message ---------- From: Santiago Vila sanvila@unex.es To: submit@bugs.debian.org Date: Fri, 23 Jan 2015 22:32:28 +0100 (CET) Subject: dovecot-imapd: corrupts mailbox after trying to retrieve it
Package: dovecot-imapd Version: 1:2.2.13-11 Severity: serious
The following mbox folder, when put in $HOME/mail, becomes corrupted after trying to retrieve it with fetchmail.
The problem may be reproduced by using the same machine as server and client:
Put "inbox-b" in $HOME/mail
Put this in $HOME/.fetchmailrc
server localhost proto imap port 143: user "someuser" pass "thepassword"
- Retrieve email using this command line:
fetchmail -a localhost --folder inbox-b -m "true"
Note: By looking at the "true" above it is clear that whatever fetchmail does with the message is not important at all.
You will see something like this:
12 messages for someuser at localhost (folder inbox-b). reading message someuser@localhost:1 of 12 (171 header octets) (3 body octets) flushed reading message someuser@localhost:2 of 12 (245 header octets) (3 body octets) flushed reading message someuser@localhost:3 of 12 (245 header octets) (3 body octets) flushed reading message someuser@localhost:4 of 12 (245 header octets) (3 body octets) flushed reading message someuser@localhost:5 of 12 (245 header octets) (3 body octets) flushed reading message someuser@localhost:6 of 12 (171 header octets) (3 body octets) flushed reading message someuser@localhost:7 of 12 (171 header octets) (3 body octets) flushed reading message someuser@localhost:8 of 12 (245 header octets) (3 body octets) flushed reading message someuser@localhost:9 of 12 (245 header octets) (3 body octets) flushed reading message someuser@localhost:10 of 12 (245 header octets) (3 body octets) flushed reading message someuser@localhost:11 of 12 (245 header octets) (3 body octets) flushed reading message someuser@localhost:12 of 12 (273 header octets)fetchmail: incorrect header line found - see manpage for bad-header option not flushed
And in fact "inbox-b" in the server is now like this:
[...]
From root@example.com Tue Jan 13 10:18:20 2015 rstuvwxyzabcdefghijklmnopqrstuvwxyz@example.com To: a@example.com Subject: a MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Message-Id: 20150113091737.B5ADA5F8B1@example.com Date: Tue, 13 Jan 2015 10:17:25 +0100 (CET) X-UID: 16035 Status: O
a
Note how the From: line has been truncated from its original state.
I have been suffering from this problem for months. At first I believed it was some misbehaving procmail/formail recipe I had on the server, but that's not the case as this example shows.
Thanks.
On 14 Feb 2015, at 16:23, Santiago Vila sanvila@unex.es wrote:
I wrote about this three weeks ago but got no answer. I'm going to officially "forward" the Debian bug this time, with all the details.
The test case is just 840 bytes long. Please give it a try. .. Package: dovecot-imapd Version: 1:2.2.13-11 Severity: serious
I can't reproduce with latest Dovecot hg. But just in case it's still not fixed, there are two important things:
Send your doveconf -n output, since there are some settings that can affect this
rm -rf ~/mail/.imap/inbox-b before testing to make sure indexes don't cause this problem.
The following mbox folder, when put in $HOME/mail, becomes corrupted after trying to retrieve it with fetchmail.
The problem may be reproduced by using the same machine as server and client:
Put "inbox-b" in $HOME/mail
Put this in $HOME/.fetchmailrc
server localhost proto imap port 143: user "someuser" pass "thepassword"
- Retrieve email using this command line:
fetchmail -a localhost --folder inbox-b -m "true"
Note: By looking at the "true" above it is clear that whatever fetchmail does with the message is not important at all.
You will see something like this:
12 messages for someuser at localhost (folder inbox-b). reading message someuser@localhost:1 of 12 (171 header octets) (3 body octets) flushed reading message someuser@localhost:2 of 12 (245 header octets) (3 body octets) flushed reading message someuser@localhost:3 of 12 (245 header octets) (3 body octets) flushed reading message someuser@localhost:4 of 12 (245 header octets) (3 body octets) flushed reading message someuser@localhost:5 of 12 (245 header octets) (3 body octets) flushed reading message someuser@localhost:6 of 12 (171 header octets) (3 body octets) flushed reading message someuser@localhost:7 of 12 (171 header octets) (3 body octets) flushed reading message someuser@localhost:8 of 12 (245 header octets) (3 body octets) flushed reading message someuser@localhost:9 of 12 (245 header octets) (3 body octets) flushed reading message someuser@localhost:10 of 12 (245 header octets) (3 body octets) flushed reading message someuser@localhost:11 of 12 (245 header octets) (3 body octets) flushed reading message someuser@localhost:12 of 12 (273 header octets)fetchmail: incorrect header line found - see manpage for bad-header option not flushed
And in fact "inbox-b" in the server is now like this:
[...]
From root@example.com Tue Jan 13 10:18:20 2015 rstuvwxyzabcdefghijklmnopqrstuvwxyz@example.com To: a@example.com Subject: a MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Message-Id: 20150113091737.B5ADA5F8B1@example.com Date: Tue, 13 Jan 2015 10:17:25 +0100 (CET) X-UID: 16035 Status: O
a
Note how the From: line has been truncated from its original state.
I have been suffering from this problem for months. At first I believed it was some misbehaving procmail/formail recipe I had on the server, but that's not the case as this example shows.
Thanks.
On Sun, 15 Feb 2015, Timo Sirainen wrote:
On 14 Feb 2015, at 16:23, Santiago Vila sanvila@unex.es wrote:
I wrote about this three weeks ago but got no answer. I'm going to officially "forward" the Debian bug this time, with all the details.
The test case is just 840 bytes long. Please give it a try.
[ Small correction: It's really 3873 bvtes long, uncompressed ].
..
Package: dovecot-imapd Version: 1:2.2.13-11 Severity: serious
I can't reproduce with latest Dovecot hg.
In such case we would love to know what is the commit that fixed this, so that we can apply it to the 2.2.13 version in Debian. We have frozen the distribution as we are about to release jessie as Debian 8, so no new upstream releases are allowed anymore.
But just in case it's still not fixed, there are two important things:
- Send your doveconf -n output, since there are some settings that can affect this
Nothing special. The default configuration from the Debian package. The bug may be reproduced without touching any file in /etc at all. This is the result of "doveconf -n":
# 2.2.13: /etc/dovecot/dovecot.conf # OS: Linux 3.16.0-4-amd64 x86_64 Debian 8.0 mail_location = mbox:~/mail:INBOX=/var/mail/%u namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = pam } protocols = " imap" ssl = no userdb { driver = passwd }
- rm -rf ~/mail/.imap/inbox-b before testing to make sure indexes don't cause this problem.
Yes, I did that. In fact, I checked that this happens on a freshly installed virtual machine, i.e. no ~/mail/.imap at all.
On 2/19/2015 4:34 PM, Santiago Vila sanvila@unex.es wrote:
In such case we would love to know what is the commit that fixed this, so that we can apply it to the 2.2.13 version in Debian. We have frozen the distribution as we are about to release jessie as Debian 8, so no new upstream releases are allowed anymore.
I have NEVER understood the rationale for doing this for MINOR release.
Major releases/updates, sure, I understand completely, but minor releases? It is far too much pain for far too little gain imnsho...
Am 20.02.2015 um 15:03 schrieb Charles Marcus:
On 2/19/2015 4:34 PM, Santiago Vila sanvila@unex.es wrote:
In such case we would love to know what is the commit that fixed this, so that we can apply it to the 2.2.13 version in Debian. We have frozen the distribution as we are about to release jessie as Debian 8, so no new upstream releases are allowed anymore.
I have NEVER understood the rationale for doing this for MINOR release.
Major releases/updates, sure, I understand completely, but minor releases? It is far too much pain for far too little gain imnsho...
that's a political decision to not break workarounds because someone removes a bug you worked around or even not break stupid software rely on the bahvior of bugs :-)
and to make web-developers lifes harder because params in PHP which help to prepare upgrade to >= 5.4 and are present for many years in 5.3.x are not available on Debian systems
Greetings.
Thanks to Jelmer Vernooij, who has just uploaded version 2.2.16 for Debian unstable, I can confirm that this bug may still be reproduced in version 2.2.16, so it has *not* been fixed yet.
Several months ago I was told "I can't reproduce it in dovecot hg", or something like that. Well, having this fixed in hg does not help very much if the bug is still present in the latest released version (2.2.16).
Would anyone here please tell which commit exactly fixed this issue?
If it's not fixed yet, I would love to provide a patch, but I'm not a programmer myself.
However, I provided a very precise and detailed explanation about how to reproduce this bug, which you will still find in the Debian BTS:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=776094
Really, it should not be so much difficult to reproduce it. Try on a Debian 8 system. I'm sure that if you follow the steps exactly as I described, you will find the problem the same way as I did.
Thanks.
Am 04.05.2015 um 21:04 schrieb Santiago Vila:
Greetings.
Thanks to Jelmer Vernooij, who has just uploaded version 2.2.16 for Debian unstable, I can confirm that this bug may still be reproduced in version 2.2.16, so it has *not* been fixed yet.
Several months ago I was told "I can't reproduce it in dovecot hg", or something like that. Well, having this fixed in hg does not help very much if the bug is still present in the latest released version (2.2.16).
Would anyone here please tell which commit exactly fixed this issue?
If it's not fixed yet, I would love to provide a patch, but I'm not a programmer myself.
However, I provided a very precise and detailed explanation about how to reproduce this bug, which you will still find in the Debian BTS:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=776094
Really, it should not be so much difficult to reproduce it. Try on a Debian 8 system. I'm sure that if you follow the steps exactly as I described, you will find the problem the same way as I did.
Thanks.
Boh fetchmail .... did you verified with getmail ?
http://pyropus.ca/software/getmail/
dig you tried fetchmail with
bad-header accept ?
what are dovecot verbose logs tell
Best Regards MfG Robert Schetterer
-- [*] sys4 AG
http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein
On Mon, May 04, 2015 at 09:23:57PM +0200, Robert Schetterer wrote:
Boh fetchmail .... did you verified with getmail ?
No, I didn't. The client should be irrelevant. Nobody should be able to corrupt a remote mailbox only by issuing imap commands.
BTW: Does getmail have an option doing the same as fetchmail --folder option at all?
dig you tried fetchmail with
bad-header accept ?
I appreciate that you are trying to help, but it seems to me that you didn't read the report, or you didn't understand it.
Answer: No. I didn't. Why should I? The mbox was fine before I tried to retrieve it. Please read the report!
Am 04.05.2015 um 22:13 schrieb Santiago Vila:
On Mon, May 04, 2015 at 09:23:57PM +0200, Robert Schetterer wrote:
Boh fetchmail .... did you verified with getmail ?
No, I didn't. The client should be irrelevant. Nobody should be able to corrupt a remote mailbox only by issuing imap commands.
BTW: Does getmail have an option doing the same as fetchmail --folder option at all?
http://pyropus.ca/software/getmail/configuration.html#retriever-parameters
All IMAP retriever types also take the following optional parameters:
mailboxes (tuple of quoted strings) — a list of mailbox paths toretrieve mail from, expressed as a Python tuple. If not specified, the default is to retrieve mail from the mail folder named INBOX. You might want to retrieve messages from several different mail folders, using a configuration like this:
mailboxes = ("INBOX", "INBOX.spam",
"mailing-lists.model-railroading")dig you tried fetchmail with
bad-header accept ?
I appreciate that you are trying to help, but it seems to me that you didn't read the report, or you didn't understand it.
Answer: No. I didn't. Why should I? The mbox was fine before I tried to retrieve it. Please read the report!
sorry i wont invest any time in fetchmail, i quit with it years ago by tons of problems i managed to forget
Best Regards MfG Robert Schetterer
-- [*] sys4 AG
http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein
On Mon, May 04, 2015 at 11:43:23PM +0200, Robert Schetterer wrote:
BTW: Does getmail have an option doing the same as fetchmail --folder option at all?
http://pyropus.ca/software/getmail/configuration.html#retriever-parameters
All IMAP retriever types also take the following optional parameters:
mailboxes (tuple of quoted strings) — a list of mailbox paths toretrieve mail from, expressed as a Python tuple. If not specified, the default is to retrieve mail from the mail folder named INBOX. You might want to retrieve messages from several different mail folders, using a configuration like this:
mailboxes = ("INBOX", "INBOX.spam", "mailing-lists.model-railroading")
Fine, if I wanted to use getmail I would have to use this:
mailboxes = ("inbox-b",)
dig you tried fetchmail with
bad-header accept ?
I appreciate that you are trying to help, but it seems to me that you didn't read the report, or you didn't understand it.
Answer: No. I didn't. Why should I? The mbox was fine before I tried to retrieve it. Please read the report!
sorry i wont invest any time in fetchmail, i quit with it years ago by tons of problems i managed to forget
But I'm not reporting a bug in fetchmail, I'm reporting a bug in dovecot.
Everything fetchmail does is to issue IMAP commands to dovecot, and it's those IMAP commands what make the remote folder to be corrupted.
Please tell me: Is there any set of forbidden IMAP commands that one "should not use" because of them being known to cause email corruption in the server?
The corruption happens in the server running dovecot, not in the client running fetchmail.
If I describe this problem with fetchmail and dovecot in the same machine is only for simpliciry about reproducing it.
I have just verified with IMAP commands. This is the procedure:
telnet localhost 143
and then type this:
A0001 CAPABILITY A0002 LOGIN "bluser" "bluser" A0003 SELECT "inbox-b" A0004 EXPUNGE A0005 FETCH 1:12 RFC822.SIZE A0006 FETCH 1 RFC822.HEADER A0007 FETCH 1 BODY.PEEK[TEXT] A0008 STORE 1 +FLAGS (\Seen \Deleted) A0009 EXPUNGE A0010 FETCH 1 RFC822.HEADER A0011 FETCH 1 BODY.PEEK[TEXT] A0012 STORE 1 +FLAGS (\Seen \Deleted) A0013 EXPUNGE A0014 FETCH 1 RFC822.HEADER A0015 FETCH 1 BODY.PEEK[TEXT] A0016 STORE 1 +FLAGS (\Seen \Deleted) A0017 EXPUNGE A0018 FETCH 1 RFC822.HEADER A0019 FETCH 1 BODY.PEEK[TEXT] A0020 STORE 1 +FLAGS (\Seen \Deleted) A0021 EXPUNGE A0022 FETCH 1 RFC822.HEADER A0023 FETCH 1 BODY.PEEK[TEXT] A0024 STORE 1 +FLAGS (\Seen \Deleted) A0025 EXPUNGE A0026 FETCH 1 RFC822.HEADER A0027 FETCH 1 BODY.PEEK[TEXT] A0028 STORE 1 +FLAGS (\Seen \Deleted) A0029 EXPUNGE A0030 FETCH 1 RFC822.HEADER A0031 FETCH 1 BODY.PEEK[TEXT] A0032 STORE 1 +FLAGS (\Seen \Deleted) A0033 EXPUNGE A0034 FETCH 1 RFC822.HEADER A0035 FETCH 1 BODY.PEEK[TEXT] A0036 STORE 1 +FLAGS (\Seen \Deleted) A0037 EXPUNGE A0038 FETCH 1 RFC822.HEADER A0039 FETCH 1 BODY.PEEK[TEXT] A0040 STORE 1 +FLAGS (\Seen \Deleted) A0041 EXPUNGE A0042 FETCH 1 RFC822.HEADER A0043 FETCH 1 BODY.PEEK[TEXT] A0044 STORE 1 +FLAGS (\Seen \Deleted) A0045 EXPUNGE A0046 FETCH 1 RFC822.HEADER A0047 FETCH 1 BODY.PEEK[TEXT] A0048 STORE 1 +FLAGS (\Seen \Deleted) A0049 EXPUNGE A0050 FETCH 1 RFC822.HEADER A0051 LOGOUT
After this, mbox folder inbox-b is corrupted, as the line saying
From: abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz@example.com
becomes
rstuvwxyzabcdefghijklmnopqrstuvwxyz@example.com
So: Could we please stop blaming fetchmail for this? It's just the messenger.
Am 05.05.2015 um 16:26 schrieb Santiago Vila:
I have just verified with IMAP commands. This is the procedure:
telnet localhost 143
and then type this:
A0001 CAPABILITY A0002 LOGIN "bluser" "bluser" A0003 SELECT "inbox-b" A0004 EXPUNGE A0005 FETCH 1:12 RFC822.SIZE A0006 FETCH 1 RFC822.HEADER A0007 FETCH 1 BODY.PEEK[TEXT] A0008 STORE 1 +FLAGS (\Seen \Deleted) A0009 EXPUNGE A0010 FETCH 1 RFC822.HEADER A0011 FETCH 1 BODY.PEEK[TEXT] A0012 STORE 1 +FLAGS (\Seen \Deleted) A0013 EXPUNGE A0014 FETCH 1 RFC822.HEADER A0015 FETCH 1 BODY.PEEK[TEXT] A0016 STORE 1 +FLAGS (\Seen \Deleted) A0017 EXPUNGE A0018 FETCH 1 RFC822.HEADER A0019 FETCH 1 BODY.PEEK[TEXT] A0020 STORE 1 +FLAGS (\Seen \Deleted) A0021 EXPUNGE A0022 FETCH 1 RFC822.HEADER A0023 FETCH 1 BODY.PEEK[TEXT] A0024 STORE 1 +FLAGS (\Seen \Deleted) A0025 EXPUNGE A0026 FETCH 1 RFC822.HEADER A0027 FETCH 1 BODY.PEEK[TEXT] A0028 STORE 1 +FLAGS (\Seen \Deleted) A0029 EXPUNGE A0030 FETCH 1 RFC822.HEADER A0031 FETCH 1 BODY.PEEK[TEXT] A0032 STORE 1 +FLAGS (\Seen \Deleted) A0033 EXPUNGE A0034 FETCH 1 RFC822.HEADER A0035 FETCH 1 BODY.PEEK[TEXT] A0036 STORE 1 +FLAGS (\Seen \Deleted) A0037 EXPUNGE A0038 FETCH 1 RFC822.HEADER A0039 FETCH 1 BODY.PEEK[TEXT] A0040 STORE 1 +FLAGS (\Seen \Deleted) A0041 EXPUNGE A0042 FETCH 1 RFC822.HEADER A0043 FETCH 1 BODY.PEEK[TEXT] A0044 STORE 1 +FLAGS (\Seen \Deleted) A0045 EXPUNGE A0046 FETCH 1 RFC822.HEADER A0047 FETCH 1 BODY.PEEK[TEXT] A0048 STORE 1 +FLAGS (\Seen \Deleted) A0049 EXPUNGE A0050 FETCH 1 RFC822.HEADER A0051 LOGOUT
After this, mbox folder inbox-b is corrupted, as the line saying
From: abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz@example.com
becomes
rstuvwxyzabcdefghijklmnopqrstuvwxyz@example.com
is this related to mbox only ? i have no mbox install online to test
did you try install dovecot from scratch with latest sources?
i used a script to install a quick dovecot test server on ubuntu
see https://sys4.de/de/blog/2013/06/06/postfix-dovecot-ceph-cluster-storage/
beyond
... Dovecot install ...
then retry with telnet , sorry no time to test myself
So: Could we please stop blaming fetchmail for this? It's just the messenger.
Best Regards MfG Robert Schetterer
-- [*] sys4 AG
http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein
On Tue, May 05, 2015 at 05:15:16PM +0200, Robert Schetterer wrote:
is this related to mbox only ?
I don't know. The bug happens as I explained, and current documentation says mbox is still supported (I really hope documentation is correct here).
i have no mbox install online to test
A local virtual machine is more than enough to reproduce this. It does not need to be online.
Also, it is not required to have client and server on different machines to reproduce this. They can be the same.
did you try install dovecot from scratch with latest sources?
This is dovecot 2.2.16-1 from Debian unstable, released yesterday, which is the most recent release I can test.
Am 05.05.2015 um 19:22 schrieb Santiago Vila:
On Tue, May 05, 2015 at 05:15:16PM +0200, Robert Schetterer wrote:
is this related to mbox only ?
I don't know. The bug happens as I explained, and current documentation says mbox is still supported (I really hope documentation is correct here).
yeah but it may be a mbox only bug
i have no mbox install online to test
A local virtual machine is more than enough to reproduce this. It does not need to be online.
i know, but i am short in time
Also, it is not required to have client and server on different machines to reproduce this. They can be the same.
i tested your telnet sequence with maildir and could not reproduce your report which means nothing corupted
did you try install dovecot from scratch with latest sources?
This is dovecot 2.2.16-1 from Debian unstable, released yesterday, which is the most recent release I can test.
install from scratch, easy with script
Best Regards MfG Robert Schetterer
-- [*] sys4 AG
http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein
Am 05.05.2015 um 20:46 schrieb Robert Schetterer:
Am 05.05.2015 um 19:22 schrieb Santiago Vila:
On Tue, May 05, 2015 at 05:15:16PM +0200, Robert Schetterer wrote:
is this related to mbox only ?
I don't know. The bug happens as I explained, and current documentation says mbox is still supported (I really hope documentation is correct here).
yeah but it may be a mbox only bug
i have no mbox install online to test
A local virtual machine is more than enough to reproduce this. It does not need to be online.
i know, but i am short in time
Also, it is not required to have client and server on different machines to reproduce this. They can be the same.
i tested your telnet sequence with maildir and could not reproduce your report which means nothing corupted
did you try install dovecot from scratch with latest sources?
This is dovecot 2.2.16-1 from Debian unstable, released yesterday, which is the most recent release I can test.
install from scratch, easy with script
Best Regards MfG Robert Schetterer
famos last words, my best bet is mbox bugs described here
https://bugzilla.redhat.com/show_bug.cgi?id=1189198
arent finally fixed
Best Regards MfG Robert Schetterer
-- [*] sys4 AG
http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein
On 05/05/2015 05:26 PM, Santiago Vila wrote:
I have just verified with IMAP commands. This is the procedure:
telnet localhost 143
and then type this:
A0001 CAPABILITY A0002 LOGIN "bluser" "bluser" A0003 SELECT "inbox-b" A0004 EXPUNGE A0005 FETCH 1:12 RFC822.SIZE A0006 FETCH 1 RFC822.HEADER A0007 FETCH 1 BODY.PEEK[TEXT] A0008 STORE 1 +FLAGS (\Seen \Deleted) A0009 EXPUNGE A0010 FETCH 1 RFC822.HEADER A0011 FETCH 1 BODY.PEEK[TEXT] A0012 STORE 1 +FLAGS (\Seen \Deleted) A0013 EXPUNGE A0014 FETCH 1 RFC822.HEADER A0015 FETCH 1 BODY.PEEK[TEXT] A0016 STORE 1 +FLAGS (\Seen \Deleted) A0017 EXPUNGE A0018 FETCH 1 RFC822.HEADER A0019 FETCH 1 BODY.PEEK[TEXT] A0020 STORE 1 +FLAGS (\Seen \Deleted) A0021 EXPUNGE A0022 FETCH 1 RFC822.HEADER A0023 FETCH 1 BODY.PEEK[TEXT] A0024 STORE 1 +FLAGS (\Seen \Deleted) A0025 EXPUNGE A0026 FETCH 1 RFC822.HEADER A0027 FETCH 1 BODY.PEEK[TEXT] A0028 STORE 1 +FLAGS (\Seen \Deleted) A0029 EXPUNGE A0030 FETCH 1 RFC822.HEADER A0031 FETCH 1 BODY.PEEK[TEXT] A0032 STORE 1 +FLAGS (\Seen \Deleted) A0033 EXPUNGE A0034 FETCH 1 RFC822.HEADER A0035 FETCH 1 BODY.PEEK[TEXT] A0036 STORE 1 +FLAGS (\Seen \Deleted) A0037 EXPUNGE A0038 FETCH 1 RFC822.HEADER A0039 FETCH 1 BODY.PEEK[TEXT] A0040 STORE 1 +FLAGS (\Seen \Deleted) A0041 EXPUNGE A0042 FETCH 1 RFC822.HEADER A0043 FETCH 1 BODY.PEEK[TEXT] A0044 STORE 1 +FLAGS (\Seen \Deleted) A0045 EXPUNGE A0046 FETCH 1 RFC822.HEADER A0047 FETCH 1 BODY.PEEK[TEXT] A0048 STORE 1 +FLAGS (\Seen \Deleted) A0049 EXPUNGE A0050 FETCH 1 RFC822.HEADER A0051 LOGOUT
After this, mbox folder inbox-b is corrupted, as the line saying
From: abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz@example.com
becomes
rstuvwxyzabcdefghijklmnopqrstuvwxyz@example.com
So: Could we please stop blaming fetchmail for this? It's just the messenger. Could you also sprovide your "dovecot -n" output and any warnings and errors in dovecot logs.
br, Teemu Huovila
On 05/06/2015 09:57 AM, Teemu Huovila wrote:
On 05/05/2015 05:26 PM, Santiago Vila wrote:
I have just verified with IMAP commands. This is the procedure:
telnet localhost 143
and then type this:
A0001 CAPABILITY A0002 LOGIN "bluser" "bluser" A0003 SELECT "inbox-b" A0004 EXPUNGE A0005 FETCH 1:12 RFC822.SIZE A0006 FETCH 1 RFC822.HEADER A0007 FETCH 1 BODY.PEEK[TEXT] A0008 STORE 1 +FLAGS (\Seen \Deleted) A0009 EXPUNGE A0010 FETCH 1 RFC822.HEADER A0011 FETCH 1 BODY.PEEK[TEXT] A0012 STORE 1 +FLAGS (\Seen \Deleted) A0013 EXPUNGE A0014 FETCH 1 RFC822.HEADER A0015 FETCH 1 BODY.PEEK[TEXT] A0016 STORE 1 +FLAGS (\Seen \Deleted) A0017 EXPUNGE A0018 FETCH 1 RFC822.HEADER A0019 FETCH 1 BODY.PEEK[TEXT] A0020 STORE 1 +FLAGS (\Seen \Deleted) A0021 EXPUNGE A0022 FETCH 1 RFC822.HEADER A0023 FETCH 1 BODY.PEEK[TEXT] A0024 STORE 1 +FLAGS (\Seen \Deleted) A0025 EXPUNGE A0026 FETCH 1 RFC822.HEADER A0027 FETCH 1 BODY.PEEK[TEXT] A0028 STORE 1 +FLAGS (\Seen \Deleted) A0029 EXPUNGE A0030 FETCH 1 RFC822.HEADER A0031 FETCH 1 BODY.PEEK[TEXT] A0032 STORE 1 +FLAGS (\Seen \Deleted) A0033 EXPUNGE A0034 FETCH 1 RFC822.HEADER A0035 FETCH 1 BODY.PEEK[TEXT] A0036 STORE 1 +FLAGS (\Seen \Deleted) A0037 EXPUNGE A0038 FETCH 1 RFC822.HEADER A0039 FETCH 1 BODY.PEEK[TEXT] A0040 STORE 1 +FLAGS (\Seen \Deleted) A0041 EXPUNGE A0042 FETCH 1 RFC822.HEADER A0043 FETCH 1 BODY.PEEK[TEXT] A0044 STORE 1 +FLAGS (\Seen \Deleted) A0045 EXPUNGE A0046 FETCH 1 RFC822.HEADER A0047 FETCH 1 BODY.PEEK[TEXT] A0048 STORE 1 +FLAGS (\Seen \Deleted) A0049 EXPUNGE A0050 FETCH 1 RFC822.HEADER A0051 LOGOUT
After this, mbox folder inbox-b is corrupted, as the line saying
From: abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz@example.com
becomes
rstuvwxyzabcdefghijklmnopqrstuvwxyz@example.com
So: Could we please stop blaming fetchmail for this? It's just the messenger. Could you also sprovide your "dovecot -n" output and any warnings and errors in dovecot logs. Ah, found the dovecot -n earlier in the thread, but the logs would still be relevant.
Teemu
Debug output:
May 6 11:23:42 qemu-sid dovecot: master: Dovecot v2.2.16 starting up for imap (core dumps disabled) May 6 11:24:06 qemu-sid dovecot: imap-login: Login: user=<bluser>, method=PLAIN, rip=192.168.122.8, lip=192.168.122.8, mpid=441, secured, session=<HtpSW2YVaQDAqHoI> May 6 11:24:06 qemu-sid dovecot: imap(bluser): Debug: Loading modules from directory: /usr/lib/dovecot/modules May 6 11:24:06 qemu-sid dovecot: imap(bluser): Debug: Module loaded: /usr/lib/dovecot/modules/lib15_notify_plugin.so May 6 11:24:06 qemu-sid dovecot: imap(bluser): Debug: Module loaded: /usr/lib/dovecot/modules/lib20_mail_log_plugin.so May 6 11:24:06 qemu-sid dovecot: imap(bluser): Debug: Effective uid=1000, gid=1000, home=/home/bluser May 6 11:24:06 qemu-sid dovecot: imap(bluser): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes location=mbox:~/mail:INBOX=/var/mail/bluser May 6 11:24:06 qemu-sid dovecot: imap(bluser): Debug: fs: root=/home/bluser/mail, index=, indexpvt=, control=, inbox=/var/mail/bluser, alt= May 6 11:24:06 qemu-sid dovecot: imap(bluser): delete: box=inbox-b, uid=15988, msgid=20150113080229.634B31FE0F@example.com, size=166 May 6 11:24:06 qemu-sid dovecot: imap(bluser): expunge: box=inbox-b, uid=15988, msgid=20150113080229.634B31FE0F@example.com, size=166 May 6 11:24:06 qemu-sid dovecot: imap(bluser): Error: Sync failed for mbox file /home/bluser/mail/inbox-b: seq=13 uid=15999 uid_broken=0 originally needed 0 bytes, now needs 23 bytes May 6 11:24:06 qemu-sid dovecot: imap(bluser): delete: box=inbox-b, uid=15989, msgid=20150113080252.84E723FB47@example.com, size=238 May 6 11:24:06 qemu-sid dovecot: imap(bluser): expunge: box=inbox-b, uid=15989, msgid=20150113080252.84E723FB47@example.com, size=238 May 6 11:24:06 qemu-sid dovecot: imap(bluser): delete: box=inbox-b, uid=15990, msgid=20150113080253.3E89D5F8B1@example.com, size=238 May 6 11:24:06 qemu-sid dovecot: imap(bluser): expunge: box=inbox-b, uid=15990, msgid=20150113080253.3E89D5F8B1@example.com, size=238 May 6 11:24:06 qemu-sid dovecot: imap(bluser): delete: box=inbox-b, uid=15991, msgid=20150113080256.702551FD75@example.com, size=238 May 6 11:24:06 qemu-sid dovecot: imap(bluser): expunge: box=inbox-b, uid=15991, msgid=20150113080256.702551FD75@example.com, size=238 May 6 11:24:06 qemu-sid dovecot: imap(bluser): delete: box=inbox-b, uid=15992, msgid=20150113081736.928595F8B1@example.com, size=238 May 6 11:24:06 qemu-sid dovecot: imap(bluser): expunge: box=inbox-b, uid=15992, msgid=20150113081736.928595F8B1@example.com, size=238 May 6 11:24:06 qemu-sid dovecot: imap(bluser): delete: box=inbox-b, uid=15993, msgid=20150113090223.890703FBC5@example.com, size=166 May 6 11:24:06 qemu-sid dovecot: imap(bluser): expunge: box=inbox-b, uid=15993, msgid=20150113090223.890703FBC5@example.com, size=166 May 6 11:24:06 qemu-sid dovecot: imap(bluser): delete: box=inbox-b, uid=15994, msgid=20150113090225.EA3BF1FDD0@example.com, size=166 May 6 11:24:06 qemu-sid dovecot: imap(bluser): expunge: box=inbox-b, uid=15994, msgid=20150113090225.EA3BF1FDD0@example.com, size=166 May 6 11:24:06 qemu-sid dovecot: imap(bluser): delete: box=inbox-b, uid=15995, msgid=20150113090235.947643FB11@example.com, size=238 May 6 11:24:06 qemu-sid dovecot: imap(bluser): expunge: box=inbox-b, uid=15995, msgid=20150113090235.947643FB11@example.com, size=238 May 6 11:24:06 qemu-sid dovecot: imap(bluser): delete: box=inbox-b, uid=15996, msgid=20150113090252.ACE3D5F8B1@example.com, size=238 May 6 11:24:06 qemu-sid dovecot: imap(bluser): expunge: box=inbox-b, uid=15996, msgid=20150113090252.ACE3D5F8B1@example.com, size=238 May 6 11:24:06 qemu-sid dovecot: imap(bluser): delete: box=inbox-b, uid=15997, msgid=20150113090252.F11061FDAE@example.com, size=238 May 6 11:24:06 qemu-sid dovecot: imap(bluser): expunge: box=inbox-b, uid=15997, msgid=20150113090252.F11061FDAE@example.com, size=238 May 6 11:24:06 qemu-sid dovecot: imap(bluser): delete: box=inbox-b, uid=15998, msgid=20150113091737.92CD33FC52@example.com, size=238 May 6 11:24:06 qemu-sid dovecot: imap(bluser): expunge: box=inbox-b, uid=15998, msgid=20150113091737.92CD33FC52@example.com, size=238 May 6 11:24:06 qemu-sid dovecot: imap(bluser): Disconnected: Logged out in=1364 out=6726
For reference, this is the "dovecot -n" output matching the previous debug output:
# 2.2.16: /etc/dovecot/dovecot.conf # OS: Linux 3.16.0-4-amd64 x86_64 Debian stretch/sid mail_debug = yes mail_location = mbox:~/mail:INBOX=/var/mail/%u mail_plugins = mail_log notify namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = pam } plugin { mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid box msgid size } protocols = " imap" ssl = no userdb { driver = passwd }
Thanks, fixed: http://hg.dovecot.org/dovecot-2.2/rev/94bd895721d8
On Thu, May 07, 2015 at 12:02:12AM +0300, Timo Sirainen wrote:
Thanks, fixed: http://hg.dovecot.org/dovecot-2.2/rev/94bd895721d8
Unfortunately, I applied single changeset 94bd895721d8 over Debian version 2.2.16-1 to create an updated Debian package and I can still reproduce the problem when using the updated package.
This is a little bit surprising considering that the patch applies cleanly over version 2.2.16.
What is the minimal set of changes from hg that I would need to apply over 2.2.16 for this to be fixed?
Or should we wait for 2.2.17?
I'm thinking about the way to fix the package in Debian unstable first, but after that it would be desirable to fix it in stable as well.
Thanks.
Am 07.05.2015 um 01:45 schrieb Santiago Vila:
On Thu, May 07, 2015 at 12:02:12AM +0300, Timo Sirainen wrote:
Thanks, fixed: http://hg.dovecot.org/dovecot-2.2/rev/94bd895721d8
Unfortunately, I applied single changeset 94bd895721d8 over Debian version 2.2.16-1 to create an updated Debian package and I can still reproduce the problem when using the updated package.
This is a little bit surprising considering that the patch applies cleanly over version 2.2.16.
What is the minimal set of changes from hg that I would need to apply over 2.2.16 for this to be fixed?
Or should we wait for 2.2.17?
I'm thinking about the way to fix the package in Debian unstable first, but after that it would be desirable to fix it in stable as well.
Thanks.
http://wiki2.dovecot.org/PrebuiltBinaries#Automatically_Built_Packages
... deb http://xi.rename-it.nl/debian/ testing-auto/dovecot-2.2 main
is up2date most of the time
Best Regards MfG Robert Schetterer
-- [*] sys4 AG
http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein
On Thu, May 07, 2015 at 05:20:31PM +0200, Robert Schetterer wrote:
http://wiki2.dovecot.org/PrebuiltBinaries#Automatically_Built_Packages
... deb http://xi.rename-it.nl/debian/ testing-auto/dovecot-2.2 main
Thanks a lot.
I installed dovecot version 2:2.2.17~rc1-1~auto+5 using that line. Changelog says:
- New revision (2054:c00f51f3c91a) in pigeonhole Mercurial repository:
- lib-sieve: storage: Fixed segfault problem in main storage initialization.
and later:
- New revision (18627:69630e6048fd) in dovecot Mercurial repository:
I can still reproduce the error, which means this was not fixed by changeset 18534:94bd895721d8 after all.
I wonder how you guys are testing for this bug. Please use something like this script to reproduce:
#!/bin/sh rm -rf $HOME/mail/.imap cp -f $HOME/etc/inbox-b $HOME/mail fetchmail -a qemu-jessie --folder inbox-b -m "true"
where $HOME/etc/inbox-b is the original test case I provided.
Thanks.
On Sun, Feb 15, 2015 at 10:55:13AM +0200, Timo Sirainen wrote:
On 14 Feb 2015, at 16:23, Santiago Vila sanvila@unex.es wrote:
I wrote about this three weeks ago but got no answer. I'm going to officially "forward" the Debian bug this time, with all the details.
The test case is just 840 bytes long. Please give it a try. .. Package: dovecot-imapd Version: 1:2.2.13-11 Severity: serious
I can't reproduce with latest Dovecot hg. But just in case it's still not fixed, there are two important things:
Would you please try with 2.2.15, which is the latest released version?
I have just tested version 2.2.15 in Debian experimental, and I can still reproduce the bug.
Thanks.
participants (6)
-
Charles Marcus
-
Reindl Harald
-
Robert Schetterer
-
Santiago Vila
-
Teemu Huovila
-
Timo Sirainen