[Dovecot] possible dovecot problem (fwd)
I'm just forwarding this verbatim from somebody who works here. Comments ?
mm
----- Forwarded message from Dave Chamberlain <davec@mv.mv.com> -----
Subject: possible dovecot problem From: "Dave Chamberlain" <davec@mv.mv.com> Date: Tue, 20 May 2003 21:29:48 -0400 (EDT) To: mem@mv.mv.com (Mark E. Mallett)
Ok, I'm kind of confused at what dovecot's imap implementation is doing.
I seem to be spending more time than I need to correctly decipher the headers of email messages.
I have included here the text from 7.4.2 as it explains the response from a fetch of an ENVELOPE. The last paragraph specifies that the FROM, SENDER and REPLYTO fields cannot be nil because some of them must be present per RFC822 and the IMAP spec will default values. I have many examples where this is not the case, where the fetch of the envelope is returning NIL for some of these values.
In the example below, the message shows that the from, replyto and sender are all set to jaentc@aukware.com (ok, a spelling error, but that was her typing it in wrong). The next field after those three is NIL, that is the TO line (more of my text after the IMAP conversation)
a uid fetch 902 ENVELOPE
- 113 EXISTS
- 1 RECENT
- 104 FETCH (UID 902 ENVELOPE ("Tue, 20 May 2003 16:05:04 -0500" "From A Friend: 'TheWMURChannel.com - News - Law Raises Age Children Must Be In Seat Restraints'" (("Janet" NIL "jaentc" "aukware.com")) (("Janet" NIL "jaentc" "aukware.com")) (("Janet" NIL "jaentc" "aukware.com")) NIL NIL NIL NIL "<200305202105.h4KL5xo20860@qbert.ibsys.com>"))
Following is the complete header from the message. There is a valid To address supplied, and it is in one of the allowed formats. I would have thought that possibly it was the format that was causing the problem. I first theorized that it didn't understand the emailaddress (personalname) format. However, the From field is in that format, and was correctly picked up (notice that "Janet" exists above as the Personal name, so it did understand that format).
I have found multple cases where the FROM was NIL, the SUBJECT was NIL and the TO is NIL, yet when looking at the header they are visible.
This is causing problems with more than just my things, OE is having a problem with it as well.
From jaentc@aukware.com Tue May 20 21:26:43 2003 UTC Received: (qmail 24772 invoked from network); 20 May 2003 17:52:17 -0400 Received: from iridium.mv.net (HELO mv.mv.com) (qmailr@199.125.85.17) by mercury.mv.net with SMTP; 20 May 2003 17:52:17 -0400 X-Peer-Info: remote-ip 199.125.85.17 local-ip 199.125.85.40 local-name mercury.mv.net Received: (qmail 21564 invoked by uid 134); 20 May 2003 17:26:53 -0400 Delivered-To: davec@mv.com Received: (qmail 3533 invoked from network); 20 May 2003 17:06:26 -0400 Received: from qbert.ibsys.com (66.187.197.24) by iridium.mv.net with SMTP; 20 May 2003 17:06:26 -0400 X-Peer-Info: remote-ip 66.187.197.24 local-ip 199.125.85.17 local-name iridium.mv.net Received: from 10.0.0.24 (cfusion3.ibsys.com [10.3.0.53]) by qbert.ibsys.com (Switch-2.1.0/Switch-2.1.0) with ESMTP id h4KL5xo20860 for <davec@mv.com (Dave)> Tue, 20 May 2003 16:05:59 -0500 (CDT) Message-Id: <200305202105.h4KL5xo20860@qbert.ibsys.com> Content-type: text/plain Date: Tue, 20 May 2003 16:05:04 -0500 From: jaentc@aukware.com (Janet) Subject: From A Friend: 'TheWMURChannel.com - News - Law Raises Age Children Must Be In Seat Restraints' To: davec@mv.com (Dave)
Section 7.4.2
ENVELOPE A parenthesized list that describes the envelope structure of a message. This is computed by the server by parsing the [RFC-2822] header into the component parts, defaulting various fields as necessary. The fields of the envelope structure are in the following order: date, subject, from, sender, reply-to, to, cc, bcc, in-reply-to, and message-id. The date, subject, in-reply-to, and message-id fields are strings. The from, sender, reply-to, to, cc, and bcc fields are parenthesized lists of address structures. An address structure is a parenthesized list that describes an electronic mail address. The fields of an address structure are in the following order: personal name, [SMTP] at-domain-list (source route), mailbox name, and host name. [RFC-2822] group syntax is indicated by a special form of address structure in which the host name field is NIL. If the mailbox name field is also NIL, this is an end of group marker (semi-colon in RFC 822 syntax). If the mailbox name field is non-NIL, this is a start of group marker, and the mailbox name field holds the group name phrase. If the Date, Subject, In-Reply-To, and Message-ID header lines are absent in the [RFC-2822] header, the corresponding member of the envelope is NIL; if these header lines are present but empty the corresponding member of the envelope is the empty string. Note: some servers may return a NIL envelope member in the "present but empty" case. Clients SHOULD treat NIL and empty string as identical. Note: [RFC-2822] requires that all messages have a valid Date header. Therefore, the date member in the envelope can not be NIL or the empty string. Note: [RFC-2822] requires that the In-Reply-To and Message-ID headers, if present, have non-empty content. Therefore, the in-reply-to and message-id members in the envelope can not be the empty string. If the From, To, cc, and bcc header lines are absent in the [RFC-2822] header, or are present but empty, the corresponding member of the envelope is NIL. If the Sender or Reply-To lines are absent in the [RFC-2822] header, or are present but empty, the server sets the corresponding member of the envelope to be the same value as the from member (the client is not expected to know to do this). Note: [RFC-2822] requires that all messages have a valid From header. Therefore, the from, sender, and reply-to members in the envelope can not be NIL.
----- End forwarded message -----
On Thu, 2003-05-22 at 18:24, Mark E. Mallett wrote:
I'm just forwarding this verbatim from somebody who works here. Comments ?
I can't reproduce this. I tested with the headers in your mail and it gave correct result. I also compared Dovecot's and uw-imap's output for mailbox with 4600 mails, and output was pretty much the same.
Maybe Dovecot is parsing the headers wrong in some conditions. Can you reproduce this in some way? The invalid envelope headers are probably cached in index file, deleting that should probably fix this for at least the old messages?
On Fri, May 23, 2003 at 12:12:00AM +0300, Timo Sirainen wrote:
On Thu, 2003-05-22 at 18:24, Mark E. Mallett wrote:
I'm just forwarding this verbatim from somebody who works here. Comments ?
I can't reproduce this. I tested with the headers in your mail and it gave correct result. I also compared Dovecot's and uw-imap's output for mailbox with 4600 mails, and output was pretty much the same.
Maybe Dovecot is parsing the headers wrong in some conditions. Can you reproduce this in some way? The invalid envelope headers are probably cached in index file, deleting that should probably fix this for at least the old messages?
Deleting the index files was the first thing I tried.
Strange.
mm
On Fri, May 23, 2003 at 12:12:00AM +0300, Timo Sirainen wrote:
On Thu, 2003-05-22 at 18:24, Mark E. Mallett wrote:
I'm just forwarding this verbatim from somebody who works here. Comments ?
I can't reproduce this. I tested with the headers in your mail and it gave correct result. I also compared Dovecot's and uw-imap's output for mailbox with 4600 mails, and output was pretty much the same.
Maybe Dovecot is parsing the headers wrong in some conditions. Can you reproduce this in some way? The invalid envelope headers are probably cached in index file, deleting that should probably fix this for at least the old messages?
I was able to reproduce it without any effort.
First let me ask the obvious thing.. when you filed the sample message from my original note, did you spot the fact that the "Subject" line was split into two lines? You have to recombine the two lines into one. (Sorry about that). Also, remove the "> " characters and, I guess, save the "From " line at the beginning (although I just tested it and it doesn't matter if it's there or not).
Here's what I did to reproduce it:
On the target system:
% rm -rf Maildir % maildirmake Maildir % cp bad.tmp Maildir/cur/0000.1111.2222.bad
"bad.tmp" is a copy of that message that I forwarded, starting with the "From" line, or with the first Received line, doesn't matter. Body of the message doesn't matter either; I kept all the RFC stuff as the body.
Now from the source system (and here is a cut and paste of what I just did, except that I changed the username/password):
% telnet target-system imap Connected to target-system Escape character is '^]'.
- OK dovecot ready. 1 login user password 1 OK Logged in. 2 select "INBOX"
- FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
- OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted.
- 1 EXISTS
- 0 RECENT
- OK [UNSEEN 1] First unseen.
- OK [UIDVALIDITY 1053640171] UIDs valid
- OK [UIDNEXT 2] Predicted next UID 2 OK [READ-WRITE] Select completed. 3 fetch 1 ENVELOPE
- 1 FETCH (ENVELOPE ("Tue, 20 May 2003 16:05:04 -0500" "From A Friend: 'TheWMURCh annel.com - News - Law Raises Age Children Must Be In Seat Restraints'" (("Janet " NIL "jaentc" "aukware.com")) (("Janet" NIL "jaentc" "aukware.com")) (("Janet" N IL "jaentc" "aukware.com")) NIL NIL NIL NIL "<200305202105.h4KL5xo20860@qbert.ibs ys.com>")) 3 OK Fetch completed. 4 logout
- BYE Logging out 4 OK Logout completed. Connection closed by foreign host.
This seems to exactly duplicate the report..
Note: the FETCH response above is wrapped.
Is there something I have got set wrong?
mm
On Fri, 2003-05-23 at 01:06, Mark E. Mallett wrote:
Maybe Dovecot is parsing the headers wrong in some conditions. Can you reproduce this in some way? The invalid envelope headers are probably cached in index file, deleting that should probably fix this for at least the old messages?
I was able to reproduce it without any effort.
Ah, yes. The "end of header" was treated as duplicate of last header except with empty value. Hmm. Maybe I should also handle differently the cases when same header occurs multiple times.. Or maybe not, RFC2822 doesn't allow them more than once anyway.. Index: message-parser.c =================================================================== RCS file: /home/cvs/dovecot/src/lib-mail/message-parser.c,v retrieving revision 1.39 diff -u -r1.39 message-parser.c --- message-parser.c 3 May 2003 17:28:20 -0000 1.39 +++ message-parser.c 22 May 2003 23:00:04 -0000 @@ -680,7 +680,8 @@ if (size == 0 || (size == 1 && msg[0] == '\r')) { /* end of headers */ line->eoh = TRUE; - line->name_len = line->value_len = 0; + line->name_len = line->value_len = line->full_value_len = 0; + line->name = ""; line->value = line->full_value = NULL; } else if (line->continued) { line->value = msg; line->value_len = size;
On Fri, May 23, 2003 at 02:03:32AM +0300, Timo Sirainen wrote:
On Fri, 2003-05-23 at 01:06, Mark E. Mallett wrote:
Maybe Dovecot is parsing the headers wrong in some conditions. Can you reproduce this in some way? The invalid envelope headers are probably cached in index file, deleting that should probably fix this for at least the old messages?
I was able to reproduce it without any effort.
Ah, yes. The "end of header" was treated as duplicate of last header except with empty value.
Hmm. Maybe I should also handle differently the cases when same header occurs multiple times.. Or maybe not, RFC2822 doesn't allow them more than once anyway..
I don't follow- what duplicates? It looks like a perfectly ordinary header to me, what am I missing?
mm
On Fri, 2003-05-23 at 04:30, Mark E. Mallett wrote:
Ah, yes. The "end of header" was treated as duplicate of last header except with empty value. I don't follow- what duplicates? It looks like a perfectly ordinary header to me, what am I missing?
Not real duplicate. But Dovecot was treating like the last header always appeared twice, but second time without value. Like:
.. headers .. To: foo@bar To:
.. body ..
On Fri, May 23, 2003 at 12:19:56PM +0300, Timo Sirainen wrote:
On Fri, 2003-05-23 at 04:30, Mark E. Mallett wrote:
Ah, yes. The "end of header" was treated as duplicate of last header except with empty value. I don't follow- what duplicates? It looks like a perfectly ordinary header to me, what am I missing?
Not real duplicate. But Dovecot was treating like the last header always appeared twice, but second time without value. Like:
.. headers .. To: foo@bar To:
.. body ..
Oh, I see what you mean. So if it happened that "To" came last, there was a problem- but if something innocuous was last, there was not?
Interesting. The patch does appear to fix the reported issue.
BTW when I first applied the patch, the imap server hung trying to fetch the body. Cleared the .imap* files and this did not happen again.
mm
participants (2)
-
Mark E. Mallett
-
Timo Sirainen