May 2003 - dovecot - dovecot.org

[Dovecot] possible dovecot problem (fwd)
by Mark E. Mallett 23 May '03

23 May '03

I'm just forwarding this verbatim from somebody who works here. Comments ? mm ----- Forwarded message from Dave Chamberlain <davec(a)mv.mv.com> ----- > Subject: possible dovecot problem > From: "Dave Chamberlain" <davec(a)mv.mv.com> > Date: Tue, 20 May 2003 21:29:48 -0400 (EDT) > To: mem(a)mv.mv.com (Mark E. Mallett) > > Ok, I'm kind of confused at what dovecot's imap implementation is doing. > > I seem to be spending more time than I need to correctly decipher the headers of > email messages. > > I have included here the text from 7.4.2 as it explains the response from a > fetch of an ENVELOPE. The last paragraph specifies that the FROM, SENDER > and REPLYTO fields cannot be nil because some of them must be present per > RFC822 and the IMAP spec will default values. I have many examples where > this is not the case, where the fetch of the envelope is returning NIL for > some of these values. > > In the example below, the message shows that the from, replyto and sender > are all set to jaentc(a)aukware.com (ok, a spelling error, but that was her > typing it in wrong). The next field after those three is NIL, that is the > TO line (more of my text after the IMAP conversation) > > a uid fetch 902 ENVELOPE > * 113 EXISTS > * 1 RECENT > * 104 FETCH (UID 902 ENVELOPE ("Tue, 20 May 2003 16:05:04 -0500" "From A Friend: > 'TheWMURChannel.com - News - Law Raises Age Children Must Be In Seat Restraints'" > (("Janet" NIL "jaentc" "aukware.com")) > (("Janet" NIL "jaentc" "aukware.com")) > (("Janet" NIL "jaentc" "aukware.com")) NIL NIL NIL NIL > "<200305202105.h4KL5xo20860(a)qbert.ibsys.com>")) > > Following is the complete header from the message. There is a valid To address > supplied, and it is in one of the allowed formats. I would have thought that > possibly it was the format that was causing the problem. I first theorized that > it didn't understand the emailaddress (personalname) format. However, the > From field is in that format, and was correctly picked up (notice that "Janet" > exists above as the Personal name, so it did understand that format). > > I have found multple cases where the FROM was NIL, the SUBJECT was NIL and > the TO is NIL, yet when looking at the header they are visible. > > This is causing problems with more than just my things, OE is having a problem > with it as well. > > > > > >From jaentc(a)aukware.com Tue May 20 21:26:43 2003 UTC > Received: (qmail 24772 invoked from network); 20 May 2003 17:52:17 -0400 > Received: from iridium.mv.net (HELO mv.mv.com) (qmailr(a)199.125.85.17) > by mercury.mv.net with SMTP; 20 May 2003 17:52:17 -0400 > X-Peer-Info: remote-ip 199.125.85.17 local-ip 199.125.85.40 > local-name mercury.mv.net > Received: (qmail 21564 invoked by uid 134); 20 May 2003 17:26:53 -0400 > Delivered-To: davec(a)mv.com > Received: (qmail 3533 invoked from network); 20 May 2003 17:06:26 -0400 > Received: from qbert.ibsys.com (66.187.197.24) > by iridium.mv.net with SMTP; 20 May 2003 17:06:26 -0400 > X-Peer-Info: remote-ip 66.187.197.24 local-ip 199.125.85.17 > local-name iridium.mv.net > Received: from 10.0.0.24 (cfusion3.ibsys.com [10.3.0.53]) > by qbert.ibsys.com (Switch-2.1.0/Switch-2.1.0) with ESMTP id h4KL5xo20860 > for <davec(a)mv.com (Dave)> Tue, 20 May 2003 16:05:59 -0500 (CDT) > Message-Id: <200305202105.h4KL5xo20860(a)qbert.ibsys.com> > Content-type: text/plain > Date: Tue, 20 May 2003 16:05:04 -0500 > From: jaentc(a)aukware.com (Janet) > Subject: From A Friend: 'TheWMURChannel.com - News - Law Raises Age Children Must Be > In Seat Restraints' > To: davec(a)mv.com (Dave) > > > Section 7.4.2 > > ENVELOPE > A parenthesized list that describes the envelope structure of a > message. This is computed by the server by parsing the > [RFC-2822] header into the component parts, defaulting various > fields as necessary. > > The fields of the envelope structure are in the following > order: date, subject, from, sender, reply-to, to, cc, bcc, > in-reply-to, and message-id. The date, subject, in-reply-to, > and message-id fields are strings. The from, sender, reply-to, > to, cc, and bcc fields are parenthesized lists of address > structures. > > An address structure is a parenthesized list that describes an > electronic mail address. The fields of an address structure > are in the following order: personal name, [SMTP] > at-domain-list (source route), mailbox name, and host name. > > [RFC-2822] group syntax is indicated by a special form of > address structure in which the host name field is NIL. If the > mailbox name field is also NIL, this is an end of group marker > (semi-colon in RFC 822 syntax). If the mailbox name field is > non-NIL, this is a start of group marker, and the mailbox name > field holds the group name phrase. > > If the Date, Subject, In-Reply-To, and Message-ID header lines > are absent in the [RFC-2822] header, the corresponding member > of the envelope is NIL; if these header lines are present but > empty the corresponding member of the envelope is the empty > string. > > Note: some servers may return a NIL envelope member in the > "present but empty" case. Clients SHOULD treat NIL and > empty string as identical. > > Note: [RFC-2822] requires that all messages have a valid > Date header. Therefore, the date member in the envelope can > not be NIL or the empty string. > > Note: [RFC-2822] requires that the In-Reply-To and > Message-ID headers, if present, have non-empty content. > Therefore, the in-reply-to and message-id members in the > envelope can not be the empty string. > > If the From, To, cc, and bcc header lines are absent in the > [RFC-2822] header, or are present but empty, the corresponding > member of the envelope is NIL. > > If the Sender or Reply-To lines are absent in the [RFC-2822] > header, or are present but empty, the server sets the > corresponding member of the envelope to be the same value as > the from member (the client is not expected to know to do > this). > > Note: [RFC-2822] requires that all messages have a valid > From header. Therefore, the from, sender, and reply-to > members in the envelope can not be NIL. ----- End forwarded message -----

2 7

[Dovecot] Dovecot and SquirrelMail
by Ken Wilson 22 May '03

22 May '03

Hello, Does anyone know how to get dovecot working with squirrelmail v1.4? We have an existing uw-imap setup, with mail stored in ~/mail and inboxes in /var/spool/mail/user. I've tried setting $imap_server_type = other in squirrelmail, but I do not see any mail in my inbox. This is my setting in dovecot.conf default_mail_env = maildir:~/mail/:INBOX=/var/spool/mail/%n Any help would be grealty appreciated, as I want to leave uw-imap and get the speed increases that dovecot offers. Thanks Ken All e-mail from Wilberforce College is governed by our legal disclaimer which is available at http://webmail.wilberforce.ac.uk/disclaimer.txt

3 2

[Dovecot] ssl still not working
by Farkas Levente 21 May '03

21 May '03

hi, I'm just download the latest cvs and try to use imaps. in mozilla I've got the following message window: ----------------------------- mail.int.bppiac.hu received a message with incorrect Message Authentication Code. If the error occurs frequently, contact the website administrator. ----------------------------- and there is only one OK button:-) and this happens always. what can be the reason and what can I do? thanks. -- Levente "Si vis pacem para bellum!"

6 11

[Dovecot] SSL problems
by Timo Sirainen 21 May '03

21 May '03

Anyone with SSL problems, please try 0.99.10-test4 or included patch. If it fixes it, I'll make it faster by caching the generated keys. http://dovecot.procontrol.fi/test/

2 1

[Dovecot] dovecot with ssl on rhl9
by Elliot Peele 20 May '03

20 May '03

Has anyone gotten dovecot to work on Red Hat Linux 9 with ssl support? It seems to work just fine on Red Hat Linux 8.0, but doesn't seem to work Red Hat Linux 9. When I try to use openssl to connect to the server i get: [elliot@localhost elliot]$ openssl s_client -connect rktest.physics.ncsu.edu:993 CONNECTED(00000003) depth=0 /OU=IMAP server/CN=imap.example.com/emailAddress=postmaster@example.com verify error:num=18:self signed certificate verify return:1 depth=0 /OU=IMAP server/CN=imap.example.com/emailAddress=postmaster@example.com verify return:1 20949:error:140943FC:SSL routines:SSL3_READ_BYTES:sslv3 alert bad record mac:s3_pkt.c:1052:SSL alert number 20 20949:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:226: I'm trying to use evolution 1.3.3 to connect via imaps. Whenever I click on the mail folder it just says 'Scanning folders in "IMAP server imap.example.com" (100%)' and then nothing happens. Anyone have any ideas? Thanks Elliot

1 0

RE: [Dovecot] 0.99.10-test1
by Maikel Verheijen 19 May '03

19 May '03

> > Does this mean you made the .subscriptions nfs-safe already in this > > version? > Well, no, but that's not maildir-specific code :) Ah, ok, My mistake! :) Regards, Maikel.

1 0

[Dovecot] 0.99.10-test1
by Timo Sirainen 19 May '03

19 May '03

http://dovecot.procontrol.fi/test/ Maildir syncing now works pretty much as described in my earlier mail. It was quite a large change, so please test this :) So the new and great features: - It's faster. - If someone renames maildir file just before we try to use it, we now resync the maildir and try to find it again. Before we just failed. It's also possible that the filename changed again while we resynced, so we try max. 10 times before failing (yes, we know if we should try resyncing so we don't always scan it 10 times). - Read-only maildirs work now. If we don't have write access to dovecot-uidlist, we'll show only those mails which are already there. - Out of quota/disk space failures are handled well now. All except updating dovecot-uidlist file which is a bit tricky. - :CONTROL=dir in default_mail_env can specify where to save dovecot-uidlist and .customflags files. Useful for above problems. I think maildir support is getting near perfect now. Only thing left is to deal with failure to write dovecot-uidlist when out of quota. Oh, and NFS safe .customflags file.

2 1

RE: [Dovecot] 0.99.10-test1
by Maikel Verheijen 19 May '03

19 May '03

Hi Timo, > I think maildir support is getting near perfect now. Only > thing left is to deal with failure to write dovecot-uidlist > when out of quota. Oh, and NFS safe .customflags file. Does this mean you made the .subscriptions nfs-safe already in this version? Kind regards, Maikel Verheijen.

2 1

[Dovecot] Maildir synchronization
by Timo Sirainen 17 May '03

17 May '03

I thought I'd send this here too, since I spent so much time writing it as comment to maildir-sync.c :) Now only thing left to do is to write the code to match it. /* Here's a description of how we handle Maildir synchronization and it's problems: We want to be as efficient as we can. The most efficient way to check if changes have occured is to stat() the new/ and cur/ directories and uidlist file - if their mtimes hasn't changed, there's no changes and we don't need to do anything. Problem 1: Multiple changes can happen within a single second - nothing guarantees that once we synced it, someone else didn't just then make a modification. Such modifications wouldn't get noticed until a new modification occured later. Problem 2: Syncing cur/ directory is much more costly than syncing new/. Moving mails from new/ to cur/ will always change mtime of cur/ causing us to sync it as well. Problem 3: We may not be able to move mail from new/ to cur/ because we're out of quota, or simply because we're accessing a read-only mailbox. MAILDIR_SYNC_SECS ----------------- Several checks below use MAILDIR_SYNC_SECS, which should be maximum clock drift between all computers accessing the maildir (eg. via NFS), rounded up to next second. Our default is 1 second, since everyone should be using NTP. Note that setting it to 0 works only if there's only one computer accessing the maildir. It's practically impossible to make two clocks _exactly_ synchronized. cur directory ------------- We have maildir_cur_dirty variable which is set to cur/ directory's mtime when it's >= time() - MAILDIR_SYNC_SECS and we _think_ we have synchronized the directory. When maildir_cur_dirty is non-zero, we don't synchronize the cur/ directory until a) cur/'s mtime changes b) opening a mail fails with ENOENT c) time() > maildir_cur_dirty + MAILDIR_SYNC_SECS This allows us to modify the maildir multiple times without having to sync it at every change. The sync will eventually be done to make sure we didn't miss any external changes. The maildir_cur_dirty is set when: - we change message flags - we expunge messages - we move mail from new/ to cur/ - we sync cur/ directory and it's mtime is >= time() - MAILDIR_SYNC_SECS It's unset when we do the final syncing, ie. when mtime is older than time() - MAILDIR_SYNC_SECS. new directory ------------- If new/'s mtime is >= time() - MAILDIR_SYNC_SECS, always synchronize it. maildir_cur_dirty-like feature might save us a few syncs, but that might break a client which saves a mail in one connection and tries to fetch it in another one. new/ directory is almost always empty, so syncing it should be very fast anyway. Normally we move all mails from new/ to cur/ whenever we sync it. If it's not possible for some reason, we set maildir_have_new flag on which instructs synchronization to check files in new/ directory as well. maildir_keep_new flag is also set which instructs syncing to not even try to move mails to cur/ anymore. If client tries to change a flag for message in new/, we try to rename() it into cur/. If it's successful, we clear the maildir_keep_new flag so at next sync we'll try to move all of them to cur/. When all of them have been moved, maildir_have_new flag is cleared as well. Expunges will also clear maildir_keep_new flag. If rename() still fails because of ENOSPC or EDQUOT, we still save the flag changes in index with dirty-flag on. When moving the mail to cur/ directory, or when we notice it's already moved there, we apply the flag changes to the filename, rename it and remove the dirty flag. If there's dirty flags, this should be tried every time after expunge or when closing the mailbox. broken clients -------------- Originally the middle identifier in Maildir filename was specified only as <process id>_<delivery counter>. That however created a problem with randomized PIDs which made it possible that the same PID was reused within one second. So if within one second a mail was delivered, MUA moved it to cur/ and another mail was delivered by a new process using same PID as the first one, we likely ended up overwriting the first mail when the second mail was moved over it. Nowadays everyone should be giving a bit more specific identifier, for example include microseconds in it which Dovecot does. There's a simple way to prevent this from happening in some cases: Don't move the mail from new/ to cur/ if it's mtime is >= time() - MAILDIR_SYNC_SECS. The second delivery's link() call then fails because the file is already in new/, and it will then use a different filename. There's a few problems with this however: - it requires extra stat() call which is unneeded extra I/O - another MUA might still move the mail to cur/ - if first file's flags are modified by either Dovecot or another MUA, it's moved to cur/ (you _could_ just do the dirty-flagging but that'd be ugly) Because this is useful only for very few people and it requires extra I/O, I decided not to implement this. It should be however quite easy to do since we need to be able to deal with files in new/ in any case. It's also possible to never accidentally overwrite a mail by using link() + unlink() rather than rename(). This however isn't very good idea as it introduces potential race conditions when multiple clients are accessing the mailbox: Trying to move the same mail from new/ to cur/ at the same time: a) Client 1 uses slightly different filename than client 2, for example one sets read-flag on but the other doesn't. You have the same mail duplicated now. b) Client 3 sees the mail between Client 1's and 2's link() calls and changes it's flag. You have the same mail duplicated now. And it gets worse when they're unlink()ing in cur/ directory: c) Client 1 changes mails's flag and client 2 changes it back between 1's link() and unlink(). The mail is now expunged. d) If you try to deal with the duplicates by unlink()ing another one of them, you might end up unlinking both of them. So, what should we do then if we notice a duplicate? First of all, it might not be a duplicate at all, readdir() might have just returned it twice because it was just renamed. What we should do is create a completely new base name for it and rename() it to that. If the call fails with ENOENT, it only means that it wasn't a duplicate after all. */

1 0

[Dovecot] Postfix locking equivalence?
by Warren Togami 17 May '03

17 May '03

/etc/postfix/main.cf.default: mailbox_delivery_lock = flock, dotlock Is flock equivalent to dovecot's fnctl locking? For mbox what type of locking settings should I be using? (No NFS, only local filesystem.) I'm using procmail to deliver mail to my mboxes. Does the above postfix setting effect procmail's locking method? I use rules like this: :0 * ^X-BeenThere: dovecot(a)procontrol.fi DOVECOT If I understand this correctly, :0: in procmailrc would explicitly use dotlocks instead of "kernel-locks" as procmail calls it. I'm not fully understanding all this locking stuff. Warren Togami warren(a)togami.com p.s. Timo, I received your test message after my prior posting attempt bounced. I had this error message: Reporting-MTA: dns; danu.procontrol.fi Arrival-Date: Sat, 17 May 2003 12:52:08 +0300 (EEST) Final-Recipient: rfc822; dovecot(a)procontrol.fi Action: failed Status: 5.0.0 Diagnostic-Code: X-Postfix; mail forwarding loop for dovecot(a)procontrol.fi

2 1