- dovecot - dovecot.org

[Dovecot] Maildir synchronization
by Timo Sirainen 17 May '03

17 May '03

I thought I'd send this here too, since I spent so much time writing it as comment to maildir-sync.c :) Now only thing left to do is to write the code to match it. /* Here's a description of how we handle Maildir synchronization and it's problems: We want to be as efficient as we can. The most efficient way to check if changes have occured is to stat() the new/ and cur/ directories and uidlist file - if their mtimes hasn't changed, there's no changes and we don't need to do anything. Problem 1: Multiple changes can happen within a single second - nothing guarantees that once we synced it, someone else didn't just then make a modification. Such modifications wouldn't get noticed until a new modification occured later. Problem 2: Syncing cur/ directory is much more costly than syncing new/. Moving mails from new/ to cur/ will always change mtime of cur/ causing us to sync it as well. Problem 3: We may not be able to move mail from new/ to cur/ because we're out of quota, or simply because we're accessing a read-only mailbox. MAILDIR_SYNC_SECS ----------------- Several checks below use MAILDIR_SYNC_SECS, which should be maximum clock drift between all computers accessing the maildir (eg. via NFS), rounded up to next second. Our default is 1 second, since everyone should be using NTP. Note that setting it to 0 works only if there's only one computer accessing the maildir. It's practically impossible to make two clocks _exactly_ synchronized. cur directory ------------- We have maildir_cur_dirty variable which is set to cur/ directory's mtime when it's >= time() - MAILDIR_SYNC_SECS and we _think_ we have synchronized the directory. When maildir_cur_dirty is non-zero, we don't synchronize the cur/ directory until a) cur/'s mtime changes b) opening a mail fails with ENOENT c) time() > maildir_cur_dirty + MAILDIR_SYNC_SECS This allows us to modify the maildir multiple times without having to sync it at every change. The sync will eventually be done to make sure we didn't miss any external changes. The maildir_cur_dirty is set when: - we change message flags - we expunge messages - we move mail from new/ to cur/ - we sync cur/ directory and it's mtime is >= time() - MAILDIR_SYNC_SECS It's unset when we do the final syncing, ie. when mtime is older than time() - MAILDIR_SYNC_SECS. new directory ------------- If new/'s mtime is >= time() - MAILDIR_SYNC_SECS, always synchronize it. maildir_cur_dirty-like feature might save us a few syncs, but that might break a client which saves a mail in one connection and tries to fetch it in another one. new/ directory is almost always empty, so syncing it should be very fast anyway. Normally we move all mails from new/ to cur/ whenever we sync it. If it's not possible for some reason, we set maildir_have_new flag on which instructs synchronization to check files in new/ directory as well. maildir_keep_new flag is also set which instructs syncing to not even try to move mails to cur/ anymore. If client tries to change a flag for message in new/, we try to rename() it into cur/. If it's successful, we clear the maildir_keep_new flag so at next sync we'll try to move all of them to cur/. When all of them have been moved, maildir_have_new flag is cleared as well. Expunges will also clear maildir_keep_new flag. If rename() still fails because of ENOSPC or EDQUOT, we still save the flag changes in index with dirty-flag on. When moving the mail to cur/ directory, or when we notice it's already moved there, we apply the flag changes to the filename, rename it and remove the dirty flag. If there's dirty flags, this should be tried every time after expunge or when closing the mailbox. broken clients -------------- Originally the middle identifier in Maildir filename was specified only as <process id>_<delivery counter>. That however created a problem with randomized PIDs which made it possible that the same PID was reused within one second. So if within one second a mail was delivered, MUA moved it to cur/ and another mail was delivered by a new process using same PID as the first one, we likely ended up overwriting the first mail when the second mail was moved over it. Nowadays everyone should be giving a bit more specific identifier, for example include microseconds in it which Dovecot does. There's a simple way to prevent this from happening in some cases: Don't move the mail from new/ to cur/ if it's mtime is >= time() - MAILDIR_SYNC_SECS. The second delivery's link() call then fails because the file is already in new/, and it will then use a different filename. There's a few problems with this however: - it requires extra stat() call which is unneeded extra I/O - another MUA might still move the mail to cur/ - if first file's flags are modified by either Dovecot or another MUA, it's moved to cur/ (you _could_ just do the dirty-flagging but that'd be ugly) Because this is useful only for very few people and it requires extra I/O, I decided not to implement this. It should be however quite easy to do since we need to be able to deal with files in new/ in any case. It's also possible to never accidentally overwrite a mail by using link() + unlink() rather than rename(). This however isn't very good idea as it introduces potential race conditions when multiple clients are accessing the mailbox: Trying to move the same mail from new/ to cur/ at the same time: a) Client 1 uses slightly different filename than client 2, for example one sets read-flag on but the other doesn't. You have the same mail duplicated now. b) Client 3 sees the mail between Client 1's and 2's link() calls and changes it's flag. You have the same mail duplicated now. And it gets worse when they're unlink()ing in cur/ directory: c) Client 1 changes mails's flag and client 2 changes it back between 1's link() and unlink(). The mail is now expunged. d) If you try to deal with the duplicates by unlink()ing another one of them, you might end up unlinking both of them. So, what should we do then if we notice a duplicate? First of all, it might not be a duplicate at all, readdir() might have just returned it twice because it was just renamed. What we should do is create a completely new base name for it and rename() it to that. If the call fails with ENOENT, it only means that it wasn't a duplicate after all. */

1 0

[Dovecot] Postfix locking equivalence?
by Warren Togami 17 May '03

17 May '03

/etc/postfix/main.cf.default: mailbox_delivery_lock = flock, dotlock Is flock equivalent to dovecot's fnctl locking? For mbox what type of locking settings should I be using? (No NFS, only local filesystem.) I'm using procmail to deliver mail to my mboxes. Does the above postfix setting effect procmail's locking method? I use rules like this: :0 * ^X-BeenThere: dovecot(a)procontrol.fi DOVECOT If I understand this correctly, :0: in procmailrc would explicitly use dotlocks instead of "kernel-locks" as procmail calls it. I'm not fully understanding all this locking stuff. Warren Togami warren(a)togami.com p.s. Timo, I received your test message after my prior posting attempt bounced. I had this error message: Reporting-MTA: dns; danu.procontrol.fi Arrival-Date: Sat, 17 May 2003 12:52:08 +0300 (EEST) Final-Recipient: rfc822; dovecot(a)procontrol.fi Action: failed Status: 5.0.0 Diagnostic-Code: X-Postfix; mail forwarding loop for dovecot(a)procontrol.fi

2 1

[Dovecot] test
by Timo Sirainen 17 May '03

17 May '03

..is this thing working.. I messed up a bit when adding spamassassin to postfix filter.

1 0

[Dovecot] Postfix locking equivalence?
by Warren Togami 17 May '03

17 May '03

/etc/postfix/main.cf.default: mailbox_delivery_lock = flock, dotlock Is flock equivalent to dovecot's fnctl locking? For mbox what type of locking settings should I be using? (No NFS, only local filesystem.) Warren Togami warren(a)togami.com

1 0

[Fwd: Re: [Dovecot] help: The current command did not succeed. The mail server responded: Invalid mask.]
by Timo Sirainen 16 May '03

16 May '03

Just FYI, I guess this message was meant here :)

1 0

[Dovecot] rawlog fails
by Mark E. Mallett 16 May '03

16 May '03

You mentioned --with-rawlog earlier, so I tried it. It fails with a select error. I believe it's because in some cases the timeval struct is not initialized in io_loop_get_wait_time (e.g. see sample diff below). -mm- diff -c src/lib/ioloop.c.orig src/lib/ioloop.c *** src/lib/ioloop.c.orig Sat Apr 12 11:00:10 2003 --- src/lib/ioloop.c Thu May 15 22:10:12 2003 *************** *** 223,230 **** --- 223,238 ---- int io_loop_get_wait_time(struct timeout *timeout, struct timeval *tv, struct timeval *tv_now) { + #if 0 /* mem 20030515 make sure tv contains valid data here */ if (timeout == NULL) return INT_MAX; + #else + if (timeout == NULL) { + tv->tv_sec = INT_MAX / 1000; + tv->tv_usec = 0; + return INT_MAX; + } + #endif if (tv_now == NULL) { if (gettimeofday(tv, NULL) < 0)

2 1

[Dovecot] help: The current command did not succeed. The mail server responded: Invalid mask.
by Brandon Hoult 15 May '03

15 May '03

Am tryin to move from uw-imapd to dovecot, using dovecot ver 0.99.9.1-1 on debian. Using Mozilla Mail v 1.3 currently using mbox format. I made some test accounts and they seem to be working fine, but my older existing accounts give error messages whenever I connect or subscribe to mailboxes within them. The message that appears on the client is: "The current command did not succeed. The mail server responded: Invalid mask." it appears three times in an alert box (on windows) then everything seems to work. There are no messages in any of the mail logs in /var/log which would indicate a problem. The test accounts I created don't have this problem so I tried renameing the home directory, deleting the user, then recreating the user. However even with a new homedir the problem remains unchanged. I have verified that the permissions in ~/mail and /var/mail are set identically for working and non-working users. Thanks in advance for any advice, Brandon Hoult dolt(a)lmistudios.homeip.net

2 1

[Dovecot] Creating and renaming mbox folders on NFS auto-installed partitions
by Robert Evans 14 May '03

14 May '03

I'm trying dovecot-0.99.9.1 in place of uw-imap which ground to a halt on a 500MHz SunBlade 100 Solaris 8 system which acts as our Departmental mail server. INBOXes are mbox files on the servers's local SCSI disk. But user directories are auto mounted from other Sun NFS servers and these may contain other mbox mailboxes in a ~/Mail directory. There's a problem with creating (and with renaming) mailboxes in the NFS mounted ~/Mail directory. For an arbitrary mailbox "newbox", say, dovecot tries to create the parent directory path /home/username/Mail before creating the newbox file. But /home/username is automounted and so creating directory /home or /home/username fails with errno 89 ENOSYS - unsupported file system operation. The solution is to check for the existence of the directory before trying to create it. This will work since /home and /home/username will always exist for any valid user. I've amended mbox-storage.c in src/lib-storage/index/mbox to call "access" to do this.... *** src/lib-storage/index/mbox/mbox-storage.c.orig Tue Apr 15 14:04:02 2003 --- src/lib-storage/index/mbox/mbox-storage.c Wed May 14 16:04:39 2003 *************** *** 38,44 **** p++; } ! if (mkdir(dir, CREATE_MODE) < 0 && errno != EEXIST) { t_pop(); return -1; } --- 38,44 ---- p++; } ! if ( (access(dir,F_OK) != 0) && mkdir(dir, CREATE_MODE) < 0 && errno != EEXIST) { t_pop(); return -1; } - - - (Could also remove the && errno == EEXIST part - it's redundant when access(dir,F_OK) is checking for existence). Robert -- Robert.Evans(a)cs.cf.ac.uk Tel:+44(0)29 2087 5518 Fax:+44(0)29 2087 4598 http://www.cs.cf.ac.uk/department/staff/Robert.Evans.shtml Computer Science, Cardiff University, PO Box 916, Cardiff, CF24 3XF, UK

2 1

Re: [dovecot] Re: List-id test
by Moe Wibble 13 May '03

13 May '03

On Mon, May 12, 2003 at 06:44:06AM +0300, Timo Sirainen wrote: > On Mon, 2003-05-12 at 04:21, Timo Sirainen wrote: > > Just enabled List-* headers, testing if this still works.. > > Hmm. Wondered about why List-Id header used wrong host name and started > looking at Ecartis' sources .. Wasn't it written in Perl? I had believed > so for quite a long time for some reason. So since it's C, I figured I'd > audit the code for security flaws. > > .. I think I'll switch switch the list manager. Mailman looks nice. tried ezmlm? :)

2 2

Re: [dovecot] Re: Design thought - Retention of reworked mbox files
by Tim Hunt 13 May '03

13 May '03

It's primarily for debugging, and I don't see it being used in a high volume environment. What triggered the thought is that I suspect I may have lost the bodies of a couple of messages during an expunge, but because I don't have the previous state of the mbox file, I have no way of verifying my concern, and certainly no way to try and recreate it. And yes, I'm using mbox files, because I have many years of mail stored in them, and I'm not quite ready to convert them to another format. Tim On Mon, 2003-05-12 at 11:01, Charlie Brady wrote: > On 12 May 2003, Timo Sirainen wrote: > > > On Mon, 2003-05-12 at 02:05, Tim Hunt wrote: > > > Would it be possible to keep a backup copy of an mbox mail file any time it > > > needs to be substantionally rebuilt - for an example before an expunge. This > > > behaviour could be controlled by the config file, since it would most likely > > > only be used in development and testing environments. > > > > If it's only for debugging, I guess I could then just save a copy of it. > > Presumably the other reason you'd want it would be for mailbox integrity. > But if you are concerned about that, you'd be using maildir, wouldn't you? > > -- > Charlie

1 0