[Dovecot] Maildir unreliability

[Geo Carncross]

This is nonsense. The problem is that the behavior of readdir() is
confusing.

Why should unlink() or rename() invalidate data that your C library
ALREADY READ from the directory?

This is like saying "I fgetc()'d a byte, but now lseek() shows that my
offset is 1024!" - it's silly.

Just put a:
		if (stat(d->d_name, &sb) == -1)continue;

After your check for the "." in the first character of the d->d_name
(about line 41) and all will be good. No amount of twiddling with
USE_UNLINK or FILES is going to affect it.


So you say, "I need to stat() each entry? That's going to create a large
number of syscalls!"

Of course. For readdir() to be atomic, it would need to do a system call
for each directory entry. This is exactly why readdir() doesn't, so that
you do one syscall for every (say) 50 entries, and if you want validity,
you'll do a stat() yourself.



Now: Maildir quite obviously wasn't designed with IMAP in mind. IMAP has
some (largely ridiculous) requirements that Maildir simply doesn't make
easy.

The largest problem (with Maildir) is this renaming of file identifiers
and moving things in and out of cur/. It's only necessary so programs
don't have to open() in order to read flags (after all, they JUST did a
readdir())...

Since the names aren't going to change in cur/, you can get away with
just doing a stat() in there [[ after all, you just rename()'d it into
cur/ if you're working on new ]]

Unfortunately, cur/ is often bigger than new/.


On Sun, 2004-10-24 at 20:02, Timo Sirainen wrote:
> Looks like maildir can't be used very realiably without quite a lot of 
> locking. Writing and scanning the directory would have to be locked, 
> but reading wouldn't (as long as the file hasn't been renamed which 
> would require scanning to find it). So much for "no locks needed"..
> 
> The problem is that opendir()/readdir() may temporarily not return some 
> files if there has been changes in the directory since the opendir(). 
> That means Dovecot thinks a message is expunged, while in fact it 
> really isn't, and the next scan would usually show it again.
> 
> Currently when that happens, Dovecot usually prints an error message 
> about it and rebuilds indexes. Of course, in real life clients aren't 
> often bombing the same mailbox with tons of changes in multiple 
> connections, which is usually needed to trigger this.
> 
> I wrote a test program which tests this:
> 
> http://dovecot.org/tmp/readdir.c
> 
> I'd like to hear if you can run it in some system without errors. I 
> tested Linux 2.4 and 2.6 with ext2, ext3, xfs and reiser3, Solaris 
> 8/ufs and OpenBSD 3.5/sparc64. Only OpenBSD passed the test, but I'm 
> not sure if it's only because the computer was so slow and didn't 
> switch between processes hard enough. I'd be especially interested 
> about FreeBSD and various NFS systems.
> 
> If it actually works properly in some systems, I guess I'll make the 
> extra locking configurable.
-- 
Geo Carncross <geocar at internetconnection.net>
Internet Connection Reliable Web Hosting
http://www.internetconnection.net/