[Dovecot] Maildir unreliability

[Miquel van Smoorenburg]

On 2004.10.25 02:02, Timo Sirainen wrote:
> Looks like maildir can't be used very realiably without quite a lot of 
> locking. Writing and scanning the directory would have to be locked, 
> but reading wouldn't (as long as the file hasn't been renamed which 
> would require scanning to find it). So much for "no locks needed"..
> 
> The problem is that opendir()/readdir() may temporarily not return some 
> files if there has been changes in the directory since the opendir(). 
> That means Dovecot thinks a message is expunged, while in fact it 
> really isn't, and the next scan would usually show it again.
> 
> Currently when that happens, Dovecot usually prints an error message 
> about it and rebuilds indexes. Of course, in real life clients aren't 
> often bombing the same mailbox with tons of changes in multiple 
> connections, which is usually needed to trigger this.
> 
> I wrote a test program which tests this:
> 
> http://dovecot.org/tmp/readdir.c

Well, there is a lockless way to do readdir, but it would mean
buffering the entire readdir output in memory.

	while(1) {
		/* Wait until directory is quiescent */
		while (1) {
			time(&t);
			/* chmod() invalidates the NFS attribute cache */
			fstat(dirhandle, &st);
			fchmod(dirhandle, st.st_mode);
			fstat(dirhandle, &st);
			if (st.st_mtime < t)
				break;
			usleep(100000);
		}
		t = st.st_mtime;

		free_list(&files);
		while (readdir(dirhandle)) {
			add_list(&files, ...);
			.. build list in memory ..
		}

		fstat(dirhandle, &st);
		fchmod(dirhandle, &st.st_mode);
		fstat(dirhandle, &st);
	} while (st.st_mtime > t);

(insert sleep() or usleep() where nessecary).

This basically just retries until you've done an entire readdir() on
the directory without mtime changes.

At least under Linux, chmod invalidates the NFS attribute cache. On normal
files, fcntl() locking should do the same, but I'm not sure you can
lock directories, and not all servers do NFS locking, so chmod is probably
the better choice. Testing would be needed, though.

Mike.