[Dovecot] Still weird UID ordering issues with maildir

[Todd Vierling]

On Sun, 11 Sep 2005, Timo Sirainen wrote:

> > Sep  1 09:25:02 server dovecot: imap-login: Login: user=<tv>, method=PLAIN, rip=192.168.1.1, lip=192.168.1.3, TLS
> > Sep  1 09:25:02 server dovecot: IMAP(tv): Maildir /home/tv/.maildir sync: UID < next_uid (1187 < 1188, file = 1125581033.7788_2.server.duh.org:2,)
> >
> > repeated every time I try to login.  I had to nuke the dovecot indexes and
> > let it reindex (which throws off the ordering of some messages).  It had
> > been working without incident since updating to 20050829, until now.
>
> Actually deleting indexes shouldn't matter at all, since Dovecot does
> that by itself after writing that error message.

No it doesn't, at least not in nightly 20050829.  The imap process exits and
I have to delete them manually; they're still sitting around.  An artifact
of this is the fact that if I don't delete the indexes manually before
reopening the mailbox, the error happens again with exactly the same UIDs:

Sep  6 15:49:01 server dovecot: IMAP(tv): Maildir /home/tv/.maildir/.groups.netbsd sync: UID < next_uid (556 < 557, file = 1126036138.8618_1.server.duh.org:2,)
Sep  6 15:49:09 server dovecot: IMAP(tv): Maildir /home/tv/.maildir/.groups.netbsd sync: UID < next_uid (556 < 557, file = 1126036138.8618_1.server.duh.org:2,)
Sep  6 16:43:32 server dovecot: IMAP(tv): Maildir /home/tv/.maildir/.groups.netbsd sync: UID < next_uid (556 < 557, file = 1126036138.8618_1.server.duh.org:2,)

> But you're still using it with NFS?

No.  It's ffs on NetBSD.

> The error message means that Dovecot saw a message 1187 that wasn't in
> index, but index said it already had seen message 1188. Probably means
> that the file was temporarily lost in some sync, but later seen again in
> another sync.. Dovecot isn't very forgiving if readdir() skips files.

I don't see how skipping files like that should happen -- HOWEVER, you do
realize that readdir(3) has no guarantee of ordering, right?  (Meaning that
two successive calls to readdir(3) can, per POSIX, return the same list of
files in a completely different order.)

If you are relying on readdir(3) ordering, then this could also explain an
oddness I have seen and dismissed as just a quirk of dovecot:  After
deleting indexes, messages sorted by "arrival" order seem to be partially
reordered in the mailbox.  That could be an artifact of ffs "soft updates"
functionality, which can reorder inode pointers in a directory file in order
to provide its metadata consistency guarantees.

If you want to guarantee ordering, the index must be responsible for that,
or else you need to readdir(3) the whole thing, stat the files, and qsort
them against st_ctime/st_ctimespec.  Any other use of readdir(3) *must not*
rely on its ordering.

> Are there other maildir clients than Dovecot updating the maildir? That
> can cause those problems.

The only other mailbox touching process is procmail, which is putting mail
into the folders.  However, this happens even when procmail is not
*actively* putting mail into the folder (it could be that mail had been
added between the last time the mailbox was closed, and when it was now
reopened).

But isn't collision-free access supposed to be a feature of using maildir --
even if I were using nfs?  :)

I've had a couple days now with no problems, and then it happened again this
morning:

Sep  9 11:25:16 server dovecot: IMAP(tv): Maildir /home/tv/.maildir/.groups.netbsd sync: UID < next_uid (633 < 634, file = 1126279502.23902_1.server.duh.org:2,)
Sep 11 10:46:22 server dovecot: IMAP(tv): Maildir /home/tv/.maildir sync: UID < next_uid (1579 < 1580, file = 1126449940.5309_2.server.duh.org:2,)

-- 
-- Todd Vierling <tv at duh.org> <tv at pobox.com> <todd at vierling.name>