[Dovecot] Bug in 0.99.14 mbox handling - mail UIDs are renumbered without bumping UIDVALIDITY

[David McBride]

Package: dovecot-imapd
Version: 0.99.14-1
Severity: important

Hi,

I think I've identified a bug in Dovecot 0.99.14 as packaged and
distributed by Debian Sarge.  I believe the bug lies in the original
Dovecot source, not with any modifications made by Debian.

Specifically, this bug deals with Dovecot's handling of message unique
identifiers (UIDs) for IMAP clients when using the mbox mailbox storage
format. 

Some background for those not familiar with the details; (I had to look
a lot of this stuff up;  see also http://www.ietf.org/rfc/rfc2060.txt):

--- 8< --------------------------------------------------------------

All IMAP servers are required to maintain unique identifiers for each
email in an IMAP mailbox that are persistent between sessions.  Dovecot
stores the unique identifiers for each email in an mbox in an X-UID
header.

Clients which maintain a local cache of already-retrieved emails use an
email's UID as a key; thus, if the UID of a particular email were to
change on a server then the client's cached state would be invalid.

To cope with the case that the UIDs need to be invalidated, whether
because the original store has been lost or for some other reason, there
is also the UIDVALIDITY number.  This 32bit integer must be increased to
some higher value to indicate that any previously cached email ids are
now invalid.  

In Dovecot-managed mbox files, this number is stored in the X-IMAPbase
header in the first message of that mbox and is typically the
uid-invalidation time expressed in unix timestamp format (ie the number
of seconds elapsed since the epoch.)

--- 8< --------------------------------------------------------------

Okay, the actual bug: Dovecot routinely (and apparently unneccessarily)
renumbers all of the UIDs in our mbox-backed mailboxes without updating
the UIDVALIDITY count.  This results in caching mail clients (such as
Evolution and Thunderbird) presenting the incorrect email to the user
when browsing their mail.

Configuration:

We are using Dovecot as a standalone daemon (ie without using inetd.) We
have configured the default mail environment with the following string:

default_mail_env = mbox:%h/Mail/:INBOX=%h/.email:INDEX=MEMORY

ie each end-user's primary INBOX is an mbox, ~/.email, and other IMAP
folders are also stored as mboxes in the directory ~/Mail/.  Indexes are
maintained in memory but are not stored on disk.

I don't think any of the rest of the configuration is relavant, but I'm
happy to detail anything else about the setup if needed.

Reproducing the bug:

I'm afraid I don't have a proper protocol dump demonstrating this bug,
so I'm just going to describe all of the end-user steps to recreate the
problem:

1) Connect to the Dovecot IMAP service with a caching mail client, such
as Evolution or Thunderbird.  Download a local copy of the IMAP server
contents of a test mbox-backed mailbox to the local cache.

For example, I have a test mbox with old root mail.  Messages are
numbered from 1 to 163.  The most recent message has an X-UID: field of
163.

2) Delete one of the messages in the middle of the mailbox.  

For example, mark the second-oldest message in the mailbox, which has a
UID of 2, for deletion.  Expunge the mailbox.

3) The UIDs of the other messages at this point have not been altered.
Disconnect from the IMAP server.  (With Evolution, I actually had to
restart the program to reproduce this bug.)

4) Reconnect to the IMAP service, and allow the client to re-sync.  You
will find that the X-UID of all of the messages after the one that was
deleted have been decremented -- and, if you're using a caching client,
all of the message *bodies* will appear to have been shifted down by
one.

Reading the headers in the mbox file directly, Dovecot has renumbered
all of the UIDs from 1 to N but not altered the UIDVALIDITY field:

dwm at kalimdor $ grep X-UID: Root
X-UID: 1
X-UID: 2
[... all numbers between 2 and 161 follow ...]
X-UID: 161
X-UID: 162
dwm at kalimdor $ grep X-IMAPbase: Root
X-IMAPbase: 1063356421 162

1063356421 == Sept 2003, when the mailbox was first created.

This is a bug.  Either Dovecot should not renumber the UIDs of each
message, or (if it really is necessary), it should increase the
UIDVALIDITY field to some higher number to invalidate the caches stored
by any remote clients.

This bug wasn't observed when using previous versions of Dovecot (we
were previously using 0.99.10.6 before this upgrade without noticing
anything broken.)

I'm hoping someone familiar with the source tree can identify and
suggest a fix relatively quickly; failing that, I'll take a look around
to see if I can work out where the problem is and implement a fix it
myself.

Cheers,
David
-- 
David McBride <dwm at tastycake.net>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://dovecot.org/pipermail/dovecot/attachments/20050722/9a6a4ac8/attachment-0001.pgp