Package: dovecot-imapd Version: 0.99.14-1 Severity: important

Hi,

I think I've identified a bug in Dovecot 0.99.14 as packaged and distributed by Debian Sarge. I believe the bug lies in the original Dovecot source, not with any modifications made by Debian.

Specifically, this bug deals with Dovecot's handling of message unique identifiers (UIDs) for IMAP clients when using the mbox mailbox storage format.

Some background for those not familiar with the details; (I had to look a lot of this stuff up; see also http://www.ietf.org/rfc/rfc2060.txt):

--- 8< --------------------------------------------------------------

All IMAP servers are required to maintain unique identifiers for each email in an IMAP mailbox that are persistent between sessions. Dovecot stores the unique identifiers for each email in an mbox in an X-UID header.

Clients which maintain a local cache of already-retrieved emails use an email's UID as a key; thus, if the UID of a particular email were to change on a server then the client's cached state would be invalid.

To cope with the case that the UIDs need to be invalidated, whether because the original store has been lost or for some other reason, there is also the UIDVALIDITY number. This 32bit integer must be increased to some higher value to indicate that any previously cached email ids are now invalid.

In Dovecot-managed mbox files, this number is stored in the X-IMAPbase header in the first message of that mbox and is typically the uid-invalidation time expressed in unix timestamp format (ie the number of seconds elapsed since the epoch.)

--- 8< --------------------------------------------------------------

Okay, the actual bug: Dovecot routinely (and apparently unneccessarily) renumbers all of the UIDs in our mbox-backed mailboxes without updating the UIDVALIDITY count. This results in caching mail clients (such as Evolution and Thunderbird) presenting the incorrect email to the user when browsing their mail.

Configuration:

We are using Dovecot as a standalone daemon (ie without using inetd.) We have configured the default mail environment with the following string:

default_mail_env = mbox:%h/Mail/:INBOX=%h/.email:INDEX=MEMORY

ie each end-user's primary INBOX is an mbox, ~/.email, and other IMAP folders are also stored as mboxes in the directory ~/Mail/. Indexes are maintained in memory but are not stored on disk.

I don't think any of the rest of the configuration is relavant, but I'm happy to detail anything else about the setup if needed.

Reproducing the bug:

I'm afraid I don't have a proper protocol dump demonstrating this bug, so I'm just going to describe all of the end-user steps to recreate the problem:

1. Connect to the Dovecot IMAP service with a caching mail client, such as Evolution or Thunderbird. Download a local copy of the IMAP server contents of a test mbox-backed mailbox to the local cache.

For example, I have a test mbox with old root mail. Messages are numbered from 1 to 163. The most recent message has an X-UID: field of 163.

2. Delete one of the messages in the middle of the mailbox.

For example, mark the second-oldest message in the mailbox, which has a UID of 2, for deletion. Expunge the mailbox.

3. The UIDs of the other messages at this point have not been altered. Disconnect from the IMAP server. (With Evolution, I actually had to restart the program to reproduce this bug.)

4. Reconnect to the IMAP service, and allow the client to re-sync. You will find that the X-UID of all of the messages after the one that was deleted have been decremented -- and, if you're using a caching client, all of the message *bodies* will appear to have been shifted down by one.

Reading the headers in the mbox file directly, Dovecot has renumbered all of the UIDs from 1 to N but not altered the UIDVALIDITY field:

dwm@kalimdor $ grep X-UID: Root X-UID: 1 X-UID: 2 [... all numbers between 2 and 161 follow ...] X-UID: 161 X-UID: 162 dwm@kalimdor $ grep X-IMAPbase: Root X-IMAPbase: 1063356421 162

1063356421 == Sept 2003, when the mailbox was first created.

This is a bug. Either Dovecot should not renumber the UIDs of each message, or (if it really is necessary), it should increase the UIDVALIDITY field to some higher number to invalidate the caches stored by any remote clients.

This bug wasn't observed when using previous versions of Dovecot (we were previously using 0.99.10.6 before this upgrade without noticing anything broken.)

I'm hoping someone familiar with the source tree can identify and suggest a fix relatively quickly; failing that, I'll take a look around to see if I can work out where the problem is and implement a fix it myself.

Cheers, David

David McBride dwm@tastycake.net