[Dovecot] critical X-UID reordering problem after upgrade from 1.1 to 1.2

[robh at cs.indiana.edu]

We upgraded our imap server hardware and went from dovecot 1.1.3 to 1.2.3
in the process and have been having a critical problem since.  I believe
I've tracked it down to X-UID reordering that dovecot seemed to do to the
mbox format (I know, I know...) inboxes.  The problem has manifest itself
in several ways with different mail clients but the general problem is
that the mail clients have gotten confused about which message is which. 
For example, thunderbird will display the wrong information about a
message in the header listing but then show the correct data in the
message display window when you actually open the message.  As you can
imagine, this is causing all kinds of user problems.

Here is a very simplified explanation of what I'm seeing.  Imagine that
the inbox had 5 messages with X-UID values 1, 3, 5, 6, and 9 before the
upgrade.  After the upgrade the X-UIDs were reordered so these same 5
messages now might have X-UID values of 1, 2, 3, 4, and 5.  Thunderbird
then shows the summary information like this:

     Subject/Sender/Date for message 1
     Subject/Sender/Date for message 2
     Subject/Sender/Date for message 2
     Subject/Sender/Date for message 4
     Subject/Sender/Date for message 3

Now, if you actually open the 5th message it shows the right message body
but the summary information is for the message that *used* to have X-UID 5
(which is now the 3rd message).  So, the thunderbird cache is out of sync
and I can fix it by removing the .msf file for that mailbox but, as you
can imagine, the prospect of getting all the users to remove their
.msf/.pst/?? cache files is daunting, to say the least.

I've seen discussions about similar things happening with mbox format
inboxes but they all seem to be in the context of some ancient 0.xx
version of dovecot with known mbox processing bugs.  But, I went from a
reasonably current 1.1.3 version to 1.2.3.  Right after seeing the
problem, I upgraded to the just released 1.2.4 but either that didn't
address the problem or, if it did, the reordering damage was already done
by version 1.2.3.

So, my 2 questions are:

1) Anyone know why this would have happened and whether it is a dovecot
problem or something I botched?  I folded all the local config changes I
had made to the 1.1.3 conf file into the stock 1.2.3 conf file so they
should have been identical modulo changes in the distro.   And, our local
config changes are really minimal.

2) Is there anything I can do now other than getting all the users to
remove their .msf/.pst/etc files?  It sure would be grand if there was
something I could do on the server side that would magically cause all the
client-side caches to be invalidated.  I'm hesitant to strip the inboxes
of all the X-IMAPbase and X-UID lines because I'm not sure what the
ramifications of that might be.

Here are some other random thoughts in case they might be related:

  - The old server was running 32bit RHEL4 and the new one 64bit RHEL5
  - I'm using the pre-built dovecot rpms from atrpms.net
  - The dovecot index cache is kept in /var/cache/dovecot/indexes and I
did not copy that over from the old server to the new assuming it would
just be rebuilt.

Any and all help is greatly appreciated!

  --Rob

=============== dovecot -n output follows ==================
# 1.2.4: /etc/dovecot.conf
# OS: Linux 2.6.18-128.4.1.el5 x86_64 Red Hat Enterprise Linux Server
release 5.3 (Tikanga)
protocols: imaps pop3s
ssl: required
ssl_cert_file: /etc/mail.pem
ssl_key_file: /etc/mail.pem
login_dir: /var/run/dovecot/login
login_executable(default): /usr/libexec/dovecot/imap-login
login_executable(imap): /usr/libexec/dovecot/imap-login
login_executable(pop3): /usr/libexec/dovecot/pop3-login
login_max_processes_count: 512
max_mail_processes: 2048
mail_max_userip_connections(default): 15
mail_max_userip_connections(imap): 15
mail_max_userip_connections(pop3): 10
first_valid_uid: 200
mail_location:
mbox:~/mail/:INBOX=/var/spool/mail/%u:INDEX=/var/cache/dovecot/indexes/%u
mail_executable(default): /usr/libexec/dovecot/imap
mail_executable(imap): /usr/libexec/dovecot/imap
mail_executable(pop3): /usr/libexec/dovecot/pop3
mail_plugin_dir(default): /usr/lib64/dovecot/imap
mail_plugin_dir(imap): /usr/lib64/dovecot/imap
mail_plugin_dir(pop3): /usr/lib64/dovecot/pop3
pop3_uidl_format(default): %08Xu%08Xv
pop3_uidl_format(imap): %08Xu%08Xv
pop3_uidl_format(pop3): %v.%u
lda:
  postmaster_address: postmaster at example.com
auth default:
  passdb:
    driver: pam
  userdb:
    driver: passwd