broken mdboxes

[Mike]

Hello,

I was forced to do an emergency server migration with about 3000
mailboxes and I botched it somewhat. Hoping to find some help here.

My symptoms are, users showing apparently deleted mail messages, some
replicated mail, and a time period of mail that is just 'missing'.

The source server is centos 6.10 with dovecot 2.2.5, and the destination
is ubuntu 16 / dovecot 2.2.22. User mailboxes are mdbox format and I did
an rsync from the old to the new. My screw up however was that my rsync
was not --delete files, and had to be run multiple times over the course
of a few weeks due to to bandwidth disparities and the sheer size of the
mail spool approaching 700GB (source server had 10mbps. Should have used
sneakernet instead). The bottom line however is that I wound up having
many extra m.* files in each user storage dir that were not actually
present on the source when the final, final sync was made and I went
live with the migration server.  I know this is my fsckup but now I have
to deal with it.

I first tried to fix the symptoms with a force-resync on the boxes not
realizing the above, and that seemed to make things worse by only
showing new mail as of the server migration date. Ugh.

New mail is being delivered and this does appear in user mail boxes. But
as I said, there seems to be some users with a problem with old mail
presumably deleted on the prior server suddenly re-appearing (probbly
due to the extraneous m.* mdbox files present). A more pressing problem
is that there is simply a time period of mail that is in user mailboxes
(on the old server) from before the migration, which is not displaying
now for these users on the new server.

I have investigated this and hacked on "mdbox-recover.pl" somewhat so
that it iterates over all mdbox files for a user and I can extract ALL
messages, and see that, yes, the mail really is present in the mdbox
files, but pop/imap never shows it. This is puzzling to me. I have a
test user that when I check imap, it shows only 21 message in inbox and
1 in sent. But if I extract the messages from the mdbox files, then I
actually have 1117 messages. 

I know I can't just rsync the source server again with the right
--delete flag, there has been lots of mail deliveries to the migrated
boxes and so I don't want to lose all of that and create yet more
problems. But I need to find some way to make these present mail
messages appear for my users again. Im thinking it's got to be a flag of
some kind since the data is there it's just ignored. Wondering what you
would reccomend?


Thanks.

Mike-