[Dovecot] Why does it appear that dovecot is deleting messages after migration?

[Bill Eccles]

Hi, y'all, long-time Mac OS X Server user Bill here with a thorny question for the mailing list since nobody at discussions.apple.com can quite put their finger on the answer. While well-aquainted with other aspects of Server, Dovecot is a bit of a mystery still to me. So here it goes:

Background

I'm struggling to migrate from MacOS X 10.6 to 10.9. I tried twice using the "live server in target disk mode" method, but after things went horribly awry (meaning I was left with no working services on the new server), I gave up on that and have been working to slowly bring services over, one at a time.

I've got both Dovecot and PostFix accepting connections from my test machine, and I am able to connect with IMAP to the new server.

What I Do...

First, I turn both mail servers off. Then I tar up the /var/spool/imap/dovecot/mail directory on the old server, move it to the new server, untar it in /Library/Server/Mail/Data/ (where it belongs), and change ownership to _dovecot:mail, all per many other suggestions found in the Apple forums. (All files appear to be in maildir format, that I can tell.)

I then turn the mail service back on on the new server.

What Dovecot Does... (or appears to do)

If I quickly connect to the new server with my IMAP client (MacOS X Mail.app) and create a "fresh user" in Mail.app, message folders appear correctly. But it's only a matter of time before Dovecot, or something, runs "doveadm index -u (usernames) (mailboxes)" on all of the mailboxes. As it does, it deletes thousands of messages, leaving, for example, the same 17 in INBOX and 3695 in "Deleted Items Archive". And 0 in others, and other seemingly-random numbers of messages in still yet others. I can watch it marching through the mailboxes, one at a time, if I "ps -ax" enough times.

What I've Done to Counteract This...

Nothing I've done so far has been able to do anything about it. Re-untar the archive, and it just does it all again. Try using dsync, and it just deletes the messages from the un-tar'd archive. In other words, there are thousands of messages in the archive which don't seem to meet Dovecot's requirement for being in that particular folder, so it deletes them. The "cur" directory in a mailbox directory may start out with >21000 messages in it, but when it's done, it'll have deleted the same messages from "cur" every time, leaving me with a fraction of what I started out with. I even used the 65-migrate_mailboxes.pl script (supplied as part of OS X 10.9 Server), and the same thing happens. (The migrate script does a lot more than just what I've noted above, but most of what it does is responsible for settings and configuration migration. Very little of the script is responsible for moving the data, and that part of the script ends up being a bunch of "cp" followed by a "chown"--essentially what I'm doing.)

The Questions

What is going on? Failing a fix, am I trying to migrate the wrong way? Is there a better way?

Random Thoughts and Observations

I've changed hostnames. The old one was shr-xs.mydomain.net, and the new one is shr-mini.mydomain.net. Is it unhappy with that for some reason?

In one of the directories where messages are deleted, there were a zillion files named like this:

1396718896.M324523P43630.shr-xs.mydomain.net,S=1209,W=1238:2,Sac

There were also a bunch of files named like this:

1211416234.cyrus.457,S=893:2,Sac

When it's all said and done, and Dovecot has done its thing, care to guess which ones are left? Only these:

1211416234.cyrus.457,S=893:2,Sac
 
Just to see if I could influence which files get deleted, I deleted all of the dovecot* files in the mailbox directory. And Dovecot recreated them faithfully, but still deleted all but the ones which said "cyrus" in the filenames.

I changed the hostname to match shr-xs.mydomain.net, and that doesn't seem to have affected things a bit, though I don't have enough data to support that I really succeeded in changing the hostname for all services (specifically for Dovecot).

Is Dovecot really filtering out messages that just don't seem to "belong" in the folder?

Thanks, y'all. I've puzzled on this one for about eight hours and it now hurts my brain. I hope somebody else has an answer sans brain hurts.