mbox to maildir migration, same server?

[kjohnson at eclypse.org]

What is the recommended procedure to migrate from mbox to maildir on a server already running dovecot?  Ideally, this procedure would convert messages in an existing mbox INBOX and in ~/mail mboxes (as defined in my dovecot configuration) to storage in a ~/Maildir.  Some of the INBOXs are rather large -- 100's of MB (one of the reasons for migrating).

What I have tried has not worked for me.  Below, I describe what I tried, how I tested before I tried, what went wrong, and the current setup (pre-migration).

What I tried:
=============

stop exim4

service dovecot stop

Make dovecot listen only locally, not to external connections.

/etc/dovecot/local.conf 

protocol imap {
    listen = 127.0.0.1:143
    ssl_listen = 127.0.0.1:993
}

protocol pop3 {
    listen = 127.0.0.1:110
    ssl_listen = 127.0.0.1:995
}

Modify dovecot configuration for maildir:
Watch capitalization!

#in 10-mail.conf:

mail_location = maildir:/home/%u/Maildir

namespace inbox {
separator = /
}

dovecot -n, to confirm results and perform
limited syntax checking.

service dovecot start

for each user, in a bash script:

dsync -v -u "$U" mirror "mbox:/home/$U/mail:INBOX=/var/mail/$U"

(if the conversion worked for all users, then reconfigure exim to use maildir and restart it; reconfigure dovecot to listen to external interfaces and restart.)


How I tested:
=============
Before trying this on the real server, I set up a matching (same versions of Debian, Exim, and Dovecot) VM using mbox, with a couple of accounts.  One using POP3, one using IMAP.  The IMAP account was accessed from Claws and from Outlook.  The test accounts had perhaps a dozen messages on the server.  I tried the conversion procedure, and everything I tried worked flawlessly.  (Yes, next time I will move some real inbox files from the live server).


What went wrong:
================
I encountered two types of problems here.  The first was several dsync warnings (username changed):

dsync(example): Warning: Mailbox changes caused a desync. You may want to run dsync again: Remote lost mailbox GUID 21bfa11a81f11763ac150000d09efc50 (maybe it was just deleted?)

As I understand it, this can result in some or all messages being downloaded again by clients.  My understanding may be wrong.  I also don't understand how the mailbox could have changed -- exim4 was stopped, dovecot is only listening on 127.0.0.1, and I am the only user who uses a local mail client on that system, and this was not my account.

The second problem I encountered seems to match Debian Bug report logs - #1011238.

In part of the discussion of that report, the reporter mentions adding -1 to the command, as shown:

dsync -v -u "$U" mirror -1 "mbox:/home/$U/mail:INBOX=/var/mail/$U"

While this seemed to eliminate the dsync panic of #1011238, as far as I can tell, no old messages were actually moved from the old mbox files to the new maildirs.  At that point I restored the backup mbox files and undid the changes to dovecot and exim4.


Current setup:
==============
Debian 11.4
Exim 4.94.2-7
dovecot 1:2.3.13+dfsg1-2

The current setup with Dovecot was a successful migration from IMAP-UW in December 2020.

Output from dovecot -n
(the list of trusted networks has been removed)
# 2.3.13 (89f716dc2): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.13 (cdd19fe3)
# OS: Linux 5.10.0-16-686-pae i686 Debian 11.4
# Hostname: localhost
auth_verbose = yes
mail_location = mbox:~/mail:INBOX=/var/mail/%u
mail_privileged_group = mail
namespace inbox {
  inbox = yes
  location =
  mailbox Drafts {
    special_use = \Drafts
  }
  mailbox Junk {
    special_use = \Junk
  }
  mailbox Sent {
    special_use = \Sent
  }
  mailbox "Sent Messages" {
    special_use = \Sent
  }
  mailbox Trash {
    special_use = \Trash
  }
  prefix =
}
passdb {
  driver = pam
}
pop3_uidl_format = %08Xv%08Xu
protocols = " imap pop3"
ssl_cert = </etc/dovecot/dovecot.pem
ssl_client_ca_dir = /etc/ssl/certs
ssl_dh = # hidden, use -P to show it
ssl_key = # hidden, use -P to show it
userdb {
  driver = passwd
}

Thank you to anyone who read this far.

Ken