mbox to maildir migration, same server?

[Aki Tuomi]

https://wiki.dovecot.org/Migration/MailFormat

If you have not yet, please read this. Conversion can be done in-place.

While doing the in-place migration, I would strongly suggest keeping the users out from the server. While it's possible to do while there are users actively using the server, it is easier when there are no changes going on.

Regards,
Aki

> On 08/09/2022 20:43 EEST kjohnson at eclypse.org wrote:
> 
>  
> What is the recommended procedure to migrate from mbox to maildir on a server already running dovecot?  Ideally, this procedure would convert messages in an existing mbox INBOX and in ~/mail mboxes (as defined in my dovecot configuration) to storage in a ~/Maildir.  Some of the INBOXs are rather large -- 100's of MB (one of the reasons for migrating).
> 
> What I have tried has not worked for me.  Below, I describe what I tried, how I tested before I tried, what went wrong, and the current setup (pre-migration).
> 
> What I tried:
> =============
> 
> stop exim4
> 
> service dovecot stop
> 
> Make dovecot listen only locally, not to external connections.
> 
> /etc/dovecot/local.conf 
> 
> protocol imap {
>     listen = 127.0.0.1:143
>     ssl_listen = 127.0.0.1:993
> }
> 
> protocol pop3 {
>     listen = 127.0.0.1:110
>     ssl_listen = 127.0.0.1:995
> }
> 
> Modify dovecot configuration for maildir:
> Watch capitalization!
> 
> #in 10-mail.conf:
> 
> mail_location = maildir:/home/%u/Maildir
> 
> namespace inbox {
> separator = /
> }
> 
> dovecot -n, to confirm results and perform
> limited syntax checking.
> 
> service dovecot start
> 
> for each user, in a bash script:
> 
> dsync -v -u "$U" mirror "mbox:/home/$U/mail:INBOX=/var/mail/$U"
> 
> (if the conversion worked for all users, then reconfigure exim to use maildir and restart it; reconfigure dovecot to listen to external interfaces and restart.)
> 
> 
> How I tested:
> =============
> Before trying this on the real server, I set up a matching (same versions of Debian, Exim, and Dovecot) VM using mbox, with a couple of accounts.  One using POP3, one using IMAP.  The IMAP account was accessed from Claws and from Outlook.  The test accounts had perhaps a dozen messages on the server.  I tried the conversion procedure, and everything I tried worked flawlessly.  (Yes, next time I will move some real inbox files from the live server).
> 
> 
> What went wrong:
> ================
> I encountered two types of problems here.  The first was several dsync warnings (username changed):
> 
> dsync(example): Warning: Mailbox changes caused a desync. You may want to run dsync again: Remote lost mailbox GUID 21bfa11a81f11763ac150000d09efc50 (maybe it was just deleted?)
> 
> As I understand it, this can result in some or all messages being downloaded again by clients.  My understanding may be wrong.  I also don't understand how the mailbox could have changed -- exim4 was stopped, dovecot is only listening on 127.0.0.1, and I am the only user who uses a local mail client on that system, and this was not my account.
> 
> The second problem I encountered seems to match Debian Bug report logs - #1011238.
> 
> In part of the discussion of that report, the reporter mentions adding -1 to the command, as shown:
> 
> dsync -v -u "$U" mirror -1 "mbox:/home/$U/mail:INBOX=/var/mail/$U"
> 
> While this seemed to eliminate the dsync panic of #1011238, as far as I can tell, no old messages were actually moved from the old mbox files to the new maildirs.  At that point I restored the backup mbox files and undid the changes to dovecot and exim4.
> 
> 
> Current setup:
> ==============
> Debian 11.4
> Exim 4.94.2-7
> dovecot 1:2.3.13+dfsg1-2
> 
> The current setup with Dovecot was a successful migration from IMAP-UW in December 2020.
> 
> Output from dovecot -n
> (the list of trusted networks has been removed)
> # 2.3.13 (89f716dc2): /etc/dovecot/dovecot.conf
> # Pigeonhole version 0.5.13 (cdd19fe3)
> # OS: Linux 5.10.0-16-686-pae i686 Debian 11.4
> # Hostname: localhost
> auth_verbose = yes
> mail_location = mbox:~/mail:INBOX=/var/mail/%u
> mail_privileged_group = mail
> namespace inbox {
>   inbox = yes
>   location =
>   mailbox Drafts {
>     special_use = \Drafts
>   }
>   mailbox Junk {
>     special_use = \Junk
>   }
>   mailbox Sent {
>     special_use = \Sent
>   }
>   mailbox "Sent Messages" {
>     special_use = \Sent
>   }
>   mailbox Trash {
>     special_use = \Trash
>   }
>   prefix =
> }
> passdb {
>   driver = pam
> }
> pop3_uidl_format = %08Xv%08Xu
> protocols = " imap pop3"
> ssl_cert = </etc/dovecot/dovecot.pem
> ssl_client_ca_dir = /etc/ssl/certs
> ssl_dh = # hidden, use -P to show it
> ssl_key = # hidden, use -P to show it
> userdb {
>   driver = passwd
> }
> 
> Thank you to anyone who read this far.
> 
> Ken