mbox to maildir migration, same server?
What is the recommended procedure to migrate from mbox to maildir on a server already running dovecot? Ideally, this procedure would convert messages in an existing mbox INBOX and in ~/mail mboxes (as defined in my dovecot configuration) to storage in a ~/Maildir. Some of the INBOXs are rather large -- 100's of MB (one of the reasons for migrating).
What I have tried has not worked for me. Below, I describe what I tried, how I tested before I tried, what went wrong, and the current setup (pre-migration).
What I tried:
stop exim4
service dovecot stop
Make dovecot listen only locally, not to external connections.
/etc/dovecot/local.conf
protocol imap { listen = 127.0.0.1:143 ssl_listen = 127.0.0.1:993 }
protocol pop3 { listen = 127.0.0.1:110 ssl_listen = 127.0.0.1:995 }
Modify dovecot configuration for maildir: Watch capitalization!
#in 10-mail.conf:
mail_location = maildir:/home/%u/Maildir
namespace inbox { separator = / }
dovecot -n, to confirm results and perform limited syntax checking.
service dovecot start
for each user, in a bash script:
dsync -v -u "$U" mirror "mbox:/home/$U/mail:INBOX=/var/mail/$U"
(if the conversion worked for all users, then reconfigure exim to use maildir and restart it; reconfigure dovecot to listen to external interfaces and restart.)
How I tested:
Before trying this on the real server, I set up a matching (same versions of Debian, Exim, and Dovecot) VM using mbox, with a couple of accounts. One using POP3, one using IMAP. The IMAP account was accessed from Claws and from Outlook. The test accounts had perhaps a dozen messages on the server. I tried the conversion procedure, and everything I tried worked flawlessly. (Yes, next time I will move some real inbox files from the live server).
What went wrong:
I encountered two types of problems here. The first was several dsync warnings (username changed):
dsync(example): Warning: Mailbox changes caused a desync. You may want to run dsync again: Remote lost mailbox GUID 21bfa11a81f11763ac150000d09efc50 (maybe it was just deleted?)
As I understand it, this can result in some or all messages being downloaded again by clients. My understanding may be wrong. I also don't understand how the mailbox could have changed -- exim4 was stopped, dovecot is only listening on 127.0.0.1, and I am the only user who uses a local mail client on that system, and this was not my account.
The second problem I encountered seems to match Debian Bug report logs - #1011238.
In part of the discussion of that report, the reporter mentions adding -1 to the command, as shown:
dsync -v -u "$U" mirror -1 "mbox:/home/$U/mail:INBOX=/var/mail/$U"
While this seemed to eliminate the dsync panic of #1011238, as far as I can tell, no old messages were actually moved from the old mbox files to the new maildirs. At that point I restored the backup mbox files and undid the changes to dovecot and exim4.
Current setup:
Debian 11.4 Exim 4.94.2-7 dovecot 1:2.3.13+dfsg1-2
The current setup with Dovecot was a successful migration from IMAP-UW in December 2020.
Output from dovecot -n (the list of trusted networks has been removed) # 2.3.13 (89f716dc2): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.13 (cdd19fe3) # OS: Linux 5.10.0-16-686-pae i686 Debian 11.4 # Hostname: localhost auth_verbose = yes mail_location = mbox:~/mail:INBOX=/var/mail/%u mail_privileged_group = mail namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = pam } pop3_uidl_format = %08Xv%08Xu protocols = " imap pop3" ssl_cert = </etc/dovecot/dovecot.pem ssl_client_ca_dir = /etc/ssl/certs ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it userdb { driver = passwd }
Thank you to anyone who read this far.
Ken
https://wiki.dovecot.org/Migration/MailFormat
If you have not yet, please read this. Conversion can be done in-place.
While doing the in-place migration, I would strongly suggest keeping the users out from the server. While it's possible to do while there are users actively using the server, it is easier when there are no changes going on.
Regards, Aki
On 08/09/2022 20:43 EEST kjohnson@eclypse.org wrote:
What is the recommended procedure to migrate from mbox to maildir on a server already running dovecot? Ideally, this procedure would convert messages in an existing mbox INBOX and in ~/mail mboxes (as defined in my dovecot configuration) to storage in a ~/Maildir. Some of the INBOXs are rather large -- 100's of MB (one of the reasons for migrating).
What I have tried has not worked for me. Below, I describe what I tried, how I tested before I tried, what went wrong, and the current setup (pre-migration).
What I tried:
stop exim4
service dovecot stop
Make dovecot listen only locally, not to external connections.
/etc/dovecot/local.conf
protocol imap { listen = 127.0.0.1:143 ssl_listen = 127.0.0.1:993 }
protocol pop3 { listen = 127.0.0.1:110 ssl_listen = 127.0.0.1:995 }
Modify dovecot configuration for maildir: Watch capitalization!
#in 10-mail.conf:
mail_location = maildir:/home/%u/Maildir
namespace inbox { separator = / }
dovecot -n, to confirm results and perform limited syntax checking.
service dovecot start
for each user, in a bash script:
dsync -v -u "$U" mirror "mbox:/home/$U/mail:INBOX=/var/mail/$U"
(if the conversion worked for all users, then reconfigure exim to use maildir and restart it; reconfigure dovecot to listen to external interfaces and restart.)
How I tested:
Before trying this on the real server, I set up a matching (same versions of Debian, Exim, and Dovecot) VM using mbox, with a couple of accounts. One using POP3, one using IMAP. The IMAP account was accessed from Claws and from Outlook. The test accounts had perhaps a dozen messages on the server. I tried the conversion procedure, and everything I tried worked flawlessly. (Yes, next time I will move some real inbox files from the live server).
What went wrong:
I encountered two types of problems here. The first was several dsync warnings (username changed):
dsync(example): Warning: Mailbox changes caused a desync. You may want to run dsync again: Remote lost mailbox GUID 21bfa11a81f11763ac150000d09efc50 (maybe it was just deleted?)
As I understand it, this can result in some or all messages being downloaded again by clients. My understanding may be wrong. I also don't understand how the mailbox could have changed -- exim4 was stopped, dovecot is only listening on 127.0.0.1, and I am the only user who uses a local mail client on that system, and this was not my account.
The second problem I encountered seems to match Debian Bug report logs - #1011238.
In part of the discussion of that report, the reporter mentions adding -1 to the command, as shown:
dsync -v -u "$U" mirror -1 "mbox:/home/$U/mail:INBOX=/var/mail/$U"
While this seemed to eliminate the dsync panic of #1011238, as far as I can tell, no old messages were actually moved from the old mbox files to the new maildirs. At that point I restored the backup mbox files and undid the changes to dovecot and exim4.
Current setup:
Debian 11.4 Exim 4.94.2-7 dovecot 1:2.3.13+dfsg1-2
The current setup with Dovecot was a successful migration from IMAP-UW in December 2020.
Output from dovecot -n (the list of trusted networks has been removed) # 2.3.13 (89f716dc2): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.13 (cdd19fe3) # OS: Linux 5.10.0-16-686-pae i686 Debian 11.4 # Hostname: localhost auth_verbose = yes mail_location = mbox:~/mail:INBOX=/var/mail/%u mail_privileged_group = mail namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = pam } pop3_uidl_format = %08Xv%08Xu protocols = " imap pop3" ssl_cert = </etc/dovecot/dovecot.pem ssl_client_ca_dir = /etc/ssl/certs ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it userdb { driver = passwd }
Thank you to anyone who read this far.
Ken
Summary: Looking for a tool to report vsize of messages by UIDL, or other ways to investigate vsize inconsistency between converted (maildir) and original (mbox) mailboxes.
I am the administrator of a mail server configured with exim4, dovecot, and mbox mail storage. The system formerly used UW-IMAP, but I made an uneventful conversion to dovecot a few years ago.
Some months ago, I attempted to convert from mbox to maildir without success. As I prepared to try again, I am testing more thoroughly. I captured the mbox files from /var/mail and from /home/<username>/mail for IMAP users, and moved them to a virtual machine to practice the conversion there.
I took these steps on the virtual machine:
Turn off Exim. /etc/exim4 stop Modify dovecot configuration and restart dovecot. The two changes needed are conf.d/10-mail.conf: mail_location = maildir:/home/%u/Maildir conf.d/10-mail.conf: separator = /
To convert from mbox to maildir, I used this command for each user:
doveadm backup -f -R -u "$U" "mbox:/home/$U/mail:INBOX=/var/mail/$U" || exit
That completed without reporting any errors.
Before attempting the conversion, I used this shell script to get status information about each mailbox. It uses 'doveadm mailbox list' to get a list of mailboxes for each user, and 'doveadm mailbox status' to provide summary information. For example (username redacted)
zxxxxxxx: INBOX messages=6 recent=0 unseen=4 vsize=7820215
================= mailbox-status.sh ===================== #!/bin/bash
USERS=$(ls -1 /var/mail | grep -v ".lock" | sort)
for U in $USERS; do doveadm mailbox list -u "$U" | while read -r BOX do echo "$U: $BOX " $(doveadm mailbox status -u "$U" -t all "$BOX") done done
========================================================
I used my mailbox status script again after the conversion to report on the maildir mailboxes.
Of the 143 mailboxes found by the script, there are three where the status does not exactly match between the old mailboxes and the new mailboxes. The difference is only in the vsize. Sometimes the new vsize is large; sometimes it is smaller.
The way I can think of to investigate this is to use a tool that will report the vsize for each message in the mailbox, identified by UIDL. Then, compare the before/after reports, and then figure out what is going on with what I hope is one or a small number of messages in the mailbox. I can probably code up such a tool in Python. Does anyone know of existing tools? Or know of a better way to investigate?
Thanks,
Ken
Before attempting the conversion, I used this shell script to get status information about each mailbox. It uses 'doveadm mailbox list' to get a list of mailboxes for each user, and 'doveadm mailbox status' to provide summary information. For example (username redacted)
zxxxxxxx: INBOX messages=6 recent=0 unseen=4 vsize=7820215
I used my mailbox status script again after the conversion to report on the maildir mailboxes.
Of the 143 mailboxes found by the script, there are three where the status does not exactly match between the old mailboxes and the new mailboxes. The difference is only in the vsize. Sometimes the new vsize is large; sometimes it is smaller.
I think you did a proper check already. How big are these differences in vsize? Could this not be related to a different way the filesystem is being used?
Greetings, kjohnson@eclypse.org!
Summary: Looking for a tool to report vsize of messages by UIDL, or other ways to investigate vsize inconsistency between converted (maildir) and original (mbox) mailboxes.
The best way to investigate the differences is, quite unsurprisingly, to use diff tool directly on the messages in question.
Based on my observation from my own conversion endeavor, the messages imported are altered in a non-intrusive, standards-compliant, but sometimes quite significant way. Which may lead to significant differences in message size.
-- With best regards, Andrey Repin Thursday, April 20, 2023 13:01:22
Sorry for my terrible english...
participants (4)
-
Aki Tuomi
-
Andrey Repin
-
kjohnson＠eclypse.org
-
Marc