[Dovecot] Script: Moving Maildirs between servers
Hi, I just migrated from dovecot 1.0.rc29 to 1.1.7, from Sparc platform to Intel, it went very well. In the process I merged folders and excluded folders from being copied. User rights on folders were kept/set. I thought I'd share my migration script with you folks. If there's an interest to put it on the wiki I can do that, but I'm not sure in what page. Use it as a starter if you need to do this yourselves. Prerequisites:
- User names are local unix accounts
- Old server's /home must be mounted with nfs
- Structure and user names must match between the two systems
- Source dovecot must be at least version 1.0.x
- Destination dovecot must be 1.1.x
- Only dovecot data folders and files are copied, neither dovecot indices nor Sieve scripts etc.
Here it goes, have fun! /Peter
#!/bin/sh # # Migrates dovecot Maildir folders and settings from older dovecot on # old server to new dovecot on new server. # Prerequisites: # * User names are local unix accounts # * Old server's /home must be mounted with nfs # * Structure and user names must match between the two systems # * Source dovecot must be at least version 1.0.x # * Destination dovecot must be 1.1.x # * Only dovecot data folders and files are copied, neither dovecot indices nor Sieve scripts etc. # # Originally created by Peter Lindgren <peter@norrskenkonsult.com> # Use this script at your own risk. No warranties whatsoever, of any kind!
# Basic settings: The first one you must definitely change, it's where you have mounted the old server's home: SOURCEBASE=/mnt/casiopea-home DESTBASE=/home
# Just an abbreviation: MD=Maildir
# Change field separator, for find's sake as there might be folders with spaces: # (doesn't play well together with the for loop otherwise) OLDIFS=${IFS} IFS=' '
# User loop:
# Depending on your source system's setup,
# you probably have non-interactive users with home directories (one of them
# is probably named _dovecot) that shall not be copied. Edit the username
# exclusion list according to your needs.
#
for username in ls $SOURCEBASE
; do
{
# This is your username exclude list:
case $username in (maildir|_dovecot) continue;; esac
# Here starts the real work: SOURCEDIR=$SOURCEBASE/$username/$MD DESTDIR=$DESTBASE/$username/$MD echo "================================" echo "Copying user "$username"..." echo "From "$SOURCEDIR" to "$DESTDIR":"
# IMAP Folder loop:
# Will create the Maildir folder too, the first echo $SOURCEDIR"/." does that. This way all the copying is in one place only.
for sourceimap in { echo $SOURCEDIR"/." ; find $SOURCEDIR -type d -name \.\* -maxdepth 1 -print ; } | sort -f
; do
{
# Folder exlusion and special actions:
# .Draft is replaced by .Drafts, so content of .Draft must be copied to .Drafts.
# (Probably there were a client in use for a while that used .Draft instead of .Drafts.)
# Junk mail folders with various names from clients using different languages shall not be copied.
# .Trash shall not be copied either.
# Folder .INBOX is a leftover from earlier versions of dovecot, and shall not be copied either.
# Add your own exceptions to the list. Be sure to quote spaces and dots.
basename=basename $sourceimap
action=copy
case $basename in (\.Junk|\.Junk\ E-mail|\.Correo\ electr\&APM-nico\ no\ deseado|\.Skr\&AOQ-ppost|\.Trash|\.INBOX) action=skip;; esac
echo -n "$basename" "$action"
if [ $action"X" == copy"X" ]; then
{
echo -n ":"
destname=$basename
# Merge contents of .Draft and .Drafts to .Drafts:
if [ $basename"X" == .Draft"X" ]; then destname=${destname}"s"; fi
destimap=${DESTDIR}/${destname}
# If this is the Maildir folder (.), a clean desination name is needed for mkdir:
if [ $basename"X" == ."X" ]; then destimap=${DESTDIR}; fi
mkdir -m 700 "${destimap}"
echo -n "."
chown ${username} "${destimap}"
echo -n "."
chmod go-rx "${destimap}"
echo -n "."
cp -p "${sourceimap}/dovecot-uidlist" "${destimap}"
echo -n "."
cp -p "${sourceimap}/dovecot-keywords" "${destimap}"
echo -n "."
cp -pR "${sourceimap}/cur" "${destimap}"
echo -n "."
cp -pR "${sourceimap}/new" "${destimap}"
echo -n "."
cp -pR "${sourceimap}/tmp" "${destimap}"
echo " done."
}
else
{
echo ""
}
fi
}; done # Copy root folder files last in user loop, since the Maildir is created in the imap loop above: cp -p $SOURCEDIR/subscriptions $DESTDIR echo "User "${username}" done." }; done
IFS=${OLDIFS}
-- Peter Lindgren http://www.norrskenkonsult.com
Peter Lindgren skrev:
I thought I'd share my migration script with you folks. If there's an interest to put it on the wiki I can do that, but I'm not sure in what page.
Slight bug: The script didn't create a Trash folder. Add the following rows before the echo " done." row in the user folder loop:
echo -n "."
mkdir -p -m 700 .Trash/tmp
mkdir -p -m 700 .Trash/new
mkdir -p -m 700 .Trash/cur
chown ${username} .Trash
chown ${username} .Trash/tmp
chown ${username} .Trash/new
chown ${username} .Trash/cur
/Peter
Peter Lindgren http://www.norrskenkonsult.com
Peter Lindgren a écrit :
Hi, I just migrated from dovecot 1.0.rc29 to 1.1.7, from Sparc platform to Intel, it went very well. In the process I merged folders and excluded folders from being copied. User rights on folders were kept/set. I thought I'd share my migration script with you folks. If there's an interest to put it on the wiki I can do that, but I'm not sure in what page. Use it as a starter if you need to do this yourselves. Prerequisites:
- User names are local unix accounts
- Old server's /home must be mounted with nfs
- Structure and user names must match between the two systems
- Source dovecot must be at least version 1.0.x
- Destination dovecot must be 1.1.x
- Only dovecot data folders and files are copied, neither dovecot indices nor Sieve scripts etc.
Here it goes, have fun! /Peter
#!/bin/sh # # Migrates dovecot Maildir folders and settings from older dovecot on # old server to new dovecot on new server. # Prerequisites: # * User names are local unix accounts # * Old server's /home must be mounted with nfs # * Structure and user names must match between the two systems # * Source dovecot must be at least version 1.0.x # * Destination dovecot must be 1.1.x # * Only dovecot data folders and files are copied, neither dovecot indices nor Sieve scripts etc. #
rsync would be simpler.
Words by mouss [Mon, Feb 02, 2009 at 11:07:19PM +0100]:
Peter Lindgren a écrit :
Hi, I just migrated from dovecot 1.0.rc29 to 1.1.7, from Sparc platform to Intel, it went very well.
rsync would be simpler.
Indeed. We always do a
rsync now rsync then rsync just another time stop everything, rsync final, start everything
with great success :)
-- Jose Celestino | http://japc.uncovering.org/files/japc-pgpkey.asc
"One man’s theology is another man’s belly laugh." -- Robert A. Heinlein
On 2/2/2009 7:30 PM, Jose Celestino wrote:
I just migrated from dovecot 1.0.rc29 to 1.1.7, from Sparc platform to Intel, it went very well.
rsync would be simpler.
Indeed. We always do a
rsync now rsync then rsync just another time stop everything, rsync final, start everything
with great success :)
So you're rsyncing a live mail system with no problems?
I've been tempted to start doing this, but I was under the impression it could be risky (cause file corruption)?
--
Best regards,
Charles
Words by Charles Marcus [Tue, Feb 03, 2009 at 10:08:46AM -0500]:
On 2/2/2009 7:30 PM, Jose Celestino wrote:
I just migrated from dovecot 1.0.rc29 to 1.1.7, from Sparc platform to Intel, it went very well.
rsync would be simpler.
Indeed. We always do a
rsync now rsync then rsync just another time stop everything, rsync final, start everything
with great success :)
So you're rsyncing a live mail system with no problems?
I've been tempted to start doing this, but I was under the impression it could be risky (cause file corruption)?
I'm rsyncing a live mail system so that I have a smaller downtime when doing the final rsync (that is always done with the system down, of course).
-- Jose Celestino | http://japc.uncovering.org/files/japc-pgpkey.asc
"One man’s theology is another man’s belly laugh." -- Robert A. Heinlein
On 2/3/2009, Jose Celestino (japc@co.sapo.pt) wrote:
I'm rsyncing a live mail system so that I have a smaller downtime when doing the final rsync (that is always done with the system down, of course).
Have you ever had issues with collisions? Meaning, rsync tries to copy a file that is being written to (saved, moved, deleted, etc) at that exact moment?
--
Best regards,
Charles
Charles Marcus wrote:
So you're rsyncing a live mail system with no problems?
I do it every day for backups. :-)
I've been tempted to start doing this, but I was under the impression it could be risky (cause file corruption)?
I can imagine that one could end up in the backup copy with an incomplete/corrupt file (but this will be fixed in the next rsync, hopefully), but I don't see how rsync could corrupt the original files in the mail server.
-- Why is it called a funny bone when it hurts so much?
Eduardo M KALINOWSKI eduardo@kalinowski.com.br http://move.to/hpkb
On T 3 Feb, 2009, at 18:40 , Eduardo M KALINOWSKI wrote:
Charles Marcus wrote:
So you're rsyncing a live mail system with no problems?
I do it every day for backups. :-)
I've been tempted to start doing this, but I was under the
impression it could be risky (cause file corruption)?I can imagine that one could end up in the backup copy with an incomplete/corrupt file (but this will be fixed in the next rsync, hopefully), but I don't see how rsync could corrupt the original files in the mail server.
I'd say that even an incomplete file is very unlikely, for message
files at least, while it might happen for files that are updated
(uidlists etc). Rsync builds a partial (in the latest versions) or
complete (in the old ones, pre 3) list of files before starting the
transfer, so by the time it starts the transfer, message files, which
existed when the list was built, are likely complete. It is quite
possible that a file will disappear (likely renamed) before rsync has
a chance to transfer it. That's not a problem though.
Consistency of the backup is another matter though, and requires a
quiescent mail system. For recovery of inadvertently deleted messages
this is not necessary.
g
On Thu, 2009-02-05 at 00:05 +0100, Giuliano Gavazzi wrote:
I can imagine that one could end up in the backup copy with an incomplete/corrupt file (but this will be fixed in the next rsync, hopefully), but I don't see how rsync could corrupt the original files in the mail server.
I'd say that even an incomplete file is very unlikely, for message
files at least,
For Maildir message files incomplete messages aren't just unlikely, they're never possible.
while it might happen for files that are updated
(uidlists etc).
True.
On T 5 Feb, 2009, at 00:13 , Timo Sirainen wrote:
For Maildir message files incomplete messages aren't just unlikely, they're never possible.
I was implying Maildir of course. I am not familiar with the
specification though. Does it require files to be written initially in
tmp to be moved in new once complete?
g
Words by Giuliano Gavazzi [Thu, Feb 05, 2009 at 12:51:28AM +0100]:
On T 5 Feb, 2009, at 00:13 , Timo Sirainen wrote:
For Maildir message files incomplete messages aren't just unlikely, they're never possible.
I was implying Maildir of course. I am not familiar with the
specification though. Does it require files to be written initially in
tmp to be moved in new once complete?
Yes. Must be atomic.
-- Jose Celestino | http://japc.uncovering.org/files/japc-pgpkey.asc
"One man’s theology is another man’s belly laugh." -- Robert A. Heinlein
participants (7)
-
Charles Marcus
-
Eduardo M KALINOWSKI
-
Giuliano Gavazzi
-
Jose Celestino
-
mouss
-
Peter Lindgren
-
Timo Sirainen