On Mon, 11 Feb 2013 21:47:57 +0100 Reindl Harald h.reindl@thelounge.net wrote:

Am 11.02.2013 21:34, schrieb David Mehler:

...

I'm having to migrate servers. Both are dovecot2 systems. I'm wanting to copy over my mail store from one system to the other. I'd like to preserve dates/times of emails. These are maildir setups on both boxes, I'd like to be as transparent to the end user as possible. I currently have the first dovecot system offline and the second dovecot system is offline.

assuming that BOTH servers are down as fro any other transfers like mysql-datadirs and any critical things which should go save and fast by preserve attributes:

rsync --force --delete-after -tPrlHpogEAXz /folder/ root@newserver:/folder/

# -z compress # -t timestamps # -P progress # -r recursive # -l links # -H hard-links # -p permissions # -o owner # -g group # -E executability # -A acls # -X xtended attributes

That's how I'd do it also. I think if you didn't mind including -D, which from my understanding is --special and --devices, neither of which I'd expect in a maildir, and if you wanted to do your deletion manually after the fact, would this be equivalent?

rsync -PaHAX /folder/

The man page says -a (--archive) is equivalent to -rlptgoD, and you have all but D listed up there.

My thinking on later manually deleting the old maildir, instead of letting rsync do it, is that if somehow, some weird thing goes wrong, I have the old one for backup. Who knows, maybe I copied the thing on top of the wrong other maildir and have to back it out -- I'd have the old maildir as a reference of which files.

Of course there are arguments for instant deletion too, like the fact that a person runs the risk of accidentally deleting the wrong one, or the fact that a huge maildir folder could take hours to delete.

Personally, I never delete til I'm sure it got to the right place and is backed up there, and sometimes I keep it on the old box for a month "just in case" if the old box won't be used anymore.

SteveT

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Mon, 11 Feb 2013, Reindl Harald wrote:

Am 11.02.2013 22:37, schrieb Steve Litt:

...

On Mon, 11 Feb 2013 21:47:57 +0100 Reindl Harald h.reindl@thelounge.net wrote:

...

Am 11.02.2013 21:34, schrieb David Mehler:

...

I'm having to migrate servers. Both are dovecot2 systems. I'm wanting to copy over my mail store from one system to the other. I'd like to preserve dates/times of emails. These are maildir setups on both boxes, I'd like to be as transparent to the end user as possible. I currently have the first dovecot system offline and the second dovecot system is offline.

assuming that BOTH servers are down as fro any other transfers like mysql-datadirs and any critical things which should go save and fast by preserve attributes:

rsync --force --delete-after -tPrlHpogEAXz /folder/ root@newserver:/folder/

# -z compress # -t timestamps # -P progress # -r recursive # -l links # -H hard-links # -p permissions # -o owner # -g group # -E executability # -A acls # -X xtended attributes

That's how I'd do it also. I think if you didn't mind including -D, which from my understanding is --special and --devices, neither of which I'd expect in a maildir, and if you wanted to do your deletion manually after the fact, would this be equivalent?

well, i have them ususally in my "rsync.sh" but did not expect such files in a maildir, but yes it does not hurt

...

rsync -PaHAX /folder/

The man page says -a (--archive) is equivalent to -rlptgoD, and you have all but D listed up there.

My thinking on later manually deleting the old maildir, instead of letting rsync do it, is that if somehow, some weird thing goes wrong, I have the old one for backup. Who knows, maybe I copied the thing on top of the wrong other maildir and have to back it out -- I'd have the old maildir as a reference of which files.

you missunderstand "--delete-after"

this is for delete files in the TARGET folder which is not or no longer in the source and IMHO very very important if you want sync folders 1:1 because old artefacts can have very bad effects

without you merge folders and if i know "hey my source contains exactly what i need, not more and lot less" this is not what i would like and never did in 10 years IT

If you use "--delete-after" you could add "--fuzzy", which theoretically could save transfers because of filename renames.

Anyway: Any form of "--delete" is required for Maildir, IMHO, because Labels, Tags, Keywords (or whatever the MUA calls it) and status information (seen, read, deleted) are reflected by the filename. Hence, if one does not "--delete", the _same_ message might pop up in the Maildir multiple times but with different status and/or tags. Same applies to new messages, because they are storred in <maildir>/new and later moved to <maildir>/cur. So one ends with the same message in "new" and in "cur".

Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux)

iQEVAwUBURn+wV3r2wJMiz2NAQJYnwgAwGfCHBFglm800nbaHFhsdWeuVkhGJjnW l0QtbmQLaSdQOkWwdVkmNEQkJLmwvfhddG0a9GblofI/zn8EBN+3EDR/CM0d5Y8f jy1yqemT26sFTVC6NlTqhDF8zq/Oi5WK7ftUfYfrt90MtmMD1rzmBQ5Q/N+Tteae TeE0jeHNL0rHl9DoAV9AkBzhJPnYYodSK2lA1oewazlzRpzJSiHYmgVh4RiSmsyk 4DAtch5ZrqLnsx1A/mUHAfDvdk66j2Os0bLXqbHu9ZAzz5Xb9bbNfpu52u1Ukg/o TfXfiGA4aRix6I1OGePdXmi/qXTzPVLBzhhIZdMiJ2CKCWNFbIYC7w== =ugv1 -----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Tue, 12 Feb 2013, David Mehler wrote:

Thanks, if I did:

rsync --force --delete-after -tPrlHpogEAXz /folder/ /backups/newmailstore/

could I then tar up the newmailstore folder with something like tar zcf without messing anything up?

keep in mind, that the backup of the Maildir with rsync is no 100% consistent snapshot, because of the filename renames; otherwise: yes.

On 2/12/13, Steffen Kaiser skdovecot@smail.inf.fh-brs.de wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Mon, 11 Feb 2013, Reindl Harald wrote:

...

Am 11.02.2013 22:37, schrieb Steve Litt:

...

On Mon, 11 Feb 2013 21:47:57 +0100 Reindl Harald h.reindl@thelounge.net wrote:

...

Am 11.02.2013 21:34, schrieb David Mehler:

...

I'm having to migrate servers. Both are dovecot2 systems. I'm wanting to copy over my mail store from one system to the other. I'd like to preserve dates/times of emails. These are maildir setups on both boxes, I'd like to be as transparent to the end user as possible. I currently have the first dovecot system offline and the second dovecot system is offline.

assuming that BOTH servers are down as fro any other transfers like mysql-datadirs and any critical things which should go save and fast by preserve attributes:

rsync --force --delete-after -tPrlHpogEAXz /folder/ root@newserver:/folder/

# -z compress # -t timestamps # -P progress # -r recursive # -l links # -H hard-links # -p permissions # -o owner # -g group # -E executability # -A acls # -X xtended attributes

That's how I'd do it also. I think if you didn't mind including -D, which from my understanding is --special and --devices, neither of which I'd expect in a maildir, and if you wanted to do your deletion manually after the fact, would this be equivalent?

well, i have them ususally in my "rsync.sh" but did not expect such files in a maildir, but yes it does not hurt

...

rsync -PaHAX /folder/

The man page says -a (--archive) is equivalent to -rlptgoD, and you have all but D listed up there.

My thinking on later manually deleting the old maildir, instead of letting rsync do it, is that if somehow, some weird thing goes wrong, I have the old one for backup. Who knows, maybe I copied the thing on top of the wrong other maildir and have to back it out -- I'd have the old maildir as a reference of which files.

you missunderstand "--delete-after"

this is for delete files in the TARGET folder which is not or no longer in the source and IMHO very very important if you want sync folders 1:1 because old artefacts can have very bad effects

without you merge folders and if i know "hey my source contains exactly what i need, not more and lot less" this is not what i would like and never did in 10 years IT

If you use "--delete-after" you could add "--fuzzy", which theoretically could save transfers because of filename renames.

Anyway: Any form of "--delete" is required for Maildir, IMHO, because Labels, Tags, Keywords (or whatever the MUA calls it) and status information (seen, read, deleted) are reflected by the filename. Hence, if one does not "--delete", the _same_ message might pop up in the Maildir multiple times but with different status and/or tags. Same applies to new messages, because they are storred in <maildir>/new and later moved to <maildir>/cur. So one ends with the same message in "new" and in "cur".

---

Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux)

iQEVAwUBURn+wV3r2wJMiz2NAQJYnwgAwGfCHBFglm800nbaHFhsdWeuVkhGJjnW l0QtbmQLaSdQOkWwdVkmNEQkJLmwvfhddG0a9GblofI/zn8EBN+3EDR/CM0d5Y8f jy1yqemT26sFTVC6NlTqhDF8zq/Oi5WK7ftUfYfrt90MtmMD1rzmBQ5Q/N+Tteae TeE0jeHNL0rHl9DoAV9AkBzhJPnYYodSK2lA1oewazlzRpzJSiHYmgVh4RiSmsyk 4DAtch5ZrqLnsx1A/mUHAfDvdk66j2Os0bLXqbHu9ZAzz5Xb9bbNfpu52u1Ukg/o TfXfiGA4aRix6I1OGePdXmi/qXTzPVLBzhhIZdMiJ2CKCWNFbIYC7w== =ugv1 -----END PGP SIGNATURE-----

Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux)

iQEVAwUBURs+tV3r2wJMiz2NAQIvngf/c37Hbg30lIq0wFPCif9IAG+BO8yDRAa3 lYY2t7DlszWopeEBvV22HjUC8SlzywECYEhBFRWhJVfqNyu55oI867of+RZ6lO0A lNcThmNixX5IsFiUnN9S9NYrTENC2qhccPdrTCAcm6A6CtaR1ydyeZDxlmvmTWe4 Za60LVqlsIVoZQ146yGE5nSIctix8JOE5kfGO5NurYXTfHt9CrPj5JvlPmRUdasp Mtn+QpxaxzFa8NGECSThKDVilMPFvqfT+JrhjWnkM1v31hNjJj4F3DYhk1L0YlAL 4wUjskXq1ytoM8+k9c8rqCy5DvM7g8N2ip0c0Buvjt+RctaHE4F9jg== =moaG -----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Wed, 13 Feb 2013, Reindl Harald wrote:

...

...

Thanks, if I did:

rsync --force --delete-after -tPrlHpogEAXz /folder/ /backups/newmailstore/

could I then tar up the newmailstore folder with something like tar zcf without messing anything up?

keep in mind, that the backup of the Maildir with rsync is no 100% consistent snapshot, because of the filename renames; otherwise: yes

who would rename them and why?

"/folder/" would be a Maildir on a production mail server. If rsync runs (without help from other funtionality, such as LVM or ZFS snapshots) and at the same time someone accesses the Maildir and:

a) "see"s a message, which lets Dovecot rename the message file from ***/new/ to ***/cur

b) tags a message with a keyword, which lets Dovecot rename the file to have some lower-case letter

c) untags a keyword -> remove that letter from the filename

d) changes Deleted, Read, Answered status --> add/remove an upper-case letter from filename

rsync might have cached a message with an old filename no longer physically present on disk. You will see a "XYZ vanished" message and that particular message is not backuped, but removed from backup. Some scripts use rsync in a loop in assumption, that no Maildir is accessed that often, that you get a clean run of rsync eventually.

Maybe, if you tag a message with a new keyword, rsync already copied dovecot-keywords without the new keyword, but copy the message, when its filename has the keyword-letter.

In fact, if I want to make 100% sure I get an exact copy, I do this:

rsync /from /to rsync /from /to mv /from /from_locked sleep 1 rsync /from_locked /to mv /from_locked /from

"/from" is the base directory of the Maildir. In my environment that causes tempfails on delivery and internal server errors on IMAP/POP3 access. But the time between the two mv's is very small, because the main differences are handled with the first two rsync's. As you seen, I copy one user after another, which breaks hardlinks between users, but keeps memory footprint of rsync low and as well as the downtime.

I do make my usual daily backups without the loop and the mv's, because it is very seldom I get the "vanished" messages. I suppose, because the script runs two hours after midnight.

Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux)

iQEVAwUBURuQIF3r2wJMiz2NAQKgjgf8CtyeO6pONFU038x8O2wggcntdvGyeg8Z Uq8KR0QFVg1n0HNDaa33OYN4IxSHX8zNvElf+wc0ejQ3NUOPVDl2mdm4iEihyOYv Veb/p2iK671Nrs8nB7USwx7OE9vY8IYoB/ZSXrXGWowqOqRQIcJWHAfZ9Ewj3Rg/ iRGMUNCn7UfEDfWl+F5yWpdp/+3xJGxoWeWaegW/yfTzlJ5nKffS/SAfJlUm7zuV u31JL4fjk25uGG7alzrCxOq0z4A3PvcpGag2nkfIRbrLLmo4Wzr+09Bd2zqmSn74 /PaHlxFS6a2uy7ugqdd5kxfiZHnOS2/d6JgV428I8qS0CWugEPuFbg== =mw5e -----END PGP SIGNATURE-----