backing up email / saving maildir on external hard drives

Kevin Laurie

8 Aug 2015 8 Aug '15

7:56 a.m.

Hello, Yesterday I tried to back up a 40GB maildir . I tried to move the maildir from home to external HDD but failed. Decided then to compress it(which took several hours). Now changing the disk format from FAT to exFAT to allow the transfer for the large compressed file.

How does one back up emails on a external drive? Some advice would be greatly appreciated.

Show replies by date

Christian Kivalo

8 Aug 8 Aug

8:06 a.m.

Am 08. August 2015 06:56:55 MESZ, schrieb Kevin Laurie superinterstellar@gmail.com:

...

Hello, Yesterday I tried to back up a 40GB maildir . I tried to move the maildir from home to external HDD but failed. Decided then to compress it(which took several hours). Now changing the disk format from FAT to exFAT to allow the transfer for the large compressed file.

How does one back up emails on a external drive? Some advice would be greatly appreciated.

i backup my maildir with rsync to my home nas.

i assume you are using linux or similar, so when backing up to an external hdd i'd also use rsync to copy over the maildir as is to the external hdd but use a different filesystem n that hdd like ext4 or xfs or anything else than fat/vfat/exfat.

when something goes wrong you can just copy the files back.

what was the error message you got when you tried to copy to the ext. hdd?

regards

Kevin Laurie

7:15 p.m.

Dear Christian, Thanks for your feedback. The HDD will not accept larger than 4GB (as its in FAT format). Its a new external HDD. Thinking of the best format(that would work with Mac , Win and Linux) .seems like a challenge. What's your view on NTFS? And why not exFAT?

Thanks Kevin

On Saturday, August 8, 2015, Christian Kivalo ml+dovecot@valo.at wrote:

...

Am 08. August 2015 06:56:55 MESZ, schrieb Kevin Laurie < superinterstellar@gmail.com javascript:;>:

...
Hello, Yesterday I tried to back up a 40GB maildir . I tried to move the maildir from home to external HDD but failed. Decided then to compress it(which took several hours). Now changing the disk format from FAT to exFAT to allow the transfer for the large compressed file.

How does one back up emails on a external drive? Some advice would be greatly appreciated.

i backup my maildir with rsync to my home nas.

i assume you are using linux or similar, so when backing up to an external hdd i'd also use rsync to copy over the maildir as is to the external hdd but use a different filesystem n that hdd like ext4 or xfs or anything else than fat/vfat/exfat.

when something goes wrong you can just copy the files back.

what was the error message you got when you tried to copy to the ext. hdd?

regards

c

Felix Zielcke

9 Aug 9 Aug

9:46 a.m.

Am Samstag, den 08.08.2015, 21:45 +0530 schrieb Kevin Laurie:

...

Dear Christian, Thanks for your feedback. The HDD will not accept larger than 4GB (as its in FAT format). Its a new external HDD. Thinking of the best format(that would work with Mac , Win and Linux) .seems like a challenge. What's your view on NTFS? And why not exFAT?

Thanks Kevin

Hi Kevin,

you can have multiple partitions even on an external HDD. Or do you really need one big partition which works on all 3 OS?

For Win and OS X I'd use exFAT. OS X AFAIK doestn't have write support on NTFS enabled by default. And then for Linux a native FS like ext4,btrfs,XFS or whatever you personally prefer.

Though the best would be indeed 3 partitions for each OS. NTFS for Windows And then HFS+ for OS X

For example I use on my backup HDD NTFS for Windows and btrfs for my Linux Systems

Kevin Laurie

5:10 p.m.

Hi Felix,

I would prefer having one HDD that works with all three OSes. That would be very convenient. Guess exFAT is my best bet. Its supported on Linux,Win and OS X. What do you reckon? Best Regards Kevin

On Sun, Aug 9, 2015 at 2:46 PM, Felix Zielcke fzielcke@z-51.de wrote:

...

Am Samstag, den 08.08.2015, 21:45 +0530 schrieb Kevin Laurie:

...
Dear Christian, Thanks for your feedback. The HDD will not accept larger than 4GB (as its in FAT format). Its a new external HDD. Thinking of the best format(that would work with Mac , Win and Linux) .seems like a challenge. What's your view on NTFS? And why not exFAT?

Thanks Kevin

Hi Kevin,

you can have multiple partitions even on an external HDD. Or do you really need one big partition which works on all 3 OS?

For Win and OS X I'd use exFAT. OS X AFAIK doestn't have write support on NTFS enabled by default. And then for Linux a native FS like ext4,btrfs,XFS or whatever you personally prefer.

Though the best would be indeed 3 partitions for each OS. NTFS for Windows And then HFS+ for OS X

For example I use on my backup HDD NTFS for Windows and btrfs for my Linux Systems

Felix Zielcke

10 Aug 10 Aug

11:09 p.m.

Am Sonntag, den 09.08.2015, 22:10 +0800 schrieb Kevin Laurie:

...

Hi Felix,

I would prefer having one HDD that works with all three OSes. That would be very convenient. Guess exFAT is my best bet. Its supported on Linux,Win and OS X. What do you reckon? Best Regards Kevin

With one HDD you can still have at least 4 different partitions on it without problems. So the question is still if you need one big partition for all 3 OS or not. I don't know how well exFAT is supported in Linux. I never used it at all.

Götz Reinicke - IT Koordinator

11 Aug 11 Aug

4:59 p.m.

Hi Kevin,

if you use maildir, I bet there is not a single mail bigger than 100 MB. So ALL mails are 40 GB, but may be hundreds and thousands small ones ...?

Or did you compress all files in one 40GB tar/gz/zip/... ?

So fat32 is good for small files.

BUT if you like to transfer as well e.g. DVD iso images larger than > 4 GB that might be a problem.

From what I think is good for you to go with, you could do a "rsync ... magic needed options" that maildir to the FAT32 Disk.

So you can read it on all systems without extra tools.

Furthermore if you want to do an other backup or "update" your files run an other rsync ... so you dont have to compress a big archive all the time.

As the others pointed out, what do you want to do, what is your requirement?

Especially in a Linux/Mac/Win World there is no one sizes fits all.

An other thing is: If you dont need to move your external disk around a lot or take it with you on the go, think of a NAS; e.g. two disks, mirrored raid so you can access it without big problems from all OSs ..

my2cents. /Götz

Am 08.08.15 um 18:15 schrieb Kevin Laurie:

...

Dear Christian, Thanks for your feedback. The HDD will not accept larger than 4GB (as its in FAT format). Its a new external HDD. Thinking of the best format(that would work with Mac , Win and Linux) .seems like a challenge. What's your view on NTFS? And why not exFAT?

Thanks Kevin

On Saturday, August 8, 2015, Christian Kivalo ml+dovecot@valo.at wrote:

...
Am 08. August 2015 06:56:55 MESZ, schrieb Kevin Laurie < superinterstellar@gmail.com javascript:;>:

...
Hello, Yesterday I tried to back up a 40GB maildir . I tried to move the maildir from home to external HDD but failed. Decided then to compress it(which took several hours). Now changing the disk format from FAT to exFAT to allow the transfer for the large compressed file.

How does one back up emails on a external drive? Some advice would be greatly appreciated.

i backup my maildir with rsync to my home nas.

i assume you are using linux or similar, so when backing up to an external hdd i'd also use rsync to copy over the maildir as is to the external hdd but use a different filesystem n that hdd like ext4 or xfs or anything else than fat/vfat/exfat.

when something goes wrong you can just copy the files back.

what was the error message you got when you tried to copy to the ext. hdd?

regards

c

Steve Litt

9 Aug 9 Aug

6:44 p.m.

On Sat, 8 Aug 2015 10:26:55 +0530 Kevin Laurie superinterstellar@gmail.com wrote:

...

Hello, Yesterday I tried to back up a 40GB maildir . I tried to move the maildir from home to external HDD but failed.

If you tried to *move* it it's an archive, not a backup. If you tried to *copy* it, with the intent of keeping the original on the original hard disk and using it further, and keeping today's copy on some other media, *that's* a backup. I'm not trying to be pedantic, but there are many distinctions between the two. Archives must be re-transferred frequently: Backups merely need to be redone at intervals.

...

Decided then to compress it(which took several hours). Now changing the disk format from FAT to exFAT to allow the transfer for the large compressed file.

Reading https://en.wikipedia.org/wiki/File_Allocation_Table#exFAT , I personally wouldn't use exFAT. Regular FAT32 has a max filesize of 2GB-1, which is 50 times the size of your whole uncompressed maildir.

...

How does one back up emails on a external drive? Some advice would be greatly appreciated.

Check this out:

================================================ slitt@mydesq2:~$ df -h ~/mail/Maildir Filesystem Size Used Avail Use% Mounted on /dev/sdb8 116G 11G 100G 10% /home/slitt/mail/Maildir slitt@mydesq2:~$

I don't have 40 GB, but * have 11, which is less than an order of magnitude away. I just back up this puppy to my backup server with my normal rsync based backup procedures, which you can read about here:

The stuff about Blu-Ray is important only if you back up to blu-ray. I like to keep some backups on write-once media, because kept in the shade at reasonable temperatures and humidities, it tends to last longer. And spinning disks that spend the majority of their time not spinning tend to have problems.

If this is a *backup*, I'd leave it uncompressed so you can take incremental backups regularly. If it's an *archive*, meaning that the data is immediately removed from your computer after copy, compression might be in order, but you should make two copies and test them both thoroughly before deleting the original, and you should test them every couple months and if either goes bad, copy the other one to something good. Archives are a PITA. For 40GB in these days of $150 2TB drives, I'd keep the data intact, back it up, and when you outgrow your hard drive, just get a bigger one.

In other parts of this thread you ask how to separate backups from different accounts from different computers. As far as accounts, I think that Maildir directory structures would take care of that. As far as different machines, just put the hostname at the front of each destination directory.

SteveT

Steve Litt July 2015 featured book: Rapid Learning for the 21st Century http://www.troubleshooters.com/rl21

Kevin Laurie

7:33 p.m.

Dear Steve, Very valuable info. Appreciate it and will be careful when using terms. Actually I think I should just use rsync without compressing. The reason why I started compressing was because the GUI gave some errors when I was trying to copy then files.

I'll just rsync the data from my laptop HDD to my external drive(without compressing)

Thanks Kevin

On Sunday, August 9, 2015, Steve Litt slitt@troubleshooters.com wrote:

...

On Sat, 8 Aug 2015 10:26:55 +0530 Kevin Laurie javascript:;> wrote:

...
Hello, Yesterday I tried to back up a 40GB maildir . I tried to move the maildir from home to external HDD but failed.

If you tried to *move* it it's an archive, not a backup. If you tried to *copy* it, with the intent of keeping the original on the original hard disk and using it further, and keeping today's copy on some other media, *that's* a backup. I'm not trying to be pedantic, but there are many distinctions between the two. Archives must be re-transferred frequently: Backups merely need to be redone at intervals.

...
Decided then to compress it(which took several hours). Now changing the disk format from FAT to exFAT to allow the transfer for the large compressed file.

Reading https://en.wikipedia.org/wiki/File_Allocation_Table#exFAT , I personally wouldn't use exFAT. Regular FAT32 has a max filesize of 2GB-1, which is 50 times the size of your whole uncompressed maildir.

...
How does one back up emails on a external drive? Some advice would be greatly appreciated.

Check this out:

================================================ slitt@mydesq2:~$ df -h ~/mail/Maildir Filesystem Size Used Avail Use% Mounted on /dev/sdb8 116G 11G 100G 10% /home/slitt/mail/Maildir slitt@mydesq2:~$

I don't have 40 GB, but * have 11, which is less than an order of magnitude away. I just back up this puppy to my backup server with my normal rsync based backup procedures, which you can read about here:

http://www.troubleshooters.com/lpm/200609/200609.htm

http://www.troubleshooters.com/linux/blu-ray-backup.htm

http://www.troubleshooters.com/lpm/201408/201408.htm

The stuff about Blu-Ray is important only if you back up to blu-ray. I like to keep some backups on write-once media, because kept in the shade at reasonable temperatures and humidities, it tends to last longer. And spinning disks that spend the majority of their time not spinning tend to have problems.

If this is a *backup*, I'd leave it uncompressed so you can take incremental backups regularly. If it's an *archive*, meaning that the data is immediately removed from your computer after copy, compression might be in order, but you should make two copies and test them both thoroughly before deleting the original, and you should test them every couple months and if either goes bad, copy the other one to something good. Archives are a PITA. For 40GB in these days of $150 2TB drives, I'd keep the data intact, back it up, and when you outgrow your hard drive, just get a bigger one.

In other parts of this thread you ask how to separate backups from different accounts from different computers. As far as accounts, I think that Maildir directory structures would take care of that. As far as different machines, just put the hostname at the front of each destination directory.

SteveT

Steve Litt July 2015 featured book: Rapid Learning for the 21st Century http://www.troubleshooters.com/rl21

Götz Reinicke - IT Koordinator

11 Aug 11 Aug

4:45 p.m.

Hi,

talking of rsync and compression is may be also a bit misleading.

On the destination there will be no compressed files if you transfer with rsync! The transfere on the network by rsync might be compressed!

/Götz

Am 09.08.15 um 18:33 schrieb Kevin Laurie:

...

Dear Steve, Very valuable info. Appreciate it and will be careful when using terms. Actually I think I should just use rsync without compressing. The reason why I started compressing was because the GUI gave some errors when I was trying to copy then files.

I'll just rsync the data from my laptop HDD to my external drive(without compressing)

Thanks Kevin

On Sunday, August 9, 2015, Steve Litt slitt@troubleshooters.com wrote:

...
On Sat, 8 Aug 2015 10:26:55 +0530 Kevin Laurie javascript:;> wrote:

...
Hello, Yesterday I tried to back up a 40GB maildir . I tried to move the maildir from home to external HDD but failed.

If you tried to *move* it it's an archive, not a backup. If you tried to *copy* it, with the intent of keeping the original on the original hard disk and using it further, and keeping today's copy on some other media, *that's* a backup. I'm not trying to be pedantic, but there are many distinctions between the two. Archives must be re-transferred frequently: Backups merely need to be redone at intervals.

...
Decided then to compress it(which took several hours). Now changing the disk format from FAT to exFAT to allow the transfer for the large compressed file.

Reading https://en.wikipedia.org/wiki/File_Allocation_Table#exFAT , I personally wouldn't use exFAT. Regular FAT32 has a max filesize of 2GB-1, which is 50 times the size of your whole uncompressed maildir.

...
How does one back up emails on a external drive? Some advice would be greatly appreciated.

Check this out:

================================================ slitt@mydesq2:~$ df -h ~/mail/Maildir Filesystem Size Used Avail Use% Mounted on /dev/sdb8 116G 11G 100G 10% /home/slitt/mail/Maildir slitt@mydesq2:~$

I don't have 40 GB, but * have 11, which is less than an order of magnitude away. I just back up this puppy to my backup server with my normal rsync based backup procedures, which you can read about here:

http://www.troubleshooters.com/lpm/200609/200609.htm

http://www.troubleshooters.com/linux/blu-ray-backup.htm

http://www.troubleshooters.com/lpm/201408/201408.htm

The stuff about Blu-Ray is important only if you back up to blu-ray. I like to keep some backups on write-once media, because kept in the shade at reasonable temperatures and humidities, it tends to last longer. And spinning disks that spend the majority of their time not spinning tend to have problems.

If this is a *backup*, I'd leave it uncompressed so you can take incremental backups regularly. If it's an *archive*, meaning that the data is immediately removed from your computer after copy, compression might be in order, but you should make two copies and test them both thoroughly before deleting the original, and you should test them every couple months and if either goes bad, copy the other one to something good. Archives are a PITA. For 40GB in these days of $150 2TB drives, I'd keep the data intact, back it up, and when you outgrow your hard drive, just get a bigger one.

In other parts of this thread you ask how to separate backups from different accounts from different computers. As far as accounts, I think that Maildir directory structures would take care of that. As far as different machines, just put the hostname at the front of each destination directory.

SteveT

Steve Litt July 2015 featured book: Rapid Learning for the 21st Century http://www.troubleshooters.com/rl21

-- Götz Reinicke IT-Koordinator

Tel. +49 7141 969 82420 E-Mail goetz.reinicke@filmakademie.de

Filmakademie Baden-Württemberg GmbH Akademiehof 10 71638 Ludwigsburg www.filmakademie.de

Eintragung Amtsgericht Stuttgart HRB 205016

Vorsitzender des Aufsichtsrats: Jürgen Walter MdL Staatssekretär im Ministerium für Wissenschaft, Forschung und Kunst Baden-Württemberg

Geschäftsführer: Prof. Thomas Schadt

Christian Schmidt

6:56 p.m.

On 11.08.2015 15:45, Götz Reinicke - IT Koordinator wrote:

...

talking of rsync and compression is may be also a bit misleading.

On the destination there will be no compressed files if you transfer with rsync!

...unless your dovecot is working with compressed maildir files. ;-)

Kind Regards, Christian Schmidt

-- No signature available.

Götz Reinicke

7:32 p.m.

...

Am 11.08.2015 um 17:56 schrieb Christian Schmidt Christian.Schmidt@chemie.uni-hamburg.de:

On 11.08.2015 15:45, Götz Reinicke - IT Koordinator wrote:

...
talking of rsync and compression is may be also a bit misleading.

On the destination there will be no compressed files if you transfer with rsync!

...unless your dovecot is working with compressed maildir files. ;-)

I was talking about the rsync file transfer, which will not compress the output on the backup server…. not about anything dovecote has configured … may be uhh he uses a filesystem compression too ? Or some rsync piping trough gzip?

kidding :)

/Götz

Kevin Laurie

15 Aug 15 Aug

8:54 a.m.

Dear everyone, I got it taken care of. Just copied the maildir to a standard FAT drive. I dont know why I was making it so complicated. Thanks tho! Appreciate it.

On Wed, Aug 12, 2015 at 12:32 AM, Götz Reinicke goetz.reinicke@filmakademie.de wrote:

...

...
Am 11.08.2015 um 17:56 schrieb Christian Schmidt Christian.Schmidt@chemie.uni-hamburg.de:

On 11.08.2015 15:45, Götz Reinicke - IT Koordinator wrote:

...
talking of rsync and compression is may be also a bit misleading.

On the destination there will be no compressed files if you transfer with rsync!

...unless your dovecot is working with compressed maildir files. ;-)

I was talking about the rsync file transfer, which will not compress the output on the backup server…. not about anything dovecote has configured … may be uhh he uses a filesystem compression too ? Or some rsync piping trough gzip?

kidding :)
    /Götz

3382

Age (days ago)

3389

Last active (days ago)

List overview

12 comments

7 participants

participants (7)

Christian Kivalo
Christian Schmidt
Felix Zielcke
Götz Reinicke
Götz Reinicke - IT Koordinator
Kevin Laurie
Steve Litt

backing up email / saving maildir on external hard drives

================================================ slitt@mydesq2:~$ df -h ~/mail/Maildir Filesystem Size Used Avail Use% Mounted on /dev/sdb8 116G 11G 100G 10% /home/slitt/mail/Maildir slitt@mydesq2:~$

================================================ slitt@mydesq2:~$ df -h ~/mail/Maildir Filesystem Size Used Avail Use% Mounted on /dev/sdb8 116G 11G 100G 10% /home/slitt/mail/Maildir slitt@mydesq2:~$

================================================ slitt@mydesq2:~$ df -h ~/mail/Maildir Filesystem Size Used Avail Use% Mounted on /dev/sdb8 116G 11G 100G 10% /home/slitt/mail/Maildir slitt@mydesq2:~$

Christian Schmidt

tags

participants (7)