Am 02.12.2013 20:37, schrieb Robert Schetterer:

Am 02.12.2013 18:47, schrieb Claus:

...

Hello,

i have to backup (tape library) a mailsystem with about 300.000 Mailboxes on 2 backends. Summary of all mailboxes are 2 TByte. The mailstore is mdbox. Is it save to do a simple filesystem backup (full and incremental) with backupsoftware? hm i wouldnt do that with mdbox, cause it does strongly depends to index, but if you use dsync you could convert to maildir at backup, maildir should be fine to store on tape, but with 2 TB this is only good for total desaster archive, i guess it would take long time to restore specially single mailboxes in acceptable time, from tape Better use cheap sata storage raids for frequent backups, and do tape snapshots sometimes.

...

What is the prefered strategy to do a backup for desaster recovery (mailsystem crash) and restoring single usermailboxes? dsync should do fine

http://wiki2.dovecot.org/Tools/Dsync

but there is no universal answer to this, perhaps filesystem snapshots strategy might better, it depened on your general setup , filesystems , mailstore etc.

wait for others report their solutions, and find the one fits best at your place

...

Regards, Claus

Best Regards MfG Robert Schetterer

thanks for your answer, the main reason we plan to use mdbox is to reduce backuptime. Now we are using Maildir on cyrus and the (full)-backup takes about 20 hours on one backend storing 150 GB of Mails in about 3 million files. Because of that we have many backends, so we can parallelize the backup prozess.

greets, Claus

On 2013-12-05 6:43 AM, Claus claus.r@bayern-mail.de wrote:

Now we are using Maildir on cyrus and the (full)-backup takes about 20 hours on one backend storing 150 GB of Mails in about 3 million files.

!?

What backup software are you using?

Not to take away from the advantages of mdbox, but...

I use rsnapshot, and am pushing the backups to an NFS mount on a cheaper NAS (QNAP) device, and the initial full backup of 300+GB of maildirs (not sure how many more millions of files, but a lot more than yours I imagine) only took about 4 or 5 hours. The subsequent runs now take anywhere from 1-2 hours...

So, maybe it is your backup software that is broken?

I am also planning on migrating to mdbox, for two reasons - much less files to backup (should reduce backup snapshot times to minutes, allowing me to add many more during the day), and I also want to enable SIS to reduce dramatically the overall storage requirements (we use a lot of large attachments, and I know for a fact that they are duplicated many times over, so the overall storage should be reduced by half or more)...

--

Best regards,

*/Charles/*

Thanks for your suggestion to use maildir, but i'm sure the key problem are the large Number of small files. the backup goes first to diskstorage and afterwards to LTO4 Tapes. We backup with EMC-Networker. The backuptime of large files (some GB) are normal.

I wonder if there is no one who operates a large Mailsystem with mdbox and has suggestions to backup this?

Claus

Am 06.12.2013 11:51, schrieb Nick Edwards:

Claus, Robert's suggestion is good, we backup maildir, with 5.9TB in 2 hours to SATA , we also run a backup of that backup, so we have two extra copies (this is on top of NAS snapshots) tape is just too slow for anything serious these days.

N ik

On 12/5/13, Claus claus.r@bayern-mail.de wrote:

...

Am 02.12.2013 20:37, schrieb Robert Schetterer:

...

Am 02.12.2013 18:47, schrieb Claus:

...

Hello,

i have to backup (tape library) a mailsystem with about 300.000 Mailboxes on 2 backends. Summary of all mailboxes are 2 TByte. The mailstore is mdbox. Is it save to do a simple filesystem backup (full and incremental) with backupsoftware? hm i wouldnt do that with mdbox, cause it does strongly depends to index, but if you use dsync you could convert to maildir at backup, maildir should be fine to store on tape, but with 2 TB this is only good for total desaster archive, i guess it would take long time to restore specially single mailboxes in acceptable time, from tape Better use cheap sata storage raids for frequent backups, and do tape snapshots sometimes.

...

What is the prefered strategy to do a backup for desaster recovery (mailsystem crash) and restoring single usermailboxes? dsync should do fine

http://wiki2.dovecot.org/Tools/Dsync

but there is no universal answer to this, perhaps filesystem snapshots strategy might better, it depened on your general setup , filesystems , mailstore etc.

wait for others report their solutions, and find the one fits best at your place

...

Regards, Claus

Best Regards MfG Robert Schetterer

thanks for your answer, the main reason we plan to use mdbox is to reduce backuptime. Now we are using Maildir on cyrus and the (full)-backup takes about 20 hours on one backend storing 150 GB of Mails in about 3 million files. Because of that we have many backends, so we can parallelize the backup prozess.

greets, Claus

Am 2013-12-07 10:04, schrieb Robert Schetterer:

Am 07.12.2013 08:53, schrieb Claus:

...

Thanks for your suggestion to use maildir, but i'm sure the key problem are the large Number of small files. the backup goes first to diskstorage and afterwards to LTO4 Tapes. We backup with EMC-Networker. The backuptime of large files (some GB) are normal.

I wonder if there is no one who operates a large Mailsystem with mdbox and has suggestions to backup this?

you are able to backup and restore mdbox with dsync ( never tested ), but its not like with maildir simply i.e rsync the mailbox or one only mail so you may find some combined strategy using both formats depending to your kind of storage, as people said i wouldnt recommand backup to tape in a direct way

Thanks Robert! As i wrote the backup goes first to diskstorage and than to tape.

why not simply do some testing with only one mdbox mailbox , should be easy with dsync

...

Claus

Am 06.12.2013 11:51, schrieb Nick Edwards:

...

Claus, Robert's suggestion is good, we backup maildir, with 5.9TB in 2 hours to SATA , we also run a backup of that backup, so we have two extra copies (this is on top of NAS snapshots) tape is just too slow for anything serious these days.

N ik

On 12/5/13, Claus claus.r@bayern-mail.de wrote:

...

Am 02.12.2013 20:37, schrieb Robert Schetterer:

...

Am 02.12.2013 18:47, schrieb Claus:

...

Hello,

i have to backup (tape library) a mailsystem with about 300.000 Mailboxes on 2 backends. Summary of all mailboxes are 2 TByte. The mailstore is mdbox. Is it save to do a simple filesystem backup (full and incremental) with backupsoftware? hm i wouldnt do that with mdbox, cause it does strongly depends to index, but if you use dsync you could convert to maildir at backup, maildir should be fine to store on tape, but with 2 TB this is only good for total desaster archive, i guess it would take long time to restore specially single mailboxes in acceptable time, from tape Better use cheap sata storage raids for frequent backups, and do tape snapshots sometimes.

...

What is the prefered strategy to do a backup for desaster recovery (mailsystem crash) and restoring single usermailboxes? dsync should do fine

http://wiki2.dovecot.org/Tools/Dsync

but there is no universal answer to this, perhaps filesystem snapshots strategy might better, it depened on your general setup , filesystems , mailstore etc.

wait for others report their solutions, and find the one fits best at your place

...

Regards, Claus

Best Regards MfG Robert Schetterer

thanks for your answer, the main reason we plan to use mdbox is to reduce backuptime. Now we are using Maildir on cyrus and the (full)-backup takes about 20 hours on one backend storing 150 GB of Mails in about 3 million files. Because of that we have many backends, so we can parallelize the backup prozess.

greets, Claus

Best Regards MfG Robert Schetterer