I use a sliding window of time to determine what to backup, so maildir fs and database changes don't matter to me. Files can move, be deleted, whatever. Doesn't matter.

I can't use tar's incremental backups because mail has the potential to move around. That and the basename of the file has the potential to change, i.e. dovecot flags S=, etc. Its not particularly fast, as it maintains a dictionary of file: {md5sum, mtime} objects. I use find to prepare a preliminary list of what to backup depending on that aforementioned sliding window of time. I then md5 those files and compare data to determine if files moved, were deleted, etc. Then i remaster the dictionary. A bit more to it than that but that's the gist.

I have a tiny vps this runs on. Usually takes around 20-30 mins to process my monthly incrementals. Not an issue for me as the load induced is very low.

On 8/4/23 13:55, eaerhaerhaehae aehraerhaeha wrote:

1. I am more concerned about the integrity of the backup, so locking maildir and database changes during the process. I accidentally replied to aki only, but after compiling maildirlock myself it works.
2. For the backup I use rsync, to minimize the time of the lock being active. Althogh your script is probably fast, too? Doesn't tar have incremental backups built in? I have never used this feature, if it exists.
3. If a file can't be written, I would assume the file system turned read-only, which can happen sometimes, and send an external alert, not log it internally *Gesendet:* Montag, 31. Juli 2023 um 00:37 Uhr *Von:* "Ben Burk via dovecot" dovecot@dovecot.org *An:* dovecot@dovecot.org *Betreff:* Re: mail-crypt long-term support? How to backup?

I have a similar setup to you. Mail_crypt enabled, user-keys. Like Aki says, its best to use tar for this. I wrote a script a year or so back to incrementally backup my mailboxes. Curious what others think.

https://gitlab.com/ben.burk/burk.tech_personal/-/raw/master/maildirincbackup...

On 7/30/23 00:29, eaerhaerhaehae aehraerhaeha wrote:

Hello everyone,
1. I was wondering whether mail-crypt is intended to be supported
in the long run.
2. Furthermore, I was wondering about the best way to backup a
whole email server with mail-crypt enabled.
(3. how do i use dsync with mail-crypt?)
Below are my thoughts so far.
- doveadm seems to work on each user individually, and it seems
that the password is required for this? At least
tachtler/dovecot-backup complains about missing the key and
password to decrypt the mails, and so does "dsync -f -u a@bc.de
backup maildir:bc.de/a"
- I was going to just tar /var/mail, but it appears that
maildirlock is deprecated and broken, so I would have to shut down
dovecot temporarily, in order to maintain file/index integrity,
which is not ideal.
- Running it in some kind of virtualized environment to be able to
take atomic snapshots might work, but I read that the "uuid list",
or whatever, is only updated "lazily". Would a shutdown even solve
that?
- I just saw that replication is going away. Maybe it would not
have supported mail-crypt anyway.
Thanks for any input.

_______________________________________________
dovecot mailing list --dovecot@dovecot.org
To unsubscribe send an email todovecot-leave@dovecot.org

_______________________________________________ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org

Its more like shoveling a growing sand pile into a bucket. The incremental sliding window works by grabbing a time window between the last time the script stopped and when the current run started. After running, I overwrite the stopped time with the start time for the current run. These live as dotfiles. One can ascertain runtime of the script by looking at the mtime differences between the stop and start file. At the end of a run, the contents of those files should be the same datetime.

In this way I apportion changes that happen during the execution of the script to the next run.

The desired use case here is creating offsite incremental backups. I have healthcheck methods in this script as well that run on the offsite system that perform verification of incremental data before deltas are applied and verification of entire mailbox tree fs against the remastered timeline file after. It works very well in my use. I haven't had a single error or mismatch in over a year.

My use case is to make incremental backups monthly. My instance is small so I'd say the incrementals are around 100MiB monthly. Could probably cut each runtime and size down significantly if run more frequently.

I wrote all this before hearing about maildirlock. I may read about that and incorporate that as part of my solution possibly.

Ben Burk BURK.TECH System Administrator

Aug 5, 2023 6:53:34 AM eaerhaerhaehae aehraerhaeha dovecotquestion@gmx.de:

But the files could always change again, until the dictionary is remastered, and so could the database? It sounds a bit like shoveling sand back into a leaking bucket.

That's why I was surprised maildirlock is broken (including in the official repos??). I wouldn't have known how to do it without it.

...

Usually takes around 20-30 mins to process my monthly incrementals. 

Monthly??

*Gesendet:* Samstag, 05. August 2023 um 02:13 Uhr *Von:* "Ben Burk via dovecot" dovecot@dovecot.org *An:* "eaerhaerhaehae aehraerhaeha" dovecotquestion@gmx.de *Cc:* dovecot@dovecot.org *Betreff:* Re: Aw: Re: mail-crypt long-term support? How to backup?

I use a sliding window of time to determine what to backup, so maildir fs and database changes don't matter to me. Files can move, be deleted, whatever. Doesn't matter.

I can't use tar's incremental backups because mail has the potential to move around. That and the basename of the file has the potential to change, i.e. dovecot flags S=, etc. Its not particularly fast, as it maintains a dictionary of file: {md5sum, mtime} objects. I use find to prepare a preliminary list of what to backup depending on that aforementioned sliding window of time. I then md5 those files and compare data to determine if files moved, were deleted, etc. Then i remaster the dictionary. A bit more to it than that but that's the gist.

I have a tiny vps this runs on. Usually takes around 20-30 mins to process my monthly incrementals. Not an issue for me as the load induced is very low.

 

On 8/4/23 13:55, eaerhaerhaehae aehraerhaeha wrote:

1. I am more concerned about the integrity of the backup, so locking maildir and database changes during the process. I accidentally replied to aki only, but after compiling maildirlock myself it works.
2. For the backup I use rsync, to minimize the time of the lock being active. Althogh your script is probably fast, too? Doesn't tar have incremental backups built in? I have never used this feature, if it exists.
3. If a file can't be written, I would assume the file system turned read-only, which can happen sometimes, and send an external alert, not log it internally

*Gesendet:* Montag, 31. Juli 2023 um 00:37 Uhr *Von:* "Ben Burk via dovecot" dovecot@dovecot.org *An:* dovecot@dovecot.org *Betreff:* Re: mail-crypt long-term support? How to backup?

I have a similar setup to you. Mail_crypt enabled, user-keys. Like Aki says, its best to use tar for this. I wrote a script a year or so back to incrementally backup my mailboxes. Curious what others think.

https://gitlab.com/ben.burk/burk.tech_personal/-/raw/master/maildirincbackup...

 

 

On 7/30/23 00:29, eaerhaerhaehae aehraerhaeha wrote: Hello everyone,

1. I was wondering whether mail-crypt is intended to be supported in the long run.
2. Furthermore, I was wondering about the best way to backup a whole email server with mail-crypt enabled. (3. how do i use dsync with mail-crypt?) Below are my thoughts so far. 

• doveadm seems to work on each user individually, and it seems that the password is required for this? At least tachtler/dovecot-backup complains about missing the key and password to decrypt the mails, and so does "dsync -f -u a@bc.de backup maildir:bc.de/a"
• I was going to just tar /var/mail, but it appears that maildirlock is deprecated and broken, so I would have to shut down dovecot temporarily, in order to maintain file/index integrity, which is not ideal.
• Running it in some kind of virtualized environment to be able to take atomic snapshots might work, but I read that the "uuid list", or whatever, is only updated "lazily". Would a shutdown even solve that?
• I just saw that replication is going away. Maybe it would not have supported mail-crypt anyway. Thanks for any input.

---

dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org _______________________________________________ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org

_______________________________________________ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org