[Dovecot] mdbox + gzip and rsync
Hi,
After reading the following paragraph from the dovecot doc, I've been wondering how it would affect rsync (when combined with gzip):
"Expunging a message only decreases the message's refcount. The space is later freed in "purge" step. This is typically done in a nightly cronjob when there's less disk I/O activity. The purging first finds all files that have refcount=0 mails. Then it goes through each file and copies the refcount>0 mails to other mdbox files (to the same files as where newly saved messages would also go), updates the map index and finally deletes the original file. So there is never any overwriting or file truncation."
How will the mailbox files (m.X) files be modified when I move or delete emails using mdbox+gzip. Will the resulting gzipped mdbox files be rsync-able or will they need a full re-upload?
If I plan on using rsync for backups, am I better off not using the gzip feature (if i can spare the extra storage)???
Thanks,
-JD
On 7.3.2012, at 5.19, Jean-Daniel Beaubien wrote:
After reading the following paragraph from the dovecot doc, I've been wondering how it would affect rsync (when combined with gzip):
"Expunging a message only decreases the message's refcount. The space is later freed in "purge" step. This is typically done in a nightly cronjob when there's less disk I/O activity. The purging first finds all files that have refcount=0 mails. Then it goes through each file and copies the refcount>0 mails to other mdbox files (to the same files as where newly saved messages would also go), updates the map index and finally deletes the original file. So there is never any overwriting or file truncation."
How will the mailbox files (m.X) files be modified when I move or delete emails using mdbox+gzip. Will the resulting gzipped mdbox files be rsync-able or will they need a full re-upload?
If I plan on using rsync for backups, am I better off not using the gzip feature (if i can spare the extra storage)???
gzipping is irrelevant, the behavior is the same with and without gzip. The purging step recreates new mail files, so the new files will need to be fully uploaded with rsync. You might want to consider using dsync instead.
Jean-Daniel Beaubien <jd.beaubien@gmail.com> writes:
After reading the following paragraph from the dovecot doc, I've been wondering how it would affect rsync (when combined with gzip):
"Expunging a message only decreases the message's refcount. The space is later freed in "purge" step. This is typically done in a nightly cronjob when there's less disk I/O activity. The purging first finds all files that have refcount=0 mails. Then it goes through each file and copies the refcount>0 mails to other mdbox files (to the same files as where newly saved messages would also go), updates the map index and finally deletes the original file. So there is never any overwriting or file truncation."
Interesting, so it would be recommended to those using mdbox format to run a 'dovadm purge -A' every night to clean up these unused files? It seems like without this, mail storage usage will just grow infinitely.
It does appear that using an rsync backup process for mdbox would not be able to detect this and backups would also grow infinitely.
micah
--
On 12.3.2012, at 17.10, Micah Anderson wrote:
Jean-Daniel Beaubien <jd.beaubien@gmail.com> writes:
After reading the following paragraph from the dovecot doc, I've been wondering how it would affect rsync (when combined with gzip):
"Expunging a message only decreases the message's refcount. The space is later freed in "purge" step. This is typically done in a nightly cronjob when there's less disk I/O activity. The purging first finds all files that have refcount=0 mails. Then it goes through each file and copies the refcount>0 mails to other mdbox files (to the same files as where newly saved messages would also go), updates the map index and finally deletes the original file. So there is never any overwriting or file truncation."
Interesting, so it would be recommended to those using mdbox format to run a 'dovadm purge -A' every night to clean up these unused files? It seems like without this, mail storage usage will just grow infinitely.
Yes.
It does appear that using an rsync backup process for mdbox would not be able to detect this and backups would also grow infinitely.
rsync --delete would delete the old files, right? Anyway, I'd avoid using rsync for mdbox unless you're doing it on a filesystem snapshot. dsync backup should work better.
participants (3)
-
Jean-Daniel Beaubien
-
Micah Anderson
-
Timo Sirainen