dsync and altpath on shared storage.
Hi,
I was wondering...
If one had mdbox ALT path set to a shared storage mount (say, on NFS) and one wanted to move a mailbox to a different host... I guess it in principle wouldn't be necessary to copy all the ALT storage through dsync, when the volume could just be mounted on the new host.
Is there anyway for dsync to avoid moving Gigabytes of data for could just be "moved" by moving the mount?
/Peter
On 2 Sep 2019, at 15.25, Peter Mogensen via dovecot dovecot@dovecot.org wrote:
Hi,
I was wondering...
If one had mdbox ALT path set to a shared storage mount (say, on NFS) and one wanted to move a mailbox to a different host... I guess it in principle wouldn't be necessary to copy all the ALT storage through dsync, when the volume could just be mounted on the new host.
Is there anyway for dsync to avoid moving Gigabytes of data for could just be "moved" by moving the mount?
Not tested but you can probably do something like this in the target server:
doveadm backup -u victim -R ssh sudouser@old-server "sudo doveadm dsync-server -o mail_location=sdbox:/location-to-your-sdbox/ -u victim"
just leave ALT storage path from the setting.
Sami
On 9/2/19 3:03 PM, Sami Ketola wrote:
On 2 Sep 2019, at 15.25, Peter Mogensen via dovecot dovecot@dovecot.org wrote: ... Is there anyway for dsync to avoid moving Gigabytes of data for could just be "moved" by moving the mount?
Not tested but you can probably do something like this in the target server:
doveadm backup -u victim -R ssh sudouser@old-server "sudo doveadm dsync-server -o mail_location=sdbox:/location-to-your-sdbox/ -u victim"
just leave ALT storage path from the setting.
I'll have to test this... but my initial guess would be that doveadm would then think the mails has disappeared. Would it then copy the index metadata for those mails to the target host anyway?
/Peter
On 3 Sep 2019, at 15.34, Peter Mogensen via dovecot dovecot@dovecot.org wrote:
On 9/2/19 3:03 PM, Sami Ketola wrote:
On 2 Sep 2019, at 15.25, Peter Mogensen via dovecot dovecot@dovecot.org wrote: ... Is there anyway for dsync to avoid moving Gigabytes of data for could just be "moved" by moving the mount?
Not tested but you can probably do something like this in the target server:
doveadm backup -u victim -R ssh sudouser@old-server "sudo doveadm dsync-server -o mail_location=sdbox:/location-to-your-sdbox/ -u victim"
just leave ALT storage path from the setting.
I'll have to test this... but my initial guess would be that doveadm would then think the mails has disappeared. Would it then copy the index metadata for those mails to the target host anyway?
Hmm. That is true. It will probably not work after all then.
Now I'm out of ideas how to do this efficiently.
Sami
On 9/3/19 2:38 PM, Sami Ketola wrote:
On 3 Sep 2019, at 15.34, Peter Mogensen via dovecot dovecot@dovecot.org wrote:
On 9/2/19 3:03 PM, Sami Ketola wrote:
On 2 Sep 2019, at 15.25, Peter Mogensen via dovecot dovecot@dovecot.org wrote: ... Is there anyway for dsync to avoid moving Gigabytes of data for could just be "moved" by moving the mount?
Not tested but you can probably do something like this in the target server:
doveadm backup -u victim -R ssh sudouser@old-server "sudo doveadm dsync-server -o mail_location=sdbox:/location-to-your-sdbox/ -u victim"
just leave ALT storage path from the setting.
I'll have to test this... but my initial guess would be that doveadm would then think the mails has disappeared. Would it then copy the index metadata for those mails to the target host anyway?
Hmm. That is true. It will probably not work after all then.
Now I'm out of ideas how to do this efficiently.
I assume it won't even work to just premount the shared storage read-only on the target side, so the mails are already there. ... since I suppose the receiving dsync reserves the right to re-pack the m.* storage files?
/Peter
So... I've done some testing.
One method which seemed to work - at least for primitive cases - was to:
- Mount the ALT storage on the destination.
- Run "doveadm force-resync \*" on the destination. (putting all the mails in ALT storage into the dovecot.map.index)
- Run dsync from source to destination.
Of course... if there was some way to avoid step 2...
/Peter
On 9/4/19 2:12 PM, Peter Mogensen wrote:
So... I've done some testing.
One method which seemed to work - at least for primitive cases - was to:
- Mount the ALT storage on the destination.
- Run "doveadm force-resync \*" on the destination. (putting all the mails in ALT storage into the dovecot.map.index)
- Run dsync from source to destination.
Of course... if there was some way to avoid step 2...
So ... I have an idea.
Assuming users mail_location is:
mdbox:~/mdbox:ALT=/alt:INDEX=~/idx
And /alt is a shard mounted storage.
Then, it suspect the following steps would make dsync avoid transfering mails on shared storage:
Create a rudimentary mdbox on the target side (just containing the dbox-alt-root link)
Mount /alt on the target host
Copy all dovecot.index and dovecot.map.index in ~/idx from source to target. That is: not the transaction (*.log) files or cache files. I suppose this needs to be done under appropriate read locking.
doveadm sync -u source doveadm dsync-server -u target ... to get the rest of the mails in primary storage and all updates sine the index files where snapshot.
It would be nice if there was a way to force dovecot*index.log files to be snapshot to index files.
If the aim is not to sync two different accounts but to simply move one account from one host to a new host where it doesn't exist in advance, are there any caveats with this?
... apart from a few missing tools.
/Peter
participants (2)
-
Peter Mogensen
-
Sami Ketola