doveadm sync backup from old to new server
So I've done quite a lot of searching on the list and on the web - and perhaps my google-fu is really bad - but I can't find any real recipes on how to sync mail from the old server to the new.
As an FYI - the old server is a CPanel/WHM setup on a VPS. The new is mailcow - which uses docker.
However, I don't think either of these platforms is what's causing the issue - but I'm certainly not sure of that.
I've tried several things - but have lost track of all the things I've tried. This seemed like the best of all the things I've tried.
This particular mailbox/user+domain is setup on both servers.
doveadm backup -D -u mc-user@abc.net ssh root@abc.net -p2200 doveadm dsync-server -u cp-user@abc.net
mc-user@abc.net is the MC/NEW mailbox/domain cp-user@abc.net is the CPanel/OLD user/domain account The SSH server of the remote system is running on port 2200.
However when I try this, I get: WARNING: The WATCHDOG_NOTIFY_EMAIL variable is not set. Defaulting to a blank string. dsync-local(user@abc.net)<v44NFyRWvF5efgAAb6OASA>: Error: read(remote) failed: EOF (version not received) doveadm(user@abc.net): Fatal: execvp(ssh) failed: No such file or directory
The process appears to hang, and a Ctrl+C stops it.
I'd love to get pointed at a reasonable recipe on how to make this work.
I don't really get/understand the docs much at all. [And either everyone else understands it just fine, and never thinks to write a document on how to do it - or, and I think this is a lot more likely - they're using something like imapsync to do it. I found numerous places where others were, essentially, "Dovecot's tool is way too complicated and I can't get it to work right, so I used imapsync." I suppose I should probably just do that too, but it does seem a shame to do that when the dovecot tool is almost certainly the best tool for the job, but I can't figure out how to use it.]
If someone can help me grok what's going on, I'm glad to write it up for the list and or a blog entry so it's more accessible.
TIA -Greg
I struggled with this at one time, and with the lack of examples around the internet. In fact, I might have written this very post myself at one point! In the end, the penny dropped when I saw that the format was
(command on local server) + (stuff you need to connect over ssh) + (command on remote server)
So what worked for me was:
doveadm backup -u user@domain.com
ssh -p 2222 (+ other ssh options) root@remoteserver.com
doveadm dsync-server -u user2@domain2.com
If you get dsync-remote(user2@domain2.com): Error: Mailbox INBOX sync: mailbox_delete failed: INBOX can't be deleted ... you'll need to clear the remote directory first! Or maybe try sync instead of backup?
I found that running as root on the remote server was necessary in my case, due to the permissions on the remote directory. You might want to check permissions on remote directory are writable by the user in your ssh command.
If still no joy you can run doveadm user user@domain.com on your local server and doveadm user user2@domain2.com on your remote server and dovecot will tell you where its synching from and to.
Finally, this solution is a 'push' from source server to target. You may 'pull' the other way if that makes more sense in your environment with -R
Those pieces were enough for me to get it to work ...
P.
PS. For the record, on the original job I was under time pressure, and yes, I did use imapsync to get it done in the end. I got this to work later on when I had more time to tinker.
On 14/05/2020 09.09, Gregory Sloop wrote:
So I've done quite a lot of searching on the list and on the web - and perhaps my google-fu is really bad - but I can't find any real recipes on how to sync mail from the old server to the new.
As an FYI - the old server is a CPanel/WHM setup on a VPS. The new is mailcow - which uses docker.
However, I don't think either of these platforms is what's causing the issue - but I'm certainly not sure of that.
I've tried several things - but have lost track of all the things I've tried. This seemed like the best of all the things I've tried.
This particular mailbox/user+domain is setup on both servers.
doveadm backup -D -u mc-user@abc.net ssh root@abc.net -p2200 doveadm dsync-server -u cp-user@abc.net
mc-user@abc.net is the MC/NEW mailbox/domain cp-user@abc.net is the CPanel/OLD user/domain account The SSH server of the remote system is running on port 2200.
However when I try this, I get: WARNING: The WATCHDOG_NOTIFY_EMAIL variable is not set. Defaulting to a blank string. dsync-local(user@abc.net)<v44NFyRWvF5efgAAb6OASA>: Error: read(remote) failed: EOF (version not received) doveadm(user@abc.net): Fatal: execvp(ssh) failed: No such file or directory
The process appears to hang, and a Ctrl+C stops it.
I'd love to get pointed at a reasonable recipe on how to make this work.
I don't really get/understand the docs much at all. [And either everyone else understands it just fine, and never thinks to write a document on how to do it - or, and I think this is a lot more likely - they're using something like imapsync to do it. I found numerous places where others were, essentially, "Dovecot's tool is way too complicated and I can't get it to work right, so I used imapsync." I suppose I should probably just do that too, but it does seem a shame to do that when the dovecot tool is almost certainly the best tool for the job, but I can't figure out how to use it.]
If someone can help me grok what's going on, I'm glad to write it up for the list and or a blog entry so it's more accessible.
TIA -Greg
So that was really helpful for me to understand that a lot more clearly. Thanks! [Many, many thanks @Plutocrat!!]
But I'm still getting a similar failure. Let me give the command I'm using.
doveadm backup -D -u user@abc.net
ssh root@po.abc.net
docker exec b3093cxxxxxx doveadm sync -D -u user@abc.net
The "local" server is where the data/mail currently is. The remote or docker container/volume is where dovecot is installed. (The data is in a different docker container.)
So, I think I'm "backing up" the data from the local machine and pushing that "backed up" data via SSH to the dovecot install in docker and attempting to "sync" that data to the remote dovecot install.
However, I immediately get "Error: read(remote) failed: EOF (version not received)"
The connection hangs. If I wait a while - perhaps 30-60s, it kills the SSH connection and aborts the process.
Is there a way I can test the remote end's ability to accept the data.
[i.e. Can I do something like
ssh root@po.abc.net
docker exec b3093cxxxxxx doveadm sync -D -u user@abc.net
And see if it would accept the data. [In short, do I have a local end problem or a remote end problem - and being able to test both parts individually would probably help me figure out what's broken. That still won't fix it, but at least I'll know which end I need to concentrate on.]
I'd guess this will all seem pretty obvious in retrospect, but for the life of me, I'm completely lost and really don't have any idea where to start to break it down better so I can see how each piece is working or not.
TIA -Greg
P> I struggled with this at one time, and with the lack of examples P> around the internet. In fact, I might have written this very post P> myself at one point! In the end, the penny dropped when I saw that the format was
P> (command on local server) + (stuff you need to connect over ssh) + (command on remote server)
P> So what worked for me was:
P> doveadm backup -u user@domain.com
P> ssh -p 2222 (+ other ssh options) root@remoteserver.com
P> doveadm dsync-server -u user2@domain2.com
P> If you get P> dsync-remote(user2@domain2.com): Error: Mailbox INBOX sync: P> mailbox_delete failed: INBOX can't be deleted P> ... you'll need to clear the remote directory first! Or maybe try sync instead of backup?
P> I found that running as root on the remote server was necessary in P> my case, due to the permissions on the remote directory. You might P> want to check permissions on remote directory are writable by the user in your ssh command.
P> If still no joy you can run P> doveadm user user@domain.com on your local server and P> doveadm user user2@domain2.com on your remote server P> and dovecot will tell you where its synching from and to.
P> Finally, this solution is a 'push' from source server to target. P> You may 'pull' the other way if that makes more sense in your environment with -R
P> Those pieces were enough for me to get it to work ...
P> P.
P> PS. For the record, on the original job I was under time pressure, P> and yes, I did use imapsync to get it done in the end. I got this P> to work later on when I had more time to tinker.
P> On 14/05/2020 09.09, Gregory Sloop wrote:
So I've done quite a lot of searching on the list and on the web - and perhaps my google-fu is really bad - but I can't find any real recipes on how to sync mail from the old server to the new.
As an FYI - the old server is a CPanel/WHM setup on a VPS. The new is mailcow - which uses docker.
However, I don't think either of these platforms is what's causing the issue - but I'm certainly not sure of that.
I've tried several things - but have lost track of all the things I've tried. This seemed like the best of all the things I've tried.
This particular mailbox/user+domain is setup on both servers.
doveadm backup -D -u mc-user@abc.net ssh root@abc.net -p2200 doveadm dsync-server -u cp-user@abc.net
mc-user@abc.net is the MC/NEW mailbox/domain cp-user@abc.net is the CPanel/OLD user/domain account The SSH server of the remote system is running on port 2200.
However when I try this, I get: WARNING: The WATCHDOG_NOTIFY_EMAIL variable is not set. Defaulting to a blank string. dsync-local(user@abc.net)<v44NFyRWvF5efgAAb6OASA>: Error: read(remote) failed: EOF (version not received) doveadm(user@abc.net): Fatal: execvp(ssh) failed: No such file or directory
The process appears to hang, and a Ctrl+C stops it.
I'd love to get pointed at a reasonable recipe on how to make this work.
I don't really get/understand the docs much at all. [And either everyone else understands it just fine, and never thinks to write a document on how to do it - or, and I think this is a lot more likely - they're using something like imapsync to do it. I found numerous places where others were, essentially, "Dovecot's tool is way too complicated and I can't get it to work right, so I used imapsync." I suppose I should probably just do that too, but it does seem a shame to do that when the dovecot tool is almost certainly the best tool for the job, but I can't figure out how to use it.]
If someone can help me grok what's going on, I'm glad to write it up for the list and or a blog entry so it's more accessible.
TIA -Greg
-- Gregory Sloop, Principal: Sloop Network & Computer Consulting Voice: 503.251.0452 x121 EMail: gregs@sloop.net http://www.sloop.net
I'm not clear myself on the difference myself between
dsync
doveadm sync
doveadm dsync-server
... perhaps someone here can explain? However what worked for me was using doveadm dsync-server on the 'receive' end. Maybe that puts it in 'listen' mode? So maybe try
doveadm backup -u user@abc.net
ssh root@po.abc.net
docker exec b3093cxxxxxx doveadm dsync-server -u user@abc.net
I've tried to get along with docker before, but it always just seems to add another level of complexity into everything, so thus far I've managed to avoid it apart from general experiments!
P.
On 15/05/2020 07.00, Gregory Sloop wrote:
So that was really helpful for me to understand that a lot more clearly. Thanks! [Many, many thanks @Plutocrat!!]
But I'm still getting a similar failure. Let me give the command I'm using.
doveadm backup -D -u user@abc.net
ssh root@po.abc.net
docker exec b3093cxxxxxx doveadm sync -D -u user@abc.netThe "local" server is where the data/mail currently is. The remote or docker container/volume is where dovecot is installed. (The data is in a different docker container.)
So, I think I'm "backing up" the data from the local machine and pushing that "backed up" data via SSH to the dovecot install in docker and attempting to "sync" that data to the remote dovecot install.
However, I immediately get "Error: read(remote) failed: EOF (version not received)"
The connection hangs. If I wait a while - perhaps 30-60s, it kills the SSH connection and aborts the process.
Is there a way I can test the remote end's ability to accept the data. [i.e. Can I do something like ssh root@po.abc.net
docker exec b3093cxxxxxx doveadm sync -D -u user@abc.netAnd see if it would accept the data. [In short, do I have a local end problem or a remote end problem - and being able to test both parts individually would probably help me figure out what's broken. That still won't fix it, but at least I'll know which end I need to concentrate on.]
I'd guess this will all seem pretty obvious in retrospect, but for the life of me, I'm completely lost and really don't have any idea where to start to break it down better so I can see how each piece is working or not.
TIA -Greg
*P> I struggled with this at one time, and with the lack of examples P> around the internet. In fact, I might have written this very post P> myself at one point! In the end, the penny dropped when I saw that the format was
P> (command on local server) + (stuff you need to connect over ssh) + (command on remote server)
P> So what worked for me was:
P> doveadm backup -u *user@domain.com mailto:user@domain.com*
P> ssh -p 2222 (+ other ssh options) *root@remoteserver.com mailto:root@remoteserver.com*
P> doveadm dsync-server -u *user2@domain2.com mailto:user2@domain2.com*P> If you get P> dsync-remote(*user2@domain2.com mailto:user2@domain2.com*): Error: Mailbox INBOX sync: P> mailbox_delete failed: INBOX can't be deleted P> ... you'll need to clear the remote directory first! Or maybe try sync instead of backup?
P> I found that running as root on the remote server was necessary in P> my case, due to the permissions on the remote directory. You might P> want to check permissions on remote directory are writable by the user in your ssh command.
P> If still no joy you can run P> doveadm user *user@domain.com mailto:user@domain.com* on your local server and P> doveadm user *user2@domain2.com mailto:user2@domain2.com* on your remote server P> and dovecot will tell you where its synching from and to.
P> Finally, this solution is a 'push' from source server to target. P> You may 'pull' the other way if that makes more sense in your environment with -R
P> Those pieces were enough for me to get it to work ...
P> P.
P> PS. For the record, on the original job I was under time pressure, P> and yes, I did use imapsync to get it done in the end. I got this P> to work later on when I had more time to tinker.
P> On 14/05/2020 09.09, Gregory Sloop wrote:
So I've done quite a lot of searching on the list and on the web - and perhaps my google-fu is really bad - but I can't find any real recipes on how to sync mail from the old server to the new.
As an FYI - the old server is a CPanel/WHM setup on a VPS. The new is mailcow - which uses docker.
However, I don't think either of these platforms is what's causing the issue - but I'm certainly not sure of that.
I've tried several things - but have lost track of all the things I've tried. This seemed like the best of all the things I've tried.
This particular mailbox/user+domain is setup on both servers.
doveadm backup -D -u *mc-user@abc.net mailto:mc-user@abc.net* ssh *root@abc.net mailto:root@abc.net* -p2200 doveadm dsync-server -u *cp-user@abc.net mailto:cp-user@abc.net
mc-user@abc.net mailto:mc-user@abc.net* is the MC/NEW mailbox/domain *>> cp-user@abc.net mailto:cp-user@abc.net* is the CPanel/OLD user/domain account The SSH server of the remote system is running on port 2200.
However when I try this, I get: WARNING: The WATCHDOG_NOTIFY_EMAIL variable is not set. Defaulting to a blank string. dsync-local(*user@abc.net mailto:user@abc.net*)<v44NFyRWvF5efgAAb6OASA>: Error: read(remote) failed: EOF (version not received) doveadm(*user@abc.net mailto:user@abc.net*): Fatal: execvp(ssh) failed: No such file or directory
The process appears to hang, and a Ctrl+C stops it.
I'd love to get pointed at a reasonable recipe on how to make this work.
I don't really get/understand the docs much at all. [And either everyone else understands it just fine, and never thinks to write a document on how to do it - or, and I think this is a lot more likely - they're using something like imapsync to do it. I found numerous places where others were, essentially, "Dovecot's tool is way too complicated and I can't get it to work right, so I used imapsync." I suppose I should probably just do that too, but it does seem a shame to do that when the dovecot tool is almost certainly the best tool for the job, but I can't figure out how to use it.]
If someone can help me grok what's going on, I'm glad to write it up for the list and or a blog entry so it's more accessible.
TIA -Greg
*/-- Gregory Sloop, Principal: Sloop Network & Computer Consulting Voice: 503.251.0452 x121 EMail: /gregs@sloop.net mailto:gregs@sloop.net http://www.sloop.net /--- /
Yeah, I tried that already - and it seems to crash more badly...
dsync-remote(use@abc.net)<mTEPAAUCvl4yFgAAb6OASA>: Panic: epoll_ctl(add, 0) failed: Operation not permitted (fd doesn't support epoll - instead of '
I found several other reports that seem similar, some on Ubuntu 18.04 which match the new platform I'm using.
And I couldn't even find details in the manpage for dsync-server - so I was totally lost - even worse than I was before. [Blind AND LOST, instead of just lost.]
doveadm sync *is* documented, so I figured it was "safer" to use.
Sigh. IMAP sync looks better and better. I'm a little puzzled why I don't just give up walking on the broken glass and use it - but it just bugs the heck out of me.
This whole process makes me feel like a total moron. I know it's not true - but the docs, and such, around doveadm sync are so obtuse and it's so non-obvious what's wrong, or what one might do to right things - well, it is hard not to feel like you ought to know, but just can't crack the davinci code or something...
Thanks again!
Hopefully someone Aki/Timo et al. will pipe up with something helpful soon. Otherwise, I guess doveadm sync/backup will continue to be unused by any mere mortals.
-Greg
P> I'm not clear myself on the difference myself between
P> dsync
P> doveadm sync
P> doveadm dsync-server
P> ... perhaps someone here can explain? However what worked for me
P> was using doveadm dsync-server on the 'receive' end. Maybe that
P> puts it in 'listen' mode? So maybe try
P> doveadm backup -u user@abc.net
P> ssh root@po.abc.net
P> docker exec b3093cxxxxxx doveadm dsync-server -u user@abc.net
P> I've tried to get along with docker before, but it always just P> seems to add another level of complexity into everything, so thus P> far I've managed to avoid it apart from general experiments!
P> P.
P> On 15/05/2020 07.00, Gregory Sloop wrote:
So that was really helpful for me to understand that a lot more clearly. Thanks! [Many, many thanks @Plutocrat!!]
But I'm still getting a similar failure. Let me give the command I'm using.
doveadm backup -D -u user@abc.net
ssh root@po.abc.net
docker exec b3093cxxxxxx doveadm sync -D -u user@abc.net
The "local" server is where the data/mail currently is. The remote or docker container/volume is where dovecot is installed. (The data is in a different docker container.)
So, I think I'm "backing up" the data from the local machine and pushing that "backed up" data via SSH to the dovecot install in docker and attempting to "sync" that data to the remote dovecot install.
However, I immediately get "Error: read(remote) failed: EOF (version not received)"
The connection hangs. If I wait a while - perhaps 30-60s, it kills the SSH connection and aborts the process.
Is there a way I can test the remote end's ability to accept the data. [i.e. Can I do something like ssh root@po.abc.net
docker exec b3093cxxxxxx doveadm sync -D -u user@abc.net
And see if it would accept the data. [In short, do I have a local end problem or a remote end problem - and being able to test both parts individually would probably help me figure out what's broken. That still won't fix it, but at least I'll know which end I need to concentrate on.]
I'd guess this will all seem pretty obvious in retrospect, but for the life of me, I'm completely lost and really don't have any idea where to start to break it down better so I can see how each piece is working or not.
TIA -Greg
*P> I struggled with this at one time, and with the lack of examples P> around the internet. In fact, I might have written this very post P> myself at one point! In the end, the penny dropped when I saw that the format was
P> (command on local server) + (stuff you need to connect over ssh) + (command on remote server)
P> So what worked for me was:
P> doveadm backup -u *user@domain.com mailto:user@domain.com*
P> ssh -p 2222 (+ other ssh options) *root@remoteserver.com mailto:root@remoteserver.com*
P> doveadm dsync-server -u *user2@domain2.com mailto:user2@domain2.com*
P> If you get P> dsync-remote(*user2@domain2.com mailto:user2@domain2.com*): Error: Mailbox INBOX sync: P> mailbox_delete failed: INBOX can't be deleted P> ... you'll need to clear the remote directory first! Or maybe try sync instead of backup?
P> I found that running as root on the remote server was necessary in P> my case, due to the permissions on the remote directory. You might P> want to check permissions on remote directory are writable by the user in your ssh command.
P> If still no joy you can run P> doveadm user *user@domain.com mailto:user@domain.com* on your local server and P> doveadm user *user2@domain2.com mailto:user2@domain2.com* on your remote server P> and dovecot will tell you where its synching from and to.
P> Finally, this solution is a 'push' from source server to target. P> You may 'pull' the other way if that makes more sense in your environment with -R
P> Those pieces were enough for me to get it to work ...
P> P.
P> PS. For the record, on the original job I was under time pressure, P> and yes, I did use imapsync to get it done in the end. I got this P> to work later on when I had more time to tinker.
P> On 14/05/2020 09.09, Gregory Sloop wrote:
So I've done quite a lot of searching on the list and on the web - and perhaps my google-fu is really bad - but I can't find any real recipes on how to sync mail from the old server to the new.
As an FYI - the old server is a CPanel/WHM setup on a VPS. The new is mailcow - which uses docker.
However, I don't think either of these platforms is what's causing the issue - but I'm certainly not sure of that.
I've tried several things - but have lost track of all the things I've tried. This seemed like the best of all the things I've tried.
This particular mailbox/user+domain is setup on both servers.
doveadm backup -D -u *mc-user@abc.net mailto:mc-user@abc.net* ssh *root@abc.net mailto:root@abc.net* -p2200 doveadm dsync-server -u *cp-user@abc.net mailto:cp-user@abc.net
mc-user@abc.net mailto:mc-user@abc.net* is the MC/NEW mailbox/domain *>> cp-user@abc.net mailto:cp-user@abc.net* is the CPanel/OLD user/domain account The SSH server of the remote system is running on port 2200.
However when I try this, I get: WARNING: The WATCHDOG_NOTIFY_EMAIL variable is not set. Defaulting to a blank string. dsync-local(*user@abc.net mailto:user@abc.net*)<v44NFyRWvF5efgAAb6OASA>: Error: read(remote) failed: EOF (version not received) doveadm(*user@abc.net mailto:user@abc.net*): Fatal: execvp(ssh) failed: No such file or directory
The process appears to hang, and a Ctrl+C stops it.
I'd love to get pointed at a reasonable recipe on how to make this work.
I don't really get/understand the docs much at all. [And either everyone else understands it just fine, and never thinks to write a document on how to do it - or, and I think this is a lot more likely - they're using something like imapsync to do it. I found numerous places where others were, essentially, "Dovecot's tool is way too complicated and I can't get it to work right, so I used imapsync." I suppose I should probably just do that too, but it does seem a shame to do that when the dovecot tool is almost certainly the best tool for the job, but I can't figure out how to use it.]
If someone can help me grok what's going on, I'm glad to write it up for the list and or a blog entry so it's more accessible.
TIA -Greg
*/-- Gregory Sloop, Principal: Sloop Network & Computer Consulting Voice: 503.251.0452 x121 EMail: /gregs@sloop.net mailto:gregs@sloop.net http://www.sloop.net /--- /
-- Gregory Sloop, Principal: Sloop Network & Computer Consulting Voice: 503.251.0452 x121 EMail: gregs@sloop.net http://www.sloop.net
On 15/05/2020 10.56, Gregory Sloop wrote:
IMAP sync looks better and better. I'm a little puzzled why I don't just give up walking on the broken glass and use it - but it just bugs the heck out of me.
If you can get the dovecot native sync to work, it seems to be a better option, as it understands all the dovecot index formats etc, and will transfer those correctly. Imapsync doesn't, and when I did that I had a few problems with POP email clients re-downloading their entire Inbox, which produced a few groans. Your mileage may vary.
Another option for you, if you have the time and energy would be to set up proper replication between the two servers. Then any changes made would be synced in real time, until the cutoff, when you just shut down the old server and the new one is already up to date. But if you can't get a simple sync happening then that might be a bit of over-reach.
My suspicions on your current situation would center around the complications introduced by docker, and on file/container permissions, environment variables, paths etc.
How about if you just try a simple doveadm command over ssh from the remote server to make sure that works. eg. from "Sending" server do
ssh root@po.abc.net
docker exec b3093cxxxxxx doveadm user user@abc.net
That should execute the doveadm user command on the remote (receiving) server. If that works, then at least you can satisfy yourself that the docker bit is working correctly. Not sure why I'm getting involved in someone else's troubleshooting nightmare! ;-)
P.
On 15/05/2020 07:10 Plutocrat plutocrat@gmail.com wrote:
On 15/05/2020 10.56, Gregory Sloop wrote:
IMAP sync looks better and better. I'm a little puzzled why I don't just give up walking on the broken glass and use it - but it just bugs the heck out of me.
If you can get the dovecot native sync to work, it seems to be a better option, as it understands all the dovecot index formats etc, and will transfer those correctly. Imapsync doesn't, and when I did that I had a few problems with POP email clients re-downloading their entire Inbox, which produced a few groans. Your mileage may vary.
Another option for you, if you have the time and energy would be to set up proper replication between the two servers. Then any changes made would be synced in real time, until the cutoff, when you just shut down the old server and the new one is already up to date. But if you can't get a simple sync happening then that might be a bit of over-reach.
My suspicions on your current situation would center around the complications introduced by docker, and on file/container permissions, environment variables, paths etc.
How about if you just try a simple doveadm command over ssh from the remote server to make sure that works. eg. from "Sending" server do
ssh root@po.abc.net
docker exec b3093cxxxxxx doveadm user user@abc.netThat should execute the doveadm user command on the remote (receiving) server. If that works, then at least you can satisfy yourself that the docker bit is working correctly. Not sure why I'm getting involved in someone else's troubleshooting nightmare! ;-)
P.
You need to have ssh installed. If you want to do sync over ssh. The docker image hasn't got ssh client installed by default.
Also see https://wiki.dovecot.org/Replication for more configuration options.
Aki
On 15/05/2020 07:10 Plutocrat plutocrat@gmail.com wrote:
On 15/05/2020 10.56, Gregory Sloop wrote:
IMAP sync looks better and better. I'm a little puzzled why I don't just give up walking on the broken glass and use it - but it just bugs the heck out of me.
If you can get the dovecot native sync to work, it seems to be a better option, as it understands all the dovecot index formats etc, and will transfer those correctly. Imapsync doesn't, and when I did that I had a few problems with POP email clients re-downloading their entire Inbox, which produced a few groans. Your mileage may vary.
Another option for you, if you have the time and energy would be to set up proper replication between the two servers. Then any changes made would be synced in real time, until the cutoff, when you just shut down the old server and the new one is already up to date. But if you can't get a simple sync happening then that might be a bit of over-reach.
My suspicions on your current situation would center around the complications introduced by docker, and on file/container permissions, environment variables, paths etc.
How about if you just try a simple doveadm command over ssh from the remote server to make sure that works. eg. from "Sending" server do
ssh root@po.abc.net
docker exec b3093cxxxxxx doveadm user user@abc.net
That should execute the doveadm user command on the remote (receiving) server. If that works, then at least you can satisfy yourself that the docker bit is working correctly. Not sure why I'm getting involved in someone else's troubleshooting nightmare! ;-)
P.
AT> You need to have ssh installed. If you want to do sync over ssh. AT> The docker image hasn't got ssh client installed by default.
AT> Also see https://wiki.dovecot.org/Replication for more configuration options.
SSH *is* installed
# ssh root@abc.net \
docker exec b3093c6200a9 ls -al
root@abc.net's password: total 124 drwxr-xr-x 1 root root 4096 Apr 15 17:30 . drwxr-xr-x 1 root root 4096 Apr 15 17:30 .. -rwxr-xr-x 1 root root 0 Apr 15 17:29 .dockerenv drwxr-xr-x 1 root root 4096 Apr 14 01:23 bin drwxr-xr-x 2 root root 4096 Feb 1 09:09 boot drwxr-xr-x 5 root root 380 May 12 11:25 dev -rwxr-xr-x 1 root root 14951 Apr 14 08:31 docker-entrypoint.sh drwxr-xr-x 1 root root 4096 Apr 15 17:30 etc drwxr-xr-x 2 root root 4096 Feb 1 09:09 home drwxr-xr-x 2 root root 4096 Apr 15 17:30 hooks drwxr-xr-x 1 root root 4096 Apr 14 01:23 lib drwxr-xr-x 2 root root 4096 Mar 26 17:00 lib64 drwxr-xr-x 2 root root 4096 Apr 15 17:30 mail_crypt drwxr-xr-x 2 root root 4096 Mar 26 17:00 media drwxr-xr-x 2 root root 4096 Mar 26 17:00 mnt drwxr-xr-x 2 root root 4096 Mar 26 17:00 opt dr-xr-xr-x 232 root root 0 May 12 11:25 proc drwx------ 1 root root 4096 Apr 14 01:23 root drwxr-xr-x 1 root root 4096 May 12 11:25 run drwxr-xr-x 1 root root 4096 Apr 14 01:23 sbin drwxr-xr-x 2 root root 4096 Mar 26 17:00 srv -rw-r--r-- 1 root root 12126 May 14 15:48 supervisord.log dr-xr-xr-x 13 root root 0 May 12 11:25 sys drwxr-xr-x 2 root root 4096 Apr 15 17:21 templates drwxrwxrwt 1 root root 4096 May 14 21:44 tmp drwxr-xr-x 1 root root 4096 Mar 26 17:00 usr drwxr-xr-x 1 root root 4096 Apr 15 17:30 var
On 15/05/2020 07:10 Plutocrat plutocrat@gmail.com wrote:
On 15/05/2020 10.56, Gregory Sloop wrote:
IMAP sync looks better and better. I'm a little puzzled why I don't just give up walking on the broken glass and use it - but it just bugs the heck out of me.
If you can get the dovecot native sync to work, it seems to be a better option, as it understands all the dovecot index formats etc, and will transfer those correctly. Imapsync doesn't, and when I did that I had a few problems with POP email clients re-downloading their entire Inbox, which produced a few groans. Your mileage may vary.
Another option for you, if you have the time and energy would be to set up proper replication between the two servers. Then any changes made would be synced in real time, until the cutoff, when you just shut down the old server and the new one is already up to date. But if you can't get a simple sync happening then that might be a bit of over-reach.
My suspicions on your current situation would center around the complications introduced by docker, and on file/container permissions, environment variables, paths etc.
How about if you just try a simple doveadm command over ssh from the remote server to make sure that works. eg. from "Sending" server do
ssh root@po.abc.net
docker exec b3093cxxxxxx doveadm user user@abc.net
That should execute the doveadm user command on the remote (receiving) server. If that works, then at least you can satisfy yourself that the docker bit is working correctly. Not sure why I'm getting involved in someone else's troubleshooting nightmare! ;-)
P.
AT> You need to have ssh installed. If you want to do sync over ssh. AT> The docker image hasn't got ssh client installed by default.
AT> Also see https://wiki.dovecot.org/Replication for more configuration options.
SSH *is* installed
# ssh root@abc.net \
docker exec b3093c6200a9 ls -al
root@abc.net's password: total 124 drwxr-xr-x 1 root root 4096 Apr 15 17:30 . drwxr-xr-x 1 root root 4096 Apr 15 17:30 .. -rwxr-xr-x 1 root root 0 Apr 15 17:29 .dockerenv drwxr-xr-x 1 root root 4096 Apr 14 01:23 bin drwxr-xr-x 2 root root 4096 Feb 1 09:09 boot drwxr-xr-x 5 root root 380 May 12 11:25 dev -rwxr-xr-x 1 root root 14951 Apr 14 08:31 docker-entrypoint.sh drwxr-xr-x 1 root root 4096 Apr 15 17:30 etc drwxr-xr-x 2 root root 4096 Feb 1 09:09 home drwxr-xr-x 2 root root 4096 Apr 15 17:30 hooks drwxr-xr-x 1 root root 4096 Apr 14 01:23 lib drwxr-xr-x 2 root root 4096 Mar 26 17:00 lib64 drwxr-xr-x 2 root root 4096 Apr 15 17:30 mail_crypt drwxr-xr-x 2 root root 4096 Mar 26 17:00 media drwxr-xr-x 2 root root 4096 Mar 26 17:00 mnt drwxr-xr-x 2 root root 4096 Mar 26 17:00 opt dr-xr-xr-x 232 root root 0 May 12 11:25 proc drwx------ 1 root root 4096 Apr 14 01:23 root drwxr-xr-x 1 root root 4096 May 12 11:25 run drwxr-xr-x 1 root root 4096 Apr 14 01:23 sbin drwxr-xr-x 2 root root 4096 Mar 26 17:00 srv -rw-r--r-- 1 root root 12126 May 14 15:48 supervisord.log dr-xr-xr-x 13 root root 0 May 12 11:25 sys drwxr-xr-x 2 root root 4096 Apr 15 17:21 templates drwxrwxrwt 1 root root 4096 May 14 21:44 tmp drwxr-xr-x 1 root root 4096 Mar 26 17:00 usr drwxr-xr-x 1 root root 4096 Apr 15 17:30 var
Doh. I wasn't thinking clearly. Yeah, I guess I'm not sure if sshd is available in the docker container itself.
But that would be *sshd* not the ssh client, right?
On 15/05/2020 07:53 Gregory Sloop gregs@sloop.net wrote:
On 15/05/2020 07:10 Plutocrat plutocrat@gmail.com wrote:
On 15/05/2020 10.56, Gregory Sloop wrote:
IMAP sync looks better and better. I'm a little puzzled why I don't just give up walking on the broken glass and use it - but it just bugs the heck out of me.
If you can get the dovecot native sync to work, it seems to be a better option, as it understands all the dovecot index formats etc, and will transfer those correctly. Imapsync doesn't, and when I did that I had a few problems with POP email clients re-downloading their entire Inbox, which produced a few groans. Your mileage may vary.
Another option for you, if you have the time and energy would be to set up proper replication between the two servers. Then any changes made would be synced in real time, until the cutoff, when you just shut down the old server and the new one is already up to date. But if you can't get a simple sync happening then that might be a bit of over-reach.
My suspicions on your current situation would center around the complications introduced by docker, and on file/container permissions, environment variables, paths etc.
How about if you just try a simple doveadm command over ssh from the remote server to make sure that works. eg. from "Sending" server do
sshroot@po.abc.net
docker exec b3093cxxxxxx doveadm useruser@abc.netThat should execute the doveadm user command on the remote (receiving) server. If that works, then at least you can satisfy yourself that the docker bit is working correctly. Not sure why I'm getting involved in someone else's troubleshooting nightmare! ;-)
P.
AT> You need to have ssh installed. If you want to do sync over ssh. AT> The docker image hasn't got ssh client installed by default.
AT> Also seehttps://wiki.dovecot.org/Replicationfor more configuration options.
SSH *is* installed
# ssh root@abc.net \
docker exec b3093c6200a9 ls -al
root@abc.net's password: total 124 drwxr-xr-x 1 root root 4096 Apr 15 17:30 . drwxr-xr-x 1 root root 4096 Apr 15 17:30 .. -rwxr-xr-x 1 root root 0 Apr 15 17:29 .dockerenv drwxr-xr-x 1 root root 4096 Apr 14 01:23 bin drwxr-xr-x 2 root root 4096 Feb 1 09:09 boot drwxr-xr-x 5 root root 380 May 12 11:25 dev -rwxr-xr-x 1 root root 14951 Apr 14 08:31 docker-entrypoint.sh drwxr-xr-x 1 root root 4096 Apr 15 17:30 etc drwxr-xr-x 2 root root 4096 Feb 1 09:09 home drwxr-xr-x 2 root root 4096 Apr 15 17:30 hooks drwxr-xr-x 1 root root 4096 Apr 14 01:23 lib drwxr-xr-x 2 root root 4096 Mar 26 17:00 lib64 drwxr-xr-x 2 root root 4096 Apr 15 17:30 mail_crypt drwxr-xr-x 2 root root 4096 Mar 26 17:00 media drwxr-xr-x 2 root root 4096 Mar 26 17:00 mnt drwxr-xr-x 2 root root 4096 Mar 26 17:00 opt dr-xr-xr-x 232 root root 0 May 12 11:25 proc drwx------ 1 root root 4096 Apr 14 01:23 root drwxr-xr-x 1 root root 4096 May 12 11:25 run drwxr-xr-x 1 root root 4096 Apr 14 01:23 sbin drwxr-xr-x 2 root root 4096 Mar 26 17:00 srv -rw-r--r-- 1 root root 12126 May 14 15:48 supervisord.log dr-xr-xr-x 13 root root 0 May 12 11:25 sys drwxr-xr-x 2 root root 4096 Apr 15 17:21 templates drwxrwxrwt 1 root root 4096 May 14 21:44 tmp drwxr-xr-x 1 root root 4096 Mar 26 17:00 usr drwxr-xr-x 1 root root 4096 Apr 15 17:30 var
Doh. I wasn't thinking clearly. Yeah, I guess I'm not sure if sshd is available in the docker container itself.
But that would be *sshd* not the ssh client, right?
You need both. sshd where you try to connect to, and ssh on where you try to connect from.
Aki
I'm no docker guru - so I'm more than a little out of my depth in this case. But clearly ssh is installed, because that's what I'm running, and the sshd allows a connect and I get a return from ls. So, it's working fine outside the container.
And I only need to execute doveadm sync inside the container. [Because that's where docecot/doveadm is installed.]
So, it sure seems to me that I've got everything in place, at least in terms of ssh/sshd. I'm not even sure how I'd ssh directly to a sshd inside the container.
AT> You need both. sshd where you try to connect to, and ssh on where you try to connect from.
AT> Aki
Uhm.
The log message earlier clearly stated you do not have binary called 'ssh' installed inside the docker container.
doveadm(user@abc.net): Fatal: execvp(ssh) failed: No such file or directory
You can install it however with docker exec -it apt install ssh
*Alternatively* you can configure doveadm server and use that.
service doveadm { inet_listener { port = 12412 } }
doveadm_port = 12412 doveadm_password = secretpass
Then you can do
doveadm backup -D -u user@name tcp:remoteip:12412
Aki
On 15/05/2020 08:02 Gregory Sloop gregs@sloop.net wrote:
I'm no docker guru - so I'm more than a little out of my depth in this case. But clearly ssh is installed, because that's what I'm running, and the sshd allows a connect and I get a return from ls. So, it's working fine outside the container.
And I only need to execute doveadm sync inside the container. [Because that's where docecot/doveadm is installed.]
So, it sure seems to me that I've got everything in place, at least in terms of ssh/sshd. I'm not even sure how I'd ssh directly to a sshd inside the container.
AT> You need both. sshd where you try to connect to, and ssh on where you try to connect from.
AT> Aki
P> On 15/05/2020 10.56, Gregory Sloop wrote:
IMAP sync looks better and better. I'm a little puzzled why I don't just give up walking on the broken glass and use it - but it just bugs the heck out of me.
P> If you can get the dovecot native sync to work, it seems to be a P> better option, as it understands all the dovecot index formats etc, P> and will transfer those correctly. Imapsync doesn't, and when I did P> that I had a few problems with POP email clients re-downloading P> their entire Inbox, which produced a few groans. Your mileage may vary.
P> Another option for you, if you have the time and energy would be P> to set up proper replication between the two servers. Then any P> changes made would be synced in real time, until the cutoff, when P> you just shut down the old server and the new one is already up to P> date. But if you can't get a simple sync happening then that might be a bit of over-reach.
P> My suspicions on your current situation would center around the P> complications introduced by docker, and on file/container P> permissions, environment variables, paths etc.
P> How about if you just try a simple doveadm command over ssh from P> the remote server to make sure that works. eg. from "Sending" server do
P> ssh root@po.abc.net
P> docker exec b3093cxxxxxx doveadm user user@abc.net
P> That should execute the doveadm user command on the remote P> (receiving) server. If that works, then at least you can satisfy P> yourself that the docker bit is working correctly. Not sure why I'm P> getting involved in someone else's troubleshooting nightmare! ;-)
doveadm user does work. i.e. ssh root@abc.net docker exec b3093c6200a9 doveadm user user@abc.net
I get the uid/gid/home/maildir of that account, etc.
I do appreciate the help you've given!
So specifically in your case:
On 14/05/2020 09.09, Gregory Sloop wrote:
doveadm backup -D -u mc-user@abc.net ssh root@abc.net -p2200 doveadm dsync-server -u cp-user@abc.net
mc-user@abc.net is the MC/NEW mailbox/domain cp-user@abc.net is the CPanel/OLD user/domain account The SSH server of the remote system is running on port 2200.
If you're running this on your source server, I think you have the accounts the wrong way around. Try
doveadm backup -u cp-user@abc.net
ssh -p 2200 root@abc.net
doveadm dsync-server -u mc-user@abc.net
Or, if you're running it on your target server, maybe you'll have to pull, using the -R switch.
P.
On 14/05/2020 04:09 Gregory Sloop gregs@sloop.net wrote:
So I've done quite a lot of searching on the list and on the web - and perhaps my google-fu is really bad - but I can't find any real recipes on how to sync mail from the old server to the new.
As an FYI - the old server is a CPanel/WHM setup on a VPS. The new is mailcow - which uses docker.
However, I don't think either of these platforms is what's causing the issue - but I'm certainly not sure of that.
I've tried several things - but have lost track of all the things I've tried. This seemed like the best of all the things I've tried.
This particular mailbox/user+domain is setup on both servers.
doveadm backup -D -u mc-user@abc.net ssh root@abc.net -p2200 doveadm dsync-server -u cp-user@abc.net
mc-user@abc.net is the MC/NEW mailbox/domain cp-user@abc.net is the CPanel/OLD user/domain account The SSH server of the remote system is running on port 2200.
However when I try this, I get: WARNING: The WATCHDOG_NOTIFY_EMAIL variable is not set. Defaulting to a blank string. dsync-local(user@abc.net)<v44NFyRWvF5efgAAb6OASA>: Error: read(remote) failed: EOF (version not received) doveadm(user@abc.net): Fatal: execvp(ssh) failed: No such file or directory
The process appears to hang, and a Ctrl+C stops it.
I'd love to get pointed at a reasonable recipe on how to make this work.
I don't really get/understand the docs much at all. [And either everyone else understands it just fine, and never thinks to write a document on how to do it - or, and I think this is a lot more likely - they're using something like imapsync to do it. I found numerous places where others were, essentially, "Dovecot's tool is way too complicated and I can't get it to work right, so I used imapsync." I suppose I should probably just do that too, but it does seem a shame to do that when the dovecot tool is almost certainly the best tool for the job, but I can't figure out how to use it.]
If someone can help me grok what's going on, I'm glad to write it up for the list and or a blog entry so it's more accessible.
TIA -Greg
Just thought I'd point out that if you want to use ssh, it should be installed on both ends:
"doveadm(user@abc.net): Fatal: execvp(ssh) failed: No such file or directory"
If you don't want to, or can't install ssh, you can also use doveadm server:
service doveadm { inet_listener { port = 12534 } }
doveadm_port = 12534 doveadm_password = secretpass
Then you can use tcp:otherhost:12534 as target/source.
Aki
participants (3)
-
Aki Tuomi
-
Gregory Sloop
-
Plutocrat