doveadm sync backup from old to new server

[Gregory Sloop]

Yeah, I tried that already - and it seems to crash more badly...

dsync-remote(use at abc.net)<mTEPAAUCvl4yFgAAb6OASA>: Panic: epoll_ctl(add, 0) failed: Operation not permitted (fd doesn't support epoll - instead of '<file', try 'cat file|')
dsync-remote(user at abc.net)<mTEPAAUCvl4yFgAAb6OASA>: Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(backtrace_append+0x3d) [0x7fa00465733d] -> /usr/lib/dovecot/libdovecot.so.0(backtrace_get+0x1e) [0x7fa00465745e] -> /usr/lib/dovecot/libdovecot.so.0(+0xeca6b) [0x7fa004660a6b] -> /usr/lib/dovecot/libdovecot.so.0(+0xecaa1) [0x7fa004660aa1] -> /usr/lib/dovecot/libdovecot.so.0(+0x4ec1e) [0x7fa0045c2c1e] -> /usr/lib/dovecot/libdovecot.so.0(+0x520b9) [0x7fa0045c60b9] -> /usr/lib/dovecot/libdovecot.so.0(+0x102278) [0x7fa004676278] -> /usr/lib/dovecot/libdovecot.so.0(io_add_istream_to+0x46) [0x7fa004676336] -> doveadm(dsync_ibc_init_stream+0x216) [0x5564c8f81f06] -> doveadm(+0x2f2a6) [0x5564c8f542a6] -> doveadm(+0x30b16) [0x5564c8f55b16] -> doveadm(+0x3170a) [0x5564c8f5670a] -> doveadm(doveadm_mail_try_run+0x21e) [0x5564c8f56ffe] -> doveadm(main+0x552) [0x5564c8f471b2] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb) [0x7fa0041fb09b] -> doveadm(_start+0x2a) [0x5564c8f472fa]
dsync-local(user at abc.net)<Gc7DAf0Bvl6yFwAAwCTZMw>: Error: read(remote) failed: EOF (version not received)
dsync-local(user at abc.net)<Gc7DAf0Bvl6yFwAAwCTZMw>: Error: Remote command returned error 134: ssh root at po.sloop.net docker exec b3093c6200a9 doveadm dsync-server -u user at abc.net dsync-server

I found several other reports that seem similar, some on Ubuntu 18.04 which match the new platform I'm using.

And I couldn't even find details in the manpage for dsync-server - so I was totally lost - even worse than I was before. [Blind AND LOST, instead of just lost.]

doveadm sync *is* documented, so I figured it was "safer" to use.

Sigh.
IMAP sync looks better and better. I'm a little puzzled why I don't just give up walking on the broken glass and use it - but it just bugs the heck out of me.

This whole process makes me feel like a total moron. I know it's not true - but the docs, and such, around doveadm sync are so obtuse and it's so non-obvious what's wrong, or what one might do to right things - well, it is hard not to feel like you ought to know, but just can't crack the davinci code or something...

Thanks again!

Hopefully someone Aki/Timo et al. will pipe up with something helpful soon. Otherwise, I guess doveadm sync/backup will continue to be unused by any mere mortals.

-Greg

P> I'm not clear myself on the difference myself between 
P> dsync
P> doveadm sync
P> doveadm dsync-server
P> ... perhaps someone here can explain? However what worked for me
P> was using doveadm dsync-server on the 'receive' end. Maybe that
P> puts it in 'listen' mode? So maybe try
P> doveadm backup -u user at abc.net \
P>   ssh root at po.abc.net \
P>   docker exec b3093cxxxxxx doveadm dsync-server -u user at abc.net

P> I've tried to get along with docker before, but it always just
P> seems to add another level of complexity into everything, so thus
P> far I've managed to avoid it apart from general experiments! 

P> P.

P> On 15/05/2020 07.00, Gregory Sloop wrote:
>> So that was really helpful for me to understand that a lot more clearly.
>> Thanks! [Many, many thanks @Plutocrat!!]

>> But I'm still getting a similar failure.
>> Let me give the command I'm using.

>> doveadm backup -D -u user at abc.net \
>> ssh root at po.abc.net \
>> docker exec b3093cxxxxxx doveadm sync -D -u user at abc.net

>> The "local" server is where the data/mail currently is.
>> The remote or docker container/volume is where dovecot is installed. (The data is in a different docker container.)

>> So, I think I'm "backing up" the data from the local machine and pushing that "backed up" data via SSH to the dovecot install in docker and attempting to "sync" that data to the remote dovecot install.

>> However, I immediately get
>> "Error: read(remote) failed: EOF (version not received)"

>> The connection hangs. If I wait a while - perhaps 30-60s, it kills the SSH connection and aborts the process.

>> Is there a way I can test the remote end's ability to accept the data.
>> [i.e. Can I do something like
>> ssh root at po.abc.net \
>> docker exec b3093cxxxxxx doveadm sync -D -u user at abc.net

>> And see if it would accept the data. [In short, do I have a local end problem or a remote end problem - and being able to test both parts individually would probably help me figure out what's broken. That still won't fix it, but at least I'll know which end I need to concentrate on.]

>> I'd guess this will all seem pretty obvious in retrospect, but for the life of me, I'm completely lost and really don't have any idea where to start to break it down better so I can see how each piece is working or not.

>> TIA
>> -Greg


>> *P> I struggled with this at one time, and with the lack of examples
>> P> around the internet. In fact, I might have written this very post
>> P> myself at one point! In the end, the penny dropped when I saw that the format was

>> P> (command on local server) + (stuff you need to connect over ssh) + (command on remote server)

>> P> So what worked for me was:

>> P> doveadm backup -u *user at domain.com <mailto:user at domain.com>* \
>> P>    ssh -p 2222 (+ other ssh options) *root at remoteserver.com <mailto:root at remoteserver.com>* \
>> P>    doveadm dsync-server -u *user2 at domain2.com <mailto:user2 at domain2.com>* 

>> P> If you get
>> P> dsync-remote(*user2 at domain2.com <mailto:user2 at domain2.com>*): Error: Mailbox INBOX sync:
>> P> mailbox_delete failed: INBOX can't be deleted
>> P> ... you'll need to clear the remote directory first! Or maybe try sync instead of backup?

>> P> I found that running as root on the remote server was necessary in
>> P> my case, due to the permissions on the remote directory. You might
>> P> want to check permissions on remote directory are writable by the user in your ssh command.

>> P> If still no joy you can run
>> P>         doveadm user *user at domain.com <mailto:user at domain.com>* on your local server and
>> P>         doveadm user *user2 at domain2.com <mailto:user2 at domain2.com>* on your remote server
>> P> and dovecot will tell you where its synching from and to.

>> P> Finally, this solution is a 'push' from source server to target.
>> P> You may 'pull' the other way if that makes more sense in your environment with -R

>> P> Those pieces were enough for me to get it to work ...

>> P> P.

>> P> PS. For the record, on the original job I was under time pressure,
>> P> and yes, I did use imapsync to get it done in the end. I got this
>> P> to work later on when I had more time to tinker.

>> P> On 14/05/2020 09.09, Gregory Sloop wrote:
>>>> So I've done quite a lot of searching on the list and on the web - and perhaps my google-fu is really bad - but I can't find any real recipes on how to sync mail from the old server to the new.

>>>> As an FYI - the old server is a CPanel/WHM setup on a VPS.
>>>> The new is mailcow - which uses docker.

>>>> However, I don't think either of these platforms is what's causing the issue - but I'm certainly not sure of that.

>>>> ---
>>>> I've tried several things - but have lost track of all the things I've tried.
>>>> This seemed like the best of all the things I've tried.

>>>> This particular mailbox/user+domain is setup on both servers.

>>>> doveadm backup -D -u *mc-user at abc.net <mailto:mc-user at abc.net>* ssh *root at abc.net <mailto:root at abc.net>* -p2200 doveadm dsync-server -u *cp-user at abc.net <mailto:cp-user at abc.net>

>>>> mc-user at abc.net <mailto:mc-user at abc.net>* is the MC/NEW mailbox/domain
>> *>> cp-user at abc.net <mailto:cp-user at abc.net>* is the CPanel/OLD user/domain account
>>>> The SSH server of the remote system is running on port 2200.

>>>> However when I try this, I get:
>>>> WARNING: The WATCHDOG_NOTIFY_EMAIL variable is not set. Defaulting to a blank string.
>>>> dsync-local(*user at abc.net <mailto:user at abc.net>*)<v44NFyRWvF5efgAAb6OASA>: Error: read(remote) failed: EOF (version not received)
>>>> doveadm(*user at abc.net <mailto:user at abc.net>*): Fatal: execvp(ssh) failed: No such file or directory

>>>> The process appears to hang, and a Ctrl+C stops it.

>>>> I'd love to get pointed at a reasonable recipe on how to make this work.

>>>> I don't really get/understand the docs much at all.
>>>> [And either everyone else understands it just fine, and never thinks to write a document on how to do it - or, and I think this is a lot more likely - they're using something like imapsync to do it. I found numerous places where others were, essentially, "Dovecot's tool is way too complicated and I can't get it to work right, so I used imapsync." I suppose I should probably just do that too, but it does seem a shame to do that when the dovecot tool is almost certainly the best tool for the job, but I can't figure out how to use it.]

>>>> If someone can help me grok what's going on, I'm glad to write it up for the list and or a blog entry so it's more accessible.

>>>> TIA
>>>> -Greg

>> */--
>> Gregory Sloop, Principal: Sloop Network & Computer Consulting
>> Voice: 503.251.0452 x121
>> EMail: /gregs at sloop.net <mailto:gregs at sloop.net>
>> http://www.sloop.net
>> /--- /

-- 
Gregory Sloop, Principal: Sloop Network & Computer Consulting
Voice: 503.251.0452 x121
EMail: gregs at sloop.net
http://www.sloop.net
---
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dovecot.org/pipermail/dovecot/attachments/20200514/9928a837/attachment-0001.html>