doveadm backup from gmail with imapc

[Durga Prasad Malyala]

On Wed, 1 Apr 2020 at 10:32, Ben Mulvihill <ben.mulvihill at gmail.com> wrote:
>
> Hello again,
>
> I am still stuck I'm afraid.
>
> I now have doveadm backup working perfectly from
> a small gmail mailbox (a few hundred messages), but
> when I try the same configuration (apart from usernames
> and passwords obviously) with a large gmail mailbox
> (around 60,000 messages), doveadm connects successfully,
> replicates the gmail folder tree, then produces no further
> output until about 30 minutes later when the server times
> out and doveadm crashes. No actual messages are downloaded.
>
> Does doveadm backup try by default to download all
> messages in a single transaction? And if so, is there
> a way to split it up?
>
> Command output and configuration are at the end of this
> message. Dovecot version is 2.3.10.
>
> Thanks in advance for any help.
>
> Ben
>
> --------------------
>
> ben at expectation:~$ doveadm backup -D -R -u ben imapc:
> dsync(ben): Info: imapc(imap.gmail.com:993): Connected to
> 74.125.133.109:993 (local MY_IP_ADDRESS:53244)
> dsync(ben): Warning: imapc(imap.gmail.com:993): Server disconnected
> with message: System Error - reconnecting (delay 0 ms)
> dsync(ben): Info: imapc(imap.gmail.com:993): Connected to
> 74.125.133.108:993 (local MY_IP_ADDRESS:34076)
> dsync(ben): Panic: file imapc-client.c: line 173
> (imapc_client_run_pre): assertion failed: (client->ioloop == NULL)
> dsync(ben): Error: Raw backtrace:
> /usr/lib/dovecot/libdovecot.so.0(backtrace_append+0x42)
> [0x7f94271610f2] ->
> /usr/lib/dovecot/libdovecot.so.0(backtrace_get+0x1e) [0x7f942716120e]
> -> /usr/lib/dovecot/libdovecot.so.0(+0xe6aee) [0x7f942716baee] ->
> /usr/lib/dovecot/libdovecot.so.0(+0xe6b31) [0x7f942716bb31] ->
> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f94270c61bb] ->
> /usr/lib/dovecot/libdovecot-storage.so.0(imapc_client_run+0x158)
> [0x7f94274c27e8] -> /usr/lib/dovecot/libdovecot-
> storage.so.0(imapc_simple_run+0x3c) [0x7f94274bf8ac] ->
> /usr/lib/dovecot/libdovecot-storage.so.0(+0x94a23) [0x7f94274b7a23] ->
> /usr/lib/dovecot/libdovecot-
> storage.so.0(imapc_list_get_mailbox_flags+0x68) [0x7f94274b8288] ->
> /usr/lib/dovecot/libdovecot-storage.so.0(+0x98e69) [0x7f94274bbe69] ->
> /usr/lib/dovecot/libdovecot-storage.so.0(+0x98fed) [0x7f94274bbfed] ->
> /usr/lib/dovecot/libdovecot-storage.so.0(+0x9bcc9) [0x7f94274becc9] ->
> /usr/lib/dovecot/libdovecot-storage.so.0(+0xa27cb) [0x7f94274c57cb] ->
> /usr/lib/dovecot/libdovecot-
> storage.so.0(imapc_connection_input_pending+0x227) [0x7f94274c7dc7] ->
> /usr/lib/dovecot/libdovecot-storage.so.0(+0xa4e1a) [0x7f94274c7e1a] ->
> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x69) [0x7f9427184999]
> -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x134)
> [0x7f9427186274] ->
> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x4c)
> [0x7f9427184a9c] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38)
> [0x7f9427184cc8] -> /usr/lib/dovecot/libdovecot-
> storage.so.0(imapc_client_run+0xbb) [0x7f94274c274b] ->
> /usr/lib/dovecot/libdovecot-
> storage.so.0(imapc_mailbox_run_nofetch+0x20) [0x7f94274bf7a0] ->
> /usr/lib/dovecot/libdovecot-storage.so.0(imapc_mail_fetch+0x199)
> [0x7f94274b9419] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x9772f)
> [0x7f94274ba72f] -> /usr/lib/dovecot/libdovecot-
> storage.so.0(mail_get_stream_because+0x64) [0x7f9427461fa4] ->
> /usr/lib/dovecot/libdovecot-
> storage.so.0(index_mail_update_access_parts_post+0x7a) [0x7f94274e6c2a]
> -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xcaf3f) [0x7f94274edf3f]
> -> /usr/lib/dovecot/libdovecot-
> storage.so.0(index_storage_search_next_nonblock+0x10d) [0x7f94274ee1cd]
> -> /usr/lib/dovecot/libdovecot-
> storage.so.0(mailbox_search_next_nonblock+0x22) [0x7f94274706e2] ->
> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_search_next+0x3d)
> [0x7f942747074d] ->
> /usr/bin/doveadm(dsync_mailbox_export_next_mail+0x57) [0x562a19bf0597]
> Aborted (core dumped)
>
> ------------------------
>
> # 2.3.10 (0da0eff44): /etc/dovecot/dovecot.conf
> # OS: Linux 4.15.0-91-generic x86_64 Ubuntu 18.04.4 LTS
> # Hostname: expectation
> imapc_features = rfc822.size gmail-migration fetch-headers
> imapc_host = imap.gmail.com
> imapc_password = # hidden, use -P to show it
> imapc_port = 993
> imapc_ssl = imaps
> imapc_ssl_verify = no
> imapc_user = MY_GMAIL_ADDRESS
> mail_location = maildir:~/Maildir
> mail_prefetch_count = 20
> namespace inbox {
>   inbox = yes
>   location =
>   mailbox Drafts {
>     special_use = \Drafts
>   }
>   mailbox Junk {
>     special_use = \Junk
>   }
>   mailbox Sent {
>     special_use = \Sent
>   }
>   mailbox "Sent Messages" {
>     special_use = \Sent
>   }
>   mailbox Trash {
>     special_use = \Trash
>   }
>   prefix =
>   separator = .
> }
> passdb {
>   driver = pam
> }
> protocols = " imap"
> service stats {
>   unix_listener stats-reader {
>     group = mail
>     mode = 0660
>     user = dovecot
>   }
>   unix_listener stats-writer {
>     group = mail
>     mode = 0660
>     user = dovecot
>   }
> }
> ssl = no
> ssl_client_ca_dir = /etc/ssl/certs
> userdb {
>   driver = passwd
> }
>

Hi,  Wouldn't a software like imapsync be a better tool here?
Rgds/DP