doveadm backup from gmail with imapc

[Ben Mulvihill]

Hello again,

I am still stuck I'm afraid.

I now have doveadm backup working perfectly from
a small gmail mailbox (a few hundred messages), but
when I try the same configuration (apart from usernames
and passwords obviously) with a large gmail mailbox
(around 60,000 messages), doveadm connects successfully,
replicates the gmail folder tree, then produces no further
output until about 30 minutes later when the server times
out and doveadm crashes. No actual messages are downloaded. 

Does doveadm backup try by default to download all
messages in a single transaction? And if so, is there
a way to split it up?

Command output and configuration are at the end of this
message. Dovecot version is 2.3.10.

Thanks in advance for any help.

Ben

--------------------

ben at expectation:~$ doveadm backup -D -R -u ben imapc:
dsync(ben): Info: imapc(imap.gmail.com:993): Connected to
74.125.133.109:993 (local MY_IP_ADDRESS:53244)
dsync(ben): Warning: imapc(imap.gmail.com:993): Server disconnected
with message: System Error - reconnecting (delay 0 ms)
dsync(ben): Info: imapc(imap.gmail.com:993): Connected to
74.125.133.108:993 (local MY_IP_ADDRESS:34076)
dsync(ben): Panic: file imapc-client.c: line 173
(imapc_client_run_pre): assertion failed: (client->ioloop == NULL)
dsync(ben): Error: Raw backtrace:
/usr/lib/dovecot/libdovecot.so.0(backtrace_append+0x42)
[0x7f94271610f2] ->
/usr/lib/dovecot/libdovecot.so.0(backtrace_get+0x1e) [0x7f942716120e]
-> /usr/lib/dovecot/libdovecot.so.0(+0xe6aee) [0x7f942716baee] ->
/usr/lib/dovecot/libdovecot.so.0(+0xe6b31) [0x7f942716bb31] ->
/usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f94270c61bb] ->
/usr/lib/dovecot/libdovecot-storage.so.0(imapc_client_run+0x158)
[0x7f94274c27e8] -> /usr/lib/dovecot/libdovecot-
storage.so.0(imapc_simple_run+0x3c) [0x7f94274bf8ac] ->
/usr/lib/dovecot/libdovecot-storage.so.0(+0x94a23) [0x7f94274b7a23] ->
/usr/lib/dovecot/libdovecot-
storage.so.0(imapc_list_get_mailbox_flags+0x68) [0x7f94274b8288] ->
/usr/lib/dovecot/libdovecot-storage.so.0(+0x98e69) [0x7f94274bbe69] ->
/usr/lib/dovecot/libdovecot-storage.so.0(+0x98fed) [0x7f94274bbfed] ->
/usr/lib/dovecot/libdovecot-storage.so.0(+0x9bcc9) [0x7f94274becc9] ->
/usr/lib/dovecot/libdovecot-storage.so.0(+0xa27cb) [0x7f94274c57cb] ->
/usr/lib/dovecot/libdovecot-
storage.so.0(imapc_connection_input_pending+0x227) [0x7f94274c7dc7] ->
/usr/lib/dovecot/libdovecot-storage.so.0(+0xa4e1a) [0x7f94274c7e1a] ->
/usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x69) [0x7f9427184999]
-> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x134)
[0x7f9427186274] ->
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x4c)
[0x7f9427184a9c] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38)
[0x7f9427184cc8] -> /usr/lib/dovecot/libdovecot-
storage.so.0(imapc_client_run+0xbb) [0x7f94274c274b] ->
/usr/lib/dovecot/libdovecot-
storage.so.0(imapc_mailbox_run_nofetch+0x20) [0x7f94274bf7a0] ->
/usr/lib/dovecot/libdovecot-storage.so.0(imapc_mail_fetch+0x199)
[0x7f94274b9419] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x9772f)
[0x7f94274ba72f] -> /usr/lib/dovecot/libdovecot-
storage.so.0(mail_get_stream_because+0x64) [0x7f9427461fa4] ->
/usr/lib/dovecot/libdovecot-
storage.so.0(index_mail_update_access_parts_post+0x7a) [0x7f94274e6c2a]
-> /usr/lib/dovecot/libdovecot-storage.so.0(+0xcaf3f) [0x7f94274edf3f]
-> /usr/lib/dovecot/libdovecot-
storage.so.0(index_storage_search_next_nonblock+0x10d) [0x7f94274ee1cd]
-> /usr/lib/dovecot/libdovecot-
storage.so.0(mailbox_search_next_nonblock+0x22) [0x7f94274706e2] ->
/usr/lib/dovecot/libdovecot-storage.so.0(mailbox_search_next+0x3d)
[0x7f942747074d] ->
/usr/bin/doveadm(dsync_mailbox_export_next_mail+0x57) [0x562a19bf0597]
Aborted (core dumped)

------------------------

# 2.3.10 (0da0eff44): /etc/dovecot/dovecot.conf
# OS: Linux 4.15.0-91-generic x86_64 Ubuntu 18.04.4 LTS
# Hostname: expectation
imapc_features = rfc822.size gmail-migration fetch-headers
imapc_host = imap.gmail.com
imapc_password = # hidden, use -P to show it
imapc_port = 993
imapc_ssl = imaps
imapc_ssl_verify = no
imapc_user = MY_GMAIL_ADDRESS
mail_location = maildir:~/Maildir
mail_prefetch_count = 20
namespace inbox {
  inbox = yes
  location =
  mailbox Drafts {
    special_use = \Drafts
  }
  mailbox Junk {
    special_use = \Junk
  }
  mailbox Sent {
    special_use = \Sent
  }
  mailbox "Sent Messages" {
    special_use = \Sent
  }
  mailbox Trash {
    special_use = \Trash
  }
  prefix =
  separator = .
}
passdb {
  driver = pam
}
protocols = " imap"
service stats {
  unix_listener stats-reader {
    group = mail
    mode = 0660
    user = dovecot
  }
  unix_listener stats-writer {
    group = mail
    mode = 0660
    user = dovecot
  }
}
ssl = no
ssl_client_ca_dir = /etc/ssl/certs
userdb {
  driver = passwd
}