On Tue, 2020-03-31 at 09:06 +0300, Aki Tuomi wrote:

...

On 30/03/2020 22:11 Ben Mulvihill ben.mulvihill@gmail.com wrote:

  I am trying to backup a gmail account (not the one I am writing from) to dovecot, using doveadm-backup and imapc, but am having ssl connection problems.

ted@expectation:~# doveadm backup -D -R -u ted imapc: dsync(ted): Info: imapc(imap.gmail.com:993): Connected to 74.125.71.108:993 (local 10.7.1.179:53852) dsync(ted): Warning: imapc(imap.gmail.com:993): Server disconnected unexpectedly: SSL_connect() failed: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure: SSL alert number 40 - reconnecting (delay 0 ms) dsync(ted): Info: imapc(imap.gmail.com:993): Connected to 74.125.71.109:993 (local 10.7.1.179:59052) dsync(ted): Error: imapc(imap.gmail.com:993): Server disconnected unexpectedly: SSL_connect() failed: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure: SSL alert number 40 - disconnecting dsync(ted): Error: User initialization failed: imapc: Login to imap.gmail.com failed: Disconnected from serv

I am using dovecot version 2.2.33.2 on ubuntu, with the configuration below. I have also enabled "allow access from unsecure apps" in my gmail settings.

My first thought looking at the error messages was has that perhaps doveadm-backup was trying to connect with ssl3, which is no longer supported by gmail or anyone else nowadays. But apparently the ssl3_read_bytes function in openssl also handles tls, so the reference to ssl3 in the message is misleading, and the real problem must be elsewhere.

If anyone can help me debug this I'd be grateful.

Many thanks,  Ben

root@expectation:/etc/dovecot# doveconf -N # 2.2.33.2 (d6601f4ec): /etc/dovecot/dovecot.conf

Hi!

This is very old version of dovecot so this could be a bug that has been fixed in more recent version.

Can you verify that you have the required CA certs with

openssl s_client -connect imap.gmail.com:993 -servername imap.gmail.com -CApath /etc/ssl/certs

and make sure the cert gets validated by openssl.

If it does, then you should probably consider upgrading to some more recent version. We provide packages at https://repo.dovecot.org if you are able to upgrade.

Aki

Hi Aki!

I checked the CA certificate and it was validated OK, so I followed your suggestion and installed dovecot version 2.3.10 from the dovecot repository (the version I was using previously was the latest available in my distribution).

The ssl handshake now completes successfully. I have hit  further problems later in the process, but I'll investigate myself first and then maybe post again with the details if I am still stuck.

Thank you for your help!

Ben

On Wed, 1 Apr 2020 at 10:32, Ben Mulvihill ben.mulvihill@gmail.com wrote:

Hello again,

I am still stuck I'm afraid.

I now have doveadm backup working perfectly from a small gmail mailbox (a few hundred messages), but when I try the same configuration (apart from usernames and passwords obviously) with a large gmail mailbox (around 60,000 messages), doveadm connects successfully, replicates the gmail folder tree, then produces no further output until about 30 minutes later when the server times out and doveadm crashes. No actual messages are downloaded.

Does doveadm backup try by default to download all messages in a single transaction? And if so, is there a way to split it up?

Command output and configuration are at the end of this message. Dovecot version is 2.3.10.

Thanks in advance for any help.

Ben

---

ben@expectation:~$ doveadm backup -D -R -u ben imapc: dsync(ben): Info: imapc(imap.gmail.com:993): Connected to 74.125.133.109:993 (local MY_IP_ADDRESS:53244) dsync(ben): Warning: imapc(imap.gmail.com:993): Server disconnected with message: System Error - reconnecting (delay 0 ms) dsync(ben): Info: imapc(imap.gmail.com:993): Connected to 74.125.133.108:993 (local MY_IP_ADDRESS:34076) dsync(ben): Panic: file imapc-client.c: line 173 (imapc_client_run_pre): assertion failed: (client->ioloop == NULL) dsync(ben): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(backtrace_append+0x42) [0x7f94271610f2] -> /usr/lib/dovecot/libdovecot.so.0(backtrace_get+0x1e) [0x7f942716120e] -> /usr/lib/dovecot/libdovecot.so.0(+0xe6aee) [0x7f942716baee] -> /usr/lib/dovecot/libdovecot.so.0(+0xe6b31) [0x7f942716bb31] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f94270c61bb] -> /usr/lib/dovecot/libdovecot-storage.so.0(imapc_client_run+0x158) [0x7f94274c27e8] -> /usr/lib/dovecot/libdovecot- storage.so.0(imapc_simple_run+0x3c) [0x7f94274bf8ac] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x94a23) [0x7f94274b7a23] -> /usr/lib/dovecot/libdovecot- storage.so.0(imapc_list_get_mailbox_flags+0x68) [0x7f94274b8288] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x98e69) [0x7f94274bbe69] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x98fed) [0x7f94274bbfed] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x9bcc9) [0x7f94274becc9] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xa27cb) [0x7f94274c57cb] -> /usr/lib/dovecot/libdovecot- storage.so.0(imapc_connection_input_pending+0x227) [0x7f94274c7dc7] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xa4e1a) [0x7f94274c7e1a] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x69) [0x7f9427184999] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x134) [0x7f9427186274] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x4c) [0x7f9427184a9c] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f9427184cc8] -> /usr/lib/dovecot/libdovecot- storage.so.0(imapc_client_run+0xbb) [0x7f94274c274b] -> /usr/lib/dovecot/libdovecot- storage.so.0(imapc_mailbox_run_nofetch+0x20) [0x7f94274bf7a0] -> /usr/lib/dovecot/libdovecot-storage.so.0(imapc_mail_fetch+0x199) [0x7f94274b9419] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x9772f) [0x7f94274ba72f] -> /usr/lib/dovecot/libdovecot- storage.so.0(mail_get_stream_because+0x64) [0x7f9427461fa4] -> /usr/lib/dovecot/libdovecot- storage.so.0(index_mail_update_access_parts_post+0x7a) [0x7f94274e6c2a] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xcaf3f) [0x7f94274edf3f] -> /usr/lib/dovecot/libdovecot- storage.so.0(index_storage_search_next_nonblock+0x10d) [0x7f94274ee1cd] -> /usr/lib/dovecot/libdovecot- storage.so.0(mailbox_search_next_nonblock+0x22) [0x7f94274706e2] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_search_next+0x3d) [0x7f942747074d] -> /usr/bin/doveadm(dsync_mailbox_export_next_mail+0x57) [0x562a19bf0597] Aborted (core dumped)

---

# 2.3.10 (0da0eff44): /etc/dovecot/dovecot.conf # OS: Linux 4.15.0-91-generic x86_64 Ubuntu 18.04.4 LTS # Hostname: expectation imapc_features = rfc822.size gmail-migration fetch-headers imapc_host = imap.gmail.com imapc_password = # hidden, use -P to show it imapc_port = 993 imapc_ssl = imaps imapc_ssl_verify = no imapc_user = MY_GMAIL_ADDRESS mail_location = maildir:~/Maildir mail_prefetch_count = 20 namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = separator = . } passdb { driver = pam } protocols = " imap" service stats { unix_listener stats-reader { group = mail mode = 0660 user = dovecot } unix_listener stats-writer { group = mail mode = 0660 user = dovecot } } ssl = no ssl_client_ca_dir = /etc/ssl/certs userdb { driver = passwd }

Hi, Wouldn't a software like imapsync be a better tool here? Rgds/DP

On 08/04/2020 19.14, Ben Mulvihill wrote:

Thanks for the suggestion. I think my next step will be to investigate imapsync, or maybe isync, which also looks promising.

I've recently migrated a server with imapsync with 150Gb of email and 200+ accounts. I found the trick with this was the run imapsync in several runs: Messages older than 1 year; messages older than 6 months; messages older than 1 week, for example. Then just keep running the full sync after that until you're ready to switch. If you've just got one email then this should be easy. With many addresses, there is a bit of scripting to do first to get it all set up.

One gotcha, which I never solved, (mainly due to the fact I had to do the whole migration in under a week as an emergency). I could never figure out how to get the dovecot indexes correct, so that when the client connected to the new server via POP, it didn't re-download all the messages. IMAP was OK though.

P.

On Thu, 9 Apr 2020, Plutocrat wrote:

I could never figure out how to get the dovecot indexes correct, so that when the client connected to the new server via POP, it didn't re-download all the messages. IMAP was OK though.

It's been a while since I migrated from the qpopper, but I believe this configuration may be the answer:

(On new server)
protocol pop3 {
	...
	pop3_reuse_xuidl = yes
}

Joseph Tam jtam.home@gmail.com

On Thu, 2020-04-09 at 13:48 +0300, Sami Ketola wrote:

...

On 31 Mar 2020, at 23.18, Ben Mulvihill ben.mulvihill@gmail.com wrote:

Hello again,

I am still stuck I'm afraid.

I now have doveadm backup working perfectly from a small gmail mailbox (a few hundred messages), but when I try the same configuration (apart from usernames and passwords obviously) with a large gmail mailbox (around 60,000 messages), doveadm connects successfully, replicates the gmail folder tree, then produces no further output until about 30 minutes later when the server times out and doveadm crashes. No actual messages are downloaded. 

It's a while since we last performed migrations from gmail. Maybe they have changed something in the meanwhile. Can you record imapc rawlogs from the session to see where it gets stuck? It might be gmail throttling the speed.

Sami

Thanks for the reply. Here are my logs, edited to remove personal information and lots of repeated lines. Dovecot configuration is  as above, with the addition of the two options imapc_rawlog_dir imapc_features = throttle:50:16000:500 Doveadm seems to be fetching lots of  message IDs but no actual messages, and then it core dumps.

Ben

=============================================================

edited-imapc-log.in

1586512882.754448 * OK Gimap ready for requests from XXX.XXX.XXX.XXX t129mb20216081wmt 1586512882.790345 * CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 XYZZY SASL-IR AUTH=XOAUTH2 AUTH=PLAIN AUTH=PLAIN-CLIENTTOKEN AUTH=OAUTHBEARER AUTH=XOAUTH 1586512882.790345 1 OK Thats all she wrote! t129mb20216081wmt 1586512883.870337 * CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE CONDSTORE ESEARCH UTF8=ACCEPT LIST-EXTENDED LIST-STATUS LITERAL- SPECIAL-USE APPENDLIMIT=35651584 1586512883.870337 2 OK MY_GMAIL_ADDRESS authenticated (Success) 1586512884.014073 * LIST (\Noselect) "/" "/" 1586512884.014073 3 OK Success 1586512884.160944 * LIST (\HasNoChildren) "/" "INBOX" 1586512884.160944 * LIST (\HasChildren \Noselect) "/" "[Google Mail]" 1586512884.160944 * LIST (\All \HasNoChildren) "/" "[Google Mail]/All Mail" 1586512884.160944 * LIST (\HasNoChildren \Trash) "/" "[Google Mail]/Bin" 1586512884.160944 * LIST (\Drafts \HasNoChildren) "/" "[Google Mail]/Drafts" 1586512884.160944 * LIST (\HasNoChildren \Important) "/" "[Google Mail]/Important" 1586512884.160944 * LIST (\HasNoChildren) "/" "[Google Mail]/MY_FOLDER_NAME" 1586512884.160944 * LIST (\HasNoChildren \Sent) "/" "[Google Mail]/Sent Mail" 1586512884.160944 * LIST (\HasNoChildren \Junk) "/" "[Google Mail]/Spam" 1586512884.160944 * LIST (\Flagged \HasNoChildren) "/" "[Google Mail]/Starred" 1586512884.161789 * LIST (\HasNoChildren) "/" "[Google Mail]/mark_as_spam" 1586512884.161789 * LIST (\HasNoChildren) "/" "[Google Mail]/mining" 1586512884.161789 4 OK Success 1586512884.317802 * STATUS "INBOX" (UIDNEXT 110434 UIDVALIDITY 602828375) 1586512884.317802 5 OK Success 1586512884.462612 * STATUS "[Google Mail]/All Mail" (UIDNEXT 231431 UIDVALIDITY 602828395) 1586512884.462612 6 OK Success 1586512884.606828 * STATUS "[Google Mail]/Bin" (UIDNEXT 21532 UIDVALIDITY 602828376) 1586512884.606828 7 OK Success 1586512884.749952 * STATUS "[Google Mail]/Drafts" (UIDNEXT 17485 UIDVALIDITY 602828378) 1586512884.749952 8 OK Success 1586512884.894350 * STATUS "[Google Mail]/Important" (UIDNEXT 29288 UIDVALIDITY 602828400) 1586512884.894350 9 OK Success 1586512885.037461 * STATUS "[Google Mail]/MY_FOLDER_NAME" (UIDNEXT 3648 UIDVALIDITY 602828399) 1586512885.037461 10 OK Success 1586512885.181415 * STATUS "[Google Mail]/Sent Mail" (UIDNEXT 14061 UIDVALIDITY 602828374) 1586512885.181415 11 OK Success 1586512885.325056 * STATUS "[Google Mail]/Spam" (UIDNEXT 98883 UIDVALIDITY 602828379) 1586512885.325056 12 OK Success 1586512885.468928 * STATUS "[Google Mail]/Starred" (UIDNEXT 21 UIDVALIDITY 602828380) 1586512885.468928 13 OK Success 1586512885.612611 * STATUS "[Google Mail]/mark_as_spam" (UIDNEXT 76 UIDVALIDITY 602828397) 1586512885.612611 14 OK Success 1586512885.755909 * STATUS "[Google Mail]/mining" (UIDNEXT 23 UIDVALIDITY 602828396) 1586512885.755909 15 OK Success 1586512885.899075 * LSUB (\HasNoChildren) "/" "INBOX" 1586512885.899075 * LSUB (\HasChildren \Noselect) "/" "[Google Mail]" 1586512885.899075 * LSUB (\All \HasNoChildren) "/" "[Google Mail]/All Mail" 1586512885.900047 * LSUB (\HasNoChildren \Trash) "/" "[Google Mail]/Bin" 1586512885.900047 * LSUB (\Drafts \HasNoChildren) "/" "[Google Mail]/Drafts" 1586512885.900047 * LSUB (\HasNoChildren \Important) "/" "[Google Mail]/Important" 1586512885.900047 * LSUB (\HasNoChildren) "/" "[Google Mail]/MY_FOLDER_NAME" 1586512885.900047 * LSUB (\HasNoChildren \Sent) "/" "[Google Mail]/Sent Mail" 1586512885.901074 * LSUB (\HasNoChildren \Junk) "/" "[Google Mail]/Spam" 1586512885.901074 * LSUB (\Flagged \HasNoChildren) "/" "[Google Mail]/Starred" 1586512885.901074 * LSUB (\HasNoChildren) "/" "[Google Mail]/mark_as_spam" 1586512885.901074 * LSUB (\HasNoChildren) "/" "[Google Mail]/mining" 1586512885.901074 16 OK Success 1586512886.990868 * LIST (\HasNoChildren) "/" "INBOX" 1586512886.990868 * LIST (\HasChildren \Noselect) "/" "[Google Mail]" 1586512886.990868 * LIST (\All \HasNoChildren) "/" "[Google Mail]/All Mail" 1586512886.990868 * LIST (\HasNoChildren \Trash) "/" "[Google Mail]/Bin" 1586512886.990868 * LIST (\Drafts \HasNoChildren) "/" "[Google Mail]/Drafts" 1586512886.990868 * LIST (\HasNoChildren \Important) "/" "[Google Mail]/Important" 1586512886.990868 * LIST (\HasNoChildren) "/" "[Google Mail]/MY_FOLDER_NAME" 1586512886.990868 * LIST (\HasNoChildren \Sent) "/" "[Google Mail]/Sent Mail" 1586512886.990868 * LIST (\HasNoChildren \Junk) "/" "[Google Mail]/Spam" 1586512886.990868 * LIST (\Flagged \HasNoChildren) "/" "[Google Mail]/Starred" 1586512886.991783 * LIST (\HasNoChildren) "/" "[Google Mail]/mark_as_spam" 1586512886.991783 * LIST (\HasNoChildren) "/" "[Google Mail]/mining" 1586512886.991783 17 OK Success 1586512887.436407 * FLAGS (\Answered \Flagged \Draft \Deleted \Seen $NotPhishing $Phishing $has_cal JUNK receipt-handled) 1586512887.436407 * OK [PERMANENTFLAGS ()] Flags permitted. 1586512887.436407 * OK [UIDVALIDITY 602828375] UIDs valid. 1586512887.436407 * 48540 EXISTS 1586512887.436407 * 0 RECENT 1586512887.436407 * OK [UIDNEXT 110434] Predicted next UID. 1586512887.436407 * OK [HIGHESTMODSEQ 8740446] 1586512887.436407 18 OK [READ-ONLY] INBOX selected. (Success) 1586512891.905633 * 1 FETCH (UID 62 FLAGS (\Seen)) 1586512891.905633 * 2 FETCH (UID 65 FLAGS (\Seen)) 1586512891.905633 * 3 FETCH (UID 69 FLAGS (\Answered \Seen)) 1586512891.905633 * 4 FETCH (UID 70 FLAGS (\Answered \Seen)) 1586512891.905633 * 5 FETCH (UID 71 FLAGS (\Answered \Seen)) 1586512891.905633 * 6 FETCH (UID 72 FLAGS (\Answered \Seen)) 1586512891.905633 * 7 FETCH (UID 73 FLAGS (\Seen)) 1586512891.905633 * 8 FETCH (UID 76 FLAGS (\Seen)) 1586512891.905633 * 9 FETCH (UID 78 FLAGS (\Answered \Seen)) 1586512891.905633 * 10 FETCH (UID 79 FLAGS (\Seen)) 1586512891.905633 * 11 FETCH (UID 80 FLAGS (\Answered \Seen)) 1586512891.905633 * 12 FETCH (UID 81 FLAGS (\Seen)) 1586512891.905633 * 13 FETCH (UID 83 FLAGS (\Answered \Seen)) 1586512891.907319 * 14 FETCH (UID 84 FLAGS ()) 1586512891.907319 * 15 FETCH (UID 85 FLAGS (\Seen)) 1586512891.907319 * 16 FETCH (UID 86 FLAGS (\Seen)) 1586512891.907319 * 17 FETCH (UID 87 FLAGS (\Answered \Seen)) 1586512891.907319 * 18 FETCH (UID 88 FLAGS ()) 1586512891.907319 * 19 FETCH (UID 89 FLAGS (\Seen)) 1586512891.907319 * 20 FETCH (UID 91 FLAGS (\Answered \Seen)) 1586512891.907319 * 21 FETCH (UID 92 FLAGS (\Seen))

....... LOTS OF SIMILAR LINES .......

1586512893.541503 * 48522 FETCH (UID 110415 FLAGS ()) 1586512893.541503 * 48523 FETCH (UID 110416 FLAGS ()) 1586512893.541503 * 48524 FETCH (UID 110417 FLAGS ()) 1586512893.541503 * 48525 FETCH (UID 110418 FLAGS ()) 1586512893.541503 * 48526 FETCH (UID 110419 FLAGS ()) 1586512893.541503 * 48527 FETCH (UID 110420 FLAGS ()) 1586512893.541503 * 48528 FETCH (UID 110421 FLAGS ()) 1586512893.541503 * 48529 FETCH (UID 110422 FLAGS ()) 1586512893.541503 * 48530 FETCH (UID 110423 FLAGS ()) 1586512893.541503 * 48531 FETCH (UID 110424 FLAGS ()) 1586512893.541503 * 48532 FETCH (UID 110425 FLAGS ()) 1586512893.541503 * 48533 FETCH (UID 110426 FLAGS ()) 1586512893.541503 * 48534 FETCH (UID 110427 FLAGS ()) 1586512893.541503 * 48535 FETCH (UID 110428 FLAGS ()) 1586512893.541503 * 48536 FETCH (UID 110429 FLAGS ()) 1586512893.541503 * 48537 FETCH (UID 110430 FLAGS ()) 1586512893.541503 * 48538 FETCH (UID 110431 FLAGS ()) 1586512893.541503 * 48539 FETCH (UID 110432 FLAGS ()) 1586512893.541503 * 48540 FETCH (UID 110433 FLAGS ()) 1586512893.541503 19 OK Success 1586512893.704625 * 1 FETCH (X-GM-MSGID 1265813037243496020 UID 62) 1586512893.704625 * 2 FETCH (X-GM-MSGID 1266142060943449934 UID 65) 1586512893.704625 * 3 FETCH (X-GM-MSGID 1266240767076216106 UID 69) 1586512893.704625 * 4 FETCH (X-GM-MSGID 1266317645039054507 UID 70) 1586512893.704625 * 5 FETCH (X-GM-MSGID 1266319104567433759 UID 71) 1586512893.704625 * 6 FETCH (X-GM-MSGID 1266338339735148225 UID 72) 1586512893.704625 * 7 FETCH (X-GM-MSGID 1266350811947568709 UID 73) 1586512893.704625 * 8 FETCH (X-GM-MSGID 1266427483536331721 UID 76) 1586512893.704625 * 9 FETCH (X-GM-MSGID 1266482766386716417 UID 78) 1586512893.704625 * 10 FETCH (X-GM-MSGID 1266512748587747731 UID 79) 1586512893.704625 * 11 FETCH (X-GM-MSGID 1266523171112424759 UID 80) 1586512893.704625 * 12 FETCH (X-GM-MSGID 1266584914541703703 UID 81) 1586512893.704625 * 13 FETCH (X-GM-MSGID 1266626775786784521 UID 83) 1586512893.704625 * 14 FETCH (X-GM-MSGID 1266663913156455749 UID 84) 1586512893.704625 * 15 FETCH (X-GM-MSGID 1266684556515340123 UID 85) 1586512893.704625 * 16 FETCH (X-GM-MSGID 1266684780091647650 UID 86) 1586512893.704625 * 17 FETCH (X-GM-MSGID 1266690080881874411 UID 87) 1586512893.704625 * 18 FETCH (X-GM-MSGID 1266697947934426922 UID 88) 1586512893.704625 * 19 FETCH (X-GM-MSGID 1266841785830673314 UID 89) 1586512893.704625 * 20 FETCH (X-GM-MSGID 1266890981732444471 UID 91) 1586512893.704625 * 21 FETCH (X-GM-MSGID 1266891244711722087 UID 92) 1586512893.704625 20 OK Success 1586512893.861241 * 22 FETCH (X-GM-MSGID 1266959738134105978 UID 94) 1586512893.861241 * 23 FETCH (X-GM-MSGID 1267022593809794009 UID 95) 1586512893.861241 * 24 FETCH (X-GM-MSGID 1267023552222412653 UID 96) 1586512893.861241 * 25 FETCH (X-GM-MSGID 1267028400626761068 UID 97) 1586512893.861241 * 26 FETCH (X-GM-MSGID 1267058789660419022 UID 98) 1586512893.861241 * 27 FETCH (X-GM-MSGID 1267060956576095193 UID 99) 1586512893.861241 * 28 FETCH (X-GM-MSGID 1267078074477602342 UID 100) 1586512893.861241 * 29 FETCH (X-GM-MSGID 1267144516612296284 UID 102) 1586512893.861241 * 30 FETCH (X-GM-MSGID 1267169233934891212 UID 103) 1586512893.861241 * 31 FETCH (X-GM-MSGID 1267181621874937138 UID 104) 1586512893.861241 * 32 FETCH (X-GM-MSGID 1267322905811934828 UID 106) 1586512893.861241 * 33 FETCH (X-GM-MSGID 1267362788296163573 UID 107) 1586512893.861241 * 34 FETCH (X-GM-MSGID 1267409013248654879 UID 110) 1586512893.861241 * 35 FETCH (X-GM-MSGID 1267429601732986948 UID 111) 1586512893.861241 * 36 FETCH (X-GM-MSGID 1267433450902833783 UID 112) 1586512893.861241 * 37 FETCH (X-GM-MSGID 1267433996158684970 UID 113) 1586512893.861241 * 38 FETCH (X-GM-MSGID 1267479985482540073 UID 114) 1586512893.861241 * 39 FETCH (X-GM-MSGID 1267512244483485892 UID 117) 1586512893.861241 * 40 FETCH (X-GM-MSGID 1267512434336402096 UID 118) 1586512893.861241 * 41 FETCH (X-GM-MSGID 1267514033433119045 UID 119) 1586512893.861241 * 42 FETCH (X-GM-MSGID 1267586382946441314 UID 120) 1586512893.861241 21 OK Success 1586512894.009036 * 43 FETCH (X-GM-MSGID 1267586414720205278 UID 121) 1586512894.009036 * 44 FETCH (X-GM-MSGID 1267594567233703154 UID 123) 1586512894.009036 * 45 FETCH (X-GM-MSGID 1267861214156448451 UID 127)

....... LOTS OF SIMILAR LINES .......

1586513176.097992 * 40414 FETCH (X-GM-MSGID 1642763475047250557 UID 99636) 1586513176.097992 * 40415 FETCH (X-GM-MSGID 1642764349836495198 UID 99637) 1586513176.097992 * 40416 FETCH (X-GM-MSGID 1642767015801735363 UID 99639) 1586513176.097992 * 40417 FETCH (X-GM-MSGID 1642768317624659015 UID 99640) 1586513176.097992 * 40418 FETCH (X-GM-MSGID 1642769644449001118 UID 99641) 1586513176.097992 * 40419 FETCH (X-GM-MSGID 1642769732523676610 UID 99642) 1586513176.097992 * 40420 FETCH (X-GM-MSGID 1642770568707862020 UID 99643) 1586513176.097992 * 40421 FETCH (X-GM-MSGID 1642771085461609988 UID 99644) 1586513176.097992 * 40422 FETCH (X-GM-MSGID 1642773510009904699 UID 99645) 1586513176.097992 * 40423 FETCH (X-GM-MSGID 1642773705798532019 UID 99646) 1586513176.097992 * 40424 FETCH (X-GM-MSGID 1642774528944320238 UID 99647) 1586513176.097992 * 40425 FETCH (X-GM-MSGID 1642801024168852473 UID 99648) 1586513176.126970 * 48541 EXISTS 1586513176.126970 1944 OK Success

=============================================================

edited-imapc-log.out

1586512882.754618 1 CAPABILITY 1586512882.790509 2 LOGIN "MY_GMAIL_ADDRESS" "MY_GMAIL_PASSWORD" 1586512883.870605 3 LIST "" "" 1586512884.014301 4 LIST "" "*" 1586512884.162105 5 STATUS "INBOX" (UIDNEXT UIDVALIDITY) 1586512884.318081 6 STATUS "[Google Mail]/All Mail" (UIDNEXT UIDVALIDITY) 1586512884.462890 7 STATUS "[Google Mail]/Bin" (UIDNEXT UIDVALIDITY) 1586512884.607101 8 STATUS "[Google Mail]/Drafts" (UIDNEXT UIDVALIDITY) 1586512884.750230 9 STATUS "[Google Mail]/Important" (UIDNEXT UIDVALIDITY) 1586512884.894624 10 STATUS "[Google Mail]/MY_FOLDER_NAME" (UIDNEXT UIDVALIDITY) 1586512885.037737 11 STATUS "[Google Mail]/Sent Mail" (UIDNEXT UIDVALIDITY) 1586512885.181692 12 STATUS "[Google Mail]/Spam" (UIDNEXT UIDVALIDITY) 1586512885.325389 13 STATUS "[Google Mail]/Starred" (UIDNEXT UIDVALIDITY) 1586512885.469204 14 STATUS "[Google Mail]/mark_as_spam" (UIDNEXT UIDVALIDITY) 1586512885.612928 15 STATUS "[Google Mail]/mining" (UIDNEXT UIDVALIDITY) 1586512885.756217 16 LSUB "" "*" 1586512886.839842 17 LIST "" "*" 1586512886.992192 18 EXAMINE "INBOX" 1586512887.436620 19 UID FETCH 1:* (FLAGS) 1586512893.560147 20 UID FETCH 62,65,69,70,71,72,73,76,78,79,80,81,83,84,85,86,87,88,89,91,92 (X-GM- MSGID) 1586512893.717661 21 UID FETCH 94,95,96,97,98,99,100,102,103,104,106,107,110,111,112,113,114,117,118,1 19,120 (X-GM-MSGID) 1586512893.861887 22 UID FETCH 121,123,127,128,129,131,132,134,136,137,139,140,141,142,143,144,145,147 ,148,149,150 (X-GM-MSGID) 1586512894.009702 23 UID FETCH 151,153,156,160,161,162,346,347,348,349,352,354,355,357,359,361,362,363 ,364,365,366 (X-GM-MSGID) 1586512894.159770 24 UID FETCH 375,376,377,379,380,382,383,384,385,392,393,394,397,400,401,402,403,404 ,406,407,411 (X-GM-MSGID) 1586512894.305890 25 UID FETCH 412,413,415,417,418,419,425,426,427,429,430,433,434,435,436,437,438,439 ,442,443,444 (X-GM-MSGID) 1586512894.450197 26 UID FETCH 447,448,449,454,455,457,459,460,462,465,468,469,470,471,472,473,479,480 ,481,482,483 (X-GM-MSGID) 1586512894.595853 27 UID FETCH 485,486,488,489,490,491,492,494,498,499,501,502,505,507,508,510,511,512 ,513,514,515 (X-GM-MSGID) 1586512894.741263 28 UID FETCH 517,518,519,520,522,523,524,525,526,532,533,534,535,537,538,540,542,543 ,544,545,548 (X-GM-MSGID)

 ....... LOTS OF SIMILAR LINES ......

1586513175.222346 1939 UID FETCH 99497,99498,99500,99501,99502,99503,99506,99508,99510,99511,99512,99515 ,99516,99517,99518,99519,99520,99521,99522,99523,99525 (X-GM-MSGID) 1586513175.368035 1940 UID FETCH 99526,99527,99528,99529,99530,99531,99532,99533,99534,99535,99536,99537 ,99538,99539,99540,99541,99542,99543,99544,99546,99547 (X-GM-MSGID) 1586513175.513463 1941 UID FETCH 99548,99550,99551,99553,99554,99555,99556,99557,99558,99559,99560,99561 ,99562,99563,99564,99565,99566,99567,99568,99569,99570 (X-GM-MSGID) 1586513175.658881 1942 UID FETCH 99571,99572,99573,99575,99576,99577,99578,99579,99580,99581,99582,99583 ,99584,99585,99586,99587,99588,99589,99590,99591,99592 (X-GM-MSGID) 1586513175.804616 1943 UID FETCH 99596,99598,99599,99601,99602,99603,99605,99606,99607,99608,99609,99611 ,99612,99613,99615,99616,99618,99619,99620,99621,99623 (X-GM-MSGID) 1586513175.951965 1944 UID FETCH 99624,99627,99628,99629,99631,99632,99633,99634,99635,99636,99637,99639 ,99640,99641,99642,99643,99644,99645,99646,99647,99648 (X-GM-MSGID) 1586513176.127109 1945 LIST "" "*"

On Fri, 2020-04-10 at 15:13 +0300, Sami Ketola wrote:

...

On 10 Apr 2020, at 14.07, Ben Mulvihill ben.mulvihill@gmail.com wrote:

last answer from gmail:

...

1586513176.126970 1944 OK Success

last commands send by dovecot:

...

1586513175.951965 1944 UID FETCH 99624,99627,99628,99629,99631,99632,99633,99634,99635,99636,99637,9 9639 ,99640,99641,99642,99643,99644,99645,99646,99647,99648 (X-GM-MSGID) 1586513176.127109 1945 LIST "" "*"

So gmail hangs and never gives response to the last LIST command.

Fetcing  X-GM-MSGID is related to POP3 UIDLs. If you don't care about POP3 and gmail labels you can remove gmail-migration from imapc_features.

also you can try to set imapc_cmd_timeout to some low value like 10s to make dovecot to reconnect in case gmail hangs. Default is to wait for 5 mins.

Sami

Thanks again! 

I tried setting imapc_cmd_timeout = 10s and that was enough to take the download process past the ID stage and on to downloading messages.  doveadm still failed the same ASSERT and core dumps, just not so soon. In four attempts  I downloaded 4200, 1800, 0 and 4200  messages respectively. That was yesterday. I tried today and have successfully downloaded a further 20000 messages without any core dumps. I ended up interrupting the process myself because I was  concerned at exceeding my ISPs fair usage limits. But by repeating the command from time to time I should gradually be able to synchronise the whole mailbox.

Ben