doveadm backup from gmail with imapc

Fri Apr 10 14:07:46 EEST 2020

On Thu, 2020-04-09 at 13:48 +0300, Sami Ketola wrote:
> 
> > 
> > On 31 Mar 2020, at 23.18, Ben Mulvihill <ben.mulvihill at gmail.com>
> > wrote:
> > 
> > Hello again,
> > 
> > I am still stuck I'm afraid.
> > 
> > I now have doveadm backup working perfectly from
> > a small gmail mailbox (a few hundred messages), but
> > when I try the same configuration (apart from usernames
> > and passwords obviously) with a large gmail mailbox
> > (around 60,000 messages), doveadm connects successfully,
> > replicates the gmail folder tree, then produces no further
> > output until about 30 minutes later when the server times
> > out and doveadm crashes. No actual messages are downloaded. 
> > 
> It's a while since we last performed migrations from gmail. Maybe
> they have changed something in the meanwhile.
> Can you record imapc rawlogs from the session to see where it gets
> stuck? It might be gmail throttling the speed.
> 
> Sami
> 

Thanks for the reply. Here are my logs, edited to remove personal
information and lots of repeated lines. Dovecot configuration is 
as above, with the addition of the two options
  imapc_rawlog_dir
  imapc_features = throttle:50:16000:500
Doveadm seems to be fetching lots of 
message IDs but no actual messages, and then it core dumps.

Ben

=============================================================

edited-imapc-log.in

1586512882.754448 * OK Gimap ready for requests from XXX.XXX.XXX.XXX
t129mb20216081wmt
1586512882.790345 * CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA
ID XLIST CHILDREN X-GM-EXT-1 XYZZY SASL-IR AUTH=XOAUTH2 AUTH=PLAIN
AUTH=PLAIN-CLIENTTOKEN AUTH=OAUTHBEARER AUTH=XOAUTH
1586512882.790345 1 OK Thats all she wrote! t129mb20216081wmt
1586512883.870337 * CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA
ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE
CONDSTORE ESEARCH UTF8=ACCEPT LIST-EXTENDED LIST-STATUS LITERAL-
SPECIAL-USE APPENDLIMIT=35651584
1586512883.870337 2 OK MY_GMAIL_ADDRESS authenticated (Success)
1586512884.014073 * LIST (\Noselect) "/" "/"
1586512884.014073 3 OK Success
1586512884.160944 * LIST (\HasNoChildren) "/" "INBOX"
1586512884.160944 * LIST (\HasChildren \Noselect) "/" "[Google Mail]"
1586512884.160944 * LIST (\All \HasNoChildren) "/" "[Google Mail]/All
Mail"
1586512884.160944 * LIST (\HasNoChildren \Trash) "/" "[Google
Mail]/Bin"
1586512884.160944 * LIST (\Drafts \HasNoChildren) "/" "[Google
Mail]/Drafts"
1586512884.160944 * LIST (\HasNoChildren \Important) "/" "[Google
Mail]/Important"
1586512884.160944 * LIST (\HasNoChildren) "/" "[Google
Mail]/MY_FOLDER_NAME"
1586512884.160944 * LIST (\HasNoChildren \Sent) "/" "[Google Mail]/Sent
Mail"
1586512884.160944 * LIST (\HasNoChildren \Junk) "/" "[Google
Mail]/Spam"
1586512884.160944 * LIST (\Flagged \HasNoChildren) "/" "[Google
Mail]/Starred"
1586512884.161789 * LIST (\HasNoChildren) "/" "[Google
Mail]/mark_as_spam"
1586512884.161789 * LIST (\HasNoChildren) "/" "[Google Mail]/mining"
1586512884.161789 4 OK Success
1586512884.317802 * STATUS "INBOX" (UIDNEXT 110434 UIDVALIDITY
602828375)
1586512884.317802 5 OK Success
1586512884.462612 * STATUS "[Google Mail]/All Mail" (UIDNEXT 231431
UIDVALIDITY 602828395)
1586512884.462612 6 OK Success
1586512884.606828 * STATUS "[Google Mail]/Bin" (UIDNEXT 21532
UIDVALIDITY 602828376)
1586512884.606828 7 OK Success
1586512884.749952 * STATUS "[Google Mail]/Drafts" (UIDNEXT 17485
UIDVALIDITY 602828378)
1586512884.749952 8 OK Success
1586512884.894350 * STATUS "[Google Mail]/Important" (UIDNEXT 29288
UIDVALIDITY 602828400)
1586512884.894350 9 OK Success
1586512885.037461 * STATUS "[Google Mail]/MY_FOLDER_NAME" (UIDNEXT 3648
UIDVALIDITY 602828399)
1586512885.037461 10 OK Success
1586512885.181415 * STATUS "[Google Mail]/Sent Mail" (UIDNEXT 14061
UIDVALIDITY 602828374)
1586512885.181415 11 OK Success
1586512885.325056 * STATUS "[Google Mail]/Spam" (UIDNEXT 98883
UIDVALIDITY 602828379)
1586512885.325056 12 OK Success
1586512885.468928 * STATUS "[Google Mail]/Starred" (UIDNEXT 21
UIDVALIDITY 602828380)
1586512885.468928 13 OK Success
1586512885.612611 * STATUS "[Google Mail]/mark_as_spam" (UIDNEXT 76
UIDVALIDITY 602828397)
1586512885.612611 14 OK Success
1586512885.755909 * STATUS "[Google Mail]/mining" (UIDNEXT 23
UIDVALIDITY 602828396)
1586512885.755909 15 OK Success
1586512885.899075 * LSUB (\HasNoChildren) "/" "INBOX"
1586512885.899075 * LSUB (\HasChildren \Noselect) "/" "[Google Mail]"
1586512885.899075 * LSUB (\All \HasNoChildren) "/" "[Google Mail]/All
Mail"
1586512885.900047 * LSUB (\HasNoChildren \Trash) "/" "[Google
Mail]/Bin"
1586512885.900047 * LSUB (\Drafts \HasNoChildren) "/" "[Google
Mail]/Drafts"
1586512885.900047 * LSUB (\HasNoChildren \Important) "/" "[Google
Mail]/Important"
1586512885.900047 * LSUB (\HasNoChildren) "/" "[Google
Mail]/MY_FOLDER_NAME"
1586512885.900047 * LSUB (\HasNoChildren \Sent) "/" "[Google Mail]/Sent
Mail"
1586512885.901074 * LSUB (\HasNoChildren \Junk) "/" "[Google
Mail]/Spam"
1586512885.901074 * LSUB (\Flagged \HasNoChildren) "/" "[Google
Mail]/Starred"
1586512885.901074 * LSUB (\HasNoChildren) "/" "[Google
Mail]/mark_as_spam"
1586512885.901074 * LSUB (\HasNoChildren) "/" "[Google Mail]/mining"
1586512885.901074 16 OK Success
1586512886.990868 * LIST (\HasNoChildren) "/" "INBOX"
1586512886.990868 * LIST (\HasChildren \Noselect) "/" "[Google Mail]"
1586512886.990868 * LIST (\All \HasNoChildren) "/" "[Google Mail]/All
Mail"
1586512886.990868 * LIST (\HasNoChildren \Trash) "/" "[Google
Mail]/Bin"
1586512886.990868 * LIST (\Drafts \HasNoChildren) "/" "[Google
Mail]/Drafts"
1586512886.990868 * LIST (\HasNoChildren \Important) "/" "[Google
Mail]/Important"
1586512886.990868 * LIST (\HasNoChildren) "/" "[Google
Mail]/MY_FOLDER_NAME"
1586512886.990868 * LIST (\HasNoChildren \Sent) "/" "[Google Mail]/Sent
Mail"
1586512886.990868 * LIST (\HasNoChildren \Junk) "/" "[Google
Mail]/Spam"
1586512886.990868 * LIST (\Flagged \HasNoChildren) "/" "[Google
Mail]/Starred"
1586512886.991783 * LIST (\HasNoChildren) "/" "[Google
Mail]/mark_as_spam"
1586512886.991783 * LIST (\HasNoChildren) "/" "[Google Mail]/mining"
1586512886.991783 17 OK Success
1586512887.436407 * FLAGS (\Answered \Flagged \Draft \Deleted \Seen
$NotPhishing $Phishing $has_cal JUNK receipt-handled)
1586512887.436407 * OK [PERMANENTFLAGS ()] Flags permitted.
1586512887.436407 * OK [UIDVALIDITY 602828375] UIDs valid.
1586512887.436407 * 48540 EXISTS
1586512887.436407 * 0 RECENT
1586512887.436407 * OK [UIDNEXT 110434] Predicted next UID.
1586512887.436407 * OK [HIGHESTMODSEQ 8740446]
1586512887.436407 18 OK [READ-ONLY] INBOX selected. (Success)
1586512891.905633 * 1 FETCH (UID 62 FLAGS (\Seen))
1586512891.905633 * 2 FETCH (UID 65 FLAGS (\Seen))
1586512891.905633 * 3 FETCH (UID 69 FLAGS (\Answered \Seen))
1586512891.905633 * 4 FETCH (UID 70 FLAGS (\Answered \Seen))
1586512891.905633 * 5 FETCH (UID 71 FLAGS (\Answered \Seen))
1586512891.905633 * 6 FETCH (UID 72 FLAGS (\Answered \Seen))
1586512891.905633 * 7 FETCH (UID 73 FLAGS (\Seen))
1586512891.905633 * 8 FETCH (UID 76 FLAGS (\Seen))
1586512891.905633 * 9 FETCH (UID 78 FLAGS (\Answered \Seen))
1586512891.905633 * 10 FETCH (UID 79 FLAGS (\Seen))
1586512891.905633 * 11 FETCH (UID 80 FLAGS (\Answered \Seen))
1586512891.905633 * 12 FETCH (UID 81 FLAGS (\Seen))
1586512891.905633 * 13 FETCH (UID 83 FLAGS (\Answered \Seen))
1586512891.907319 * 14 FETCH (UID 84 FLAGS ())
1586512891.907319 * 15 FETCH (UID 85 FLAGS (\Seen))
1586512891.907319 * 16 FETCH (UID 86 FLAGS (\Seen))
1586512891.907319 * 17 FETCH (UID 87 FLAGS (\Answered \Seen))
1586512891.907319 * 18 FETCH (UID 88 FLAGS ())
1586512891.907319 * 19 FETCH (UID 89 FLAGS (\Seen))
1586512891.907319 * 20 FETCH (UID 91 FLAGS (\Answered \Seen))
1586512891.907319 * 21 FETCH (UID 92 FLAGS (\Seen))

....... LOTS OF SIMILAR LINES .......

1586512893.541503 * 48522 FETCH (UID 110415 FLAGS ())
1586512893.541503 * 48523 FETCH (UID 110416 FLAGS ())
1586512893.541503 * 48524 FETCH (UID 110417 FLAGS ())
1586512893.541503 * 48525 FETCH (UID 110418 FLAGS ())
1586512893.541503 * 48526 FETCH (UID 110419 FLAGS ())
1586512893.541503 * 48527 FETCH (UID 110420 FLAGS ())
1586512893.541503 * 48528 FETCH (UID 110421 FLAGS ())
1586512893.541503 * 48529 FETCH (UID 110422 FLAGS ())
1586512893.541503 * 48530 FETCH (UID 110423 FLAGS ())
1586512893.541503 * 48531 FETCH (UID 110424 FLAGS ())
1586512893.541503 * 48532 FETCH (UID 110425 FLAGS ())
1586512893.541503 * 48533 FETCH (UID 110426 FLAGS ())
1586512893.541503 * 48534 FETCH (UID 110427 FLAGS ())
1586512893.541503 * 48535 FETCH (UID 110428 FLAGS ())
1586512893.541503 * 48536 FETCH (UID 110429 FLAGS ())
1586512893.541503 * 48537 FETCH (UID 110430 FLAGS ())
1586512893.541503 * 48538 FETCH (UID 110431 FLAGS ())
1586512893.541503 * 48539 FETCH (UID 110432 FLAGS ())
1586512893.541503 * 48540 FETCH (UID 110433 FLAGS ())
1586512893.541503 19 OK Success
1586512893.704625 * 1 FETCH (X-GM-MSGID 1265813037243496020 UID 62)
1586512893.704625 * 2 FETCH (X-GM-MSGID 1266142060943449934 UID 65)
1586512893.704625 * 3 FETCH (X-GM-MSGID 1266240767076216106 UID 69)
1586512893.704625 * 4 FETCH (X-GM-MSGID 1266317645039054507 UID 70)
1586512893.704625 * 5 FETCH (X-GM-MSGID 1266319104567433759 UID 71)
1586512893.704625 * 6 FETCH (X-GM-MSGID 1266338339735148225 UID 72)
1586512893.704625 * 7 FETCH (X-GM-MSGID 1266350811947568709 UID 73)
1586512893.704625 * 8 FETCH (X-GM-MSGID 1266427483536331721 UID 76)
1586512893.704625 * 9 FETCH (X-GM-MSGID 1266482766386716417 UID 78)
1586512893.704625 * 10 FETCH (X-GM-MSGID 1266512748587747731 UID 79)
1586512893.704625 * 11 FETCH (X-GM-MSGID 1266523171112424759 UID 80)
1586512893.704625 * 12 FETCH (X-GM-MSGID 1266584914541703703 UID 81)
1586512893.704625 * 13 FETCH (X-GM-MSGID 1266626775786784521 UID 83)
1586512893.704625 * 14 FETCH (X-GM-MSGID 1266663913156455749 UID 84)
1586512893.704625 * 15 FETCH (X-GM-MSGID 1266684556515340123 UID 85)
1586512893.704625 * 16 FETCH (X-GM-MSGID 1266684780091647650 UID 86)
1586512893.704625 * 17 FETCH (X-GM-MSGID 1266690080881874411 UID 87)
1586512893.704625 * 18 FETCH (X-GM-MSGID 1266697947934426922 UID 88)
1586512893.704625 * 19 FETCH (X-GM-MSGID 1266841785830673314 UID 89)
1586512893.704625 * 20 FETCH (X-GM-MSGID 1266890981732444471 UID 91)
1586512893.704625 * 21 FETCH (X-GM-MSGID 1266891244711722087 UID 92)
1586512893.704625 20 OK Success
1586512893.861241 * 22 FETCH (X-GM-MSGID 1266959738134105978 UID 94)
1586512893.861241 * 23 FETCH (X-GM-MSGID 1267022593809794009 UID 95)
1586512893.861241 * 24 FETCH (X-GM-MSGID 1267023552222412653 UID 96)
1586512893.861241 * 25 FETCH (X-GM-MSGID 1267028400626761068 UID 97)
1586512893.861241 * 26 FETCH (X-GM-MSGID 1267058789660419022 UID 98)
1586512893.861241 * 27 FETCH (X-GM-MSGID 1267060956576095193 UID 99)
1586512893.861241 * 28 FETCH (X-GM-MSGID 1267078074477602342 UID 100)
1586512893.861241 * 29 FETCH (X-GM-MSGID 1267144516612296284 UID 102)
1586512893.861241 * 30 FETCH (X-GM-MSGID 1267169233934891212 UID 103)
1586512893.861241 * 31 FETCH (X-GM-MSGID 1267181621874937138 UID 104)
1586512893.861241 * 32 FETCH (X-GM-MSGID 1267322905811934828 UID 106)
1586512893.861241 * 33 FETCH (X-GM-MSGID 1267362788296163573 UID 107)
1586512893.861241 * 34 FETCH (X-GM-MSGID 1267409013248654879 UID 110)
1586512893.861241 * 35 FETCH (X-GM-MSGID 1267429601732986948 UID 111)
1586512893.861241 * 36 FETCH (X-GM-MSGID 1267433450902833783 UID 112)
1586512893.861241 * 37 FETCH (X-GM-MSGID 1267433996158684970 UID 113)
1586512893.861241 * 38 FETCH (X-GM-MSGID 1267479985482540073 UID 114)
1586512893.861241 * 39 FETCH (X-GM-MSGID 1267512244483485892 UID 117)
1586512893.861241 * 40 FETCH (X-GM-MSGID 1267512434336402096 UID 118)
1586512893.861241 * 41 FETCH (X-GM-MSGID 1267514033433119045 UID 119)
1586512893.861241 * 42 FETCH (X-GM-MSGID 1267586382946441314 UID 120)
1586512893.861241 21 OK Success
1586512894.009036 * 43 FETCH (X-GM-MSGID 1267586414720205278 UID 121)
1586512894.009036 * 44 FETCH (X-GM-MSGID 1267594567233703154 UID 123)
1586512894.009036 * 45 FETCH (X-GM-MSGID 1267861214156448451 UID 127)

....... LOTS OF SIMILAR LINES .......

1586513176.097992 * 40414 FETCH (X-GM-MSGID 1642763475047250557 UID
99636)
1586513176.097992 * 40415 FETCH (X-GM-MSGID 1642764349836495198 UID
99637)
1586513176.097992 * 40416 FETCH (X-GM-MSGID 1642767015801735363 UID
99639)
1586513176.097992 * 40417 FETCH (X-GM-MSGID 1642768317624659015 UID
99640)
1586513176.097992 * 40418 FETCH (X-GM-MSGID 1642769644449001118 UID
99641)
1586513176.097992 * 40419 FETCH (X-GM-MSGID 1642769732523676610 UID
99642)
1586513176.097992 * 40420 FETCH (X-GM-MSGID 1642770568707862020 UID
99643)
1586513176.097992 * 40421 FETCH (X-GM-MSGID 1642771085461609988 UID
99644)
1586513176.097992 * 40422 FETCH (X-GM-MSGID 1642773510009904699 UID
99645)
1586513176.097992 * 40423 FETCH (X-GM-MSGID 1642773705798532019 UID
99646)
1586513176.097992 * 40424 FETCH (X-GM-MSGID 1642774528944320238 UID
99647)
1586513176.097992 * 40425 FETCH (X-GM-MSGID 1642801024168852473 UID
99648)
1586513176.126970 * 48541 EXISTS
1586513176.126970 1944 OK Success

=============================================================

edited-imapc-log.out

1586512882.754618 1 CAPABILITY
1586512882.790509 2 LOGIN "MY_GMAIL_ADDRESS" "MY_GMAIL_PASSWORD"
1586512883.870605 3 LIST "" ""
1586512884.014301 4 LIST "" "*"
1586512884.162105 5 STATUS "INBOX" (UIDNEXT UIDVALIDITY)
1586512884.318081 6 STATUS "[Google Mail]/All Mail" (UIDNEXT
UIDVALIDITY)
1586512884.462890 7 STATUS "[Google Mail]/Bin" (UIDNEXT UIDVALIDITY)
1586512884.607101 8 STATUS "[Google Mail]/Drafts" (UIDNEXT UIDVALIDITY)
1586512884.750230 9 STATUS "[Google Mail]/Important" (UIDNEXT
UIDVALIDITY)
1586512884.894624 10 STATUS "[Google Mail]/MY_FOLDER_NAME" (UIDNEXT
UIDVALIDITY)
1586512885.037737 11 STATUS "[Google Mail]/Sent Mail" (UIDNEXT
UIDVALIDITY)
1586512885.181692 12 STATUS "[Google Mail]/Spam" (UIDNEXT UIDVALIDITY)
1586512885.325389 13 STATUS "[Google Mail]/Starred" (UIDNEXT
UIDVALIDITY)
1586512885.469204 14 STATUS "[Google Mail]/mark_as_spam" (UIDNEXT
UIDVALIDITY)
1586512885.612928 15 STATUS "[Google Mail]/mining" (UIDNEXT
UIDVALIDITY)
1586512885.756217 16 LSUB "" "*"
1586512886.839842 17 LIST "" "*"
1586512886.992192 18 EXAMINE "INBOX"
1586512887.436620 19 UID FETCH 1:* (FLAGS)
1586512893.560147 20 UID FETCH
62,65,69,70,71,72,73,76,78,79,80,81,83,84,85,86,87,88,89,91,92 (X-GM-
MSGID)
1586512893.717661 21 UID FETCH
94,95,96,97,98,99,100,102,103,104,106,107,110,111,112,113,114,117,118,1
19,120 (X-GM-MSGID)
1586512893.861887 22 UID FETCH
121,123,127,128,129,131,132,134,136,137,139,140,141,142,143,144,145,147
,148,149,150 (X-GM-MSGID)
1586512894.009702 23 UID FETCH
151,153,156,160,161,162,346,347,348,349,352,354,355,357,359,361,362,363
,364,365,366 (X-GM-MSGID)
1586512894.159770 24 UID FETCH
375,376,377,379,380,382,383,384,385,392,393,394,397,400,401,402,403,404
,406,407,411 (X-GM-MSGID)
1586512894.305890 25 UID FETCH
412,413,415,417,418,419,425,426,427,429,430,433,434,435,436,437,438,439
,442,443,444 (X-GM-MSGID)
1586512894.450197 26 UID FETCH
447,448,449,454,455,457,459,460,462,465,468,469,470,471,472,473,479,480
,481,482,483 (X-GM-MSGID)
1586512894.595853 27 UID FETCH
485,486,488,489,490,491,492,494,498,499,501,502,505,507,508,510,511,512
,513,514,515 (X-GM-MSGID)
1586512894.741263 28 UID FETCH
517,518,519,520,522,523,524,525,526,532,533,534,535,537,538,540,542,543
,544,545,548 (X-GM-MSGID)

 ....... LOTS OF SIMILAR LINES ......

1586513175.222346 1939 UID FETCH
99497,99498,99500,99501,99502,99503,99506,99508,99510,99511,99512,99515
,99516,99517,99518,99519,99520,99521,99522,99523,99525 (X-GM-MSGID)
1586513175.368035 1940 UID FETCH
99526,99527,99528,99529,99530,99531,99532,99533,99534,99535,99536,99537
,99538,99539,99540,99541,99542,99543,99544,99546,99547 (X-GM-MSGID)
1586513175.513463 1941 UID FETCH
99548,99550,99551,99553,99554,99555,99556,99557,99558,99559,99560,99561
,99562,99563,99564,99565,99566,99567,99568,99569,99570 (X-GM-MSGID)
1586513175.658881 1942 UID FETCH
99571,99572,99573,99575,99576,99577,99578,99579,99580,99581,99582,99583
,99584,99585,99586,99587,99588,99589,99590,99591,99592 (X-GM-MSGID)
1586513175.804616 1943 UID FETCH
99596,99598,99599,99601,99602,99603,99605,99606,99607,99608,99609,99611
,99612,99613,99615,99616,99618,99619,99620,99621,99623 (X-GM-MSGID)
1586513175.951965 1944 UID FETCH
99624,99627,99628,99629,99631,99632,99633,99634,99635,99636,99637,99639
,99640,99641,99642,99643,99644,99645,99646,99647,99648 (X-GM-MSGID)
1586513176.127109 1945 LIST "" "*"