Syncing emails from external server like Gmail
Greeting
I've managed to backup my emails from external email server via dsync and imapc: protocol like this:
doveadm -Dv -o imapc_user='user@gmail.com' -o imapc_password="pass" -o imapc_host=imap.gmail.com backup -R -u duke@ejoom.com imapc:
taken from https://wiki.dovecot.org/Migration/Dsync
I wish to do regular syncing of 5min interval to substitute 'getmail' program.
When I repeat the command It doesn't download all the attachments(!good) but It seems that at retrieve all the the GUIDs and UIDs(!bad) again that take quite long considering 30K emails. The question is how to speed it up and retrieve only new emails? Does anyone have a solution for regular synchronization? What is the best way?
Logs look like this: https://gist.githubusercontent.com/vnagara/87e0da1b21b51f68fd74270b3ec653b3/...
It wrights that "Mailboxes are equal" but still verifies all the UIDs again.
Thanks.
On 11. Jul 2020, at 14.22, Vitalii <vnagara@yandex.com> wrote:
Greeting
I've managed to backup my emails from external email server via dsync and imapc: protocol like this:
doveadm -Dv -o imapc_user='user@gmail.com' -o imapc_password="pass" -o imapc_host=imap.gmail.com backup -R -u duke@ejoom.com imapc:
It basicly does it to verify mail -> UID pairing. You can try to save imapc indexes locally by defining imapc location as: imapc:~/.imapc to see if it can be skipped.
Sami
On Sat, 11 Jul 2020 20:36:44 +0300 Sami Ketola <sami.ketola@dovecot.fi> wrote:
On 11. Jul 2020, at 14.22, Vitalii <vnagara@yandex.com> wrote:
Greeting
I've managed to backup my emails from external email server via dsync and imapc: protocol like this:
doveadm -Dv -o imapc_user='user@gmail.com' -o imapc_password="pass" -o imapc_host=imap.gmail.com backup -R -u duke@ejoom.com imapc:
It basicly does it to verify mail -> UID pairing. You can try to save imapc indexes locally by defining imapc location as: imapc:~/.imapc to see if it can be skipped.
Sami
That's awesome Sami, thanks a lot. For gmail.com account with 43000 emails it went down from 6 mins to 15seconds (24 times faster). For yandex.com account with 15000 emails it went down from 32min to 28seconds (60 times faster).
I started testing it more and found out that it works as a charm with new emails and move operations. However, it fails when I delete (expunge) email on gmail server. On backup I get this error:
Panic: file imapc-sync.c: line 328 (imapc_initial_sync_check): assertion failed: (mail_index_is_expunged(view, lseq) || seq_range_exists(&ctx->mbox->delayed_expunged_uids, luid))
(Attached a bigger log)
On 12. Jul 2020, at 12.54, Vitalii <vnagara@yandex.com> wrote:
Panic: file imapc-sync.c: line 328 (imapc_initial_sync_check): assertion failed: (mail_index_is_expunged(view, lseq) || seq_range_exists(&ctx->mbox->delayed_expunged_uids, luid))
what dovecot version is this? I think there was bug around this in some version.
Sami
On 12. Jul 2020, at 23.38, Sami Ketola <sami.ketola@dovecot.fi> wrote:
On 12. Jul 2020, at 12.54, Vitalii <vnagara@yandex.com> wrote:
Panic: file imapc-sync.c: line 328 (imapc_initial_sync_check): assertion failed: (mail_index_is_expunged(view, lseq) || seq_range_exists(&ctx->mbox->delayed_expunged_uids, luid))
what dovecot version is this? I think there was bug around this in some version.
There was. Fixed in 2.2.36.
Sami
On Sun, 12 Jul 2020 23:45:07 +0300 Sami Ketola <sami.ketola@dovecot.fi> wrote:
On 12. Jul 2020, at 23.38, Sami Ketola <sami.ketola@dovecot.fi> wrote:
On 12. Jul 2020, at 12.54, Vitalii <vnagara@yandex.com> wrote:
Panic: file imapc-sync.c: line 328 (imapc_initial_sync_check): assertion failed: (mail_index_is_expunged(view, lseq) || seq_range_exists(&ctx->mbox->delayed_expunged_uids, luid))
what dovecot version is this? I think there was bug around this in some version.
There was. Fixed in 2.2.36.
Sami
I have v2.3.4.1 (f79e8e7e4). Then supposed to be fixed :(.
Thanks, Vitalii
On Mon, 13 Jul 2020 09:24:26 +0300 (EEST) Aki Tuomi <aki.tuomi@open-xchange.com> wrote:
2.2 and 2.3 had parallel versions for a while, so can you try with latest dovecot from https://repo.dovecot.org/ ? </div> Aki Tuomi</pre>
Thanks Aki, now it works perfectly. Seems latest debian 10 has packages released in Nov 2018. Didn't expect that.
For completeness of my work and public record I provide here full command that I use now for syncing Gmail (and it may work in both ways actually, if you don't delete INBOX somehow as I did :D)
#!/bin/bash
doveadm -Dv -o imapc_user=user@gmail.com -o
imapc_password="mypassword" -o imapc_host=imap.gmail.com
backup -a 'virtual/All' -O '-$GmailHaveLabels' -x '\Important'
-R -u user@gmail.com imapc:~/.imapc
Plus config:
imapc_features = rfc822.size gmail-migration imapc_features = $imapc_features fetch-headers mail_prefetch_count = 20 # for SSL: imapc_port = 993 imapc_ssl = imaps ssl_client_ca_dir = /etc/ssl/certs imapc_ssl_verify = yes
Tested on Dovecot v2.3.4.1 & v2.3.10.1(partially)
I don't exclude Starred folder ('\Flagged') as it is small and it's easier to look up flagged BY ME emails (versus 'Important' folder). And yes, I can have local user "user@gmail.com" it doesn't cause conflicts even so It may be confusing.
Thanks, Vitalii Nagara
On 13. Jul 2020, at 1.47, Vitalii <vnagara@yandex.com> wrote:
On Sun, 12 Jul 2020 23:45:07 +0300 Sami Ketola <sami.ketola@dovecot.fi> wrote:
On 12. Jul 2020, at 23.38, Sami Ketola <sami.ketola@dovecot.fi> wrote:
On 12. Jul 2020, at 12.54, Vitalii <vnagara@yandex.com> wrote:
Panic: file imapc-sync.c: line 328 (imapc_initial_sync_check): assertion failed: (mail_index_is_expunged(view, lseq) || seq_range_exists(&ctx->mbox->delayed_expunged_uids, luid))
what dovecot version is this? I think there was bug around this in some version.
There was. Fixed in 2.2.36.
Sami
I have v2.3.4.1 (f79e8e7e4). Then supposed to be fixed :(.
dovecot 2.2.x and 2.3.x had separate development cycles at that time already. There is fixes around this at least in releases 2.3.3 and 2.3.7. Maybe even later versions. Can you try latest release from https://repo.dovecot.org/ <https://repo.dovecot.org/>
Sami
Am 11.07.2020 um 13:22 schrieb Vitalii:
Greeting
I've managed to backup my emails from external email server via dsync and imapc: protocol like this:
doveadm -Dv -o imapc_user='user@gmail.com' -o imapc_password="pass" -o imapc_host=imap.gmail.com backup -R -u duke@ejoom.com imapc:
taken from https://wiki.dovecot.org/Migration/Dsync
I wish to do regular syncing of 5min interval to substitute 'getmail' program.
When I repeat the command It doesn't download all the attachments(!good) but It seems that at retrieve all the the GUIDs and UIDs(!bad) again that take quite long considering 30K emails. The question is how to speed it up and retrieve only new emails? Does anyone have a solution for regular synchronization? What is the best way?
Logs look like this: https://gist.githubusercontent.com/vnagara/87e0da1b21b51f68fd74270b3ec653b3/...
It wrights that "Mailboxes are equal" but still verifies all the UIDs again.
Thanks.
perhaps this help
getmail should use /usr/lib/dovecot/deliver
https://blog.sys4.de/abholdienst-fur-mail-de.html
and/or imapsync is your friend
https://blog.sys4.de/mail-migration-mit-imapsync-de.html
-- [*] sys4 AG
https://sys4.de, +49 (89) 30 90 46 64 Schleißheimer Straße 26/MG,80333 München
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Marc Schiffbauer, Wolfgang Stief Aufsichtsratsvorsitzender: Florian Kirstein
On Mon, 13 Jul 2020 10:50:09 +0200 Robert Schetterer <rs@sys4.de> wrote:
perhaps this help
getmail should use /usr/lib/dovecot/deliver
That is what I used before and wanted to switch off. Here is the 2 main reasons:
- It add extra header overhead similar to: Received by localost at time, etc....)
- It put emails in local delivery queue so syncing becomes slow.
- only backup, no syncing back.
https://blog.sys4.de/abholdienst-fur-mail-de.html
and/or imapsync is your friend
Don't know about imapsync but I believe it has the same disadvantages as it isn't native dovecot.
Am 13.07.2020 um 11:23 schrieb Vitalii:
On Mon, 13 Jul 2020 10:50:09 +0200 Robert Schetterer <rs@sys4.de> wrote:
perhaps this help
getmail should use /usr/lib/dovecot/deliver
That is what I used before and wanted to switch off. Here is the 2 main reasons:
- It add extra header overhead similar to: Received by localost at time, etc....)
- It put emails in local delivery queue so syncing becomes slow.
- only backup, no syncing back.
https://blog.sys4.de/abholdienst-fur-mail-de.html
and/or imapsync is your friend
Don't know about imapsync but I believe it has the same disadvantages as it isn't native dovecot.
its pure imap fetching, and it works like charme
-- [*] sys4 AG
https://sys4.de, +49 (89) 30 90 46 64 Schleißheimer Straße 26/MG,80333 München
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Marc Schiffbauer, Wolfgang Stief Aufsichtsratsvorsitzender: Florian Kirstein
participants (4)
-
Aki Tuomi
-
Robert Schetterer
-
Sami Ketola
-
Vitalii