doveadm sync duplicating emails

[Aki Tuomi]

On 23.4.2021 21.40, Paul Robinson wrote:
> I'm upgrading a mail server using postfix 2.9.6 and dovecot 2.0.19 to
> a new server running postfix 3.4.13 and dovecot 2.3.7.2 (using Maildir
> format on both sides).
>
> I had trouble getting doveadm backup to work for one user, while all
> the others worked straight away. It would complain about problems on
> the old server in the Maildir/dovecot* files. I managed to get it to
> work by removing all the dovecot* files under Maildir for that one
> user (accepting that this would mean clients having to redownload emails).
>
> From that point, a doveadm backup worked.
>
> Using roundcube to look at the new mail server, I could see everything
> arrived.
>
> At this point, some mail generated on the new mail server would go to
> the new mail server, but all other mail is going to the old mail server.
>
> I then tried to use doveadm sync -1R to update the new mail server
> so the new server contains all the mail from the old server plus the
> new mail that has been locally delivered. 
>
> What then happened was that all the new emails delivered to the old
> server since the backup were copied to the new server, and all
> the emails that were locally delivered on the new mail server were
> still there.
>
> (So far so good).
>
> But I also saw that every email that was on the old server before the
> backup now appeared twice on the new mail server.
>
> Running sync again would mean I had three copies of email.
>
> The sync ran quickly - much too quickly for these extra copies to have
> been sent over the network. It appears that the sync made extra local
> copies on the new server.
>
> I see no errors in logs that can explain things.
>
> Looking at the email, the files stored in the cur directories are
> there multiple times; they have different names, but identical content.
>
> Trying "doveadm deduplicate -F user-list ALL" ran quickly and to no
> effect (where user-list is a file with a list of users). Trying
> "doveadm deduplicate -F user-list -m ALL" takes 100% cpu and a long
> time to run and still running right now after 4 hours. Judging by the
> disk space usage freed up, this is going to take a very long time to
> complete.
>
> Has anybody got any idea what might cause this and what change I
> should make?
>
> This was the sync command line:
> doveadm -vc /etc/dovecot/dovecot-migration.conf sync -1RF user-list imapc:
>
> For the backup and sync, I used this as the config:
>
> imapc_features = rfc822.size fetch-headers
> imapc_host = <snipped>
> imapc_ssl = imaps
> imapc_port = 993
> imapc_user = %n
> imapc_master_user = master
> imapc_password = <snipped>
>
> mail_prefetch_count = 20
> ssl_cipher_list =
> EECDH+AESGCM+AES128:EECDH+AESGCM+AES256:EECDH+CHACHA20:EDH+AESGCM+AES128:EDH+AESGCM+AES256:EDH+CHACHA20:EECDH+SHA256+AES128:EECDH+SHA384+AES256:EDH+SHA256+AES128:EDH+SHA256+AES256:EECDH+SHA1+AES128:EECDH+SHA1+AES256:EDH
> +SHA1+AES128:EDH+SHA1+AES256:EECDH+HIGH:EDH+HIGH:AESGCM+AES128:AESGCM+AES256:CHACHA20:SHA256+AES128:SHA256+AES256:SHA1+AES128:SHA1+AES256:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK:!KRB5:!aECDH:!DH
>
>
> mail_location = maildir:~/Maildir
> mail_home = /var/vmail/%d/%n/
> mail_fsync=never
>
> Thanks,
> Paul


Hi!

Can you try running with `doveadm -D` to get debug logs?

Aki

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dovecot.org/pipermail/dovecot/attachments/20210427/db6369e8/attachment.html>