doveadm sync duplicating emails

[Paul Robinson]

I'm upgrading a mail server using postfix 2.9.6 and dovecot 2.0.19 to a new
server running postfix 3.4.13 and dovecot 2.3.7.2 (using Maildir format on
both sides).

I had trouble getting doveadm backup to work for one user, while all the
others worked straight away. It would complain about problems on the old
server in the Maildir/dovecot* files. I managed to get it to work by
removing all the dovecot* files under Maildir for that one user (accepting
that this would mean clients having to redownload emails).

>From that point, a doveadm backup worked.

Using roundcube to look at the new mail server, I could see everything
arrived.

At this point, some mail generated on the new mail server would go to the
new mail server, but all other mail is going to the old mail server.

I then tried to use doveadm sync -1R to update the new mail server so the
new server contains all the mail from the old server plus the new mail that
has been locally delivered.

What then happened was that all the new emails delivered to the old server
since the backup were copied to the new server, and all the emails that
were locally delivered on the new mail server were still there.

(So far so good).

But I also saw that every email that was on the old server before the
backup now appeared twice on the new mail server.

Running sync again would mean I had three copies of email.

The sync ran quickly - much too quickly for these extra copies to have been
sent over the network. It appears that the sync made extra local copies on
the new server.

I see no errors in logs that can explain things.

Looking at the email, the files stored in the cur directories are there
multiple times; they have different names, but identical content.

Trying "doveadm deduplicate -F user-list ALL" ran quickly and to no effect
(where user-list is a file with a list of users). Trying "doveadm
deduplicate -F user-list -m ALL" takes 100% cpu and a long time to run and
still running right now after 4 hours. Judging by the disk space usage
freed up, this is going to take a very long time to complete.

Has anybody got any idea what might cause this and what change I should
make?

This was the sync command line:
doveadm -vc /etc/dovecot/dovecot-migration.conf sync -1RF user-list imapc:

For the backup and sync, I used this as the config:

imapc_features = rfc822.size fetch-headers
imapc_host = <snipped>
imapc_ssl = imaps
imapc_port = 993
imapc_user = %n
imapc_master_user = master
imapc_password = <snipped>

mail_prefetch_count = 20
ssl_cipher_list =
EECDH+AESGCM+AES128:EECDH+AESGCM+AES256:EECDH+CHACHA20:EDH+AESGCM+AES128:EDH+AESGCM+AES256:EDH+CHACHA20:EECDH+SHA256+AES128:EECDH+SHA384+AES256:EDH+SHA256+AES128:EDH+SHA256+AES256:EECDH+SHA1+AES128:EECDH+SHA1+AES256:EDH
+SHA1+AES128:EDH+SHA1+AES256:EECDH+HIGH:EDH+HIGH:AESGCM+AES128:AESGCM+AES256:CHACHA20:SHA256+AES128:SHA256+AES256:SHA1+AES128:SHA1+AES256:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK:!KRB5:!aECDH:!DH


mail_location = maildir:~/Maildir
mail_home = /var/vmail/%d/%n/
mail_fsync=never

Thanks,
Paul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dovecot.org/pipermail/dovecot/attachments/20210423/0278cfd9/attachment.html>