Hello

Short version:

Dovecot’s dsync from release 2.3.7.2 (default install from focal repository) running under Ubuntu 20.04.6 LTS server is not deleting mails on the identical remote server.

Background:

I have installed this configuration using automated scripts several times for customers. Functionally identical installations done on Ubuntu LTS Servers 14.04, 16.04 and 18.04 are running for years without problem. Now I have tried several times to do the same with Ubuntu 20.04 and can’t get dsync to work properly. Last year I could fall back onto Ubuntu 18.04. This time I can’t and must install the latest versions.

Setup:

- Two VM’s on ESXI 6.7U3 with 36 GB of disk space and 8 GB ram, 10k spinning disks or SSD with HW RAID1

- Mariadb for mail users (master – master synched)

- Postfixadmin to manage the users

- Postfix

- Dovecot

- Some with Roundcube

- The clients can connect to either one server and connect SMTPS (465 or 587) or POPS (995) to the servers.

- The two servers connect on a dedicated network without routing (SSL, port > 49000). The synchronization network has no mail traffic except the one that synchronizes the servers.

- Additionally the two servers are ntp peers within a few ms - but that is only relevant to know when we start comparing logs from the machines

Problem description:

- When I send one mail to one account, it is immediately synched to the other server. This is also seen in the dovecot log and I can connect to any one of the two servers to download it.

- When I read and delete the mail on one server, it is NOT deleted on the other server. I do not see a trigger in the dovecot log that kicks on the synchronization (like I do when sending mails). As a result, the mail can be downloaded and deleted from both servers.

- When I send a new mail to one of the two servers, the synchronization takes place and the mail that I read and deleted is also gone on the remote (backup) server.

- Changing users, rights in dsynch.conf , SSL on or off, or connecting to PLAIN ports 25, 110 or 143 is never improved it. Iptables or ufw are not in use.

- In my case I have systems using the server that’ don’t like duplicate mails and the servers fail the acceptance test criteria.

- This causes duplicate mail if one client is connecting to the other server. Since a new mail fixes that duplicate mail problem, it is likely that up to now nobody detects it and complains about this. Especially if the servers would run in a primary – backup mode. I assume that’s why I could not find reportings on the web until now.

Dovecot –n:

See attachments

Log from my automated test system using Ubuntu 20.01 and focal default dovecot package from repository (dovecot 2.3.7.2):

2021-11-22 18:20:06.30 - --------------------------------------------------------------------------------------------------------------------------------------------------

2021-11-22 18:20:06.81 - SMTP Server: mail-pri Port 465 TLS - User: user@test.ch - To: user@test.ch - Subject: Test 2021-11-22 18:20:06, Msg 00001

2021-11-22 18:20:14.90 - --------------------------------------------------------------------------------------------------------------------------------------------------

2021-11-22 18:20:15.37 - POP3 Server: mail-pri Port 995 TLS - User: user@test.ch - From: user@test.ch - Subject: Test 2021-11-22 18:20:06, Msg 00001 (retr no delete)

2021-11-22 18:20:17.32 - --------------------------------------------------------------------------------------------------------------------------------------------------

2021-11-22 18:20:17.77 - POP3 Server: mail-sec Port 995 TLS - User: user@test.ch - From: user@test.ch - Subject: Test 2021-11-22 18:20:06, Msg 00001 (retr no delete) -> synchronization worked when sending mails

2021-11-22 18:20:27.52 - --------------------------------------------------------------------------------------------------------------------------------------------------

2021-11-22 18:20:27.96 - POP3 Server: mail-pri Port 995 TLS - User: user@test.ch - From: user@test.ch - Subject: Test 2021-11-22 18:20:06, Msg 00001 (retr and delete)

2021-11-22 18:20:30.72 - --------------------------------------------------------------------------------------------------------------------------------------------------

2021-11-22 18:20:31.17 - POP3 Server: mail-sec Port 995 TLS - User: user@test.ch - From: user@test.ch - Subject: Test 2021-11-22 18:20:06, Msg 00001 (retr and delete) -> synchronization failed after POP3 connection with delete

2021-11-22 18:20:53.06 - --------------------------------------------------------------------------------------------------------------------------------------------------

2021-11-22 18:20:53.66 - SMTP Server: mail-sec Port 465 TLS - User: user@test.ch - To: user@test.ch - Subject: Test 2021-11-22 18:20:53, Msg 00002

2021-11-22 18:20:57.91 - --------------------------------------------------------------------------------------------------------------------------------------------------

2021-11-22 18:20:58.33 - POP3 Server: mail-pri Port 995 TLS - User: user@test.ch - From: user@test.ch - Subject: Test 2021-11-22 18:20:53, Msg 00002 (retr no delete)

2021-11-22 18:21:00.82 - --------------------------------------------------------------------------------------------------------------------------------------------------

2021-11-22 18:21:01.27 - POP3 Server: mail-sec Port 995 TLS - User: user@test.ch - From: user@test.ch - Subject: Test 2021-11-22 18:20:53, Msg 00002 (retr no delete)

2021-11-22 18:21:04.65 - --------------------------------------------------------------------------------------------------------------------------------------------------

2021-11-22 18:21:05.11 - POP3 Server: mail-pri Port 995 TLS - User: user@test.ch - From: user@test.ch - Subject: Test 2021-11-22 18:20:53, Msg 00002 (retr and delete)

2021-11-22 18:21:08.95 - --------------------------------------------------------------------------------------------------------------------------------------------------

2021-11-22 18:21:09.40 - POP3 Server: mail-sec Port 995 TLS - User: user@test.ch - From: user@test.ch - Subject: Test 2021-11-22 18:20:53, Msg 00002 (retr and delete)

2021-11-22 18:21:39.50 - --------------------------------------------------------------------------------------------------------------------------------------------------

Reference log with the same installation using Ubuntu 18.04 and bionic default package form repository (dovecot 1:2.2.33.2-1ubuntu4.7):

2021-11-23 13:55:06.93 - --------------------------------------------------------------------------------------------------------------------------------------------------

2021-11-23 13:54:49.86 - SMTP Server: mail-pri Port 465 TLS - User: user@test.ch - To: user@test.ch - Subject: Test 2021-11-23 13:54:49, Msg 00001

2021-11-23 13:55:06.93 - --------------------------------------------------------------------------------------------------------------------------------------------------

2021-11-23 13:55:07.47 - POP3 Server: mail-pri Port 995 TLS - User: user@test.ch - From: user@test.ch - Subject: Test 2021-11-23 13:54:49, Msg 00001 (retr no delete)

2021-11-23 13:55:13.10 - --------------------------------------------------------------------------------------------------------------------------------------------------

2021-11-23 13:55:13.66 - POP3 Server: mail-sec Port 995 TLS - User: user@test.ch - From: user@test.ch - Subject: Test 2021-11-23 13:54:49, Msg 00001 (retr no delete) -> synchronization OK for sending mails

2021-11-23 13:55:22.32 - --------------------------------------------------------------------------------------------------------------------------------------------------

2021-11-23 13:55:22.87 - POP3 Server: mail-pri Port 995 TLS - User: user@test.ch - From: user@test.ch - Subject: Test 2021-11-23 13:54:49, Msg 00001 (retr and delete)

2021-11-23 13:55:27.91 - --------------------------------------------------------------------------------------------------------------------------------------------------

2021-11-23 13:55:28.35 - POP3 Server: mail-sec Port 995 TLS - User: user@test.ch - no mails found -> synchronization OK for deleting mails

2021-11-23 13:55:36.48 - --------------------------------------------------------------------------------------------------------------------------------------------------

2021-11-23 13:55:37.00 - SMTP Server: mail-sec Port 465 TLS - User: user@test.ch - To: user@test.ch - Subject: Test 2021-11-23 13:55:36, Msg 00002

2021-11-23 13:55:45.94 - --------------------------------------------------------------------------------------------------------------------------------------------------

2021-11-23 13:55:46.48 - POP3 Server: mail-pri Port 995 TLS - User: user@test.ch - From: user@test.ch - Subject: Test 2021-11-23 13:55:36, Msg 00002 (retr no delete)

2021-11-23 13:55:55.59 - --------------------------------------------------------------------------------------------------------------------------------------------------

2021-11-23 13:55:56.14 - POP3 Server: mail-sec Port 995 TLS - User: user@test.ch - From: user@test.ch - Subject: Test 2021-11-23 13:55:36, Msg 00002 (retr no delete)

2021-11-23 13:56:05.70 - --------------------------------------------------------------------------------------------------------------------------------------------------

2021-11-23 13:56:06.26 - POP3 Server: mail-pri Port 995 TLS - User: user@test.ch - From: user@test.ch - Subject: Test 2021-11-23 13:55:36, Msg 00002 (retr and delete)

2021-11-23 13:56:38.70 - --------------------------------------------------------------------------------------------------------------------------------------------------

2021-11-23 13:56:39.12 - POP3 Server: mail-sec Port 995 TLS - User: user@test.ch - no mails found

P.S. I compared the dovecot and postfix config files between these two versions and both servers and they are functionally identical except where they need to be different.

I would appreciate any help here. If needed I can create log files that show that nothing is triggered as well as produce wireshark captures (port 25 and 110) or the link for the synch (currently with ssl but I could change that if needed).

Thanks in advance

Otto