Dsync deleting mailboxes due to duplicate UIDs

[Andreas Thienemann]

Hi Sami,

On Sun, 9 Sep 2018, Sami Ketola wrote:

> Is it possible to get imapc rawlogs to analyze? create a directory that 
> is writeable and add -o imapc_rawlog_dir=/path/to/directory to command 
> line.

Tried that. Thanks for the suggestion, that is actually really helpful.
If anyone wants to play along at home, the files in that directory are 
created with the permissions of mail_uid/mail_gid. The dir should be 
world-writable to make it easy.

> Also with recent dovecot release you can tell dsync which header fields 
> to hash when matching mails. Add dsync_hashed_headers=Message-ID to 
> config and dovecot will only mach mails using Message-ID header fileld.

I had seen that parameter but thought it would be prudent to not _just_ 
rely on the Message-ID but instead keep the default "Date Message-ID".

I _think_ I might have found an issue when looking at the log:

1536525811.026071 * 39 FETCH (UID 39 BODY[HEADER.FIELDS (Date Message-ID)] {159}
1536525811.026071 Message-ID: <Pine.LNX.4.30.0207091925420.433-100000 at trinity.knopfdruck.org>
1536525811.026071 Date: Tue, 9 Jul 2002 19:26:04 +0200 (MEST)
1536525811.026071 Date: Tue, 9 Jul 2002 12:36:13 +0200
1536525811.026071
1536525811.026071 )

A duplicate Date header is certainly unexpected. Having a look at the 
original mail on disk on the Cyrus server shows the following Mail:

>From [redacted] Tue Jul  9 19:26:04 2002
X-Sieve: cmu-sieve 1.3
Return-Path: [redacted]
Received: from bender.bawue.de (localhost [127.0.0.1])
         by bender.bawue.de (Postfix) with ESMTP
         id C79CE48CBC; Tue,  9 Jul 2002 19:29:08 +0200 (CEST)
Delivered-To: [redacted]
Received: from [redacted]
         by bender.bawue.de (Postfix) with ESMTP id CFA4848CAA
         for [redacted]; Tue,  9 Jul 2002 19:28:43 +0200 (CEST)
From: [redacted]
X-Sender:  [redacted]
To: [redacted]
Message-ID: <Pine.LNX.4.30.0207091925420.433-100000 at trinity.knopfdruck.org>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN;
   charset=ISO-8859-1
Content-Transfer-Encoding: 8BIT
Subject: [redacted]
Date: Tue, 9 Jul 2002 19:26:04 +0200 (MEST)
Status: RO
X-Status:
X-Keywords:
X-UID: 38

---------- Forwarded message ----------
Date: Tue, 9 Jul 2002 12:36:13 +0200
Subject: Gewinn

[redacted]


So it looks like the Cyrus imapd is incorrectly parsing the mail and 
returning the body content as a header. Comparing the content of UID 39 on 
the remote server will then hash differently than the UID 39 on the local 
dovecot server, which behaves better and only returns a single date 
header.

Following the robustness principle it feels to me that it would make sense 
for dsync to disregard a duplicate header from a remote server and only 
use the first occurence.

Would that be a good approach to the problem? Now that I understand the 
problem I am having, I can just workaround it but it seems to me that 
dsync should handle this case better.

Anyway, thanks a lot for the imapc_rawlog_dir hint, that helped a lot.

cheers,
  Andreas