Pigeonhole redirect is adding a message-id header when it already exists

[hi at zakaria.website]

On 2022-10-01 22:59, michael.zork at feierfighter.de wrote:
> Hi Zakaria,
> 
> I'm very happy about your email to the mailing list, I thought I'm
> alone with my problem, and it's nice to see that the problem might be
> a bug/problem in pigeonhole. I was not sure about it because I didn't
> get a technical answer from that German provider, so I didn't know it
> it's a bug in pigeonhole or not. I didn't have time to investigate,
> thanks for your time to investigate and pinpoint to the line of code
> where the problem might be!
> 
> Back then, a few months ago, I disabled the "Message-Id duplicate
> check" because of this problem.
> 
> It's just a problem of <0.001% of the emails, most systems generate
> valid Message-IDs. Maybe Alibaba/Aliexpress is the biggest one who
> generates invalid Message-Ids...
> 
> It would be awesome if you could also try to contact
> Alibaba/Aliexpress, and tell them the problem. Maybe they fix it if
> multiple people report it. But I doubt it.
> 
> Let's see what the Dovecot guys say to this problem. Maybe the best
> solution is to remove the invalid Message-Id before adding a new one.
> 
> Michael
> 
> Am 01-Oct-2022 18:24:05 +0200 schrieb hi at zakaria.website:
> 
>> On 2022-10-01 12:49, michael.zork at feierfighter.de wrote:
>>> Hi there,
>>> 
>>> I can confirm this behavior. A few months ago I introduced a
>> milter
>>> which is checking for multiple headers when the RFC says that
>> there
>>> just should be one of them For example "Message-Id".
>>> 
>>> I found the described problem in an email coming from Alibaba,
>> which
>>> had an invalid "Message-Id" header. It didn't contain an "@" sign
>> or
>>> similar. It was RFC-invalid.
>>> 
>>> This email was sent from Alibaba to a German email provider. There
>> was
>>> a redirect at that email provider, pointing to my mailserver.
>>> 
>>> My server rejected the email because there were 2 "Message-Id"
>>> headers: The original invalid "Message-Id" header from Alibaba,
>> and a
>>> new "Message-Id" header from the German provider, which seems to
>> have
>>> been added during the redirect. There were "Dovecot-sieve" headers
>> in
>>> that mail, so my guess was that it happened because of
>>> Dovecot-sieve/pigeonhole implementation.
>>> 
>>> I contacted the email provider, asking for help. Asking if it
>> really
>>> is a bug in pigeonhole (or maybe some other system at that
>> provider,
>>> who knows). And I contacted Alibaba, so they fix the invalid
>>> "Message-Id". I got responses from both, but until now, as far as
>> I
>>> can see, it has not been fixed.
>>> 
>>> The best fix would be (if it really is a bug in pigeonhole), if
>>> pigeonhole fixes the problem, then it's fixed for all users of
>>> Dovecot. I guess Alibaba is not the only sender with an invalid
>>> "Message-ID" header, but that's the only one I saw.
>>> 
>>> Michael
>>> 
>>> Am 01-Oct-2022 14:00:45 +0200 schrieb sriccio at swisscenter.com:
>>> 
>>>>> You wrote in the original email the message was rejected. Sorry
>> I
>>>> don't have login access to my gmail test account anymore since
>> the
>>>> google @#$%@#$% wanted to have me add a phone number.
>>>> 
>>>> In my original post I said that gmail was rejecting the forwards
>>>> because
>>>> of duplicate headers, and that the duplicate header seems to be a
>>>> Message-ID added by pigeonhole when it's "not happy" with the
>>>> original
>>>> mail Message-ID.
>>>> 
>>>> I probably failed to explain the issue clearly and sorry for
>> that.
>>>> 
>>>> Thank you anyway for trying to help :)
>> 
>> Hi Michael,
>> 
>> I just wanted to say sorry, that I sent this empty email by mistake
>> intending nothing else.
>> 
>> Anyhow, thanks so much for your valuable input, it's very much
>> appreciated.
>> 
>> Zakaria.

Good Morning Michael,

No problem. Although, to be clear I didnt point out anything, but its 
Sébastien and the credit shall go for him.

Also, by the way in EXIM MTA, there is verify syntax condition in data 
acl, which checks all email headers and validate that they are standards 
compliant and reject sending any email beforehand to your issue which 
has non-compliant header and offers the ability to remove such headers 
with condition handling e.g. if contained any Sieve header, to request 
removing message id headers with headers_remove and add compliant one 
with add_header, and I guess this will mitigate any mail service 
provider rejection issue.

If you are postfix user and considering switching, let me know, I might 
be able to help.

Zakaria.