Pigeonhole redirect is adding a message-id header when it already exists
Hello,
We are using sieve filter to redirect incoming mails to an external mailbox (gmail) an we are seeing some forwarded e-mails being rejected by gmail because of duplicate headers.
I think was able to identify the probable cause of the duplicate headers: It seems that when the original message's "Message-ID" is not compliant, pigeonhole acts like if the header is not existing and adds it's own Message-ID.
This result in having two "Message-ID" headers being submitted to the destination mail server.
Original Message-ID: Message-ID: 1883biz_pay_after_purchase:0:0_572392900$ae7ed6e4d53b424c84aaf83b30c507e7
Pigeon hole adds his own Message-ID: Message-ID: <dovecot-sieve-1664386049-21512-0@510faa1ff2d9>
Gmail answer: 550-5.7.1 [94.103.100.17] Our system has detected that this message is not RFC 550-5.7.1 5322 compliant: duplicate headers. To reduce the amount of spam sent 550-5.7.1 to Gmail, this message has been blocked. Please review 550 5.7.1 RFC 5322 specifications for more information.
While I understand the original "Message-ID" seems to be all but compliant with RFCs, shouldn't a "Message-ID" be added only when "Message-ID" is really missing ? When redirecting a message, If the original "Message-ID" exists but is bogus shouldn't it still be forwarded untouched ?
Also, maybe if pigeonhole is adding it's own "Message-ID", shouldn't it remove any previous "Message-ID" to avoid generating duplicate headers ? (but maybe it could break some DKIM stuff?)
Any ideas how to workaround this issue ?
Thanks a lot for your feedbacks!
Sébastien RICCIO SYSTEM ADMINISTRATOR P +41 840 888 888 F +41 840 888 000 M sriccio@swisscenter.com<mailto:sriccio@swisscenter.com>
We are using sieve filter to redirect incoming mails to an external mailbox (gmail) an we are seeing some forwarded e-mails being rejected by gmail because of duplicate headers.
How are you redirecting, like this[1]? I tested this ages ago. I am mostly testing with own gmail account, so can't imagine this not working. It was an older 2.2 server I was testing this on.
[1] if false # true { redirect :copy "shit@gmail.com"; }
How are you redirecting, like this[1]? I tested this ages ago. I am mostly testing with own gmail account, so can't imagine this not working. It was an older 2.2 server I was testing this on.
[1] if false # true { redirect :copy "shit@gmail.com"; }
Hello Marc,
It's redirected using the rule added by SOGo when you enable mail forwarding for the mailbox. Basically it's only this line:
redirect "stuff@gmail.com";
The redirect works, the problem is gmail changed their anti spam policies and are since a few months rejecting messages that have duplicate mail headers and in this particular case, Message-ID.
So all the incoming mails that have a non-RFC compliant Message-ID are being added a secondary Message-ID by pigeonhole, before it is relayed to the target mailbox.
To reproduce the issue you need to have:
- An incoming message with a non-RFC compliant Message-ID to your mailbox
- A forward of your mailbox to a gmail account
I guess the double Message-ID thing in pigeonhole has always been there, but now it is more visible since gmail check for duplcate headers..
I would say the culprit is around here: https://github.com/dovecot/pigeonhole/blob/1cb177cc2329cd565023c8f7a19872403...
and more precisely here: https://github.com/dovecot/pigeonhole/blob/1cb177cc2329cd565023c8f7a19872403...
Even though the original message has a Message-ID, somehow msgdata->id seems to be empty so it adds one. Maybe the function populating msgdata->id (whch TBH i wan't able to locate) is using some pattern to validate the Message-ID against the format specified in RFC and it doesn't match, so variable is then null. But that's only my assumptions.
Kind regards.
How are you redirecting, like this[1]? I tested this ages ago. I am
mostly testing with own gmail account, so can't imagine this not working. It was an older 2.2 server I was testing this on.
[1] if false # true { redirect :copy "shit@gmail.com"; }
Hello Marc,
It's redirected using the rule added by SOGo when you enable mail forwarding for the mailbox. Basically it's only this line:
redirect "stuff@gmail.com";
The redirect works, the problem is gmail changed their anti spam policies and are since a few months rejecting messages that have duplicate mail headers and in this particular case, Message-ID.
So all the incoming mails that have a non-RFC compliant Message-ID are being added a secondary Message-ID by pigeonhole, before it is relayed to the target mailbox.
Oct 1 13:31:46 xxxx sendmail[30321]: 291BVjjx030318: to=<xxxx@gmail.com>, delay=00:00:01, xdelay=00:00:01, mailer=esmtp, pri=122536, relay=gmail-smtp-in.l.google.com. [142.250.102.27], dsn=2.0.0, stat=Sent (OK 1664623906 gs19-20020a1709072d1300b00777a40d515dsi4096082ejc.456 - gsmtp)
I just tested for you, enabled the sieve forward, send test mail and the forward is being accepted by google.
Oct 1 13:31:46 xxxx sendmail[30321]: 291BVjjx030318: to=<xxxx@gmail.com>, delay=00:00:01, xdelay=00:00:01, mailer=esmtp, pri=122536, relay=gmail-smtp-in.l.google.com. [142.250.102.27], dsn=2.0.0, stat=Sent (OK 1664623906 gs19-20020a1709072d1300b00777a40d515dsi4096082ejc.456 - gsmtp)
I just tested for you, enabled the sieve forward, send test mail and the forward is being accepted by google.
Thanks for the test. However, does your test mail had a "bogus" Message-ID header in it like I tried to explain ?
For example try to send a mail that has the following Message-ID header:
Message-ID: 1883biz_pay_after_purchase:0:0_572392900$ae7ed6e4d53b424c84aaf83b30c507e7
Oct 1 13:31:46 xxxx sendmail[30321]: 291BVjjx030318: to=<xxxx@gmail.com>, delay=00:00:01, xdelay=00:00:01, mailer=esmtp, pri=122536, relay=gmail-smtp-in.l.google.com. [142.250.102.27], dsn=2.0.0, stat=Sent (OK 1664623906 gs19- 20020a1709072d1300b00777a40d515dsi4096082ejc.456 - gsmtp)
I just tested for you, enabled the sieve forward, send test mail and the forward is being accepted by google.
Thanks for the test. However, does your test mail had a "bogus" Message-ID header in it like I tried to explain ?
You wrote in the original email the message was rejected. Sorry I don't have login access to my gmail test account anymore since the google @#$%@#$% wanted to have me add a phone number.
You wrote in the original email the message was rejected. Sorry I don't have login access to my gmail test account anymore since the google @#$%@#$% wanted to have me add a phone number.
In my original post I said that gmail was rejecting the forwards because of duplicate headers, and that the duplicate header seems to be a Message-ID added by pigeonhole when it's "not happy" with the original mail Message-ID.
I probably failed to explain the issue clearly and sorry for that.
Thank you anyway for trying to help :)
Hi there, I can confirm this behavior. A few months ago I introduced a milter which is checking for multiple headers when the RFC says that there just should be one of them For example "Message-Id". I found the described problem in an email coming from Alibaba, which had an invalid "Message-Id" header. It didn't contain an "@" sign or similar. It was RFC-invalid. This email was sent from Alibaba to a German email provider. There was a redirect at that email provider, pointing to my mailserver. My server rejected the email because there were 2 "Message-Id" headers: The original invalid "Message-Id" header from Alibaba, and a new "Message-Id" header from the German provider, which seems to have been added during the redirect. There were "Dovecot-sieve" headers in that mail, so my guess was that it happened because of Dovecot-sieve/pigeonhole implementation. I contacted the email provider, asking for help. Asking if it really is a bug in pigeonhole (or maybe some other system at that provider, who knows). And I contacted Alibaba, so they fix the invalid "Message-Id". I got responses from both, but until now, as far as I can see, it has not been fixed. The best fix would be (if it really is a bug in pigeonhole), if pigeonhole fixes the problem, then it's fixed for all users of Dovecot. I guess Alibaba is not the only sender with an invalid "Message-ID" header, but that's the only one I saw. Michael
Am 01-Oct-2022 14:00:45 +0200 schrieb sriccio@swisscenter.com:
You wrote in the original email the message was rejected. Sorry I don't have login access to my gmail test account anymore since the google @#$%@#$% wanted to have me add a phone number.
In my original post I said that gmail was rejecting the forwards because of duplicate headers, and that the duplicate header seems to be a Message-ID added by pigeonhole when it's "not happy" with the original mail Message-ID.
I probably failed to explain the issue clearly and sorry for that.
Thank you anyway for trying to help :)
Hi there,
I can confirm this behavior. A few months ago I introduced a milter which is checking for multiple headers when the RFC says that there just should be one of them For example "Message-Id".
I found the described problem in an email coming from Alibaba, which had an invalid "Message-Id" header. It didn't contain an "@" sign or similar. It was RFC-invalid.
This email was sent from Alibaba to a German email provider. There was a redirect at that email provider, pointing to my mailserver.
My server rejected the email because there were 2 "Message-Id" headers: The original invalid "Message-Id" header from Alibaba, and a new "Message-Id" header from the German provider, which seems to have been added during the redirect. There were "Dovecot-sieve" headers in that mail, so my guess was that it happened because of Dovecot-sieve/pigeonhole implementation.
I contacted the email provider, asking for help. Asking if it really is a bug in pigeonhole (or maybe some other system at that provider, who knows). And I contacted Alibaba, so they fix the invalid "Message-Id". I got responses from both, but until now, as far as I can see, it has not been fixed.
The best fix would be (if it really is a bug in pigeonhole), if pigeonhole fixes the problem, then it's fixed for all users of Dovecot. I guess Alibaba is not the only sender with an invalid "Message-ID" header, but that's the only one I saw.
Michael
Hello Michael,
Thanks a lot for your feedback. It comfort me that I'm maybe on the right track about this issue.
The mail I used to troubleshoot the issue was a mail from aliexpress containing a funky Message-ID. Probably the same people behind Aliexpress/Alibaba. But for sure there are probably other mail senders that generate RFC-invalide Message-ID.
If Pigeonhole consider the Message-ID is bad and add it's own, it should maybe at least remove the original one... ?
Kind regards
Hi,
After reading a bit the code and trying to understand it, here is what I think happens here:
Given a bogus Message-ID, for example (notice it's missing angle brackets < >:
Message-ID: 1883biz_pay_after_purchase:0:0_572392900$ae7ed6e4d53b424c84aaf83b30c507e7
Dovecot is parsing Message-ID headers and is looking for the angle bracket as the begining of the Message-ID: https://github.com/dovecot/core/blob/d2ff32792ac052610cea7d65f30de1ee139cb55...
As none is found it will act as if there was no Message-ID header in the mail (even that the header is present).
Then, pigeonhole's redirect function is told to generate a new Message-ID if none was previously detected: https://github.com/dovecot/pigeonhole/blob/5a3f4bd672cc2fb9e755a4b09c4753ac8...
The result is the mail being forwarded, in this case, is now having dual Message-ID and is not RFC 5322 compliant anymore and can be rejected for this reason (hi, gmail?) https://www.spamresource.com/2022/08/gmail-weird-rfc-5322-bounces-and-what.h...
Some thoughts:
First, to be honest, I'm not sure gmail would accept the original mail with the bogus Message-ID sent directly to their servers, but if it was refused, I would assume that these senders would have fixed the issue on their side so their message are delivered (unless there is some whitelisting going on?)
What options could we have to resolve this?
a) Having dovecot core to remove the Message-ID header line from the mail if it is not going to consider it valid ? (So there is no dupe headers when pigeonhole adds one?) b) Having pigeonhole check, when adding a new valid Message-ID, if there is already one existing, and remove the bogus one ?
For now, to workaround this, I'm trying to find a way in the mail flow on our servers to keep only the top most Message-ID when more than one exists. Maybe using: https://www.postfix.org/postconf.5.html#smtp_header_checks but I'm not sure how to achieve it yet or even if it's possible.
Kind regards
Sébastien RICCIO SYSTEM ADMINISTRATOR P +41 840 888 888 F +41 840 888 000 M sriccio@swisscenter.com<mailto:sriccio@swisscenter.com>
------ Message d'origine ------ De "michael.zork@feierfighter.de<mailto:michael.zork@feierfighter.de>" <michael.zork@feierfighter.de<mailto:michael.zork@feierfighter.de>> À "dovecot@dovecot.org<mailto:dovecot@dovecot.org>" <dovecot@dovecot.org<mailto:dovecot@dovecot.org>> Date 01.10.2022 14:49:13 Objet Re: Re[6]: Pigeonhole redirect is adding a message-id header when it already exists
Hi there,
I can confirm this behavior. A few months ago I introduced a milter which is checking for multiple headers when the RFC says that there just should be one of them For example "Message-Id".
I found the described problem in an email coming from Alibaba, which had an invalid "Message-Id" header. It didn't contain an "@" sign or similar. It was RFC-invalid.
This email was sent from Alibaba to a German email provider. There was a redirect at that email provider, pointing to my mailserver.
My server rejected the email because there were 2 "Message-Id" headers: The original invalid "Message-Id" header from Alibaba, and a new "Message-Id" header from the German provider, which seems to have been added during the redirect. There were "Dovecot-sieve" headers in that mail, so my guess was that it happened because of Dovecot-sieve/pigeonhole implementation.
I contacted the email provider, asking for help. Asking if it really is a bug in pigeonhole (or maybe some other system at that provider, who knows). And I contacted Alibaba, so they fix the invalid "Message-Id". I got responses from both, but until now, as far as I can see, it has not been fixed.
The best fix would be (if it really is a bug in pigeonhole), if pigeonhole fixes the problem, then it's fixed for all users of Dovecot. I guess Alibaba is not the only sender with an invalid "Message-ID" header, but that's the only one I saw.
Michael
Am 01-Oct-2022 14:00:45 +0200 schrieb sriccio@swisscenter.com<mailto:sriccio@swisscenter.com>:
You wrote in the original email the message was rejected. Sorry I don't have login access to my gmail test account anymore since the google @#$%@#$% wanted to have me add a phone number.
In my original post I said that gmail was rejecting the forwards because of duplicate headers, and that the duplicate header seems to be a Message-ID added by pigeonhole when it's "not happy" with the original mail Message-ID.
I probably failed to explain the issue clearly and sorry for that.
Thank you anyway for trying to help :)
Le 02/10/2022 à 06:35, Sébastien Riccio a écrit :
Hi,
After reading a bit the code and trying to understand it, here is what I think happens here:
Given a bogus Message-ID, for example (notice it's missing angle brackets < >:
Message-ID: 1883biz_pay_after_purchase:0:0_572392900$ae7ed6e4d53b424c84aaf83b30c507e7
Dovecot is parsing Message-ID headers and is looking for the angle bracket as the begining of the Message-ID: https://github.com/dovecot/core/blob/d2ff32792ac052610cea7d65f30de1ee139cb55...
As none is found it will act as if there was no Message-ID header in the mail (even that the header is present).
Then, pigeonhole's redirect function is told to generate a new Message-ID if none was previously detected: https://github.com/dovecot/pigeonhole/blob/5a3f4bd672cc2fb9e755a4b09c4753ac8...
The result is the mail being forwarded, in this case, is now having dual Message-ID and is not RFC 5322 compliant anymore and can be rejected for this reason (hi, gmail?) https://www.spamresource.com/2022/08/gmail-weird-rfc-5322-bounces-and-what.h...
Some thoughts:
First, to be honest, I'm not sure gmail would accept the original mail with the bogus Message-ID sent directly to their servers, but if it was refused, I would assume that these senders would have fixed the issue on their side so their message are delivered (unless there is some whitelisting going on?)
What options could we have to resolve this?
a) Having dovecot core to remove the Message-ID header line from the mail if it is not going to consider it valid ? (So there is no dupe headers when pigeonhole adds one?) b) Having pigeonhole check, when adding a new valid Message-ID, if there is already one existing, and remove the bogus one ?
Hello, In my opinion, an option c) should be considered. Bogus or not, if a Message-ID is present in the header you should not touch it. For it own use, dovecot should continue to not use it as it is bogus.The only thing to do is to warn in logs in the dovecot core. In a forwarding scenario, you should not alter the existing Message-ID, you should not take the responsibility to fix someone else problem. The bogus message ID key is ok for duplicate detection. The only risk is false trigger by another equal bogus Message-ID header. Bogus messages are bogus so ... For the present case, it means that the check for the header presence should be done by pigeonhole directly, not relying on the dovecot core Message-ID validity check. In the missing/not implemented/not drafted redirect-as-new (inline or attached) case which exiting in some systems, you will generate a completely new message with a fresh/compliant Message-ID.
Emmanuel.
Hello, In my opinion, an option c) should be considered. Bogus or not, if a Message-ID is present in the header you should not touch it. For it own use, dovecot should continue to not use it as it is bogus.The only thing to do is to warn in logs in the dovecot core. In a forwarding scenario, you should not alter the existing Message-ID, you should not take the responsibility to fix someone else problem. The bogus message ID key is ok for duplicate detection. The only risk is false trigger by another equal bogus Message-ID header. Bogus messages are bogus so ... For the present case, it means that the check for the header presence should be done by pigeonhole directly, not relying on the dovecot core Message-ID validity check. In the missing/not implemented/not drafted redirect-as-new (inline or attached) case which exiting in some systems, you will generate a completely new message with a fresh/compliant Message-ID.
Emmanuel.
Hello Emmanuel,
Yes, this sounds to me like a reasonable approach.
Kind regards
Am 02.10.22 um 11:37 schrieb Emmanuel Fusté:
Le 02/10/2022 à 06:35, Sébastien Riccio a écrit :
- What options could we have to resolve this?
a) Having dovecot core to remove the Message-ID header line from the mail if it is not going to consider it valid ? (So there is no dupe headers when pigeonhole adds one?) b) Having pigeonhole check, when adding a new valid Message-ID, if there is already one existing, and remove the bogus one ?
Hello, In my opinion, an option c) should be considered. Bogus or not, if a Message-ID is present in the header you should not touch it.
I agree. Pigeonhole should not modify content on forwarding.
Andreas
This should almost be an RFC discussion, rather than a dovecot discussion, for clarity on what to do with a malformed Message-Id.
For the record, if you start modifying it by deleting the bad message id, and adding your own, you can start breaking other things, such as DKIM signing etc..
IMHO, Dovecot should simply refuse to accept or deliver a message with a 'bad' message id, so that the sending system can identify and correct the problem.
That way Dovecot doesn't need to address/modify the email message.
-- Michael --On 2022-10-01 21:35, Sébastien Riccio wrote:
Hi,
After reading a bit the code and trying to understand it, here is what I think happens here:
Given a bogus Message-ID, for example (notice it's missing angle brackets < >:
Message-ID: 1883biz_pay_after_purchase:0:0_572392900$ae7ed6e4d53b424c84aaf83b30c507e7
Dovecot is parsing Message-ID headers and is looking for the angle bracket as the begining of the Message-ID: https://github.com/dovecot/core/blob/d2ff32792ac052610cea7d65f30de1ee139cb55... <https://github.com/dovecot/core/blob/d2ff32792ac052610cea7d65f30de1ee139cb55c/src/lib-mail/message-id.c#L75>
As none is found it will act as if there was no Message-ID header in the mail (even that the header is present).
Then, pigeonhole's redirect function is told to generate a new Message-ID if none was previously detected: https://github.com/dovecot/pigeonhole/blob/5a3f4bd672cc2fb9e755a4b09c4753ac8... <https://github.com/dovecot/pigeonhole/blob/5a3f4bd672cc2fb9e755a4b09c4753ac86e15f99/src/lib-sieve/cmd-redirect.c#L569>
The result is the mail being forwarded, in this case, is now having dual Message-ID and is not RFC 5322 compliant anymore and can be rejected for this reason (hi, gmail?) https://www.spamresource.com/2022/08/gmail-weird-rfc-5322-bounces-and-what.h... <https://www.spamresource.com/2022/08/gmail-weird-rfc-5322-bounces-and-what.html>
Some thoughts:
First, to be honest, I'm not sure gmail would accept the original mail with the bogus Message-ID sent directly to their servers, but if it was refused, I would assume that these senders would have fixed the issue on their side so their message are delivered (unless there is some whitelisting going on?)
What options could we have to resolve this?
a) Having dovecot core to remove the Message-ID header line from the mail if it is not going to consider it valid ? (So there is no dupe headers when pigeonhole adds one?) b) Having pigeonhole check, when adding a new valid Message-ID, if there is already one existing, and remove the bogus one ? For now, to workaround this, I'm trying to find a way in the mail flow on our servers to keep only the top most Message-ID when more than one exists. Maybe using: https://www.postfix.org/postconf.5.html#smtp_header_checks <https://www.postfix.org/postconf.5.html#smtp_header_checks> but I'm not sure how to achieve it yet or even if it's possible.
Kind regards
*Sébastien RICCIO*
*SYSTEM ADMINISTRATOR*
*P* +41 840 888 888
*F***+41 840 888 000
*M****sriccio@swisscenter.com <mailto:sriccio@swisscenter.com>*
------ Message d'origine ------ De "michael.zork@feierfighter.de <mailto:michael.zork@feierfighter.de>" <michael.zork@feierfighter.de <mailto:michael.zork@feierfighter.de>> À "dovecot@dovecot.org <mailto:dovecot@dovecot.org>" <dovecot@dovecot.org <mailto:dovecot@dovecot.org>> Date 01.10.2022 14:49:13 Objet Re: Re[6]: Pigeonhole redirect is adding a message-id header when it already exists
Hi there, I can confirm this behavior. A few months ago I introduced a milter which is checking for multiple headers when the RFC says that there just should be one of them For example "Message-Id". I found the described problem in an email coming from Alibaba, which had an invalid "Message-Id" header. It didn't contain an "@" sign or similar. It was RFC-invalid. This email was sent from Alibaba to a German email provider. There was a redirect at that email provider, pointing to my mailserver. My server rejected the email because there were 2 "Message-Id" headers: The original invalid "Message-Id" header from Alibaba, and a new "Message-Id" header from the German provider, which seems to have been added during the redirect. There were "Dovecot-sieve" headers in that mail, so my guess was that it happened because of Dovecot-sieve/pigeonhole implementation. I contacted the email provider, asking for help. Asking if it really is a bug in pigeonhole (or maybe some other system at that provider, who knows). And I contacted Alibaba, so they fix the invalid "Message-Id". I got responses from both, but until now, as far as I can see, it has not been fixed. The best fix would be (if it really is a bug in pigeonhole), if pigeonhole fixes the problem, then it's fixed for all users of Dovecot. I guess Alibaba is not the only sender with an invalid "Message-ID" header, but that's the only one I saw. Michael
Am 01-Oct-2022 14:00:45 +0200 schrieb sriccio@swisscenter.com <mailto:sriccio@swisscenter.com>:
>You wrote in the original email the message was rejected. Sorry I don't have login access to my gmail test account anymore since the google @#$%@#$% wanted to have me add a phone number. In my original post I said that gmail was rejecting the forwards because of duplicate headers, and that the duplicate header seems to be a Message-ID added by pigeonhole when it's "not happy" with the original mail Message-ID. I probably failed to explain the issue clearly and sorry for that. Thank you anyway for trying to help :)
-- "Catch the Magic of Linux..."
Michael Peddemors, President/CEO LinuxMagic Inc. Visit us at http://www.linuxmagic.com @linuxmagic A Wizard IT Company - For More Info http://www.wizard.ca "LinuxMagic" a Registered TradeMark of Wizard Tower TechnoServices Ltd.
604-682-0300 Beautiful British Columbia, Canada
This email and any electronic data contained are confidential and intended solely for the use of the individual or entity to which they are addressed. Please note that any views or opinions presented in this email are solely those of the author and are not intended to represent those of the company.
"Michael" == Michael Peddemors <michael@linuxmagic.com> writes:
This should almost be an RFC discussion, rather than a dovecot discussion, for clarity on what to do with a malformed Message-Id.
Seems simple, just rename it to "Message-Id-Orig: ..." and insert your own. Fixes the problem, still let's you trace the message, etc.
For the record, if you start modifying it by deleting the bad message id, and adding your own, you can start breaking other things, such as DKIM signing etc..
IMHO, Dovecot should simply refuse to accept or deliver a message with a 'bad' message id, so that the sending system can identify and correct the problem.
That way Dovecot doesn't need to address/modify the email message.
-- Michael --
On 2022-10-01 21:35, Sébastien Riccio wrote:
Hi,
After reading a bit the code and trying to understand it, here is what I think happens here:
Given a bogus Message-ID, for example (notice it's missing angle brackets < >:
Message-ID: 1883biz_pay_after_purchase:0:0_572392900$ae7ed6e4d53b424c84aaf83b30c507e7
Dovecot is parsing Message-ID headers and is looking for the angle bracket as the begining of the Message-ID: https://github.com/dovecot/core/blob/d2ff32792ac052610cea7d65f30de1ee139cb55... <https://github.com/dovecot/core/blob/d2ff32792ac052610cea7d65f30de1ee139cb55c/src/lib-mail/message-id.c#L75>
As none is found it will act as if there was no Message-ID header in the mail (even that the header is present).
Then, pigeonhole's redirect function is told to generate a new Message-ID if none was previously detected: https://github.com/dovecot/pigeonhole/blob/5a3f4bd672cc2fb9e755a4b09c4753ac8... <https://github.com/dovecot/pigeonhole/blob/5a3f4bd672cc2fb9e755a4b09c4753ac86e15f99/src/lib-sieve/cmd-redirect.c#L569>
The result is the mail being forwarded, in this case, is now having dual Message-ID and is not RFC 5322 compliant anymore and can be rejected for this reason (hi, gmail?) https://www.spamresource.com/2022/08/gmail-weird-rfc-5322-bounces-and-what.h... <https://www.spamresource.com/2022/08/gmail-weird-rfc-5322-bounces-and-what.html>
Some thoughts:
First, to be honest, I'm not sure gmail would accept the original mail with the bogus Message-ID sent directly to their servers, but if it was refused, I would assume that these senders would have fixed the issue on their side so their message are delivered (unless there is some whitelisting going on?)
What options could we have to resolve this?
a) Having dovecot core to remove the Message-ID header line from the mail if it is not going to consider it valid ? (So there is no dupe headers when pigeonhole adds one?) b) Having pigeonhole check, when adding a new valid Message-ID, if there is already one existing, and remove the bogus one ? For now, to workaround this, I'm trying to find a way in the mail flow on our servers to keep only the top most Message-ID when more than one exists. Maybe using: https://www.postfix.org/postconf.5.html#smtp_header_checks <https://www.postfix.org/postconf.5.html#smtp_header_checks> but I'm not sure how to achieve it yet or even if it's possible.
Kind regards
*Sébastien RICCIO*
*SYSTEM ADMINISTRATOR*
*P* +41 840 888 888
*F***+41 840 888 000
*M****sriccio@swisscenter.com <mailto:sriccio@swisscenter.com>*
------ Message d'origine ------ De "michael.zork@feierfighter.de <mailto:michael.zork@feierfighter.de>" <michael.zork@feierfighter.de <mailto:michael.zork@feierfighter.de>> À "dovecot@dovecot.org <mailto:dovecot@dovecot.org>" <dovecot@dovecot.org <mailto:dovecot@dovecot.org>> Date 01.10.2022 14:49:13 Objet Re: Re[6]: Pigeonhole redirect is adding a message-id header when it already exists
Hi there, I can confirm this behavior. A few months ago I introduced a milter which is checking for multiple headers when the RFC says that there just should be one of them For example "Message-Id". I found the described problem in an email coming from Alibaba, which had an invalid "Message-Id" header. It didn't contain an "@" sign or similar. It was RFC-invalid. This email was sent from Alibaba to a German email provider. There was a redirect at that email provider, pointing to my mailserver. My server rejected the email because there were 2 "Message-Id" headers: The original invalid "Message-Id" header from Alibaba, and a new "Message-Id" header from the German provider, which seems to have been added during the redirect. There were "Dovecot-sieve" headers in that mail, so my guess was that it happened because of Dovecot-sieve/pigeonhole implementation. I contacted the email provider, asking for help. Asking if it really is a bug in pigeonhole (or maybe some other system at that provider, who knows). And I contacted Alibaba, so they fix the invalid "Message-Id". I got responses from both, but until now, as far as I can see, it has not been fixed. The best fix would be (if it really is a bug in pigeonhole), if pigeonhole fixes the problem, then it's fixed for all users of Dovecot. I guess Alibaba is not the only sender with an invalid "Message-ID" header, but that's the only one I saw. Michael
Am 01-Oct-2022 14:00:45 +0200 schrieb sriccio@swisscenter.com <mailto:sriccio@swisscenter.com>:
You wrote in the original email the message was rejected. Sorry I don't have login access to my gmail test account anymore since the google @#$%@#$% wanted to have me add a phone number.
In my original post I said that gmail was rejecting the forwards because of duplicate headers, and that the duplicate header seems to be a Message-ID added by pigeonhole when it's "not happy" with the original mail Message-ID.
I probably failed to explain the issue clearly and sorry for that.
Thank you anyway for trying to help :)
-- "Catch the Magic of Linux..."
Michael Peddemors, President/CEO LinuxMagic Inc. Visit us at http://www.linuxmagic.com @linuxmagic A Wizard IT Company - For More Info http://www.wizard.ca "LinuxMagic" a Registered TradeMark of Wizard Tower TechnoServices Ltd.
604-682-0300 Beautiful British Columbia, Canada
This email and any electronic data contained are confidential and intended solely for the use of the individual or entity to which they are addressed. Please note that any views or opinions presented in this email are solely those of the author and are not intended to represent those of the company.
Hi All
Has there been any progress with this issue please?
I am also seeing the problem, users with a "redirect "xx";" in their .dovecot.sieve file are seeing some emails bouncing:
Aug 2 23:21:29 ocean dovecot: lmtp(xxx)<45389><X+oMBunWymRNsQAANqdT4A>: sieve: msgid=unspecified: redirect action: forwarded to <xxx@googlemail.com> Aug 2 23:21:29 ocean postfix/qmgr[37858]: 1DB297E032: from=<DCMSERVICE.BOUNCE@lloydsbank.co.uk>, size=65630, nrcpt=1 (queue active) Aug 2 23:21:29 ocean postfix/cleanup[45380]: 3A1C47E033: message-id=<dovecot-sieve-1691014889-102901-0@ocean.unseen.org>
Aug 2 23:21:29 ocean postfix/smtp[45393]: Trusted TLS connection established to gmail-smtp-in.l.google.com[74.125.133.26]:25: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-256) server-digest SHA256 Aug 2 23:21:29 ocean postfix/smtp[45393]: 3A1C47E033: to=<xxx@googlemail.com>, relay=gmail-smtp-in.l.google.com[74.125.133.26]:25, delay=0.66, delays=0.07/0.03/0.24/0.32, dsn=5.7.1, status=bounced (host gmail-smtp-in.l.google.com[74.125.133.26] said: 550-5.7.1 This message is not RFC 5322 compliant. There are multiple Message-ID 550-5.7.1 headers. To reduce the amount of spam sent to Gmail, this message has 550-5.7.1 been blocked. Please visit 550 5.7.1 https://support.google.com/mail/?p=RfcMessageNonCompliant d13-20020adff2cd000000b0031788453e97si6656282wrp.124 - gsmtp (in reply to end of DATA command))
Any workarounds on how to fix this please?
Thanks very much Kevin
These are real people with bank accounts? Get paid? Have money for breakfast lunch dinner and a roof over their heads?
Just asking because my own bank account stupidly enough requires a phone number to log in online whether or not I even have an email address.
And the POTS (Plain Old Telephone Service) system, including cell phone service, unlike the internet at large which is based on open standards, is a highly proprietary closed-source-only multinational corporate fee-for-service system based on billing and debt collections for long distance calls, mostly owned by fraudsters, blackmailers, thieves, extortioners and hackers, and subject to strict intellectual property restrictions and intelligence surveillance by various governments and nation-states as well as court orders relating to domestic violence restraining orders, no-contact orders and various other "established" service of process in local small town court systems to obstruct or deny access.
AT&T and friends have been around since the early railroad days, and there are people who need to be SERVED here like no one has ever been served in over 150 years in the United States.
On October 1, 2022 3:52:43 AM AKDT, Marc <Marc@f1-outsourcing.eu> wrote:
Oct 1 13:31:46 xxxx sendmail[30321]: 291BVjjx030318: to=<xxxx@gmail.com>, delay=00:00:01, xdelay=00:00:01, mailer=esmtp, pri=122536, relay=gmail-smtp-in.l.google.com. [142.250.102.27], dsn=2.0.0, stat=Sent (OK 1664623906 gs19- 20020a1709072d1300b00777a40d515dsi4096082ejc.456 - gsmtp)
I just tested for you, enabled the sieve forward, send test mail and the forward is being accepted by google.
Thanks for the test. However, does your test mail had a "bogus" Message-ID header in it like I tried to explain ?
You wrote in the original email the message was rejected. Sorry I don't have login access to my gmail test account anymore since the google @#$%@#$% wanted to have me add a phone number.
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Oct 1 13:31:46 xxxx sendmail[30321]: 291BVjjx030318: to=<xxxx@gmail.com>, delay=00:00:01, xdelay=00:00:01, mailer=esmtp, pri=122536, relay=gmail-smtp-in.l.google.com. [142.250.102.27], dsn=2.0.0, stat=Sent (OK 1664623906 gs19-20020a1709072d1300b00777a40d515dsi4096082ejc.456 - gsmtp)
I just tested for you, enabled the sieve forward, send test mail and the forward is being accepted by google.
It looks to me you're dismissing the fact that it only concerns the forwarding of mails that have originally a "bogus" Message-ID.
That's the only way to trigger the issue. Other mails (that are missing Message-ID or having RFC compliant Message-ID are forwarded and accepted just fine by gmail :)
participants (9)
-
A. Schulze
-
Emmanuel Fusté
-
John Stoffel
-
justina colmena ~biz
-
kevin＠unseen.org
-
Marc
-
Michael Peddemors
-
michael.zork＠feierfighter.de
-
Sébastien Riccio