Pigeonhole Sieve Vacation Reply-To peculiarity with inbound AWS-SES
To begin with, usage of Amazons Simple Email Service (SES) is mandatory for outgoing mails from AWS-EC2 instances. The Pro is, it does work and we may utilize email relays with high reputation for our outgoing mails at no additional costs. The Con's are that (A) SES does not relay mails with empty <> MAIL FROM addresses and (B) it fiddles with the envelope addresses of the relayed mails.
Anyway this is out of our control, and while I could avoid (A) by switching the e-mail service provider, I still have no control over (B) because this affects replying to emails from others using AWS-SES.
Problem A) The empty-MAIL-FROM may be effectively addressed by setting Pigeonhole’s sieve_vacation_send_from_recipient to ‚yes‘.
Problem B) Pigeonhole Vacation utilizes the senders envelope address of the inbound message as the reply-address.
The problem here is that inbound mails from third parties utilizing AWS-SES come in with an unpersonalized envelope address and SES takes returns to this as bounce messages and changes the body's From: to „MAILER-DAEMON@xx-zzzz-1.amazonses.com“, which is not even our MAILER-DAEMON but the one of the receiver of our reply. So the receiver gets no chance to know from the headers the identity of whom replied - he may assume it from the context the actual message, though.
I found the setting sieve_vacation_to_header_ignore_envelope, which was introduced in 2017 (https://dovecot.org/list/dovecot-cvs/2017-August/028917.html). My finding is, that this resolves the problem only half way.
For a local workaround, I changed Line 1119 of cmd-vacation.c https://github.com/dovecot/pigeonhole/blob/34431d7a67474c00d5b0554f5d45b8675...
to:
struct smtp_address real_to; if (smtp_address_init_from_msg(&real_to, &reply_to) >= 0) sctx = sieve_smtp_start_single(senv, &real_to, smtp_from, &output); else sctx = sieve_smtp_start_single(senv, smtp_to, smtp_from, &output);
That in place and setting sieve_vacation_to_header_ignore_envelope to ‚yes‘, effectively resolves problem B as well.
Best regards
Rolf Jansen
On 7/2/23 22:01, Dr. Rolf Jansen wrote:
To begin with, usage of Amazons Simple Email Service (SES) is mandatory for outgoing mails from AWS-EC2 instances.
I run AWS-EC2 instances using postfix to send a receive mail. They can send direct assuming I set up suitable SPF, but they typically forward mail to another host under my control that is not on AWS to use as the outgoing server.
Jeremy
Am 07.02.2023 um 17:54 schrieb jeremy ardley <jeremy@ardley.org>:
On 7/2/23 22:01, Dr. Rolf Jansen wrote:
To begin with, usage of Amazons Simple Email Service (SES) is mandatory for outgoing mails from AWS-EC2 instances.
I run AWS-EC2 instances using postfix to send a receive mail. They can send direct assuming I set up suitable SPF, but they typically forward mail to another host under my control that is not on AWS to use as the outgoing server.
OK, that’s another use case. Many do use a full fledged Postfix/Dovecot installation. However the outgoing port 25 into the internet is blocked by AWS, and therefore we may either use a third party relay for our outgoing emails or may use SES, which is not that bad - except some unusual peculiarities.
Of course, if I wanted to, I could change, this is something which is under my control.
That would however change nothing with the other issue, of my Vacation plugin responding with a nonsense MAIL TO address to third parties using SES, this is not under our control. There is even no channel to argue about it.
Best regards
Rolf Jansen
Am 07.02.2023 um 17:54 schrieb jeremy ardley <jeremy@ardley.org>:
On 7/2/23 22:01, Dr. Rolf Jansen wrote:
To begin with, usage of Amazons Simple Email Service (SES) is mandatory for outgoing mails from AWS-EC2 instances.
I run AWS-EC2 instances using postfix to send a receive mail. They can send direct assuming I set up suitable SPF, but they typically forward mail to another host under my control that is not on AWS to use as the outgoing server.
OK, that’s another use case. Many do use a full fledged Postfix/Dovecot installation. However the outgoing port 25 into the internet is blocked by AWS, and therefore we may either use a third party relay for our outgoing emails or may use SES, which is not that bad - except some unusual peculiarities.
Of course, if I wanted to, I could change, this is something which is under my control.
That would however change nothing with the other issue, of my Vacation plugin responding with a nonsense MAIL TO address to third parties using SES, this is not under our control. There is even no channel to argue about it.
Best regards
Rolf Jansen
On 8/2/23 05:08, Dr. Rolf Jansen wrote:
Am 07.02.2023 um 17:54 schrieb jeremy ardley<jeremy@ardley.org>:
On 7/2/23 22:01, Dr. Rolf Jansen wrote:
To begin with, usage of Amazons Simple Email Service (SES) is mandatory for outgoing mails from AWS-EC2 instances. I run AWS-EC2 instances using postfix to send a receive mail. They can send direct assuming I set up suitable SPF, but they typically forward mail to another host under my control that is not on AWS to use as the outgoing server. OK, that’s another use case. Many do use a full fledged Postfix/Dovecot installation. However the outgoing port 25 into the internet is blocked by AWS, and therefore we may either use a third party relay for our outgoing emails or may use SES, which is not that bad - except some unusual peculiarities.
This is off topic, but to be precise:
- AWS throttles but does not block traffic to a *destination* port 25.
- The *origin* port on the EC2 instance is an unprivilged port, not port 25
- If you use a relayhost you typically send from an unprivilged EC2 port to port 587 on the relay host
Jeremy
On 2023-02-07 13:33, jeremy ardley wrote:
On 8/2/23 05:08, Dr. Rolf Jansen wrote:
Am 07.02.2023 um 17:54 schrieb jeremy ardley<jeremy@ardley.org>:
On 7/2/23 22:01, Dr. Rolf Jansen wrote:
To begin with, usage of Amazons Simple Email Service (SES) is mandatory for outgoing mails from AWS-EC2 instances. I run AWS-EC2 instances using postfix to send a receive mail. They can send direct assuming I set up suitable SPF, but they typically forward mail to another host under my control that is not on AWS to use as the outgoing server. OK, that’s another use case. Many do use a full fledged Postfix/Dovecot installation. However the outgoing port 25 into the internet is blocked by AWS, and therefore we may either use a third party relay for our outgoing emails or may use SES, which is not that bad - except some unusual peculiarities.
This is off topic, but to be precise:
- AWS throttles but does not block traffic to a *destination* port 25.
- The *origin* port on the EC2 instance is an unprivilged port, not port 25
- If you use a relayhost you typically send from an unprivilged EC2 port to port 587 on the relay host
Jeremy
And if you DO intend to send out to port 25, remember to update the PTR on your EC2 instance.
-- "Catch the Magic of Linux..."
Michael Peddemors, President/CEO LinuxMagic Inc. Visit us at http://www.linuxmagic.com @linuxmagic A Wizard IT Company - For More Info http://www.wizard.ca "LinuxMagic" a Registered TradeMark of Wizard Tower TechnoServices Ltd.
604-682-0300 Beautiful British Columbia, Canada
This email and any electronic data contained are confidential and intended solely for the use of the individual or entity to which they are addressed. Please note that any views or opinions presented in this email are solely those of the author and are not intended to represent those of the company.
On 2/8/23 05:33, jeremy ardley wrote:
- AWS throttles but does not block traffic to a *destination* port 25.
I set one up last week and can confirm that port 25 outbound was definitely blocked rather than throttled. However there's a form you can fill in to get it unblocked, and it only takes a day or so.
P.
https://doc.dovecot.org/settings/pigeonhole-ext/vacation/#pigeonhole_setting...
On 7-2-2023 15:01, Dr. Rolf Jansen wrote:
To begin with, usage of Amazons Simple Email Service (SES) is mandatory for outgoing mails from AWS-EC2 instances. The Pro is, it does work and we may utilize email relays with high reputation for our outgoing mails at no additional costs. The Con's are that (A) SES does not relay mails with empty <> MAIL FROM addresses and (B) it fiddles with the envelope addresses of the relayed mails.
Anyway this is out of our control, and while I could avoid (A) by switching the e-mail service provider, I still have no control over (B) because this affects replying to emails from others using AWS-SES.
Problem A) The empty-MAIL-FROM may be effectively addressed by setting Pigeonhole’s sieve_vacation_send_from_recipient to ‚yes‘.
Problem B) Pigeonhole Vacation utilizes the senders envelope address of the inbound message as the reply-address.
The problem here is that inbound mails from third parties utilizing AWS-SES come in with an unpersonalized envelope address and SES takes returns to this as bounce messages and changes the body's From: to „MAILER-DAEMON@xx-zzzz-1.amazonses.com“, which is not even our MAILER-DAEMON but the one of the receiver of our reply. So the receiver gets no chance to know from the headers the identity of whom replied - he may assume it from the context the actual message, though.
I found the setting sieve_vacation_to_header_ignore_envelope, which was introduced in 2017 (https://dovecot.org/list/dovecot-cvs/2017-August/028917.html). My finding is, that this resolves the problem only half way.
For a local workaround, I changed Line 1119 of cmd-vacation.c https://github.com/dovecot/pigeonhole/blob/34431d7a67474c00d5b0554f5d45b8675...
to:
struct smtp_address real_to; if (smtp_address_init_from_msg(&real_to, &reply_to) >= 0) sctx = sieve_smtp_start_single(senv, &real_to, smtp_from, &output); else sctx = sieve_smtp_start_single(senv, smtp_to, smtp_from, &output);That in place and setting sieve_vacation_to_header_ignore_envelope to ‚yes‘, effectively resolves problem B as well.
Best regards
Rolf Jansen
Am 08.02.2023 um 08:37 schrieb Stephan Bosch <stephan@rename-it.nl>:
https://doc.dovecot.org/settings/pigeonhole-ext/vacation/#pigeonhole_setting...
On 7-2-2023 15:01, Dr. Rolf Jansen wrote:
To begin with, usage of Amazons Simple Email Service (SES) is mandatory for outgoing mails from AWS-EC2 instances. The Pro is, it does work and we may utilize email relays with high reputation for our outgoing mails at no additional costs. The Con's are that (A) SES does not relay mails with empty <> MAIL FROM addresses and (B) it fiddles with the envelope addresses of the relayed mails.
Anyway this is out of our control, and while I could avoid (A) by switching the e-mail service provider, I still have no control over (B) because this affects replying to emails from others using AWS-SES.
Problem A) The empty-MAIL-FROM may be effectively addressed by setting Pigeonhole’s sieve_vacation_send_from_recipient to ‚yes‘.
Yes that’s important for resolving problem A, and I wrote this already (s. above).
Without my patch, problem B remains unresolved, though.
On 8-2-2023 13:10, Dr. Rolf Jansen wrote:
Am 08.02.2023 um 08:37 schrieb Stephan Bosch <stephan@rename-it.nl>:
https://doc.dovecot.org/settings/pigeonhole-ext/vacation/#pigeonhole_setting...
On 7-2-2023 15:01, Dr. Rolf Jansen wrote:
To begin with, usage of Amazons Simple Email Service (SES) is mandatory for outgoing mails from AWS-EC2 instances. The Pro is, it does work and we may utilize email relays with high reputation for our outgoing mails at no additional costs. The Con's are that (A) SES does not relay mails with empty <> MAIL FROM addresses and (B) it fiddles with the envelope addresses of the relayed mails.
Anyway this is out of our control, and while I could avoid (A) by switching the e-mail service provider, I still have no control over (B) because this affects replying to emails from others using AWS-SES.
Problem A) The empty-MAIL-FROM may be effectively addressed by setting Pigeonhole’s sieve_vacation_send_from_recipient to ‚yes‘. Yes that’s important for resolving problem A, and I wrote this already (s. above).
Without my patch, problem B remains unresolved, though.
Oh, right..let's say it's early here. Your patch seems a bit risky though. It will try to send replies to whatever is in sender, resent-from, or from header (whichever is found first in that order), without considering where it might have actually come from. Just hoping that MTA in between checks whether headers match the envelope, before it is mangled by AWS.
Regards,
Stephan.
Am 08.02.2023 um 09:27 schrieb Stephan Bosch <stephan@rename-it.nl>:
On 8-2-2023 13:10, Dr. Rolf Jansen wrote:
Am 08.02.2023 um 08:37 schrieb Stephan Bosch <stephan@rename-it.nl>:
https://doc.dovecot.org/settings/pigeonhole-ext/vacation/#pigeonhole_setting...
On 7-2-2023 15:01, Dr. Rolf Jansen wrote:
To begin with, usage of Amazons Simple Email Service (SES) is mandatory for outgoing mails from AWS-EC2 instances. The Pro is, it does work and we may utilize email relays with high reputation for our outgoing mails at no additional costs. The Con's are that (A) SES does not relay mails with empty <> MAIL FROM addresses and (B) it fiddles with the envelope addresses of the relayed mails.
Anyway this is out of our control, and while I could avoid (A) by switching the e-mail service provider, I still have no control over (B) because this affects replying to emails from others using AWS-SES.
Problem A) The empty-MAIL-FROM may be effectively addressed by setting Pigeonhole’s sieve_vacation_send_from_recipient to ‚yes‘. Yes that’s important for resolving problem A, and I wrote this already (s. above).
Without my patch, problem B remains unresolved, though.
Oh, right..let's say it's early here. Your patch seems a bit risky though. It will try to send replies to whatever is in sender, resent-from, or from header (whichever is found first in that order), without considering where it might have actually come from. Just hoping that MTA in between checks whether headers match the envelope, before it is mangled by AWS.
There is this yet undocumented commit from from 2017:
https://dovecot.org/list/dovecot-cvs/2017-August/028917.html
My guess is that this was intended to resolve problem B, but it was not effective, because the envelope sender always made it into the variable smtp_to.
My patch only finishes this new setting from 2017. With the patch in place we my switch between the two options by said new setting from 2017:
-- sieve_vacation_to_header_ignore_envelope = yes
reply to whatever is in sender, resent-from, or from header.
-- sieve_vacation_to_header_ignore_envelope = no (the default option)
reply to the envelope sender
That said, I did not do an in-deep analysis of the code of Pigeonhole. There may well be better places for fixing of sieve_vacation_to_header_ignore_envelope is falling short.
participants (6)
-
Dr. Rolf Jansen
-
Dr. Rolf Jansen
-
jeremy ardley
-
Michael Peddemors
-
Plutocrat
-
Stephan Bosch