2.3.13 broken submission relay smtp parser
I have a new install of dovecot 2.3.13, along with exim 4.94, in an Azure hosted FreeBSD 12.2 VM. I have been running exim on local hardware with FreeBSD for 15+ years, but dovecot and Azure are a new "learning experience". I am getting an error response in dovecot.log when trying to use the submission relay function, which is apparently new in 2.3... It would appear the parser is either broken or has a character set limitation that no other smtp implementation has. I finally gave up trying to figure out what I might have done wrong in setting up exim and pointed dovecot at mailjet and got the same error.
Jun 08 19:39:42 submission(testing@dispatch.tndh.net)<89538><lOfAL0zEFNmsOCrh>: Warning: smtp-client: conn in-v3.mailjet.com:587 (104.199.96.85:587) [1]: Received invalid EHLO response line: Unexpected character in EHLO keyword Jun 08 19:39:42 submission(testing@dispatch.tndh.net)<89538><lOfAL0zEFNmsOCrh>: Warning: smtp-client: conn in-v3.mailjet.com:587 (104.199.96.85:587) [1]: Received invalid EHLO response line: Unexpected character in EHLO keyword
I didn't try the mailjet path with telnet, but I had done that earlier with the local exim server and I can't see any invalid characters, even in the tcpdump pcap file.
Jun 08 10:49:42 submission(testing@dispatch.tndh.net)<29791><j8NnyETEqV2sOCq3>: Warning: smtp-client: conn 127.0.0.1:58 [1]: Received invalid EHLO response line: Unexpected character in EHLO keyword # telnet localhost 58 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. 220 secure smtp server ehlo dovecot.tndh.net 250-exim.tndh.net Hello dovecot.tndh.net [127.0.0.1] 250-SIZE 536870912 250-8BITMIME 250-VRFY 250-PIPELINING 250-X_PIPE_CONNECT 250-AUTH CRAM-MD5 250-CHUNKING 250-SMTPUTF8 250 HELP
This might be some confusion about starttls on the mailjet path, but if that is true the error message is wrong; and it wouldn't be true for the local exim open smtp port. If it really is smtp, it would be most helpful if the error message actually reported what string it is taking issue with.
I have the dovecot-sysreport, but I am not encouraged about sending it when stdout presented: # dovecot-sysreport Gathering configurations ... grep: The -P option is not supportedgrep: The -P option is not supported grep: The -P option is not supported Gathering system informations ... Creating archive ... All done! Please report file dovecot-sysreport-TNDH-mail-1623209001.tar.gz Removing temp files at /tmp/tmp.kphlba44 ... #
While dovecot -n stdout presented the line: ssl_key = # hidden, use -P to show it
expecting people to put sensitive configuration on a public mail list without knowing what the tool is including is a challenge, but when the tool is errantly using the command line option that is also used for exposing the private data by a related tool, it is even less likely that I want to do that. While the dovecot -n option did hide passwords, it did not hide the username associated with that. I will put dovecot -n (redacted) here, but until I have time to see exactly what the sysreport included, I am not releasing that.
# 2.3.13 (89f716dc2): /usr/local/etc/dovecot/dovecot.conf # Pigeonhole version 0.5.13 (cdd19fe3) # OS: FreeBSD 12.2-RELEASE-p4 amd64 ufs # Hostname: TNDH-mail.g4msrgoph2uevil3ys5jvbbpza.jx.internal.cloudapp.net auth_debug = yes auth_debug_passwords = yes auth_verbose = yes debug_log_path = /var/log/dovecot-debug.log disable_plaintext_auth = no first_valid_uid = 220 hostname = dispatch.tndh.net imap_idle_notify_interval = 20 mins info_log_path = /var/log/dovecot-info.log last_valid_uid = 220 log_debug = (event=* AND cat=*) log_path = /var/log/dovecot.log login_greeting = tndh.net Mailer Server Ready ... login_trusted_networks = 127.0.0.1 10.0.0.4 mail_debug = yes mail_location = maildir:/usr/local/var/dovecot/vhosts/%d/%n mail_plugins = mail_log notify notify_status managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext namespace inbox { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } mailbox virtual/Flagged { auto = subscribe special_use = \Flagged } prefix = separator = / type = private } passdb { args = username_format=%n /usr/local/var/dovecot/db/%d/passwd driver = passwd-file } plugin { expire = Trash mail_home = /usr/local/var/dovecot/vhosts/%d/%n mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid box msgid size recipient_delimiter = + sieve = /usr/local/var/dovecot/vhosts/%d/%n/sieve/.dovecot.sieve sieve_after = /usr/local/var/dovecot/vhosts/%d/%n/sieve/sieve-after.d sieve_before = /usr/local/var/dovecot/vhosts/%d/%n/sieve/sieve-before.d sieve_dir = /usr/local/var/dovecot/vhosts/%d/%n/sieve sieve_global_path = /usr/local/var/dovecot/vhosts/sieve/default.sieve } pop3_client_workarounds = outlook-no-nuls oe-ns-eoh protocols = imap pop3 lmtp submission service auth-worker { user = vmail } service imap-login { inet_listener imap { port = 143 } inet_listener imaps { port = 993 ssl = yes } } service stats { unix_listener stats-writer { mode = 0666 } } service submission-login { inet_listener submission { port = 465 ssl = yes } } ssl_cert = **-as-if-I-want-this-on-a-public-list-** userdb { args = username_format=%n /usr/local/var/dovecot/db/%d/passwd driver = passwd-file } verbose_ssl = yes protocol lmtp { mail_fsync = optimized mail_plugins = mail_log notify notify_status sieve } protocol imap { mail_max_userip_connections = 10 mail_plugins = mail_log notify notify_status imap_sieve } protocol pop3 { mail_max_userip_connections = 10 mail_plugins = mail_log notify notify_status } protocol lda { mail_fsync = optimized mail_plugins = mail_log notify notify_status sieve }
On 09/06/2021 08:57 Tony Hain tony@tndh.net wrote:
I have a new install of dovecot 2.3.13, along with exim 4.94, in an Azure hosted FreeBSD 12.2 VM. I have been running exim on local hardware with FreeBSD for 15+ years, but dovecot and Azure are a new "learning experience". I am getting an error response in dovecot.log when trying to use the submission relay function, which is apparently new in 2.3... It would appear the parser is either broken or has a character set limitation that no other smtp implementation has. I finally gave up trying to figure out what I might have done wrong in setting up exim and pointed dovecot at mailjet and got the same error.
Jun 08 19:39:42 submission(testing@dispatch.tndh.net)<89538><lOfAL0zEFNmsOCrh>: Warning: smtp-client: conn in-v3.mailjet.com:587 (104.199.96.85:587) [1]: Received invalid EHLO response line: Unexpected character in EHLO keyword Jun 08 19:39:42 submission(testing@dispatch.tndh.net)<89538><lOfAL0zEFNmsOCrh>: Warning: smtp-client: conn in-v3.mailjet.com:587 (104.199.96.85:587) [1]: Received invalid EHLO response line: Unexpected character in EHLO keyword
I didn't try the mailjet path with telnet, but I had done that earlier with the local exim server and I can't see any invalid characters, even in the tcpdump pcap file.
Jun 08 10:49:42 submission(testing@dispatch.tndh.net)<29791><j8NnyETEqV2sOCq3>: Warning: smtp-client: conn 127.0.0.1:58 [1]: Received invalid EHLO response line: Unexpected character in EHLO keyword # telnet localhost 58 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. 220 secure smtp server ehlo dovecot.tndh.net 250-exim.tndh.net Hello dovecot.tndh.net [127.0.0.1] 250-SIZE 536870912 250-8BITMIME 250-VRFY 250-PIPELINING 250-X_PIPE_CONNECT 250-AUTH CRAM-MD5 250-CHUNKING 250-SMTPUTF8 250 HELP
This might be some confusion about starttls on the mailjet path, but if that is true the error message is wrong; and it wouldn't be true for the local exim open smtp port. If it really is smtp, it would be most helpful if the error message actually reported what string it is taking issue with.
Hi!
Can you provide the rawlogs?
Aki
On 09/06/2021 08:57 Tony Hain tony@tndh.net wrote:
<snip/>
I have the dovecot-sysreport, but I am not encouraged about sending it when stdout presented: # dovecot-sysreport Gathering configurations ... grep: The -P option is not supportedgrep: The -P option is not supported grep: The -P option is not supported Gathering system informations ... Creating archive ... All done! Please report file dovecot-sysreport-TNDH-mail-1623209001.tar.gz Removing temp files at /tmp/tmp.kphlba44 ... #
While dovecot -n stdout presented the line: ssl_key = # hidden, use -P to show it
expecting people to put sensitive configuration on a public mail list without knowing what the tool is including is a challenge, but when the tool is errantly using the command line option that is also used for exposing the private data by a related tool, it is even less likely that I want to do that. While the dovecot -n option did hide passwords, it did not hide the username associated with that. I will put dovecot -n (redacted) here, but until I have time to see exactly what the sysreport included, I am not releasing that.
To alleviate your concerns:
from man grep
-P, --perl-regexp Interpret PATTERNS as Perl-compatible regular expressions (PCREs). This option is experimental when combined with the -z (--null-data) option, and grep -P may warn of unimplemented features.
dovecot-sysreport is a shell script, so you can easily verify that it is not attempting to call doveconf -nP, but instead, is trying to pass it to grep.
Aki
.... <snip/>
To alleviate your concerns:
from
man grep-P, --perl-regexp Interpret PATTERNS as Perl-compatible regular expressions (PCREs). This option is experimental when combined with the -z (--null-data) option, and grep -P may warn of unimplemented features.
dovecot-sysreport is a shell script, so you can easily verify that it is not attempting to call
doveconf -nP, but instead, is trying to pass it to grep.
I had looked at the script, but the point was that the process for reporting bugs is not particularly appropriate for protecting sensitive information, and the combination of messages just reinforces that issue.
I did finally get time to unpack the archive and look at everything being sent. Since the submission_relay_user is included, the next time I need to report something I will have to comment that out before running sysreport.
Tony
On 09/06/2021 07:57, Tony Hain wrote:
I have a new install of dovecot 2.3.13, along with exim 4.94, in an Azure hosted FreeBSD 12.2 VM. I have been running exim on local hardware with FreeBSD for 15+ years, but dovecot and Azure are a new "learning experience". I am getting an error response in dovecot.log when trying to use the submission relay function, which is apparently new in 2.3... It would appear the parser is either broken or has a character set limitation that no other smtp implementation has. I finally gave up trying to figure out what I might have done wrong in setting up exim and pointed dovecot at mailjet and got the same error.
Jun 08 19:39:42 submission(testing@dispatch.tndh.net)<89538><lOfAL0zEFNmsOCrh>: Warning: smtp-client: conn in-v3.mailjet.com:587 (104.199.96.85:587) [1]: Received invalid EHLO response line: Unexpected character in EHLO keyword Jun 08 19:39:42 submission(testing@dispatch.tndh.net)<89538><lOfAL0zEFNmsOCrh>: Warning: smtp-client: conn in-v3.mailjet.com:587 (104.199.96.85:587) [1]: Received invalid EHLO response line: Unexpected character in EHLO keyword
I didn't try the mailjet path with telnet, but I had done that earlier with the local exim server and I can't see any invalid characters, even in the tcpdump pcap file.
Jun 08 10:49:42 submission(testing@dispatch.tndh.net)<29791><j8NnyETEqV2sOCq3>: Warning: smtp-client: conn 127.0.0.1:58 [1]: Received invalid EHLO response line: Unexpected character in EHLO keyword # telnet localhost 58 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. 220 secure smtp server ehlo dovecot.tndh.net 250-exim.tndh.net Hello dovecot.tndh.net [127.0.0.1] 250-SIZE 536870912 250-8BITMIME 250-VRFY 250-PIPELINING 250-X_PIPE_CONNECT 250-AUTH CRAM-MD5 250-CHUNKING 250-SMTPUTF8 250 HELP
There is your problem. We should probably allow this in Dovecot (seen this problem before), but the underscore in the X_PIPE_CONNECT capability is not allowed in SMTP.
Regards,
Stephan.
This might be some confusion about starttls on the mailjet path, but if that is true the error message is wrong; and it wouldn't be true for the local exim open smtp port. If it really is smtp, it would be most helpful if the error message actually reported what string it is taking issue with.
I have the dovecot-sysreport, but I am not encouraged about sending it when stdout presented: # dovecot-sysreport Gathering configurations ... grep: The -P option is not supportedgrep: The -P option is not supported grep: The -P option is not supported Gathering system informations ... Creating archive ... All done! Please report file dovecot-sysreport-TNDH-mail-1623209001.tar.gz Removing temp files at /tmp/tmp.kphlba44 ... #
While dovecot -n stdout presented the line: ssl_key = # hidden, use -P to show it
expecting people to put sensitive configuration on a public mail list without knowing what the tool is including is a challenge, but when the tool is errantly using the command line option that is also used for exposing the private data by a related tool, it is even less likely that I want to do that. While the dovecot -n option did hide passwords, it did not hide the username associated with that. I will put dovecot -n (redacted) here, but until I have time to see exactly what the sysreport included, I am not releasing that.
# 2.3.13 (89f716dc2): /usr/local/etc/dovecot/dovecot.conf # Pigeonhole version 0.5.13 (cdd19fe3) # OS: FreeBSD 12.2-RELEASE-p4 amd64 ufs # Hostname: TNDH-mail.g4msrgoph2uevil3ys5jvbbpza.jx.internal.cloudapp.net auth_debug = yes auth_debug_passwords = yes auth_verbose = yes debug_log_path = /var/log/dovecot-debug.log disable_plaintext_auth = no first_valid_uid = 220 hostname = dispatch.tndh.net imap_idle_notify_interval = 20 mins info_log_path = /var/log/dovecot-info.log last_valid_uid = 220 log_debug = (event=* AND cat=*) log_path = /var/log/dovecot.log login_greeting = tndh.net Mailer Server Ready ... login_trusted_networks = 127.0.0.1 10.0.0.4 mail_debug = yes mail_location = maildir:/usr/local/var/dovecot/vhosts/%d/%n mail_plugins = mail_log notify notify_status managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext namespace inbox { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } mailbox virtual/Flagged { auto = subscribe special_use = \Flagged } prefix = separator = / type = private } passdb { args = username_format=%n /usr/local/var/dovecot/db/%d/passwd driver = passwd-file } plugin { expire = Trash mail_home = /usr/local/var/dovecot/vhosts/%d/%n mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid box msgid size recipient_delimiter = + sieve = /usr/local/var/dovecot/vhosts/%d/%n/sieve/.dovecot.sieve sieve_after = /usr/local/var/dovecot/vhosts/%d/%n/sieve/sieve-after.d sieve_before = /usr/local/var/dovecot/vhosts/%d/%n/sieve/sieve-before.d sieve_dir = /usr/local/var/dovecot/vhosts/%d/%n/sieve sieve_global_path = /usr/local/var/dovecot/vhosts/sieve/default.sieve } pop3_client_workarounds = outlook-no-nuls oe-ns-eoh protocols = imap pop3 lmtp submission service auth-worker { user = vmail } service imap-login { inet_listener imap { port = 143 } inet_listener imaps { port = 993 ssl = yes } } service stats { unix_listener stats-writer { mode = 0666 } } service submission-login { inet_listener submission { port = 465 ssl = yes } } ssl_cert = **-as-if-I-want-this-on-a-public-list-** userdb { args = username_format=%n /usr/local/var/dovecot/db/%d/passwd driver = passwd-file } verbose_ssl = yes protocol lmtp { mail_fsync = optimized mail_plugins = mail_log notify notify_status sieve } protocol imap { mail_max_userip_connections = 10 mail_plugins = mail_log notify notify_status imap_sieve } protocol pop3 { mail_max_userip_connections = 10 mail_plugins = mail_log notify notify_status } protocol lda { mail_fsync = optimized mail_plugins = mail_log notify notify_status sieve }
Thanks Stephan,
That appears to be a new feature of 4.94 in response to CVE-2020-28018. One could argue that it is an Exim bug, and they really need to fix it. At the same time that character is not likely to cause any parsers to expose security holes so it is unclear why it is precluded in an smtp protocol response other than 40 year old historic syntax conventions.
I will see what I can do to turn that off in exim, but it would be good if the dovecot team reconsidered Postel's mantra: "be conservative in what you send and liberal in what you accept". Granted security considerations moderate how liberal one can be, but pedantic parsing rules that make no difference only reduce the utility of the software. In any case the logging could be more helpful if it would include the objectionable string.
Tony
-----Original Message----- From: dovecot [mailto:dovecot-bounces@dovecot.org] On Behalf Of Stephan Bosch Sent: Wednesday, June 09, 2021 5:21 AM To: Tony Hain; dovecot@dovecot.org Subject: Re: 2.3.13 broken submission relay smtp parser
On 09/06/2021 07:57, Tony Hain wrote:
I have a new install of dovecot 2.3.13, along with exim 4.94, in an Azure hosted FreeBSD 12.2 VM. I have been running exim on local hardware with FreeBSD for 15+ years, but dovecot and Azure are a new "learning experience". I am getting an error response in dovecot.log when trying to use the submission relay function, which is apparently new in 2.3... It would appear the parser is either broken or has a character set limitation that no other smtp implementation has. I finally gave up trying to figure out what I might have done wrong in setting up exim and pointed dovecot at mailjet and got the same error.
Jun 08 19:39:42 submission(testing@dispatch.tndh.net)<89538><lOfAL0zEFNmsOCrh>: Warning: smtp-client: conn in-v3.mailjet.com:587 (104.199.96.85:587) [1]: Received invalid EHLO response line: Unexpected character in EHLO keyword Jun 08 19:39:42 submission(testing@dispatch.tndh.net)<89538><lOfAL0zEFNmsOCrh>: Warning: smtp-client: conn in-v3.mailjet.com:587 (104.199.96.85:587) [1]: Received invalid EHLO response line: Unexpected character in EHLO keyword
I didn't try the mailjet path with telnet, but I had done that earlier with the local exim server and I can't see any invalid characters, even in the tcpdump pcap file.
Jun 08 10:49:42 submission(testing@dispatch.tndh.net)<29791><j8NnyETEqV2sOCq3>: Warning: smtp-client: conn 127.0.0.1:58 [1]: Received invalid EHLO response line: Unexpected character in EHLO keyword # telnet localhost 58 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. 220 secure smtp server ehlo dovecot.tndh.net 250-exim.tndh.net Hello dovecot.tndh.net [127.0.0.1] 250-SIZE 536870912 250-8BITMIME 250-VRFY 250-PIPELINING 250-X_PIPE_CONNECT 250-AUTH CRAM-MD5 250-CHUNKING 250-SMTPUTF8 250 HELP
There is your problem. We should probably allow this in Dovecot (seen this problem before), but the underscore in the X_PIPE_CONNECT capability is not allowed in SMTP.
Regards,
Stephan.
This might be some confusion about starttls on the mailjet path, but if that is true the error message is wrong; and it wouldn't be true for the local exim open smtp port. If it really is smtp, it would be most helpful if the error message actually reported what string it is taking issue with.
I have the dovecot-sysreport, but I am not encouraged about sending it
when
stdout presented: # dovecot-sysreport Gathering configurations ... grep: The -P option is not supportedgrep: The -P option is not supported grep: The -P option is not supported Gathering system informations ... Creating archive ... All done! Please report file dovecot-sysreport-TNDH-mail- 1623209001.tar.gz Removing temp files at /tmp/tmp.kphlba44 ... #
While dovecot -n stdout presented the line: ssl_key = # hidden, use -P to show it
expecting people to put sensitive configuration on a public mail list without knowing what the tool is including is a challenge, but when the tool is errantly using the command line option that is also used for exposing the private data by a related tool, it is even less likely that I want to do that. While the dovecot -n option did hide passwords, it did not hide the username associated with that. I will put dovecot -n (redacted) here, but until I have time to see exactly what the sysreport included, I am not releasing that.
# 2.3.13 (89f716dc2): /usr/local/etc/dovecot/dovecot.conf # Pigeonhole version 0.5.13 (cdd19fe3) # OS: FreeBSD 12.2-RELEASE-p4 amd64 ufs # Hostname: TNDH- mail.g4msrgoph2uevil3ys5jvbbpza.jx.internal.cloudapp.net auth_debug = yes auth_debug_passwords = yes auth_verbose = yes debug_log_path = /var/log/dovecot-debug.log disable_plaintext_auth = no first_valid_uid = 220 hostname = dispatch.tndh.net imap_idle_notify_interval = 20 mins info_log_path = /var/log/dovecot-info.log last_valid_uid = 220 log_debug = (event=* AND cat=*) log_path = /var/log/dovecot.log login_greeting = tndh.net Mailer Server Ready ... login_trusted_networks = 127.0.0.1 10.0.0.4 mail_debug = yes mail_location = maildir:/usr/local/var/dovecot/vhosts/%d/%n mail_plugins = mail_log notify notify_status managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded- character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext namespace inbox { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } mailbox virtual/Flagged { auto = subscribe special_use = \Flagged } prefix = separator = / type = private } passdb { args = username_format=%n /usr/local/var/dovecot/db/%d/passwd driver = passwd-file } plugin { expire = Trash mail_home = /usr/local/var/dovecot/vhosts/%d/%n mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid box msgid size recipient_delimiter = + sieve = /usr/local/var/dovecot/vhosts/%d/%n/sieve/.dovecot.sieve sieve_after = /usr/local/var/dovecot/vhosts/%d/%n/sieve/sieve-after.d sieve_before = /usr/local/var/dovecot/vhosts/%d/%n/sieve/sieve- before.d sieve_dir = /usr/local/var/dovecot/vhosts/%d/%n/sieve sieve_global_path = /usr/local/var/dovecot/vhosts/sieve/default.sieve } pop3_client_workarounds = outlook-no-nuls oe-ns-eoh protocols = imap pop3 lmtp submission service auth-worker { user = vmail } service imap-login { inet_listener imap { port = 143 } inet_listener imaps { port = 993 ssl = yes } } service stats { unix_listener stats-writer { mode = 0666 } } service submission-login { inet_listener submission { port = 465 ssl = yes } } ssl_cert = **-as-if-I-want-this-on-a-public-list-** userdb { args = username_format=%n /usr/local/var/dovecot/db/%d/passwd driver = passwd-file } verbose_ssl = yes protocol lmtp { mail_fsync = optimized mail_plugins = mail_log notify notify_status sieve } protocol imap { mail_max_userip_connections = 10 mail_plugins = mail_log notify notify_status imap_sieve } protocol pop3 { mail_max_userip_connections = 10 mail_plugins = mail_log notify notify_status } protocol lda { mail_fsync = optimized mail_plugins = mail_log notify notify_status sieve }
participants (3)
-
Aki Tuomi
-
Stephan Bosch
-
Tony Hain