-------- Original-Nachricht -------- Betreff: STARTTLS bug - background story Datum: Mon, 7 Mar 2011 15:08:09 -0500 (EST) Von: Wietse Venema wietse@porcupine.org An: Postfix users postfix-users@postfix.org
CERT/CC announces a flaw today in multiple STARTTLS implementations. This problem was silently fixed in Postfix 2.8 and 2.9. Updates for Postfix 2.[4-7] are made available via the usual channels.
WietsePlaintext injection in multiple implementations of STARTTLS
This is a writeup about a flaw that I found recently, and that existed in multiple implementations of SMTP (Simple Mail Transfer Protocol) over TLS (Transport Layer Security) including my Postfix open source mailserver. I give an overview of the problem and its impact, technical background, how to find out if a server is affected, fixes, and draw lessons about where we can expect similar problems now or in the future. A time line is at the end.
On-line information is/will be available at: http://www.kb.cert.org/vuls/id/555316 http://www.postfix.org/CVE-2011-0411.html
Problem overview and impact
The TLS protocol encrypts communication and protects it against modification by other parties. This protection exists only if a) software is free of flaws, and b) clients verify the server's TLS certificate, so that there can be no "man in the middle" (servers usually don't verify client certificates).
The problem discussed in this writeup is caused by a software flaw. The flaw allows an attacker to inject client commands into an SMTP session during the unprotected plaintext SMTP protocol phase (more on that below), such that the server will execute those commands during the SMTP-over-TLS protocol phase when all communication is supposed to be protected.
The injected commands could be used to steal the victim's email or SASL (Simple Authentication and Security Layer) username and password.
This is not as big a problem as it may appear to be. The reason is that many SMTP client applications don't verify server TLS certificates. These SMTP clients are always vulnerable to command injection and other attacks. Their TLS sessions are only encrypted but not protected.
A similar plaintext injection flaw may exist in the way SMTP clients handle SMTP-over-TLS server responses, but its impact is less interesting than the server-side flaw.
SMTP is not the only protocol with a mid-session switch from plaintext to TLS. Other examples are POP3, IMAP, NNTP and FTP. Implementations of these protocols may be affected by the same flaw as discussed here.
Technical background: SMTP over TLS
For a precise description of SMTP over TLS, see RFC 3207, on-line at http://www.ietf.org/rfc/rfc3207.txt.
SMTP over TLS uses the same TLS protocol that is also used to encrypt traffic between web clients and web servers. But, there is a subtle difference in the way TLS is used, and that makes this flaw possible.
SMTP sessions over TLS begin with an SMTP protocol handshake in plaintext. Plaintext means no encryption (thus no privacy), and no protection against modification (no integrity). The plaintext handshake is needed because SMTP has always worked this way. Simply skipping this plaintext phase would seriously break internet email.
During the plaintext handshake phase, the SMTP server announces whether it is willing to use TLS. If both SMTP client and server support TLS, the client sends a "STARTTLS" request to turn on TLS. Once TLS is turned on, all further traffic is encrypted and protected from modification. The client and server repeat the entire SMTP protocol handshake, and the client starts sending mail.
Demonstration
The problem is easy to demonstrate with a one-line change to the OpenSSL s_client command source code (I would prefer scripting, but having to install Perl CPAN modules and all their dependencies is more work than downloading a .tar.gz file from openssl.org, adding eight characters to one line, and doing "./config; make").
(The OpenSSL s_client command can make a connection to servers that support straight TLS, SMTP over TLS, or a handful other protocols over TLS. The demonstration here focuses on SMTP over TLS only.)
The demonstration with SMTP over TLS involves a one-line change in the OpenSSL s_client source code (with OpenSSL 1.0.0, at line 1129 of file apps/s_client.c).
Old: BIO_printf(sbio,"STARTTLS\r\n"); New: BIO_printf(sbio,"STARTTLS\r\nRSET\r\n");
With this change, the s_client command sends the plaintext STARTTLS command ("let's turn on TLS") immediately followed by an RSET command (a relatively harmless protocol "reset"). Both commands are sent as plaintext in the same TCP/IP packet, and arrive together at the server. The "\r\n" are the carriage-return and newline characters; these are necessary to terminate an SMTP command.
When an SMTP server has the plaintext injection flaw, it reads the STARTTLS command first, switches to SMTP-over-TLS mode, and only then the server reads the RSET command. Note, the RSET command was transmitted during the plaintext SMTP phase when there is no protection, but the server reads the command as if it was received over the TLS-protected channel.
Thus, when the SMTP server has the flaw, the s_client command output will show two "250" SMTP server responses instead of one. The first "250" response is normal, and is present even when the server is not flawed. The second "250" response is for the RSET command, and indicates that the SMTP server has the plaintext injection flaw.
$ apps/openssl s_client -quiet -starttls smtp -connect server:port [some server TLS certificate details omitted] 250 some text here <=== Normal response, also with "good" server. 250 more text here <=== RSET response, only with flawed server.
How would an attacker exploit this? It would play man-in-the-middle on the connection between SMTP client and server, perhaps using ARP spoofing at a public WIFI access point. Instead of adding a harmless RSET command, it could steal email or authentication credentials.
Anatomy of the flaw
The flaw is made possible by two ingredients: I already discussed the switch mid-session from plaintext SMTP to SMTP over TLS. This allows an attacker to piggy-back commands onto the SMTP client's plaintext STARTTLS ("let's turn on TLS") request, such that the server may read those commands after the switch to TLS is completed, as if the commands arrived through the TLS-encrypted session.
The second ingredient is a layered software architecture through which those piggy-backed commands bubble up from the network to the application. In the case of SMTP, we have the following major layers before and after the switch to TLS:
Before switch to TLS After switch to TLS
SMTP protocol engine SMTP protocol engine
|| ||
TCP/IP protocol engine TLS protocol engine
|| ||
Internet TCP/IP protocol engine
||
InternetEach layer hides details of what is happening inside. Layering makes it easy to switch from plaintext to TLS, with minimal changes to existing code.
To insert the TLS layer between the SMTP engine and the O/S TCP/IP engine, simply adjust the plumbing between the layers, and make all information flow through the TLS layer.
It's all about the plumbing
Whether a program may have the plaintext injection flaw depends on how it adjusts the plumbing as it inserts the TLS protocol layer in-between the SMTP protocol layer and the O/S TCP/IP protocol layer. I illustrate this with examples from three open source MTAs: Postfix, Sendmail and Exim.
The diagram below zooms in on the plumbing between the SMTP and TLS layers, and shows how different MTAs handle the switch from plaintext to TLS. It is best viewed with a fixed-width font (for example, from the Courier family).
Postfix MTA Sendmail MTA Exim MTA
before/after before/after before/afterswitch to TLS switch to TLS switch to TLS
SMTP SMTP SMTP SMTP SMTP SMTP <= SMTP layer
|| || || || || ||stream stream stream stream' || || buffers buffers buffers buffers' rw r'w' <= stream layer rw r'w' rw r'w' || || || || || || || || || TLS || TLS || TLS <= TLS layer || || || || || || O/S O/S O/S O/S O/S O/S <= TCP/IP layer
As shown in the diagram, both Postfix and Sendmail use an application- level stream abstraction, where each stream has properties such as read/write buffers, read/write functions (indicated with rw), and other properties that are omitted for brevity.
When Postfix switches to SMTP over TLS, it replaces the plaintext read/write functions (rw) with the TLS read/write functions (r'w'). Postfix does not modify any of the other stream properties including the read/write buffers. A patch for qmail that introduces TLS support uses the same approach. This approach of replacing only the stream read/write functions, but not the buffers or other stream properties, can introduce the plaintext injection flaw.
When Sendmail switches to SMTP over TLS, it replaces the entire stream, along with its read/write buffers and read/write functions. Exim. on the other hand, does not have a stream abstraction like Postfix, Sendmail or qmail. Instead of replacing streams or stream properties, Exim replaces plaintext read/write functions with TLS read/write functions. Because of their program structure, Sendmail and Exim didn't suffer from the plaintext injection flaw.
Switching world views
When the switch from plaintext to TLS mode is made, all layers above the TLS layer need to purge all information that they have received through the plaintext session. This "world view switch" needs to be implemented consistently. Otherwise, information that was sent as unprotected plaintext may slip through the cracks.
In the case of Postfix and other affected MTAs, this "world view switch" was incomplete. Postfix it did not account for information that lingered in stream buffers at the boundary between two layers. This allowed an attacker to piggy-back commands onto the plaintext "let's turn on TLS" request, such that the commands could be read after the switch to TLS was completed, as if they had arrived through the TLS-encrypted session.
Fixing the problem
There are two solutions to address the flaw, and both solutions can be used together.
Report an error when unexpected plaintext is received after the STARTTLS command. As documented in RFC 3207, STARTTLS must be the last command in a pipelined group. If plaintext commands are received after STARTTLS, then that is a protocol violation.
This measure can also be implemented outside the MTA, for example in a protocol-aware firewall.
If a program uses the same input buffer before and after the switch to TLS, it should discard the contents of the input buffer, just like it discards SMTP protocol information that it received during the plaintext protocol phase.
Conclusion
This plaintext injection problem is likely to recur when some development moves the plaintext-to-ciphertext switch outside the application: for example, into the kernel, into the local hardware, into a proxy, or into other infrastructure. This encourages applications to use the same application-level streams and buffers and read/write functions before and after the switch to ciphertext. When this migration happens, plaintext injection becomes once more a possibility.
Postfix did not reject plaintext commands that were piggy-backed onto the plaintext "let's turn on TLS" request. This reflects what once was the primary mission of Postfix: to deliver mail, not to force other systems to implement all the Internet RFCs correctly. Nowadays, strict protocol compliance is becoming a requirement for senders to get their email delivered. As this episode shows, stricter protocol enforcement by receivers can bring security benefits, besides blocking spambots.
Time line
Jan 5 2011: While finishing Postfix for its annual release, I found this flaw in the server and client implementations of SMTP over TLS. It had been sitting there for six years ever since TLS support was adopted into Postfix. Not wanting to delay the release schedule, I silently fixed the problem and sent an email to co-developer Victor Duchovni.
Jan 6-10 2011: As we investigated the scope of the problem, Victor discovered quickly that many other SMTP over TLS implementations were also affected. Among those affected were email service providers, email anti-spam/virus service providers, anti-spam/virus appliances, as well as other mail server implementations. It was clear that this problem's resolution was going to involve many organizations.
Jan 11 2011: Contact CERT/CC to help coordinate with the problem's resolution.
Mar 7 2011: Public announcement.