Email transport security: STARTTLS, MTA-STS, DANE

30 May 2026 · 12 min read

Why mail in transit is the weak link nobody talks about

DMARC, SPF and DKIM get most of the attention because they answer the question every postmaster cares about: is this message really from who it claims to be? They are authentication mechanisms. They prove identity and they let you reject spoofed mail. What they do not do is protect the message while it is travelling from one mail server to another across the open internet.

That gap matters. When a message leaves your mail server, it hops across the public network to the recipient's inbound mail server. By default, that hop can happen in plaintext. Anyone sitting on the path, a compromised router, a malicious Wi-Fi access point, a hostile transit provider, can read the contents and, worse, can tamper with the connection to strip away any encryption that was on offer. Transport security is the set of standards that closes this gap. It is a different problem from authentication, and it needs its own set of tools.

This article walks through the four pieces that work together to secure inbound mail in transit: opportunistic STARTTLS, MTA-STS, DANE and TLS-RPT. By the end you will understand what each one does, where each one falls short on its own, how they combine, and how to deploy the realistic subset of them for your own domain. If you want to check your current posture as you read, the MTA-STS checker and the full free tools suite will show you exactly what the world sees today.

SMTP started life with no encryption at all

To understand why these standards exist, you have to start with SMTP itself. The Simple Mail Transfer Protocol was designed in the early 1980s for a small, trusted network. There was no encryption in the original specification because there was no threat model that required it. Mail servers connected on port 25 and exchanged messages as readable text.

Encryption was bolted on later through an extension called STARTTLS. The way it works is straightforward. The sending server connects on port 25, the two servers exchange capability advertisements through the EHLO handshake, and if the receiving server advertises STARTTLS, the sender can issue the STARTTLS command to upgrade the plaintext connection to an encrypted TLS session. From that point the rest of the conversation, including the message body, is protected.

This is genuinely useful, and the good news is that it is now extremely widespread. The major providers all support it. The problem is the word that always precedes it in practice: opportunistic.

The fatal flaw of opportunistic STARTTLS

Opportunistic STARTTLS means the sending server will use encryption if it is offered, but will happily fall back to plaintext if it is not. There is no requirement, no enforcement and no memory. Each connection is evaluated fresh, and a connection with no encryption looks, to the sender, exactly like a server that simply does not support it.

This design makes STARTTLS trivially defeatable by an active attacker. Consider what happens when a man-in-the-middle sits between the two servers:

• The sending server connects and sends EHLO.
• The receiving server's real response would advertise STARTTLS.
• The attacker intercepts that response and strips the STARTTLS line out of it before passing it on.
• The sending server, seeing no STARTTLS advertised, concludes the receiver does not support encryption and continues in plaintext.
• The attacker now reads and relays the entire message in the clear.

This is called a STARTTLS stripping or downgrade attack, and it requires no certificate forgery, no cryptographic break, nothing clever. It just deletes one line of text from a handshake. Because opportunistic STARTTLS has no way to know that encryption was supposed to be available, it cannot detect that it was robbed of it.

A second, quieter weakness is certificate validation. Even when STARTTLS does negotiate successfully, most sending servers historically did not validate the receiving server's certificate. They would encrypt to whatever certificate was presented, including a self-signed one or one with a completely wrong hostname. That defends against passive eavesdropping but not against an active attacker who simply presents their own certificate and terminates the TLS session themselves.

So opportunistic STARTTLS gives you encryption against a passive listener and almost nothing against an active one. To close that hole you need a way to tell sending servers, in advance and through a trusted channel, "this domain requires TLS, and here is how to verify the server you are talking to is the real one." That is precisely what MTA-STS and DANE provide, by two different routes.

MTA-STS: a policy published over HTTPS

MTA-STS stands for SMTP MTA Strict Transport Security. It is the more widely deployed of the two enforcement mechanisms, in large part because it does not depend on DNSSEC, which we will get to shortly. MTA-STS lets a domain owner publish a policy that says, in effect, "when you send mail to me, you must use TLS, the certificate must be valid, and the hostname must match one of these patterns. If you cannot satisfy that, do not deliver in plaintext."

It is built from three components working together.

1. A DNS TXT record that signals a policy exists. You publish a record at _mta-sts.yourdomain.com that looks like this:

_mta-sts.example.com. IN TXT "v=STSv1; id=20260622000000Z"

The id is a version string that changes whenever you update the policy. Sending servers cache the policy and use the id to detect when to refetch it.

2. A policy file served over HTTPS. The actual rules live at a well-known URL on a dedicated subdomain:

https://mta-sts.example.com/.well-known/mta-sts.txt

A typical policy file reads:

version: STSv1
mode: enforce
mx: mail.example.com
mx: *.example.com
max_age: 604800

The mode is the important field. It has three values:

• none removes any previously published policy.
• testing tells senders to evaluate the policy and send failure reports through TLS-RPT, but to deliver the mail anyway if TLS fails. This is your safe rollout setting.
• enforce tells senders to refuse to deliver if the TLS requirements are not met.

The mx lines list the hostname patterns your inbound mail servers are allowed to use, and max_age is how long, in seconds, a sender should cache the policy.

3. Trust anchored in the web PKI. The clever part is that the policy file is fetched over HTTPS, with full certificate validation against the public certificate authority system, the same trust system your browser uses. An attacker cannot forge the policy without also forging a valid TLS certificate for mta-sts.example.com, which is exactly the thing the web PKI is designed to make hard. Once a sender has fetched and cached the policy, the STARTTLS stripping attack stops working: the sender knows encryption is mandatory and will not fall back.

MTA-STS does have a genuine limitation worth stating honestly. It relies on trust-on-first-use caching. The very first time a sender contacts you, before it has cached your policy, an attacker who can block access to your HTTPS policy endpoint could prevent the policy from ever being fetched. In practice this is a much narrower window than the wide-open downgrade that opportunistic STARTTLS allows, and once the policy is cached it holds for the full max_age. For most organisations MTA-STS is the right first step because it is deployable today without any DNSSEC dependency.

You can validate every part of this setup, the TXT record, the policy file and the MX patterns, with the MTA-STS checker, and read the deployment-focused walkthrough on the MTA-STS product page.

DANE: trust anchored in DNSSEC instead

DANE, DNS-based Authentication of Named Entities, solves the same downgrade and impersonation problem as MTA-STS, but it anchors trust in a completely different place. Instead of relying on the web PKI and HTTPS, DANE publishes the receiving server's certificate fingerprint directly in DNS, in a record type called TLSA, and protects that record with DNSSEC.

A TLSA record for a mail server looks like this:

_25._tcp.mail.example.com. IN TLSA 3 1 1 <hash-of-the-certificate-public-key>

The name encodes the port (_25) and protocol (_tcp). The three numbers before the hash describe how to interpret it: which certificate it pins, whether it pins the full certificate or just the public key, and which hash algorithm produced the digest. A sending server that supports DANE looks up this record, and when it connects over STARTTLS it checks that the certificate the receiver presents matches the published hash.

The strength of DANE is that there is no trust-on-first-use gap. The requirement to use TLS, and the identity of the correct certificate, are both stated authoritatively in DNS from the very first connection. There is no policy to cache, no HTTPS endpoint to reach, and no certificate authority to trust. An attacker cannot strip the requirement because the TLSA record's mere existence, validated by DNSSEC, tells the sender that TLS is mandatory and that any other certificate is wrong.

The catch, and it is a significant one, is that DANE is only as trustworthy as DNSSEC. Without DNSSEC, the TLSA record could be forged by the same attacker you are trying to defend against, so DANE without DNSSEC is meaningless. That makes DANE harder to deploy: your domain's zone must be DNSSEC-signed, your DNS provider must support it correctly, and you take on the operational discipline of keeping TLSA records in lockstep with certificate renewals. Rotate a certificate without updating the TLSA record first and you will start bouncing legitimate mail.

This is the core trade-off between the two mechanisms. DANE has no first-contact weakness but demands DNSSEC. MTA-STS demands no DNSSEC but has a narrow trust-on-first-use window. They are not mutually exclusive. A domain can publish both, and many large mail operators do, so that senders supporting either mechanism get protection.

TLS-RPT: the feedback loop that makes enforcement safe

If you turn on enforcement, whether through MTA-STS enforce mode or DANE, you have created a way for mail to fail to deliver. A misconfigured certificate, an expired one, a mismatched MX hostname or a broken TLSA record can now cause senders to refuse delivery rather than fall back to plaintext. You need visibility into that, and you need it before you flip the switch to enforce. That is the job of TLS-RPT, SMTP TLS Reporting.

TLS-RPT is a DNS TXT record that asks sending servers to send you a daily aggregate report of their TLS connection outcomes to your domain:

_smtp._tls.example.com. IN TXT "v=TLSRPTv1; rua=mailto:tls-reports@example.com"

The reports arrive as JSON, summarising successful and failed TLS sessions and, crucially, why the failures happened: certificate hostname mismatch, expired certificate, STARTTLS not supported, policy fetch failure, TLSA mismatch and so on. This is the transport-layer analogue of the aggregate reports you already get from DMARC. Where DMARC RUA reports tell you who is sending mail as your domain, TLS-RPT tells you whether mail to your domain is being protected in transit and where the encryption is breaking.

The deployment discipline mirrors the DMARC rollout you may already know from moving from p=none to p=reject. You do not jump straight to enforcement. You:

1. Publish TLS-RPT first, so you have a feedback channel.
2. Publish your MTA-STS policy in testing mode (or your DANE records in a watched state).
3. Collect reports for a week or two and confirm that legitimate senders are negotiating TLS cleanly with no failures.
4. Only then move the MTA-STS policy to enforce.

Without TLS-RPT you are enforcing blind. With it, you get told about problems by the senders themselves before those problems turn into silently bounced mail.

How the four pieces fit together

It helps to see the layers as a stack, each one covering the gap left by the one below.

• STARTTLS provides the encryption mechanism. It is necessary but, on its own, optional and strippable.
• MTA-STS makes that encryption mandatory and verifiable, with trust anchored in the web PKI and a small trust-on-first-use caveat.
• DANE makes it mandatory and verifiable with trust anchored in DNSSEC, with no first-contact caveat but a DNSSEC prerequisite.
• TLS-RPT gives you the telemetry to deploy either enforcement mechanism safely and to keep it healthy.

Walk through a single inbound message to see them interact. A sender wants to deliver to you@example.com. It looks up your MX records and finds mail.example.com. If it supports MTA-STS, it checks _mta-sts.example.com, fetches and caches your policy over HTTPS, and sees mode: enforce with mx: mail.example.com. If it supports DANE, it instead checks for a DNSSEC-validated TLSA record at _25._tcp.mail.example.com. Either way it now knows TLS is required. It connects on port 25, issues STARTTLS, and validates the presented certificate against the MTA-STS hostname pattern or the DANE TLSA hash. If validation passes, the message is delivered encrypted. If it fails, the sender refuses to deliver in plaintext and, the next day, files a TLS-RPT report telling you exactly what went wrong.

The downgrade attack we described earlier no longer works. Even if an attacker strips STARTTLS from the handshake, the sender already knows from the cached policy or the TLSA record that encryption is mandatory, so it treats the missing STARTTLS as a hard failure rather than a reason to fall back.

A practical deployment order

Here is the order I would recommend for a typical domain, from lowest risk to highest effort.

1. Confirm STARTTLS is working on your inbound servers. Every mainstream provider supports it, but verify your own MX hosts present a valid certificate whose hostname matches the MX record. This is the foundation everything else builds on.
2. Publish TLS-RPT. It is a single TXT record and it carries zero delivery risk. Start collecting reports immediately so you have baseline data.
3. Deploy MTA-STS in testing mode. Stand up the mta-sts. subdomain with a valid HTTPS certificate, serve the policy file, and publish the _mta-sts TXT record. Watch your TLS-RPT reports for a couple of weeks.
4. Move MTA-STS to enforce once reports show clean TLS negotiation from real senders and no policy or certificate failures.
5. Consider DANE if your zone is DNSSEC-signed. If you already run DNSSEC, adding TLSA records gives you stronger, first-contact protection and complements MTA-STS. If you do not run DNSSEC, MTA-STS alone is a perfectly defensible posture and far less operationally fragile.

Throughout, keep two operational rules in mind. First, your MTA-STS mx patterns and your TLSA records must always match your real MX hosts and their certificates; a certificate renewal that changes the key requires a TLSA update first. Second, transport security is about inbound mail, the mail other people send to you. It is a separate axis from authenticating the mail you send out, which is what SPF, DKIM and DMARC handle. A complete email security posture needs both axes, which is why it is worth reading them together with what DMARC is and the broader requirements overview.

The practical takeaway

Authentication proves who sent a message. Transport security protects that message while it crosses the network. They are complementary, not interchangeable, and a domain that has deployed DMARC at p=reject but still allows plaintext STARTTLS downgrades has only solved half the problem.

The good news is that the high-value steps are also the low-risk ones. Publishing TLS-RPT and rolling out MTA-STS in testing mode costs you nothing in deliverability and immediately tells you how well your inbound mail is being protected. From there, moving to enforce, and optionally adding DANE if you run DNSSEC, hardens the connection against active attackers who would otherwise strip your encryption silently.

Start by seeing what the world sees today. Run your domain through the MTA-STS checker and browse the rest of the free diagnostic tools to spot gaps in TLS, MX and certificate configuration. If you would rather not hand-roll the subdomain, the HTTPS policy endpoint, the certificate lifecycle and the report parsing yourself, the done-for-you MTA-STS service and ongoing monitoring with change alerts set all of this up and watch it for you, so your inbound mail stays encrypted and your enforcement never quietly breaks.