December 2014
Please note that republishing this article in full or in part is only allowed under the conditions described here.
Quantifying the Quality of SMTP TLS Support
This study analyzes, how many mail server offer a TLS upgrade and examines the quality of the STARTTLS implementation or setup. It checks the quality of protocol version and ciphers, but also determines how widespread are some setup or implementation errors I've seen in practice. To make the results somewhat more relevant it uses the top 1 million domains based on the studies by Alexa. It also provides the necessary tools to do such study with your own domain list.
Some background to mail delivery
Mail is transferred from sender to recipient with multiple hops. The initial hop is usually from senders MUA (Mail User Agent) to the MTA (Mail Transfer Agent) of the senders domain using the SMTP (Simple Mail Transfer Protocol) protocol. Then there is at least on hop from there to the MTA of the recipients domain, but in more complex setups like in companies there can be multiple hops or there can be some middleboxes like firewalls in-between.
While the MTA for the sender is usually configured within the senders MUA, the MTA for the recipients domain is determined by looking at the MX (Mail Exchange) records in the DNS (Domain Name System). Per definition the recipients MTA is contacted by plain, unencrypted SMTP on port 25. Only after issuing a STARTTLS command the connection can be upgraded to TLS.
What this study is not about
This study does not check if the recipients MTA provides a valid certificate, which matches the recipients domain and is signed by a trusted certificate agency. Such certificates are needed with do detect active man-in-the-middle attacks. Unfortunatly, due to the overall design of the STARTTLS process an active attacker can just simply strip the STARTTLS support from the communication instead of mounting the much harder man-in-the-middle attack against TLS. Thus STARTTLS like currently used is only able to protect the connection against passive sniffing and not active attacks.
What happens if the initial TLS upgrade fails?
Contrary to most desktop browsers MTA will usually not retry with a downgraded TLS protocol or without TLS later, but will simply retry the same way again at some later time. Thus the delivery will fail again and usually after some days the mail is finally considered undeliverable. Unfortunatly, all one will get as a feedback is usually a "TLS handshake failed" message which most system administrators will either ignore or will need to explicitly add an delivery exception for this specific domain.
Summary of results
- From the domains checked about 84% provide a mail server and from these about 72% provide support for STARTTLS, i.e. about 60% of all domains.
- Nearly all STARTTLS enabled servers are configured in a sane way, that is 68% offer TLS1.2 and 32% TLS1.0. Only very few servers still need SSL3.0. Interestingly, TLS1.2 is less in use amomg the top sites, i.e only 59% of the top 5k sites offer TLS1.2 while it is 68% among the top 1M sites.
- The offered ciphered are mostly fine. More than 80% of the servers uses ciphers with Forward Secrecy by default, either with DHE or ECDHE ciphers. The use of the faster ECDHE is more prominent among the top servers, even if TLS1.2 is less common at these top sites.
- A surprisingly large number especially of the top sites has problems with MD5 client certificates, see below. Other problems I've seen in practice (see below) are not that widespread.
Problems seen in practice
Even if the large majority of the mail servers are not broken, there are still enough servers out there which cause real problems. Only about 30% of the servers I had problem reports for were even in the top 1M list and even then at the lower end of the list, where the most problems (apart from MD5 client certificates) can be seen.
Croaks on client certificate signed with MD5
Lots of sites (9% of the top 5k, 4% of all) simply close the connection if the client provides an client certificate signed with MD5 within a TLS1.2 connection. This problem does not happen if the certificate is signed with SHA-1 or others signature algorithms, or if a TLS protocol lower than TLS1.2 is used. Interestingly it also works if the client does not provide a certificate at all, because the server only requests but not requires client certificates. And while MD5 signatures should not be used anymore it is not uncommon that they are just kept if older systems get upgraded.
This problem shows up with sites where *.mail.protection.outlook.com is configured as MX. It can also be seen with other sites and often a hint to some "AntiSpamProxy" can be seen inside the servers certificate. Because of the strange symptoms the underlying cause of delivery errors is hard to find unless one specifically looks out for this problem.
Only TLS1.2 ciphers
Due to a misunderstanding of the POODLE attack some sites disabled all SSL3.0 ciphers instead of only disabling the SSL3.0 protocol version. Since SSL3.0 ciphers are also needed for TLS1.0 and TLS1.1 and only TLS1.2 defines some new ciphers, this effectivly makes these server TLS1.2 only and thus inaccesible for about 30..40% of the mail server which can do only TLS1.0. While not very common this error is much more prominent among the less used sites.
TLS1.1+ required
Some sites only allow TLS1.1+ for some yet unknown reason. One reason is the disabling of SSL3.0 ciphers as described in the previous issue, but there seem to be other causes too. The impact is about the same as with the previous problems, that is it makes the server inaccessible for 30..40% of the MTAs's worldwide.
Detailed results
The test was done with the mx_starttls_bulk Perl program and the summary was created with another Perl program and some editing.
----------------------------------------------------------------------------------------------------
| | | |
Total domains: | 5000 | 20000 | 100000 | 998549
| | | |
----------------------------------------------------------------------------------------------------
| | | |
No MX record: | 647 (12.9%) | 2762 (13.8%) | 16793 (16.8%) | 161555 (16.2%)
No STARTTLS: | 1214 (24.3%) | 4295 (21.5%) | 21124 (21.1%) | 199902 (20%)
TLS available: | 3058 (61.2%) | 12338 (61.7%) | 58101 (58.1%) | 601647 (60.3%)
| | | |
----------------------------------------------------------------------------------------------------
Relative to TLS sites: | | | |
TLS initial success: | 3058 (100%) | 12335 (100%) | 58080 (100%) | 601451 (100%)
TLS after downgrade: | 0 (0%) | 3 (0.024%) | 16 (0.028%) | 157 (0.026%)
TLS total fail: | 0 (0%) | 0 (0%) | 4 (0.0069%) | 32 (0.0053%)
| | | |
----------------------------------------------------------------------------------------------------
TLS common problems: | | | |
TLS1.1+ required: | 1 (0.033%) | 1 (0.0081%) | 4 (0.0069%) | 83 (0.014%)
Only TLS1.2 ciphers: | 0 (0%) | 0 (0%) | 4 (0.0069%) | 84 (0.014%)
Croak on MD5 client cert: | 261 (8.53%) | 792 (6.42%) | 3120 (5.37%) | 22810 (3.79%)
| | | |
----------------------------------------------------------------------------------------------------
TLS protocols used: | | | |
TLS1.2: | 1798 (58.8%) | 7847 (63.6%) | 38401 (66.1%) | 410073 (68.2%)
TLS1.1: | 4 (0.13%) | 17 (0.14%) | 106 (0.18%) | 1245 (0.21%)
TLS1.0: | 1255 (41%) | 4473 (36.3%) | 19586 (33.7%) | 190261 (31.6%)
SSL3.0: | 1 (0.033%) | 1 (0.0081%) | 3 (0.0052%) | 29 (0.0048%)
| | | |
----------------------------------------------------------------------------------------------------
Ciphers used: | | | |
ECDHE-* (FS) | 1408 (46%) | 5836 (47.3%) | 24879 (42.8%) | 197756 (32.9%)
DHE-* (FS) | 1036 (33.9%) | 4270 (34.6%) | 23426 (40.3%) | 310913 (51.7%)
RC4 (bad) | 19 (0.62%) | 100 (0.81%) | 510 (0.88%) | 8150 (1.35%)