In the recent years a number of security-oriented client-side controls for web browsers appeared in the scene in form of security headers. These headers can be used to improve the security of the user experience when interacting with a web application with little additional effort and negligible performance overhead -- essentially, they can serve as a safety net mitigation for a number of well-known web exploits.
Despite all benefits brought by these headers, judging by the web security assessments we have performed it seemed to us that their ubiquitousness is far less pervasive than we wanted it to be.
With that in mind, we sought to answer the question of how popular HTTP security headers are, and to quantify their pervasiveness in Brazil and Estonia, where our presence is.
HTTP Security Headers
X-XSS-Protection: activates the built-in anti-XSS filter of IE and Chrome.
X-Content-Type-Options: prevents the browser from performing content type sniffing.
HTTP Strict Transport Security: forces the browser to only access the HTTPS version of a site.
X-Frame-Options: disallows that particular website to be shown inside a frame, protecting against clickjacking attacks.
Content Security Policy: enforces a series of restrictions in how content is fetched and loaded and scripts executed - reduces the risk of successful exploitation of cross-site scripting, mixed content, etc.
Public key pinning: based on the model of trust-on-first-use, it is a security feature that tells the browser to pin the hashes of certificates for a particular site. This prevents man-in-the-middle attacks, including the ones where an attacker has compromised a trusted CA.
Where the idea of the survey came from
A few months ago while we were setting up our web-facing infrastructure we wanted to leave no room for the old saying "the shoemaker's son always goes barefoot". After all, we are an information security consultancy firm and we have to practice what we preach -- it would make no sense at all to advise clients to use security headers in their web servers when we, the trusted advisors, do not.
In fact, unfortunately many IT security companies do not even take advantage of these headers themselves, but this discussion is out of scope for this post.
Even though setting up most of HTTP security headers is, in most cases, fairly straightforward and despite the existence of step-by-step tutorials on the Internet, in many web application security engagements we delivered we noticed that we rarely saw these headers being used.
Why Brazil and Estonia?
Internet security is our business and having a bigger picture of the current state of affairs of awareness and usage of security controls in the countries where we are incorporated is our interest.
On top of that, these countries have a strong Internet penetration rate: as of 2016, 66.4% in Brazil (more than 100 million people) and an astonishing 91.4% in Estonia.
Setting up the experiment
We started out by downloading a list from Alexa Top 1 million visited websites worldwide. From this list we extracted all domains ending in ".ee" and ".br" and removed duplicates from domains like "blogspot.com.br", etc. This left us with 539 Estonian URLs and 15,434 Brazilian web sites to analyze.
As usual, we wrote a Python script to automate the survey.
In a nutshell, the process consists in sending a simple GET request to the web server and analyze the headers that come in the response. For each security header found in the response, a score counter is incremented by 1. The results are saved in a local SQLite3 database.
It is important to stress that no malicious requests had to be issued in order to obtain these headers; only a normal connection to the webserver. The response analysis works passively and is performed off-line.
At the time of the analysis we managed to collect data on 14,334 Brazilian websites and 514 from Estonia. A small amount of the web sites could not be reached at all during the experiment.
The code written for this analysis merely checks for the presence of HTTP security headers but does not make any judgement on the effectiveness or correctness of the configurations or rules implemented by the server.
For example, it does not verify the validity of HSTS pins or if X-XSS-Protection is set to 0.
The survey began with Estonia as it contained a much smaller number of URLs to query so we could quickly validate our hypothesis and to see whether we were in the right direction as well as fine-tune our script and fix any bugs that could cause hiccups when performing the same exercise against a larger data set.
The script and data used to conduct the experiment can be found in our Github.
In Brazil we found out that 12,462 out of 14,334 websites, that is 86.94% of the most visited websites in the country did not use even a single of the HTTP security headers. In Estonia the number was 448, accounting for 87.15% of the most popular sites.
Approximately 13.05% of the sites in Brazil employ at least one security header, whereas in Estonia 12.84% do.
Finally, none of the sites in both countries had enabled all security headers in scope for this survey. Only five sites, four in Brazil and one in Estonia, had five of the headers enabled.
Although HTTP security headers have been around for a couple of years, it seems that it still has not gained the popularity it deserves. This denies browser-enforced protection to hundreds of thousands of users, leaving them exposed to potential vulnerabilities in the applications hosted by these web servers.
This weekend research project revealed that the adoption of such headers is still very weak and far from ubiquitous.
We hope the results of the survey brings more awareness to system administrators and application developers in order to improve web security as a whole.