Have you heard? A tiny bug in Cloudflare’s code has led an unknown quantity of data—including passwords, personal information, messages, cookies, and more—to leak all over the internet. If you haven’t heard of the so-called Cloudbleed vulnerability, keep reading. This is a scary big deal.
Let’s start with the good news. Cloudflare, one of the world’s largest internet security companies, acted fast when security researcher Tavis Ormandy of Google’s Project Zero identified the vulnerability.
The bad news is that the Cloudflare-backed websites had been leaking data for months before Ormandy noticed the bug. Cloudflare says the earliest data leak dates back to September 2016. It’s so far unclear if blackhat hackers had already found the vulnerability and exploited it secretly before Cloudflare fixed its code. Cloudflare’s clients include huge companies like Uber, OKCupid, 1Password (Update: 1Password claims its user data is safe), and FitBit. That means a holy fuck ton of sensitive data has potentially been compromised.
As with any major security vulnerability, it will take some time before we can fully comprehend the level of destruction caused by Cloudbleed. For now, you should change your passwords—all of them—and implement two-factor authentication everywhere you can. You’ll figure out why this is a good idea when you read about how this nasty little security disaster unfolded.
You might not be familiar with Cloudflare itself, but the company’s technology is running on a lot of your favorite websites. Cloudflare describes itself as a “web performance and security company.” Originally an app for tracking down the source of spam, the company now offers a whole menu of products to websites, including performance-based services like content delivery services; reliability-focused offerings like domain name server (DNS) services; and security services like protection against direct denial of service (DDoS) attacks.
The fact that Cloudflare is a security company makes the dustup around this new vulnerability supremely ironic. After all, countless companies pay Cloudflare to help keep their user data safe. The Cloudbleed blunder did the opposite of that.
“I’ve informed Cloudflare what I’m working on. I’m finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings,” Tavis Ormandy wrote in an advisory. “We’re talking full https requests, client IP addresses, full responses, cookies, passwords, keys, data, everything.” Ormandy also said that the Cloudbleed vulnerability leaked data across 3,438 unique domains during a five-day period in February.
For you geeks out there, Cloudbleed is especially interesting because a single character in Cloudflare’s code lead to the vulnerability. It appears to be a simple coding error, though we’ve reached out to Cloudflare for information on what exactly happened. (Update: Cloudflare called us back and explained some things.) Based on what’s been reported, it appears that Cloudbleed works a bit like Heartbleed in how it leaks information during certain processes. The scale of Cloudbleed also looks like it could impacts as many users as Heartbleed, as it affects a common security service used by many websites.
According to a Cloudflare blog post, the issue stems from the company’s decision to use a new HTML parser called cf-html. An HTML parser is an application that scans code to pull out relevant information like start tags and end tags. This makes it easier to modify that code.
Cloudflare ran into trouble when formatting the source code of cf-html and its old parser Ragel to work with its own software. An error in the code created something called a buffer overrun vulnerability. (The error involved a “==” in the code where there should have been a “>=”.) This means that when the software was writing data to a buffer, a limited amount of space for temporary data, it would fill up the buffer and then keep writing code somewhere else. (If you’re dying for a more technical explanation, Cloudflare laid it all out in a blog post.)
In plain English, Cloudflare’s software tried to save user data in the right place. That place got full. So Cloudflare’s software ended up storing that data elsewhere, like on a completely different website. Again, the data included everything from API keys to private messages. The data was also cached by Google and other sites, which means that Cloudflare now has to hunt it all down before hackers find it.
It’s unclear who exactly has been pwned. Cloudlfare claims that only a very small number of requests led to leaked data, but since the vulnerability has been almost six months, who knows how much information is out in the wild. Furthermore, the fact that so much of that data was cached across different sites means that, while Cloudflare’s initial patch stopped the leaking, the company needs to do lots of hunting around the web to ensure that all of the leaked data gets scrubbed. And even worse, even sites that don’t use Cloudflare’s service—but have a lot of Cloudflare users—might have compromised data on their servers.
Entrepreneur and security expert Ryan Lackey offered some good advice in a blog post. And Lackey knows what he’s talking about, since his company CryptoSeal was acquired by Cloudflare in 2014.
“Cloudflare is behind many of the largest consumer web services (Uber, Fitbit, OKCupid, …), so rather than trying to identify which services are on Cloudflare, it’s probably most prudent to use this as an opportunity to rotate ALL passwords on all of your sites,” Lackey wrote. “Users should also log out and log in to their mobile applications after this update. While you’re at it, if it’s possible to use 2FA or 2SV with sites you consider important.”
Changing your passwords sucks, but you should be doing it on a semi-regular basis anyways. As we’ve argued in the past, you might as well enable two-factor authentication on everything, too, since it’s your best first defense against hackers. That said, nothing is ever truly secure on the internet, and Cloudbleed might compromise some accounts using. two-factor authentication.
This is all to say: you can’t control what happens under the hood of websites and companies like Cloudflare that power them. But you can watch your own ass—and pray to the hacker gods to keep you safe. Whatever works.