Behind the Scenes: The Crazy 72 Hours Leading Up to the Heartbleed Discovery
From Finland to Silicon Valley, a small team of bug hunters identified and prepared for the worst security flaw in Internet history
Today Heartbleed is a household name—every person who uses the web is terrified of the security glitch. But David Chartier knew about it almost a week ago, before just about anyone else on the planet.
The call came from Finland early Friday morning, just as Chartier had arrived to work at his Silicon Valley office. It was a typical overcast day in San Francisco, and Chartier, the chief executive of a security firm called Codenomicon, picked up the phone. On the other end of the line was one of Codenomicon’s lead security engineers. While testing a new security product, the engineer’s team had found a potentially serious bug in the world’s biggest open-source encryption service, which is used by pretty much every major site, including places like Google and Facebook, to keep personal information secure.
Chartier knew the finding was serious, but at that moment, he had only an inkling of what would come next.
Codenomicon, founded in 2001 by Finnish IT experts, is a global security research firm with offices in a half-dozen countries around the world. Basically, they’re bug bounty hunters. Teams of Codenomicon engineers are constantly testing software for bugs, and writing patches to fix them. They’re good at it, too. Some of their clients include Verizon, Microsoft and Adobe.
Chartier, the company’s CEO, has over 20 years of experience in the field, so he’s encountered thousands of bugs before, but this one was different. Bruce Schneier, the well-known computer security researcher, would later call the bug a “catastrophic” security glitch that affects nearly everyone on the web. “On the scale of 1 to 10, this is an 11,” he wrote.
Before hanging up, Chartier instructed one of the Finnish engineers to write an exploit code to take advantage of Codenomicon‘s own site. Basically, Chartier wanted to see what, exactly, a hacker could get if they knew about the bug.
“We attacked ourselves,” Chartier says. The results freaked him out. The team realized they were able to access a user’s memory, encryption keys, usernames and passwords—”plus a lot of other stuff that we don’t want to mention,” Chartier says. “We saw how serious it was.”
As with any potential security flaw, Chartier knew that the next 24 hours were critical. Most of all, secrecy was tantamount. Using the company’s internally developed encrypted messaging service, Chartier set up a task force of Finnish engineers to create a patch for the bug.
“We kept a high level of secrecy on this,” he says. “We didn’t want anyone to leak this—and we wanted to make sure no one was eavesdropping.”
Chartier, working alone from the Silicon Valley office, instructed the Finnish team on what to do next.
The first thing that needed to be done was to report the bug to the Finnish National Security Cyber Center, which is commonly called “CERT” in security circles. Because the bug was found in the vast OpenSSL encryption service, on Saturday morning, CERT notified the OpenSSL Project, a team of about a dozen volunteer developers located throughout the world. CERT instructed them to begin updating their systems and to prepare a patch to be released to the public as soon as possible.
Unbeknownst to Chartier, a little-known security researcher at Google, Neel Mehta, had discovered and reported the OpenSSL bug on the same day. Considering the bug had actually existed since March 2012, the odds of the two research teams, working independently, finding and reporting the bug at the same time was highly surprising. (Mehta declined to be interviewed for this article.)
In any case, Chartier and the team of engineers got to work. Chartier knew that the advisory wouldn’t be particularly informative or user-friendly, so he began thinking about ways to create a campaign around this particular bug to get the word out.
“The challenge with advisories is that they come out every day,” Chartier says. “So if you’re the IT manager, how do you know which advisory is important and which isn’t? We thought we’d put a name on it and answer some questions and let people know that it’s one of the more serious bugs that’s been discovered in the last few years.”
Up until that point, on Friday evening, Heartbleed wasn’t actually called Heartbleed—it was simply referred to as “CVE-2014-0160,” for the line of code that contained the glitch. On Saturday morning, Ossi Herrala, a system administrator at Codenomicon working from the company’s office in Helsinki, came up with the name Heartbleed.
“There’s an extension on OpenSSL called Heartbeat,” Chartier explains. “[Ossi] thought it was fitting to call it Heartbleed because it was bleeding out the important information from the memory.”
Another Codenomicon employee, Marko Laaso, bought the Heartbleed.com domain name early Saturday morning. In 2008, it had been a popular site for emo kids to share song lyrics and links to other emo websites. (“I feel like im [sic] falling into the emptyness [sic]. …The things I can hold on to are so far away,” read one of posts.)
The team moved quickly. A company artist began work on the logo—a heart with blood dripping from its sides. Once domain was purchased and the logo was created, the company’s security researchers began crafting the frequently asked questions that would appear on Heartbleed.com.
The Helsinki office was buzzing. “We knew it would be a major, significant bug,” says Chartien. “We were stoked.”
On Sunday, the dozen or so Codenomicon employees chatted over the encrypted messaging service and Chartien continued to monitor the web, making sure the bug hadn’t leaked out. By Sunday evening, all the marketing material was in place, and the team was waiting on OpenSSL to release its patch before the launch of Heartbleed.com.
“You don’t want to release information before there’s patch because then there’s no way for people to protect themselves,” Chartier says. “Otherwise, it’s defeating the purpose.”
On Monday afternoon, Heartbleed.com went live, and immediately went crazy with traffic, as the media began picking up the story. Mashable called it a nightmare scenario. Vox released an explainer on how to protect yourself. Pretty much every major news outlet, from CNN to the Washington Post to The New Yorker, had a story up about the bug. In less than 48 hours, by Wednesday afternoon, the site had about 1.4 million unique visitors. Today that will likely be closer to 2 million. Chartier was pleased they were able to make an impact.
“Our mission is to make the Internet safer,” Chartier adds. “I’m happy to see the overall community response. The IT security community has really taken this and done a lot with it. I think it’s a tremendous community effort here.”