Single customer triggered mass internet outage, says Fastly

The infrastructure provider behind the internet blackout that took some of the biggest websites in the world offline last week has said that the problem was caused by a single customer updating their settings.

Nick Rockwell, head of engineering and infrastructure at Fastly, said that a bug in the company’s code that was initially pushed in mid-May had not been triggered until Tuesday morning last week.

When the unnamed client updated their system, the bug was triggered, taking out 85% of Fastly’s network in the process.

“On May 12, we began a software deployment that introduced a bug that could be triggered by a specific customer configuration under specific circumstances,” said Rockwell. “Early June 8, a customer pushed a valid configuration change that included the specific circumstances that triggered the bug, which caused 85% of our network to return errors.

“We detected the disruption within one minute, then identified and isolated the cause, and disabled the configuration. Within 49 minutes, 95% of our network was operating as normal.

“Even though there were specific conditions that triggered this outage, we should have anticipated it. We provide mission-critical services, and we treat any action that can cause service issues with the utmost sensitivity and priority. We apologize to our customers and those who rely on them for the outage and sincerely thank the community for its support.”

Fastly’s content delivery network (CDN) is one of the largest on the internet. The service makes the internet faster and more reliable for users by allowing them to connect to servers that are physically closer to them, and optimised for taking on high traffic.

The incident caused Fastly’s stock price to jump by 12% over the course of Tuesday. The counterintuitive rise in price could be due to the company’s effective response to the incident, or simply because investors became more aware of how large Fastly’s customer base actually is.

However, Fastly’s customers will have felt the cost of being taken offline. According to SEO agency Reboot, the outage could have cost Amazon $32m in sales.

“Although it seems they weren’t down for long, the impact it would have had will be huge, especially on e-commerce sites,” said the agency’s managing director, Naomi Aharony.

“With our research estimating Amazon could have potentially lost $6,803 every second it was down, it’s clear an investigation will want to be made to find out what happened.”