Worldwide Internet Outage Caused By Single Configuration Error

The June 8th Internet outage which affected websites in dozens of countries across the Americas, Europe and Asia, as well as South Africa has been traced to a single configuration error at a little known but important IT infrastructure company, Fastly. 

Amazon, eBay and Boots were among retailers whose websites disappeared during the outage. Amazon and Boots alone report a combined £25bn in annual sales in the UK, meaning they would typically earn nearly £3m between them in the course of an average hour. Payment provider PayPal which processed $936bn of transactions in 2020, approximately $106m for every hour, was also temporarily unavailable.

Fastly said it had identified a service configuration that triggered disruptions across its servers and has now disabled that configuration.

The network run by Fastly had the outage for more than an hour and in most cases was occurring mid-morning London time. Investigation reveals the incident was caused by a cloud configuration fault affecting governments and businesses around the world in sectors ranging from media to online retail and telecoms. “We experienced a global outage due to an undiscovered software bug that surfaced on June 8 when it was triggered by a valid customer configuration change. We detected the disruption within one minute, then identified and isolated the cause, and disabled the configuration.... Within 49 minutes, 95% of our network was operating as normal”, said Fastly’s SVP  of Engineering & Infrastructure in a statement. 

The basic issue appears to be that Fastly had mitakenly taken down its own network with a bad software update, resulting in a blockage affecting millions of dollars in revenue for numerous corporations around the world.

All UK government websites using the gov.uk domain were swept up in the outage, which temporarily knocked out crucial services such as the online system by which taxpayers can file annual returns with HM Revenue and Customs. The British  Government said it was investigating reports that users were unable to book Covid-19 tests online as a “matter of urgency”.
Visitors to the official website of the White House were also greeted with a message likely to have been seen by hundreds of millions of internet users: “Error 503 service unavailable”.

According to analysis of data from Google’s AdSense platform, outlets lost approximately $300,000 in revenue during the period, as they missed out on clicks that would usually translate into payments from advertisers. The calculation, made by marketing agency connective3, covers lost revenue at affected advertising-reliant media sites including the The BBC, The Guardian New York Times, the Financial Times, CNN, Al Jazeera, Bloomberg, the Independent and the Evening Standard., were all among the websites that crashed. 

Fastly is a content delivery network (CDN), which maintains a network of servers that transfer content quickly from websites to users. It provides a layer of support between Internet companies and customers trying to access the various online platforms it services, when it goes down, access to those platforms can be blocked entirely. 

One of the reasons that the outage was so wide is that cloud computing service companies like Fastly are consolidating, leaving websites dependent on a shrinking number of providers, a critical vulnerability demonstrated recently by a similar large scale outage affecting Amazon Web Services in December 2020.

Fastly:      CNN:     Vox:     Computing:   Guardian:

You Might Also Read:

SANS Institute book: Practical Guide To Security In The AWS Cloud: