Worldwide Internet Outage Caused By Single Configuration Error

The June 8th Internet outage which affected websites in dozens of countries across the Americas, Europe and Asia, as well as South Africa has been traced to a single configuration error at a little known but important IT infrastructure company, Fastly

Amazon, eBay and Boots were among retailers whose websites disappeared during the outage. Amazon and Boots alone report a combined £25bn in annual sales in the UK, meaning they would typically earn nearly £3m between them in the course of an average hour. Payment provider PayPal which processed $936bn of transactions in 2020, approximately $106m for every hour, was also temporarily unavailable.

Fastly said it had identified a service configuration that triggered disruptions across its servers and has now disabled that configuration.

The network run by Fastly had the outage for more than an hour and in most cases was occurring mid-morning London time. Investigation reveals the incident was caused by a cloud configuration fault affecting governments and businesses around the world in sectors ranging from media to online retail and telecoms. “We experienced a global outage due to an undiscovered software bug that surfaced on June 8 when it was triggered by a valid customer configuration change. We detected the disruption within one minute, then identified and isolated the cause, and disabled the configuration.... Within 49 minutes, 95% of our network was operating as normal”, said Fastly’s SVP  of Engineering & Infrastructure in a statement. 

The basic issue appears to be that Fastly had mitakenly taken down its own network with a bad software update, resulting in a blockage affecting millions of dollars in revenue for numerous corporations around the world.

All UK government websites using the gov.uk domain were swept up in the outage, which temporarily knocked out crucial services such as the online system by which taxpayers can file annual returns with HM Revenue and Customs. The British  Government said it was investigating reports that users were unable to book Covid-19 tests online as a “matter of urgency”.
Visitors to the official website of the White House were also greeted with a message likely to have been seen by hundreds of millions of internet users: “Error 503 service unavailable”.

According to analysis of data from Google’s AdSense platform, outlets lost approximately $300,000 in revenue during the period, as they missed out on clicks that would usually translate into payments from advertisers. The calculation, made by marketing agency connective3, covers lost revenue at affected advertising-reliant media sites including the The BBC, The Guardian New York Times, the Financial Times, CNN, Al Jazeera, Bloomberg, the Independent and the Evening Standard., were all among the websites that crashed. 

Fastly is a content delivery network (CDN), which maintains a network of servers that transfer content quickly from websites to users. It provides a layer of support between Internet companies and customers trying to access the various online platforms it services, when it goes down, access to those platforms can be blocked entirely. 

One of the reasons that the outage was so wide is that cloud computing service companies like Fastly are consolidating, leaving websites dependent on a shrinking number of providers, a critical vulnerability demonstrated recently by a similar large scale outage affecting Amazon Web Services in December 2020.

Fastly:      CNN:     Vox:     Computing:   Guardian:

You Might Also Read:

SANS Institute book: Practical Guide To Security In The AWS Cloud:

 

« Beware Of Credentials Phishing
AI Dominance On The Battlefield »

CyberSecurity Jobsite
Check Point

Directory of Suppliers

NordLayer

NordLayer

NordLayer is an adaptive network access security solution for modern businesses — from the world’s most trusted cybersecurity brand, Nord Security. 

Alvacomm

Alvacomm

Alvacomm offers holistic VIP cybersecurity services, providing comprehensive protection against cyber threats. Our solutions include risk assessment, threat detection, incident response.

Authentic8

Authentic8

Authentic8 transforms how organizations secure and control the use of the web with Silo, its patented cloud browser.

BackupVault

BackupVault

BackupVault is a leading provider of automatic cloud backup and critical data protection against ransomware, insider attacks and hackers for businesses and organisations worldwide.

Syxsense

Syxsense

Syxsense brings together endpoint management and security for greater efficiency and collaboration between IT management and security teams.

Hewlett Packard Enterprise (HPE)

Hewlett Packard Enterprise (HPE)

HPE is an information technology company focused on Enterprise networking, Services and Support.

evoila

evoila

evoila GmbH is one of the leading providers in consulting, analysis, implementation and management of cloud infrastructure.

International School of IT Security (ISITS)

International School of IT Security (ISITS)

The International School of IT Security (ISITS) is a leading provider of professional training in the field of IT Security.

Granite Partners

Granite Partners

Granite is a cloud service for the development of business risk management, cyber security and privacy and occupational safety and health.

NetFort

NetFort

NetFort provides software products to monitor activity on virtual and physical networks.

Innotec Security

Innotec Security

Innotec Security is a Spanish company specializing in cybersecurity-as-a-service, cyber resilience and cyber risk management.

Network Box

Network Box

Network Box is one of the world's leading Managed Security Service Providers.

Adlumin

Adlumin

Adlumin Inc. provides the enterprise-grade security operations platform and managed detection and response services that keep mid-market organizations secure.

Beyond Identity

Beyond Identity

Beyond Identity employs an elegantly simple concept, the personal certificate authority and self signed certificates, to replace passwords.

CentricalCyber

CentricalCyber

CentricalCyber is a cyber risk consultancy and NIST CSF specialist set up to help business leaders better understand and manage cyber risk.

Extreme Engineering Solutions (X-ES)

Extreme Engineering Solutions (X-ES)

Extreme Engineering Solutions is a leader in the design, manufacture, testing, and support of hardware and software solutions for the embedded computing market.

CampusGuard

CampusGuard

CampusGuard focuses on the cybersecurity and compliance needs of campus-based organizations including higher education, healthcare, and state and local government.

Avalor

Avalor

Avalor are on a mission to help security teams make faster, more accurate decisions by making sense of their data. With Avalor you can bring in data from anywhere, normalize it and analyze it.

Theos Cyber Solutions

Theos Cyber Solutions

Theos Cyber provides service-first cybersecurity solutions to digital businesses in Asia.

Velaspan

Velaspan

Velaspan design, deploy, and manage enterprise wireless networks and cybersecurity solutions for leading businesses and brands.

Netarx

Netarx

Netarx provide real-time deepfake detection for enterprise voice, video and email.