The Cause Of Amazon’s Cloud Outage

Amazon Web Services (AWS) has explained the cause of their outage, which took down thousands of third-party online services for hours. Amazon say that, “the new capacity had caused all of the servers in the fleet to exceed the maximum number of threads allowed by an operating system configuration... As this limit was being exceeded, cache construction was failing to complete and front-end servers were ending up with useless shard-maps that left them unable to route requests to back-end clusters.” 

While dozens of services were affected, AWS says the outage occurred in its Northern Virginia, US-East-1, region. It happened after a "small addition of capacity" to its front-end fleet of Kinesis servers. 

Amazon Kinesis enables real-time processing of streaming data. In addition to its direct use by customers, Kinesis is used by several other AWS services and these services also saw impact during the shutdown. Kinesis is used by developers, as well as other AWS services like CloudWatch and Cognito authentication, to capture data and video streams and run them through AWS machine-learning platforms.  

The Kinesis service's front-end handles authentication, throttling, and distributes workloads to its back-end "workhorse" cluster via a database mechanism called sharding.  

Amazon’s additions to capacity triggered the outage but wasn't the root cause of it. AWS was adding capacity for an hour after 2:44am PST, and after that all the servers in Kinesis front-end fleet began to exceed the maximum number of threads allowed by its current operating system configuration.  The first alarm was triggered at 5:15am PST and AWS engineers spent the next five hours trying to resolve the issue. Kinesis was fully restored at 10:23pm PST. 

Amazon explains how the front-end servers distribute data across its Kinesis back-end: "Each server in the front-end fleet maintains a cache of information, including membership details and shard ownership for the back-end clusters, called a shard-map." According to AWS, that information is obtained through calls to a micro service vending the membership information, retrieval of configuration information from DynamoDB and continuous processing of messages from other Kinesis front-end servers. For Kinesis communication, each front-end server creates operating system threads for each of the other servers in the front-end fleet. Upon any addition of capacity, the servers that are already operating members of the fleet will learn of new servers joining and establish the appropriate threads. It takes up to an hour for any existing front-end fleet member to learn of new participants." 

As the number of threads exceeded the OS configuration, the front-end servers ended up with "useless shard-maps" and were unable to route requests to Kinesis back-end clusters. AWS had already rolled back the additional capacity that triggered the event but had reservations about boosting the thread limit in case it delayed the recovery.  

As a first step, AWS has moved to larger CPU and memory servers, as well as reduced the total number of servers and threads required by each server to communicate across the fleet.  It's also testing an increase in thread count limits in its operating system configuration and working to "radically improve the cold-start time for the front-end fleet".  

CloudWatch and other large AWS services will move to a separate, partitioned front-end fleet. AWS is also working on a broader project to isolate failures in one service from affecting other services.  

AWS has also acknowledged the delays in updating its Service Health Dashboard during the incident, but says that was because the tool its support engineers use to update the public dashboard was affected by the outage. During that time, it was updating customers via the Personal Health Dashboard.   Amazon has apologised for the impact this event caused its customers.

Amazon:        Down Detector:       ZDNet

You Might Also Read:

The Risks &  Benefits Of Cloud Security:

 

« We Live In A Transient Internet
Orca Security Wants To Streamline Cloud Computing »

CyberSecurity Jobsite
Perimeter 81

Directory of Suppliers

Cyber Security Supplier Directory

Cyber Security Supplier Directory

Our Supplier Directory lists 6,000+ specialist cyber security service providers in 128 countries worldwide. IS YOUR ORGANISATION LISTED?

Authentic8

Authentic8

Authentic8 transforms how organizations secure and control the use of the web with Silo, its patented cloud browser.

Resecurity, Inc.

Resecurity, Inc.

Resecurity is a cybersecurity company that delivers a unified platform for endpoint protection, risk management, and cyber threat intelligence.

NordLayer

NordLayer

NordLayer is an adaptive network access security solution for modern businesses — from the world’s most trusted cybersecurity brand, Nord Security. 

Clayden Law

Clayden Law

Clayden Law advise global businesses that buy and sell technology products and services. We are experts in information technology, data privacy and cybersecurity law.

CIO

CIO

CIO provides technology and business leaders with insight and analysis on information technology trends

Libraesva

Libraesva

Libraesva specialize in Email Security. From Email Security, Phishing Awareness and Email Archiver. We can assist you with any email issues you may have.

Swiss Re

Swiss Re

Swiss Re Group is a leading wholesale provider of reinsurance, insurance and other insurance-based forms of risk transfer including cyber risk.

SynerComm

SynerComm

SynerComm is an IT solution provider specializing in network and security infrastructure, enterprise mobility, remote access, wireless solutions, audit, pentesting and information assurance.

DFI

DFI

DFI is a global leading provider of high-performance computing technology across multiple embedded industries.

Smart Protection

Smart Protection

Smart Protection are experts in brand and trademark protection - we fight against counterfeits and unauthorized usages of brands with machine learning technology.

AuthLite

AuthLite

With AuthLite, you can keep using all your existing software, with added two-factor authentication security placed exactly where you need it.

CYRISMA

CYRISMA

CYRISMA is a revolutionary cybersecurity platform that helps organizations manage risk without the usual headaches associated with enterprise cybersecurity tools.

Thistle Technologies

Thistle Technologies

Thistle Technologies is building tools that help connected device manufacturers build security resiliency into devices.

Venari Security

Venari Security

Venari is an award-winning cybersecurity SaaS provider that has developed an ETA (Encrypted Traffic Analysis) platform which fundamentally changes the way encrypted traffic is analysed.

Readynez

Readynez

Readynez is the digital skills concierge service that helps you ensure your workforce has the tech skills and resources needed to stay ahead of the digital curve.

WillJam Ventures

WillJam Ventures

WillJam Ventures are a private equity firm focused on investing in world-class cybersecurity companies that will become the next generation of leaders in protecting the world’s digital assets.

Insurica

Insurica

INSURICA is a full-service insurance agency built upon a tradition of integrity, industry leadership, and excellence.

Synagex

Synagex

Synagex Modern IT is a simple IT and cybersecurity solution for businesses.

Disecto Technologies

Disecto Technologies

At Disecto, we provide SaaS based Data Discovery, Classification and a remediation solution for data privacy compliance.

Quantum Dice

Quantum Dice

Quantum Dice is an award-winning venture-backed spinout from Oxford University’s world-renowned quantum optics laboratory.