Wednesday, December 22, 2021

AWS hit by loss of power within a data center in US-EAST-1 Region

 At 4:35 AM PST on December 22, Amazon Web Services reported increased EC2 launch failures and networking connectivity issues for some instances in its US-EAST-1 Region. Shortly after, AWS confirmed a loss of power within a single data center within a single Availability Zone (USE1-AZ4) in its US-EAST-1 Region. 

The outage impacted availability and connectivity to EC2 instances that are part of the affected data center within the affected Availability Zone. AWS also reported elevated RunInstance API error rates for launches within the affected Availability Zone. 

By 5:39 AM PST, AWS restored power to all instances and network devices within the affected data center. Network connectivity within the affected Availability Zone returned to normal levels.

As of 9:28 AM PST, AWS was still working to restore connectivity issues between some remaining EC2 instances and EBS volumes in the affected data center. AWS also noted increased error rates for some customers using Directory Services AD Connector or Managed AD with Amazon SSO in US-EAST-1 Region.

AWS attributes outage to surge from automated scaling of internal network

At 7:30 AM PST on December 7th, 2021, an automated activity at AWS Northern Virginia (US-EAST-1) Region that is used to scale capacity of services hosted in the main AWS network triggered an unexpected behavior from a large number of clients inside the internal network. The unexpected behavior resulted in a large surge of connection activity that overwhelmed the networking devices between the internal network and the main AWS network, resulting in...

AWS hit by outage in US-EAST-1 region

Amazon Web Services (AWS) experienced an outage started on Dec 7, 2021 at approximately 15:40 UTC impacting various regions worldwide.The AWS status page reported API and console issues in the US-EAST-1 Region, which is used to host the AWS global console landing page. The trouble included elevated error rates for EC2 APIs in the US-EAST-1 region. At 12:34 EST, the company said the root cause was "impairment of several network devices," confirmed...