Monday, June 3, 2019

Google Cloud outage attributed to configuration change

The widescale outage experienced by Google Cloud Platform on 02-June-2019 was caused by a configuration change that was intended for a small number of servers in a single region but was mistakenly applied to a larger number of servers across several neighboring regions, according to a blog posting by Benjamin Treynor Sloss, VP, 24x7, Google.

Google says the disruption caused a 10% drop in global views of YouTube during the incident, while Google Cloud Storage measured a 30% reduction in traffic.

https://cloud.google.com/blog/topics/inside-google-cloud/an-update-on-sundays-service-disruption

In a follow-up report tracking the Google Cloud outage, ThousandEyes said its monitoring data indicate that connectivity and packet loss issues impacted Google network locations in the eastern US, including Ashburn, Atlanta and Chicago. High packet loss conditions radiated out to Google’s network edge.  However, most GCP regions were unaffected by the outage, including regions in the US as well as in Europe and elsewhere.

More: https://blog.thousandeyes.com/google-cloud-platform-outage-analysis/