Cloud Outage Reports

The detailed summaries of outages from cloud vendors are comprehensive and the response to each highlights many lessons in how to build robust distributed systems. For outages that significantly affected Netflix, the Netflix techblog report gives insight into how to effectively build reliable services on top of AWS. I've included some Google and Azure outages here because they illustrate different failure modes that should be taken into account. Recent AWS and Azure outage reports have far more detail than Google outage reports.

I plan to collect reports here over time, and welcome links to other write-ups of outages and how to survive them. My naming convention is {vendor} {primary scope} {cause}. The scope may be global, a specific region, or a zone in the region. In some cases there are secondary impacts with a wider scope but shorter duration such as regional control planes becoming unavailable for a short time during a zone outage.

This post was written while researching my AWS Re:Invent talk.
Slides: http://www.slideshare.net/AmazonWebServices/arc203-netflixha
Video: http://www.youtube.com/watch?v=dekV3Oq7pH8

---

November 18th, 2014 - Azure Global Storage Outage

Microsoft Reports

http://azure.microsoft.com/blog/2014/11/19/update-on-azure-storage-service-interruption/

http://azure.microsoft.com/blog/2014/12/17/final-root-cause-analysis-and-improvement-areas-nov-18-azure-storage-service-interruption/

---

January 10th, 2014 - Dropbox Global Outage

Dropbox Report

https://tech.dropbox.com/2014/01/outage-post-mortem/

---

April 20th, 2013 - Google Global API Outage

Google Report

http://googledevelopers.blogspot.com/2013/05/google-api-infrastructure-outage.html

---

February 22nd, 2013 - Azure Global Outage Cert Expiry

Azure Report

http://blogs.msdn.com/b/windowsazure/archive/2013/03/01/details-of-the-february-22nd-2013-windows-azure-storage-disruption.aspx

---

December 24th, 2012 - AWS US-East Partial Regional ELB State Overwritten

AWS Service Event Report

http://aws.amazon.com/message/680587/

Netflix Techblog Report

http://techblog.netflix.com/2012/12/a-closer-look-at-christmas-eve-outage.html

---

October 26th, 2012 - Google AppEngine Network Router Overload

Google Outage Report

http://googleappengine.blogspot.com/2012/10/about-todays-app-engine-outage.html

---

October 22, 2012 - AWS US-East Zone EBS Data Collector Bug

AWS Outage Report

http://aws.amazon.com/message/680342/

Netflix Techblog Report

http://techblog.netflix.com/2012/10/post-mortem-of-october-222012-aws.html

---

June 29th 2012 - AWS US-East Zone Power Outage During Storm 

AWS Outage Report

http://aws.amazon.com/message/67457/

Netflix Techblog Report

http://techblog.netflix.com/2012/07/lessons-netflix-learned-from-aws-storm.html

---

June 13th, 2012 - AWS US-East SimpleDB Region Outage

AWS Outage Report

http://aws.amazon.com/message/65649/

---

February 29th, 2012 - Microsoft Azure Global Leap-Year Outage

Azure Outage Report

http://blogs.msdn.com/b/windowsazure/archive/2012/03/09/summary-of-windows-azure-service-disruption-on-feb-29th-2012.aspx

---

August 17th, 2011 - AWS EU-West Zone Power Outage

AWS Outage Report

http://aws.amazon.com/message/2329B7/

---

April 2011 - AWS US-East Zone EBS Outage

AWS Outage Report

http://aws.amazon.com/message/65648/

Netflix Techblog Report

http://techblog.netflix.com/2011/04/lessons-netflix-learned-from-aws-outage.html

---

February 24th, 2010 - Google App Engine Power Outage

Google Forum Report

https://groups.google.com/forum/#!topic/google-appengine/p2QKJ0OSLc8

---

July 20th, 2008 - AWS Global S3 Gossip Protocol Corruption

AWS Outage Report

http://status.aws.amazon.com/s3-20080720.html