May 15th Outage: Post Mortem

Thursday, May 15th, at approximately 3:30pm Mountain Time, our customers experienced 45 minutes of unexpected downtime. The root cause of the problem was a misconfigured load balancer. A planned maintenance event increased the traffic load for a short period of time causing the misconfigured load balancer to behave poorly and eventually fail. This resulted in a cascading event as each load balancer crashed when traffic was diverted to the next available load balancer. Read more...