[Stage] Degraded performance

Incident Report for WePay

Resolved

This incident has been resolved.

Posted Aug 18, 2016 - 11:59 PDT

Investigating

We are investigating a service degradation on the *Stage* website/API that occurred overnight. One of our Memcache servers crashed around 8pm Pacific on Wednesday evening and all requests were routed to MySQL directly, causing latency across all layers of the *Stage* service. We are presently investigating the impact, and reviewing our monitoring and alerting systems to understand why our Ops team missed this issue overnight.

Posted Aug 18, 2016 - 08:45 PDT

This incident affected: API and Website.