API error spike & production login issues
Incident Report for WePay
Postmortem

On Monday, December 11th, at 9:01am PST, WePay experienced an outage which impacted all incoming API and iFrame traffic. The incident lasted until resolved at 9:53am PST. The issue was the result of a database reboot by our infrastructure provider. A database replication glitch caused by the reboot impacted our ability to recover automatically and was the reason for the extended downtime.

We have fixed the replication glitch and are now operating at full capacity. We understand the root cause for the replication glitch and we are working on the ultimate fix.

We are performing a full assessment of all database replication to ensure it is working properly following the event and to ensure that no additional site events will occur.

Posted Dec 11, 2017 - 14:54 PST

Resolved
We have continued to monitor the behavior of our services and are confident that there is no further disruption. This issue is resolved.
Posted Dec 11, 2017 - 11:02 PST
Monitoring
We have introduced a resolution for the errors being experienced. We are currently seeing API traffic flowing normally as of 9:57am PST. We will continue to closely monitor for any further disruption.
Posted Dec 11, 2017 - 10:00 PST
Identified
We have identified the issue and are in the process of working to resolve as quickly as possible. We apologize for any impact you are currently experiencing. We will continue to provide updates on the incident until resolved.
Posted Dec 11, 2017 - 09:36 PST
Investigating
We have identified a spike in API errors, as well as difficulty with users logging into their WePay account. We are currently experiencing a momentary downtime and are actively investigating. We will provide an update shortly as further information becomes available.
Posted Dec 11, 2017 - 09:16 PST
This incident affected: API, Website, US Processing, Canadian Processing, and Settlements/Withdrawals.