We are sorry for the transactions your customers could not complete during the time span our service was not fully functional. Our upstream service provider has determined that an unfortunate chain of events led to the incident. While redirecting traffic from their primary to their secondary site in preparation for maintenance, they were not aware that one or more network ports on the secondary site were out of order, causing traffic to be lost. Under normal circumstances they would have been alerted about the condition, however, alerting for the secondary site were experiencing various issues leading up to the incident. They are aware that the chain of events is completely unacceptable and have introduced additional checks and balances to prevent it from occurring in the future.

As noted in the incident updates, we discovered a way to improve transparency of such incidents, namely to display failing transactions in our dashboard. This improvement is currently being implemented and is intended to be in place within a few weeks. The investigation of this incident identified additional, potential improvements (regarding the uncertainty about the transaction state of a failed transaction) which even touches upon extending our API with “idempotency”. If you are a partner and have an opinion or request in regards to this, please feel free to reach out to us, perhaps even through an issue on our public documentation.

Sincerely,
your operations team