Our upstream partner has resolved their issue and our monitoring shows that the connection has been stable and without problems since our last update. We will investigate mitigating actions and the root cause of this incident and will publish a postmortem once we have a complete overview from the upstream partner.
To summarize, we saw on average around 90 % of authorizations impacted during the time period 2024-09-12T03:30 - 2024-09-12T11:45 UTC. All impacted authorizations were responded with status code 50000 by our gateway API for transaction processing.
We sincerely regret the inconvenience this serious incident has caused!
Posted Sep 12, 2024 - 20:39 UTC
Monitoring
We have seen a significant increase in approval rates and the connectivity is now back to normal. Due to the severity of this issue and the previous sudden change in approval rate we are still actively monitoring this situation.
Posted Sep 12, 2024 - 12:04 UTC
Update
We are again seeing an improvement in approval rate starting 11:46 UTC and are continuing to monitor the situation as we previously have seen such improvement temporarily.
Posted Sep 12, 2024 - 11:51 UTC
Update
We are awaiting further communication from our provider. We are actively monitoring the situation. There is no ETA for when this issue is resolved. We will update as soon as we know more.
Posted Sep 12, 2024 - 10:47 UTC
Update
We saw an improvement in approval for a short period of time from around 08:47 UTC to 08:59 UTC. Unfortunately, this has worsened again, and we are back on approval rates below 15 %.
Posted Sep 12, 2024 - 09:06 UTC
Update
Unfortunately, we still see very high failure rates on authorizations. Our upstream provider is still trying to mitigate this critical incident. We are seeing examples where amounts are being reserved on cardholders' accounts despite the authorization failing. Therefore we recommend that subsequent-in-series recurring and unscheduled (e.g. subscriptions) are not attempted until after this incident is resolved. Furthermore, we are exploring options to potentially release these reservations.
Posted Sep 12, 2024 - 08:41 UTC
Identified
We have identified that the issue is with our upstream provider. We have attempted mitigating actions but unfortunately with very limited impact. We are still seeing very low approval rates that fluctuates approximately from 0 % to 15 %. Our upstream provider is trying to resolve the connectivity issue.
Posted Sep 12, 2024 - 06:34 UTC
Update
Our upstream provider has confirmed and is aware of a connectivity issue. Currently almost all authorizations, voids and Mastercard credits are impacted.
Posted Sep 12, 2024 - 05:05 UTC
Investigating
Since 2024-09-12T03.39Z we have observed connectivity issues towards an upstream provider. We have reached out and are investigating with the provider.
Posted Sep 12, 2024 - 04:42 UTC
This incident affected: Payment Processing APIs (Gateway API (gateway.clearhaus.com)).