A configuration change caused Google Pay transactions to fail. The change was made to clean up configuration inconsistencies between environments. Google Pay had an implicit dependency on one of these configurations. Before moving traffic to a new deployment, we always run automated test transactions. However, Google Pay is not tested automatically but only per need, as it is a convoluted payment method and difficult to automate. Since this deployment did not include any changes to Google Pay and was not expected to affect it, no additional effort was made to test Google Pay.
Google Pay has significantly less traffic than other payment methods so it hits below our alerting threshold, and because of this we were slow to notice the issue.
We have improved our alerting to catch failures affecting a small share of traffic. To avoid a significant number of false positives, we’ve done thorough analysis and backtesting to create advanced alerts that check to confirm that errors don't only happen on individual merchant accounts or for individual issuers. This allows us to be alerted per payment method — even for smaller ones like Google Pay — but also per transaction type — even for smaller ones like credits. With these new alerts in place we would have been alerted to the outage within, at most, 15 minutes.