Every MasterLinux API answers 503, in all three regions at once

Every API of the MasterLinux school (enrolment, payments, course platform, mobile app, branch kiosks) goes through in-house API gateways (ml-passerelle-01 to -03, one per region). Before each request, the gateway asks the internal service ml-quotas: “is this client allowed, and within which limit?”.

ml-quotas (Go) runs on GKE in three regions (europe-west1, europe-west9, europe-southwest1). It loads its policies from a multi-region Cloud Spanner database (eur3, leader in europe-west1): any row written there is visible everywhere within seconds. Version 2.14.0 (“per-agency quotas”), released two weeks ago, has run without incident since.

Since 07:41, every API answers 503. 1,200 students taking an online exam are blocked, the branches' payments are refused. The network team checked the VPN and the interconnects: nothing. The CTO wants to know whether to “fail over to another region”; a developer suggests “going back to 2.13 right now”.

You are the on-call SRE engineer, on ml-bastion: gcloud, kubectl (contexts ew1, ew9, esw1), Ansible (~/ansible) and the operations runbooks (~/runbooks), with SSH access to the gateways.