Zellige Market: 45% of customers see “503” at checkout, but every dashboard is green

Zellige Market (handicraft marketplace, 1.2 million accounts) serves www.zellige-market.ma from two GKE clusters of the same fleet:

• gke-ew1 (europe-west1, Belgium) — it is also the config cluster of the multi-cluster Gateway boutique-gw;
• gke-mad (europe-southwest1, Madrid) — the region closest to Morocco and Spain.

The global load balancer sends each customer to the closest healthy region. Each region has its own Memorystore session and cart cache; the boutique-api API exposes /healthz/live (the process is alive) and /healthz/ready (Redis and the database answer).

Since 10:40, support has been swamped with “503 error at checkout”, almost all from Morocco and Spain. At 11:02, Karim, from the platform team, “failed Madrid over” by setting its backends' capacity to 0 in the console: it worked… for three minutes. The noon flash sale has been announced to 400,000 subscribers.

You are on the on-call workstation (gcloud, project zm-prod-boutique). The clusters are private: they are driven from bastion-ew1 and bastion-mad (IAP), where kubectl is configured.