Rihla Voyages: booking payments fail with 503 and the weekend's GCP bill exceeds a whole year's

Rihla Voyages is an online travel agency from Marrakech (riads, desert tours, flights to Mecca for the Umrah). On the GCP project rihla-prod-2291 (region europe-west1):

• the Cloud Run service api-paiements takes the payments for bookings on www.rihla.ma;
• each booking publishes an event to the Pub/Sub topic reservations-evenements;
• the Cloud Function (2nd gen) fn-enrichir-reservations, triggered by Eventarc on that topic, adds the weather, the exchange rate and the airport transfer, and writes to Cloud SQL sql-rihla-prod.

Saturday 17/10 at 21:38, Yassir Ouazzani deployed a new version of the function (“publish the enriched event for the notification service”). On Sunday, the site was slow at times.

This Monday since 06:50, api-paiements has been returning 503s and database errors right in the booking peak for the school holidays. At 07:15, the bank called the manager, Latifa Benjelloun: a Google Cloud charge of more than 38,000 USD for the weekend (the usual monthly bill is 1,900 USD). The accountant wants to “cut the project's billing right now”. A developer wants to “delete that overflowing Pub/Sub topic”.

You are the on-call SRE engineer (Kenza), on poste-sre (gcloud, Editor role on the project). You can open an IAP session on bastion-sql (psql to Cloud SQL) and on runner-build (functions repository, deployments), and open Cloud Shell with the billing account (read access to the BigQuery export). Stop the bleeding without losing a single booking, then put the service back in order.