Concert ticket sales open: the API returns bursts of 500 errors

Darb Tickets, an online ticketing company in Casablanca, opened sales at 10:00 for the big end-of-year concert at the sports complex. Its API api-billets runs on Cloud Run (europe-west1) and relies on a Cloud SQL PostgreSQL 15 database pg-darb-prod (machine type db-custom-2-4096).

On Friday, anticipating the rush, Ilyas (developer) raised the Cloud Run maximum number of instances from 10 to 40. The tests the day before, under low traffic, were perfect.

Since 10:03, the waiting queue has been filling up, payments are failing and social media is ablaze. The DBA thinks a bigger server is needed; Ilyas thinks even more instances are needed.

You are on your workstation, with gcloud configured on darbtickets-prod-2741. An access token can be obtained with gcloud auth print-access-token.