INC-3262 · P1
Enrolment day: one request in two fails with a 429 error
Rabat · project masterlinux-prod · europe-southwest1 · MasterLinux school · Monday 08:20
The MasterLinux school opens enrolment for the winter session today at 08:00: 2,400 places,
allocated on a first-come, first-served basis. The Cloud Run service inscriptions (region europe-southwest1)
receives the form, generates the signed PDF pre-enrolment certificate, then stores the application
in Cloud SQL PostgreSQL (ml-sql-inscriptions).
Since 08:01, the support team has been receiving screenshots: “Error 429 – try again later”. On social media, applicants accuse the school of favouring those with “connections”. The director of studies wants an answer before 08:45.
Nothing was deployed this morning. Last week, the developer fixed a PDF signing bug (“certificates mixed up between two applicants”). The technical lead suspects “a Google quota reached” or “the database saturating”.
You are the on-call engineer, on ml-poste-01, with gcloud configured on masterlinux-prod and the region
europe-southwest1, and psql access to the database.
This simulation is part of a production simulation pack.
Unlock every production simulation
AWS, Azure, Google Cloud, on-premise and multi-cloud, from Beginner to Extreme: the real incidents of IT service companies and large IT organisations, available for a year.
Try the free demo simulation first
Already enrolled? Log in.