Incident simulations
Exit

INC-3262 · P1

Enrolment day: one request in two fails with a 429 error

Rabat · project masterlinux-prod · europe-southwest1 · MasterLinux school · Monday 08:20

The MasterLinux school opens enrolment for the winter session today at 08:00: 2,400 places, allocated on a first-come, first-served basis. The Cloud Run service inscriptions (region europe-southwest1) receives the form, generates the signed PDF pre-enrolment certificate, then stores the application in Cloud SQL PostgreSQL (ml-sql-inscriptions).

Since 08:01, the support team has been receiving screenshots: “Error 429 – try again later”. On social media, applicants accuse the school of favouring those with “connections”. The director of studies wants an answer before 08:45.

Nothing was deployed this morning. Last week, the developer fixed a PDF signing bug (“certificates mixed up between two applicants”). The technical lead suspects “a Google quota reached” or “the database saturating”.

You are the on-call engineer, on ml-poste-01, with gcloud configured on masterlinux-prod and the region europe-southwest1, and psql access to the database.

This simulation is part of a production simulation pack.

Unlock every production simulation

AWS, Azure, Google Cloud, on-premise and multi-cloud, from Beginner to Extreme: the real incidents of IT service companies and large IT organisations, available for a year.

Try the free demo simulation first

Already enrolled? Log in.