MasterLinux: the whole GCP production vanished overnight, cluster, database and disks, backups included

The whole MasterLinux platform (shop, enrolments, the school's lab rooms) runs on Google Cloud, in the masterlinux-prod project (europe-southwest1): the GKE cluster gke-ml-prod, the Cloud SQL PostgreSQL 16 database sql-ml-prod and the applications' persistent disks.

Environments are created by mlprov, the platform team's internal provisioning tool (server ml-provision-01): the CI calls its API, it creates the project and its resources from a template, then a periodic process, mlprov-reaper, deletes temporary environments that have reached their end date. The production infrastructure is also described in Terraform (state kept in a bucket of the masterlinux-provision project, which was not touched).

Since 02:00, external monitoring has been red. At 06:00, the on-call engineer finds the project empty. The shop is closed, the school's 2,300 students no longer have access to their labs, and enrolments for the November session close on Thursday.

The good news: for the past year, the operations team has been copying independent backups outside Google Cloud, to a locked S3 bucket (Object Lock) on AWS eu-west-3: hourly pg_dump exports of the database and daily Velero backups of the Kubernetes workloads.

Management puts you in charge of the crisis cell: understand what happened, stop it from continuing, then rebuild and restore production. You are on ml-bastion (gcloud, Terraform, kubectl, velero); you can hop to ml-provision-01 (account admin) and to ml-restauration-01 (account dba, AWS CLI with read access to the backup bucket, PostgreSQL tools).