The whole MasterLinux platform is offline: the production GKE cluster no longer exists

The whole platform of the MasterLinux school (API, course site, labs, payments) runs on the regional GKE cluster ml-gke-prod in the masterlinux-prod project (region europe-southwest1). The infrastructure is described in Terraform (repository infra-gke, state in the gs://masterlinux-tfstate bucket, one workspace per environment: preprod and prod). Cluster backups are taken every night at 03:00 by Backup for GKE (separate Terraform stack plateforme-sauvegardes).

This morning, a developer was preparing to add a node pool in pre-production from his workstation. At 10:12, monitoring turned red: no service responds any more, and the 2,300 connected students lost their lab. In the console, the ml-gke-prod cluster no longer appears.

In the incident channel, some talk about an attack (“someone deleted the cluster”), others about a Google outage. The technical lead suggests “running terraform apply again right now to recreate everything”.

You are the on-call DevOps engineer, on ml-bastion (gcloud authenticated with an ops account, a copy of the repository in ~/infra-gke). The developer sent you his terminal log in ~/incident/INC-3266/.