Incident simulations
Exit

INC-3261 · P2

Monday morning, nobody on the operations team can SSH into the web server any more

Fez · GCP project masterlinux-prod (europe-southwest1-a) · MasterLinux Fez office · Monday 08:45

The MasterLinux Fez office manages the web server ml-web-01 (Debian 12, Compute Engine, europe-southwest1-a, public IP 34.175.88.21) that hosts the training website. It has always been the case that the operations team connects with its SSH keys, stored in the instance metadata (ssh-keys).

On Friday evening, the security team applied a new policy to the masterlinux-prod project. This morning, it is impossible to connect: an urgent security fix must be applied to nginx before noon, and the site must be restarted during the morning.

The site responds normally. You are on your workstation ml-poste-01, with the gcloud CLI signed in as ops@masterlinux.lab (member of the grp-ops@masterlinux.lab group, with IAM administration rights delegated on the project for on-call duty).

This simulation is part of a production simulation pack.

Unlock every production simulation

AWS, Azure, Google Cloud, on-premise and multi-cloud, from Beginner to Extreme: the real incidents of IT service companies and large IT organisations, available for a year.

Try the free demo simulation first

Already enrolled? Log in.