Incident simulations
Exit

INC-3361 · P1

No kubectl command works any more on the Rabat Kubernetes cluster, deployments are blocked

Rabat · datacenter of the MasterLinux school · ml-k8s-cp-01 · Tuesday 08:14

The MasterLinux school runs its internal applications (enrolments, automatic lab grading, trainers' dashboard) on a Kubernetes cluster installed with kubeadm in its Rabat datacenter: one control plane node ml-k8s-cp-01 and three nodes ml-k8s-wk-01 to ml-k8s-wk-03. The cluster was installed last year by a contractor, and nobody has touched it since: “it runs by itself”.

Since yesterday morning, every release fails in the CI pipeline, and this morning no administrator can run a single kubectl command. Pods already started are still running, but an urgent fix for the lab grading application is waiting to be deployed.

The contractor mentions “a network problem towards port 6443”, a colleague suspects etcd or a clock drift.

You are the on-call Kubernetes administrator, connected over SSH to ml-k8s-cp-01 with the admin account (sudo).

This simulation is part of a production simulation pack.

Unlock every production simulation

AWS, Azure, Google Cloud, on-premise and multi-cloud, from Beginner to Extreme: the real incidents of IT service companies and large IT organisations, available for a year.

Try the free demo simulation first

Already enrolled? Log in.