INC-3361 · P1
No kubectl command works any more on the Rabat Kubernetes cluster, deployments are blocked
Rabat · datacenter of the MasterLinux school · ml-k8s-cp-01 · Tuesday 08:14
The MasterLinux school runs its internal applications (enrolments, automatic lab grading,
trainers' dashboard) on a Kubernetes cluster installed with kubeadm in its Rabat
datacenter: one control plane node ml-k8s-cp-01 and three nodes ml-k8s-wk-01 to ml-k8s-wk-03. The cluster
was installed last year by a contractor, and nobody has touched it since: “it runs by itself”.
Since yesterday morning, every release fails in the CI pipeline, and this morning no administrator can
run a single kubectl command. Pods already started are still running, but an urgent fix for
the lab grading application is waiting to be deployed.
The contractor mentions “a network problem towards port 6443”, a colleague suspects etcd or a clock drift.
You are the on-call Kubernetes administrator, connected over SSH to ml-k8s-cp-01 with the
admin account (sudo).
This simulation is part of a production simulation pack.
Unlock every production simulation
AWS, Azure, Google Cloud, on-premise and multi-cloud, from Beginner to Extreme: the real incidents of IT service companies and large IT organisations, available for a year.
Try the free demo simulation first
Already enrolled? Log in.