Incident simulations
Exit

INC-3169 · P2

Last night's upgrade of the production AKS cluster failed: cluster in “Failed” state, nodes on two versions

Rabat · Azure subscription masterlinux-prod (francecentral) · MasterLinux school · Wednesday 08:10

The MasterLinux school runs its platform (enrolments, lab rooms, course payments) on the AKS cluster aks-ml-prod (resource group rg-aks-prod, francecentral), with two node pools: systempool (3 nodes) and userpool (4 nodes) for the applications.

Version 1.29 is reaching end of support: yesterday at 23:00, during the maintenance window, the platform team started the upgrade to 1.30.6 (az aks upgrade). This morning, the command ended with an error, the portal shows the cluster as Failed, and monitoring reports a node in SchedulingDisabled for six hours.

The sites respond, but the cluster is in a shaky state: no other operation (scaling, new pool) is possible while it is failed, and enrolments for the new term start on Monday. The platform manager asks you to find what blocked the upgrade and to finish it cleanly, without interrupting payments.

You are on ml-bastion, with the Azure CLI and a kubectl already connected to the cluster.

This simulation is part of a production simulation pack.

Unlock every production simulation

AWS, Azure, Google Cloud, on-premise and multi-cloud, from Beginner to Extreme: the real incidents of IT service companies and large IT organisations, available for a year.

Try the free demo simulation first

Already enrolled? Log in.