INC-3169 · P2
Last night's upgrade of the production AKS cluster failed: cluster in “Failed” state, nodes on two versions
Rabat · Azure subscription masterlinux-prod (francecentral) · MasterLinux school · Wednesday 08:10
The MasterLinux school runs its platform (enrolments, lab rooms, course payments) on the
AKS cluster aks-ml-prod (resource group rg-aks-prod, francecentral), with two node pools:
systempool (3 nodes) and userpool (4 nodes) for the applications.
Version 1.29 is reaching end of support: yesterday at 23:00, during the maintenance window, the platform team
started the upgrade to 1.30.6 (az aks upgrade). This morning, the command ended with an error,
the portal shows the cluster as Failed, and monitoring reports a node in SchedulingDisabled for
six hours.
The sites respond, but the cluster is in a shaky state: no other operation (scaling, new pool) is possible while it is failed, and enrolments for the new term start on Monday. The platform manager asks you to find what blocked the upgrade and to finish it cleanly, without interrupting payments.
You are on ml-bastion, with the Azure CLI and a kubectl already connected to the cluster.
This simulation is part of a production simulation pack.
Unlock every production simulation
AWS, Azure, Google Cloud, on-premise and multi-cloud, from Beginner to Extreme: the real incidents of IT service companies and large IT organisations, available for a year.
Try the free demo simulation first
Already enrolled? Log in.