INC-3164 · P1
In the middle of an order peak, the shop's AKS cluster adds no more nodes and pods stay Pending
Casablanca · Azure subscription “masterlinux-prod” francecentral · the MasterLinux shop · Tuesday 11:20
The MasterLinux shop (Raspberry Pi kits, books, training vouchers) runs on the AKS cluster
aks-boutique-prod (resource group rg-boutique-prod, region francecentral). The user node pool hosts the
commandes API; the cluster autoscaler is enabled on it (3 to 12 nodes) and the HPA adds pods when CPU
goes above 70%.
This Tuesday, the “back to school” campaign was sent at 10:45. Since 11:00, traffic has tripled and the HPA asks for 18 replicas instead of 6: about ten pods stay Pending, payment response time is above 9 seconds and customer service is receiving the first complaints.
On the team side: nothing has been deployed to the cluster since Friday. The governance team, for its part, “cleaned up costs” on Monday at the end of the day.
You are the on-call engineer, on your workstation with Azure CLI and kubectl (context
aks-boutique-prod). The nodes must arrive, fast, without breaking governance.
This simulation is part of a production simulation pack.
Unlock every production simulation
AWS, Azure, Google Cloud, on-premise and multi-cloud, from Beginner to Extreme: the real incidents of IT service companies and large IT organisations, available for a year.
Try the free demo simulation first
Already enrolled? Log in.