Incident simulations
Exit

INC-3164 · P1

In the middle of an order peak, the shop's AKS cluster adds no more nodes and pods stay Pending

Casablanca · Azure subscription “masterlinux-prod” francecentral · the MasterLinux shop · Tuesday 11:20

The MasterLinux shop (Raspberry Pi kits, books, training vouchers) runs on the AKS cluster aks-boutique-prod (resource group rg-boutique-prod, region francecentral). The user node pool hosts the commandes API; the cluster autoscaler is enabled on it (3 to 12 nodes) and the HPA adds pods when CPU goes above 70%.

This Tuesday, the “back to school” campaign was sent at 10:45. Since 11:00, traffic has tripled and the HPA asks for 18 replicas instead of 6: about ten pods stay Pending, payment response time is above 9 seconds and customer service is receiving the first complaints.

On the team side: nothing has been deployed to the cluster since Friday. The governance team, for its part, “cleaned up costs” on Monday at the end of the day.

You are the on-call engineer, on your workstation with Azure CLI and kubectl (context aks-boutique-prod). The nodes must arrive, fast, without breaking governance.

This simulation is part of a production simulation pack.

Unlock every production simulation

AWS, Azure, Google Cloud, on-premise and multi-cloud, from Beginner to Extreme: the real incidents of IT service companies and large IT organisations, available for a year.

Try the free demo simulation first

Already enrolled? Log in.