Tawsil Food: Friday night order peak, the app slows down and pods stay Pending

Tawsil Food, a meal delivery app for Rabat, Salé and Kenitra, runs its order API on the AKS cluster aks-tawsil-prod (resource group rg-tawsil-prod, region francecentral). The apps pool (Standard_D8s_v5) has the cluster autoscaler enabled, from 3 to 20 nodes.

Friday night, the national team is playing: orders have tripled since 20:00. The HPA has scaled api-commandes from 12 to 30 replicas, but 11 pods have stayed Pending for 25 minutes. Response time exceeds 4 s, and couriers receive their deliveries late.

On Tuesday, the data team created, in the same subscription, a compute cluster of 4 D8s_v5 VMs for its forecasting models. You are on call, with the Azure CLI and kubectl configured.