Incident simulations
Exit

INC-3015 · P2

Booking peak: new pods stay stuck at startup

Tangier · AWS account eu-west-1 · Rihla Transport · 19:10

Rihla Transport sells coach tickets online between Tangier, Tetouan, Fez and Casablanca. Tonight, bookings open for the school holidays: traffic is multiplied by 4.

The reservation-api API runs on EKS (cluster eks-rihla-prod, Kubernetes 1.30, node group ng-general on m5.xlarge, 6 nodes in two private subnets). The HPA scaled the deployment from 12 to 40 replicas in a few minutes, and the Cluster Autoscaler has already added 2 nodes.

Yet more than half of the new pods do not start and response times exceed 6 s. The team lead suggests “adding even more nodes”.

You have kubectl (context eks-rihla-prod, administrator rights) and the AWS CLI v2 (region eu-west-1).

This simulation is reserved for students enrolled in a course.

Already enrolled? Log in.