Black Friday at Bazar Zellige: the site returns 503 as soon as the sale opens, although Auto Scaling has quadrupled the servers

Bazar Zellige sells decoration, handicrafts and household appliances online (www.bazarzellige.ma, 1.2 million customer accounts). Tonight is Black Friday: the sale opens at 21:00 (Casablanca time, 20:00 UTC) with a TV and SMS campaign. The company makes a quarter of its yearly revenue on it.

The architecture is classic: an ALB alb-bazar-prod, an Auto Scaling group asg-bazar-web (Node.js instances, scaling on the number of requests per target), an ElastiCache Redis cache cache-bazar-prod in front of the RDS PostgreSQL database db-bazar-prod. To absorb the peak, the team planned a cache warm-up by the scheduled task rechauffe-catalogue on batch-01.

At 21:00:03, the first 503 errors appeared; by 21:09, Auto Scaling had launched 18 more instances, with no improvement at all. The sales director, Mounir Squalli, calls every two minutes. The DBA, Ilham, suggests “moving the database to an instance twice as big”; a developer wants to “flush Redis to start clean”.

You are Anas, the on-call cloud engineer, on poste-cloud (AWS CLI v2, region eu-west-3, SSM Session Manager access to the instances, psql client through the VPN). Sample application instance: i-0c3f8a21d94e7b605; batch server: batch-01 (i-07b1e9d4c2a5f3816).