INC-3168 · P2
Back-to-school sale: the shop API returns 500 errors with “RedisTimeoutException” as soon as traffic rises
Casablanca · Azure subscription masterlinux-prod (francecentral) · MasterLinux shop · Tuesday 11:20
The MasterLinux shop (books, Raspberry Pi kits, training vouchers) launches its back-to-school sale this morning.
Its API api.boutique.masterlinux.lab is a .NET 10 application hosted on Azure App Service (Linux,
plan asp-ml-boutique-p2v3, 3 instances). Carts, sessions and regional prices are read from
Azure Cache for Redis redis-ml-boutique (Standard C2 tier) with the StackExchange.Redis library.
Since 10:00, traffic has quadrupled and about 16% of requests fail with 500 after
5 seconds: the logs are full of StackExchange.Redis.RedisTimeoutException. Last Friday, the development
team shipped a new regional pricing feature.
In the incident channel, two proposals are already going around: move the cache to Premium P2 (about four times the price) or restart it. The technical lead wants a proven cause before spending anything. The sale runs until Sunday.
You are on your workstation ml-poste-01, with the Azure CLI signed in to the masterlinux-prod subscription and a
copy of the masterlinux/boutique-api repository in ~/boutique-api.
This simulation is part of a production simulation pack.
Unlock every production simulation
AWS, Azure, Google Cloud, on-premise and multi-cloud, from Beginner to Extreme: the real incidents of IT service companies and large IT organisations, available for a year.
Try the free demo simulation first
Already enrolled? Log in.