INC-3504 · P1
Meal delivery: emptied baskets and “Error 500” at checkout since last night's reboot
Marrakech · Sidi Ghanem server room · bastion-ch · Monday 11:40
Chhiwat Livraison (a fictional company, Marrakech) delivers meals from 600 restaurants. Sessions,
baskets and payment locks live in Redis 7: one master and two replicas (redis-ch-01..03), watched by
three Sentinels that fail over automatically in case of failure. Two Node.js servers (app-ch-01,
app-ch-02) run the API.
Last night at 04:30, redis-ch-01 (the master) rebooted for a kernel patch. It came back online at
04:33. Since then, baskets empty themselves and order checkout fails. The lunch rush starts at
12:00. The on-call engineer has already checked: “Redis answers PING on all three servers”.
You are a level-3 production engineer on bastion-ch (root SSH, redis-cli available everywhere).
Mission: restore writes without losing the baskets or creating two masters.
This simulation is reserved for students enrolled in a course.
Already enrolled? Log in.