Incident simulations
Exit

INC-3504 · P1

Meal delivery: emptied baskets and “Error 500” at checkout since last night's reboot

Marrakech · Sidi Ghanem server room · bastion-ch · Monday 11:40

Chhiwat Livraison (a fictional company, Marrakech) delivers meals from 600 restaurants. Sessions, baskets and payment locks live in Redis 7: one master and two replicas (redis-ch-01..03), watched by three Sentinels that fail over automatically in case of failure. Two Node.js servers (app-ch-01, app-ch-02) run the API.

Last night at 04:30, redis-ch-01 (the master) rebooted for a kernel patch. It came back online at 04:33. Since then, baskets empty themselves and order checkout fails. The lunch rush starts at 12:00. The on-call engineer has already checked: “Redis answers PING on all three servers”.

You are a level-3 production engineer on bastion-ch (root SSH, redis-cli available everywhere). Mission: restore writes without losing the baskets or creating two masters.

This simulation is reserved for students enrolled in a course.

Already enrolled? Log in.