INC-3370 · P1
Every cart and every session of the shop disappeared last night
Marrakesh · MasterLinux server room in Sidi Ghanem · MasterLinux shop · 08:20
The MasterLinux shop keeps customer sessions and their carts in Redis: a master
ml-redis-01 and two replicas ml-redis-02 and ml-redis-03, watched by three Sentinels (one per node).
The application only reads from and writes to the master designated by Sentinel.
This morning, customer service is swamped with messages: every customer has been logged out and the
carts are empty, including those prepared last night for the 09:00 promotion. The key count graph
shows a vertical drop from 2.1 million to 0 at 03:02. No alert went off: every node
answers PING and Sentinel reported no failure.
A nightly copy exists: every day at 02:00, a scheduled task runs a BGSAVE on a replica and
copies the file to /sauvegardes/redis (NFS share from ml-backup-01, mounted on all three nodes).
You are logged in as ops on ml-redis-01, with sudo. Understand what happened, restore the
data before the promotion, and make sure it does not happen again.
This simulation is part of a production simulation pack.
Unlock every production simulation
AWS, Azure, Google Cloud, on-premise and multi-cloud, from Beginner to Extreme: the real incidents of IT service companies and large IT organisations, available for a year.
Try the free demo simulation first
Already enrolled? Log in.