Incident simulations
Exit

INC-3370 · P1

Every cart and every session of the shop disappeared last night

Marrakesh · MasterLinux server room in Sidi Ghanem · MasterLinux shop · 08:20

The MasterLinux shop keeps customer sessions and their carts in Redis: a master ml-redis-01 and two replicas ml-redis-02 and ml-redis-03, watched by three Sentinels (one per node). The application only reads from and writes to the master designated by Sentinel.

This morning, customer service is swamped with messages: every customer has been logged out and the carts are empty, including those prepared last night for the 09:00 promotion. The key count graph shows a vertical drop from 2.1 million to 0 at 03:02. No alert went off: every node answers PING and Sentinel reported no failure.

A nightly copy exists: every day at 02:00, a scheduled task runs a BGSAVE on a replica and copies the file to /sauvegardes/redis (NFS share from ml-backup-01, mounted on all three nodes).

You are logged in as ops on ml-redis-01, with sudo. Understand what happened, restore the data before the promotion, and make sure it does not happen again.

This simulation is part of a production simulation pack.

Unlock every production simulation

AWS, Azure, Google Cloud, on-premise and multi-cloud, from Beginner to Extreme: the real incidents of IT service companies and large IT organisations, available for a year.

Try the free demo simulation first

Already enrolled? Log in.