Incident simulations
Exit

INC-3362 · P1

The MasterLinux shop is frozen: no order can be displayed or saved any more

Casablanca · MasterLinux server room · ml-db-01 · 10:09

The MasterLinux shop (sale of courses, lab vouchers and certificates) relies on the MySQL 8.0 database boutique, hosted on ml-db-01 (primary) and replicated to ml-db-02. The commandes (orders) table holds 14 million rows.

This morning, a developer had to add a code_promo column for the back-to-school campaign. He ran his ALTER TABLE in production at 10:02, “an instant change in MySQL 8”, according to him. Since then, no order page responds, payment is blocked and the application's connection pool is almost full. The developer has closed his terminal, but his query still shows up.

The project manager wants to restart MySQL; a colleague talks about an InnoDB deadlock.

You are the on-call DBA, connected over SSH to ml-db-01 with the dba account (mysql client configured in ~/.my.cnf: you can type mysql -e "…" or the SQL query directly).

This simulation is part of a production simulation pack.

Unlock every production simulation

AWS, Azure, Google Cloud, on-premise and multi-cloud, from Beginner to Extreme: the real incidents of IT service companies and large IT organisations, available for a year.

Try the free demo simulation first

Already enrolled? Log in.