Incident simulations
Exit

INC-3063 · P1

Since 06:12, no enrolment is saved on the shop any more: every payment ends in a 500 error

Fez · AWS account eu-west-3 · the MasterLinux shop · Tuesday 06:40

The MasterLinux shop sells enrolments in the school's courses. Its API saves every order in the Amazon RDS for MySQL 8.0 database ml-boutique-db (database boutique), in Multi-AZ.

Since 06:12, every order fails: the customer pays, the payment gateway confirms, then the API returns 500 and the order does not exist in the database. Customer service is receiving the first messages from customers charged without an enrolment. The November enrolment campaign starts at 08:00.

Nothing has been shipped since Friday. The on-call developer thinks of "a lock blocking the table" and suggests restarting the RDS instance.

You are the database administrator (account dba), on ml-bastion with the AWS CLI v2 and the mysql client.

This simulation is part of a production simulation pack.

Unlock every production simulation

AWS, Azure, Google Cloud, on-premise and multi-cloud, from Beginner to Extreme: the real incidents of IT service companies and large IT organisations, available for a year.

Try the free demo simulation first

Already enrolled? Log in.