The MasterLinux shop can no longer reserve stock: every order has been stuck “pending” since the night

The MasterLinux shop sells the school's lab kits, books and certification vouchers online. Every order goes through the API ml-commandes-01, which reserves the stock with the internal API api.stock.interne.masterlinux.lab before confirming the payment. If that fails, the order goes back into the SQS queue commandes-a-reserver and will be retried.

The name api.stock.interne.masterlinux.lab is published in a Route 53 private zone by an in-house DNS automation:

• a planner (ml-dnsplan-01) watches the healthy instances of the stock service and produces numbered plans (generations) stored in the S3 bucket masterlinux-dnsauto-plans;
• two independent enactors, ml-dnsauto-01 (zone eu-west-3a) and ml-dnsauto-02 (zone eu-west-3b), apply each new plan in Route 53;
• a cleaner (on ml-dnsauto-02) deletes, every 5 minutes, plans that are too old and the records they own.

Last night, between 02:50 and 03:08, the stock team progressively replaced the instances of the stock API (new version). At 06:50, customer service reports that no order is confirmed any more; the queue exceeds 48,000 messages and the first customers are cancelling. Management wants the service back before the shops open at 09:00, and above all wants to understand, so that it never happens again.

You are on ml-bastion (AWS CLI v2, region eu-west-3), with SSH access to ml-commandes-01, ml-dnsauto-01 and ml-dnsauto-02.