Incident simulations
Exit

INC-3523 · P1

Migration to AWS: customers can no longer find orders placed since 02:14

Tangier · Gzenaya datacenter + AWS account eu-west-3 · Tijara Nord · Monday 08:20

Tijara Nord is an online wholesaler from Tangier (supplies for grocery stores and cafés, 1,900 business customers). Its MySQL 8.0 database commandes still runs in the rack in the Gzenaya zone (db-onprem, 10.20.4.15). Since 11/10, it has been migrating to Aurora MySQL (aurora-commandes, region eu-west-3) through an AWS DMS task tache-commandes-cdc (full load then CDC), over the site-to-site VPN between the vpn-gw gateway (strongSwan) and AWS.

First step of the cutover, done on Friday: the public order tracking API (app-01, EC2) now reads from Aurora (SSM parameter /tijara/prod/commandes/source-lecture = aurora). Writes stay on db-onprem until the final cutover planned for next Saturday.

Since opening time this morning, the call centre has been swamped: customers get “commande introuvable” (order not found) for orders placed last night and this morning. The network manager, Anouar, is convinced that “it's the VPN again” and wants to restart it. The CTO, Driss Tazi, on the contrary, wants to “switch 100 % to AWS right now and be done with it”.

You are the on-call cloud engineer (Nadia), on poste-cloud (AWS CLI v2, profile tijara-prod by default). You have SSH access to app-01, vpn-gw and db-onprem. Restore a correct service and a complete replication, without losing a single order.

This simulation is reserved for students enrolled in a course.

Already enrolled? Log in.