Incident simulations
Exit

INC-3508 · P1

Agencies and the night batch cut off from the pricing database, while the online portal works

Rabat · Azure France Central · hub-spoke · Nour Mutuelle · Monday 07:40

Nour Mutuelle (fictional company, Rabat) insures 380,000 members. The pricing database (SQL Server 2022 on vm-sql-01, 10.120.2.10, port 1433) lives in Azure, in the spoke vnet-spoke-data. Hub-spoke topology: all traffic between the site and the spokes, and between spokes, must go through the Azure Firewall afw-hub-nm (10.100.1.4) — an ACAPS requirement after the 2025 audit. Head office is connected to the hub by a site-to-site VPN; the hub is fully described in Terraform (repository ~/infra-hub, automatic apply pipeline every hour).

Last night, a refactor of the hub's routing module was merged. Since 01:48, head office's ETL batch (etl-siege-01) and the pricing software of the 42 agencies (thick client connected directly to SQL Server) end in timeout. The web portal portail.nourmutuelle.ma (in the other spoke), however, works perfectly with the same database. The agencies open at 08:30.

You are a level-3 cloud engineer, on bastion-siege (172.16.20.5) with az CLI logged in to the subscription sub-nm-connectivite (7c2e9b41-3d8a-4f6e-9a15-0b8d2c4e6f71), SSH access to the head office servers and to the Azure VMs, and the Terraform repository cloned. Firewall Log Analytics workspace: 4e1a7c93-58b2-4d0f-a6e1-9c3b7d2f8a05.

This simulation is reserved for students enrolled in a course.

Already enrolled? Log in.