Incident simulations
Exit

INC-3569 · P1

Since last night, the AWS applications can no longer reach headquarters, while Azure works

Casablanca · MasterLinux headquarters in Aïn Sebaâ + AWS eu-west-3 + Azure francecentral · ml-bastion · Tuesday 07:15

The MasterLinux headquarters in Casablanca hosts the ERP (erp.masterlinux.lab, 10.20.12.10) and the directory. Its core router ml-rt-coeur-01 (FRRouting, AS 65010) advertises over BGP the networks of headquarters and branches to two clouds:

  • AWS (eu-west-3, VPC masterlinux-prod 10.60.0.0/16) through Direct Connect 1 Gbit/s (private VIF vif-prod-siege), with an AWS site-to-site VPN as backup (two BGP tunnels);
  • Azure (francecentral, VNet vnet-prod-frc 10.70.0.0/16) through ExpressRoute (private peering).

Last night at 23:30, change CHG-2241 was applied on ml-rt-coeur-01: new VLANs for the new headquarters floor and a “clean-up” of the BGP configuration. The change was declared successful at 23:40: the sessions to Azure were established.

Since 23:34, the enrolment application ml-app-aws-01 (AWS) can no longer reach the ERP: all of the night's enrolments are queued, and the branches open at 08:00. The trainers' portal ml-app-azure-01 (Azure) works. The link provider states that the fibre and the equipment are healthy.

You are the on-call network engineer, on ml-bastion (AWS CLI v2 and Azure CLI configured, SSH access to the router and the applications). Restore the path to AWS and make sure it does not happen again.

This simulation is part of a production simulation pack.

Unlock every production simulation

AWS, Azure, Google Cloud, on-premise and multi-cloud, from Beginner to Extreme: the real incidents of IT service companies and large IT organisations, available for a year.

Try the free demo simulation first

Already enrolled? Log in.