Incident simulations
Exit

INC-3510 · P1

Tanja Fret: after last night's network maintenance, parcel tracking returns 504, the sync to GCP is frozen and the Casablanca warehouse no longer sees AWS

Tangier · AWS eu-west-3 + GCP europe-west1 + Casablanca datacenter · Tanja Fret · 06:40

Tanja Fret is a freight forwarder at the port of Tanger Med (1,400 containers tracked per day). Its architecture is multi-cloud and hybrid:

  • in AWS (eu-west-3): the tracking site suivi.tanjafret.ma (instances app-suivi-*) and the shipments' RDS PostgreSQL database, behind a Transit Gateway tgw-tanjafret (Amazon ASN 64512);
  • in Google Cloud (europe-west1): the pricing API (GKE) and the data warehouse fed from RDS by the relay relais-sync;
  • the two clouds are linked by a GCP HA VPN ↔ two AWS Site-to-Site VPN connections (vpn-gcp-a, vpn-gcp-b, 4 tunnels, dynamic BGP, Cloud Router cr-tanjafret);
  • the Casablanca datacenter (warehouse WMS) is linked to GCP by a Partner Interconnect and reaches AWS by transiting through GCP.

Last night, from 01:00 to 01:30, change CHG-2210 was applied: “rotation of the tunnels' pre-shared keys + migration to the Terraform module vpn-aws-gcp v3”. The ticket is closed “OK, tunnels UP”.

Since 01:14: parcel tracking shows “tarif indisponible” (504), the synchronisation of shipments to GCP is frozen, and the Casablanca WMS no longer prints the pickup notes (it reads RDS). The first trucks arrive at the port at 07:30. You are on call on poste-astreinte (AWS CLI v2 and gcloud logged in, repository ~/infra-reseau, SSH access to the machines). The maintenance ticket is in ~/INC-3510/CHG-2210.txt.

This simulation is reserved for students enrolled in a course.

Already enrolled? Log in.