Incident simulations
Exit

INC-3527 · P1

Sahm Logistique: the billing fix must reach production before 10:00, but CI fails, Terraform answers 403 and ArgoCD no longer syncs anything

Casablanca · Sidi Maârouf (datacenter) + GitHub + k8s-ops cluster · Sahm Logistique · Monday 08:50

Sahm Logistique is a Casablanca carrier (B2B parcel delivery, 340 trucks, 4 hubs between Casablanca, Tangier, Agadir and Oujda). Its software delivery chain is fully automated:

  • GitHub Actions on self-hosted runners (runner-ci-01 to -04, Sidi Maârouf) builds and publishes the images, then updates the deploiements repository;
  • Atlantis comments the Terraform plans on infrastructure PRs;
  • ArgoCD (cluster k8s-ops) syncs the deploiements repository to production.

To avoid managing dozens of tokens, the platform team created a single GitHub App, sahm-plateforme-bot (installation 41872093), whose private key is used by CI, Atlantis and ArgoCD. All outbound traffic goes through the proxy proxy-sortie-01 (Squid).

This morning, a billing bug has been overcharging customers on volumetric weight since 03:00. The fix (PR #1187, facturation v4.21.3) is approved: it must be in production before 10:00, the time when the month's invoices are sent automatically to 2,100 customers. Since 08:05, nothing goes through. The project manager, Youssef Chraibi, keeps rerunning the jobs “until it works”. One developer suggests “putting 20 retries everywhere”, another one creating personal tokens to get around it.

You are the on-call DevOps engineer (Anas), on poste-devops (gh CLI logged in with your account). You can SSH to runner-ci-03, proxy-sortie-01 and bastion-k8s (kubectl and argocd to k8s-ops). Unblock the delivery without opening a security hole.

This simulation is reserved for students enrolled in a course.

Already enrolled? Log in.