INC-3527 · P1
Sahm Logistique: the billing fix must reach production before 10:00, but CI fails, Terraform answers 403 and ArgoCD no longer syncs anything
Casablanca · Sidi Maârouf (datacenter) + GitHub + k8s-ops cluster · Sahm Logistique · Monday 08:50
Sahm Logistique is a Casablanca carrier (B2B parcel delivery, 340 trucks, 4 hubs between Casablanca, Tangier, Agadir and Oujda). Its software delivery chain is fully automated:
- GitHub Actions on self-hosted runners (
runner-ci-01to-04, Sidi Maârouf) builds and publishes the images, then updates thedeploiementsrepository; - Atlantis comments the Terraform plans on infrastructure PRs;
- ArgoCD (cluster
k8s-ops) syncs thedeploiementsrepository to production.
To avoid managing dozens of tokens, the platform team created a single GitHub App,
sahm-plateforme-bot (installation 41872093), whose private key is used by CI, Atlantis and
ArgoCD. All outbound traffic goes through the proxy proxy-sortie-01 (Squid).
This morning, a billing bug has been overcharging customers on volumetric weight since 03:00. The
fix (PR #1187, facturation v4.21.3) is approved: it must be in production before 10:00,
the time when the month's invoices are sent automatically to 2,100 customers. Since 08:05, nothing goes through.
The project manager, Youssef Chraibi, keeps rerunning the jobs “until it works”. One developer suggests
“putting 20 retries everywhere”, another one creating personal tokens to get around it.
You are the on-call DevOps engineer (Anas), on poste-devops (gh CLI logged in with your
account). You can SSH to runner-ci-03, proxy-sortie-01 and bastion-k8s
(kubectl and argocd to k8s-ops). Unblock the delivery without opening a security hole.
This simulation is reserved for students enrolled in a course.
Already enrolled? Log in.