Incident simulations

Real outages replayed in a simulated terminal. Read the ticket, investigate, make your diagnosis, fix it.

Infrastructure & systems

INC-2515P2 · Major

At Hotel Tafoukt, the front desk PCs no longer get an IP address

Since the festival weekend, workstations show a 169.254.x.x address. Yet the DHCP server seems to be running.

Windows Server 2022 · DHCP · PowerShell15 minFree INC-2417P1 · Critical

Atlas Shop returns “502 Bad Gateway” in the middle of the night

The online store went down at 03:12. nginx is responding, but the application behind it has stopped working.

Ubuntu 24.04 · nginx · Node.js20 minFree INC-2516P1 · Critical

At Bennani & Associés, nobody can sign in to the domain anymore

Since 08:30, workstations show a date and time error at sign-in. The passwords, however, have not changed.

Windows Server 2022 · Active Directory · Kerberos25 min INC-2518P2 · Major

Gharb Agro's ERP crawls: a load of 38 on an 8-core server

Since the offices opened, every ERP screen takes a minute to load. Yet the processor looks almost idle.

Rocky Linux 9 · Odoo · PostgreSQL20 min INC-2512P1 · Critical

The cooperative's file server no longer boots: “You are in emergency mode”

After Monday morning's reboot, the server is stuck in emergency mode. None of the shares are reachable.

Debian 12 · systemd · fstab20 minFree INC-2513P1 · Critical

Oriental Logistique's ERP server vanished from the network after switching to a static IP

A colleague replaced the DHCP configuration with a static address using netplan. Since then, the server answers nothing.

Ubuntu 24.04 · netplan · systemd-networkd15 min INC-2511P2 · Major

The Riad Voyages website returns “403 Forbidden” after moving to /srv

The site was copied to a new disk mounted on /srv. Apache is running, permissions look fine… and yet every request is denied.

Rocky Linux 9 · Apache httpd · SELinux20 min INC-2517P2 · Major

The Marrakech branch VPN is “connected”… but nothing gets through

The WireGuard tunnel between the branch and the Casablanca head office is up, the handshake is recent, and yet the ERP is unreachable.

Debian 12 · WireGuard · iptables25 min INC-2514P2 · Major

The Ibn Sina Pharmacies' nightly backups stopped without a single alert

The 02:30 cron job “runs” every night, but no backup has been produced since 6 October.

Ubuntu 24.04 · cron · Bash20 min INC-2433P2 · Major

Nobody can log in over SSH after the hardening

A colleague “secured” SSH on Friday evening. On Monday morning, every connection is refused.

Rocky Linux 9 · OpenSSH · systemd15 min

Containers & orchestration

INC-2451P1 · Critical

The delivery API keeps restarting after the production release

A new version was deployed with Docker Compose: the api container never stays up for more than 5 seconds.

Docker · Docker Compose · Node.js20 min INC-2647P2 · Major

The payment API is unreachable even though its pods are running

After the manifests were refactored, the ingress returns 503 on /api. The pods are Running and Ready, the logs are clean.

Kubernetes 1.31 · ingress-nginx · kube-proxy20 min INC-2641P1 · Critical

The billing service restarts in a loop after release 3.2

The facturation pods go into CrashLoopBackOff a few seconds after every start. No invoices are being generated.

Kubernetes 1.31 · containerd · Node.js20 minFree INC-2653P2 · Major

Migration to a new namespace: the pods are stuck in ImagePullBackOff

The port container-tracking app moves to the suivi-prod namespace. No pod starts: the image will not download.

Kubernetes 1.31 · containerd · Harbor15 min

DevOps

INC-2662P3 · Minor

The “disk full” alert fires constantly on a half-empty server

For three days DisqueRempli has fired on web-03. The team silences it every morning. Real problem or false alarm?

Prometheus · Alertmanager · node_exporter15 min INC-2658P2 · Major

The GitHub Actions pipeline has been failing at deployment for three days

Tests and build pass, but the deploy job fails on every push to main. Three fixes are waiting to reach production.

GitHub Actions · gh CLI · Docker20 min

Cloud

INC-2669P1 · Critical

Migration to AWS: the new website cannot be reached from the Internet

The EC2 instance is running, nginx has started, DNS points to the right place… and yet the browser just spins.

AWS EC2 · VPC · AWS CLI v220 min

Development

INC-2674P1 · Critical

The API returns 500 errors a few minutes after going to production

Version 4.1 deployed at 11:00: all fine for ten minutes, then every request waits five seconds and fails with 500.

Node.js 22 · Express · node-postgres (pg)20 minFree INC-2686P2 · Major

The booking site has crawled since the search by phone number went live

Pages take more than 10 seconds to load and the database CPU is at 98 %. No server changed, only a new feature.

PostgreSQL 16 · psql · pg_stat_statements20 min