Incident simulations
Exit

INC-3511 · P1

Sahel Assurances: since last night, policyholders see yesterday's contracts, claim declarations fail and paid contracts are suspended

Rabat · AWS eu-west-3 (Paris) + eu-central-1 (Frankfurt) · Sahel Assurances · 07:05

Sahel Assurances, a car and home mutual insurer from Rabat (310,000 policyholders), serves its mobile app and its web portal from api.sahel-assurances.ma:

  • primary region eu-west-3 (Paris): ALB protected by AWS WAF waf-sahel-public, instances app-paris-*, RDS PostgreSQL database sahel-prod;
  • recovery region eu-central-1 (Frankfurt): instances app-francfort-* in “secours-lecture” (read-only recovery) mode, connected to the cross-region replica sahel-dr-replica;
  • Route 53 with failover routing: PRIMARY = Paris (health check hc-paris on /health), SECONDARY = Frankfurt;
  • the payment gateway PayMaghreb notifies each payment by webhook; at 06:00, the job relances-impayes suspends the contracts whose instalment has not been paid.

Yesterday at 18:30, after a bot attack, the SOC added a geo-restriction WAF rule (Morocco and France only) in Count mode; it was to switch to Block “after one night of observation”.

Since dawn: policyholders see stale data, claim declarations fail (500), agents report customers suspended for non-payment although they have paid. The night team noticed an automatic failover to Frankfurt at 03:08 and suggests promoting the replica to “get writes back”. You are on call (AWS CLI v2, the partner's paymaghreb CLI, SSH access to the machines). The night's logbook is in ~/INC-3511/main-courante.txt.

This simulation is reserved for students enrolled in a course.

Already enrolled? Log in.