Incident simulations
Exit

INC-3571 · P1

The database came back two hours ago, but nobody can log in any more: mobile app, branches and partner blocked

Casablanca · Bouskoura data centre + AWS eu-west-3 + GCP europe-west1 · MasterLinux · Tuesday 08:15

Every MasterLinux login goes through the authentication service ml-auth (auth.masterlinux.lab):

  • on site, in Bouskoura: the nginx API gateway ml-gw-01 (active virtual address, keepalived) and its standby ml-gw-02; it relays to AWS over Direct Connect;
  • in AWS (eu-west-3): the ml-auth Deployment on EKS (namespace auth, autoscaling from 6 to 40 pods) and its RDS PostgreSQL Multi-AZ database ml-auth-db;
  • the clients: the students' mobile app, the terminals of the 14 branches (retry settings distributed by AWS AppConfig) and the partner synchronisation batch, a Cloud Run job in the GCP project masterlinux-batch.

At 06:09, an AWS maintenance triggered a Multi-AZ failover of ml-auth-db: 3 minutes without a database. Since 06:12, RDS has been available. But authentication never recovered: the gateway returns 502s and 504s, autoscaling went up to 40 pods that keep restarting, incoming traffic is ten times higher than usual. At 08:00, the night team already “restarted all the pods” and “added nodes”, with no effect. The operations director suggests “doing another database failover, just in case”; the application team wants to “go up to 80 pods”.

You are the on-call site reliability engineer (SRE), on ml-bastion (kubectl on the cluster eks-masterlinux-prod, AWS CLI v2, gcloud, read-only psql), with SSH access to ml-gw-01 and ml-gw-02. The gateway load-shedding runbook is called RB-AUTH-07.

This simulation is part of a production simulation pack.

Unlock every production simulation

AWS, Azure, Google Cloud, on-premise and multi-cloud, from Beginner to Extreme: the real incidents of IT service companies and large IT organisations, available for a year.

Try the free demo simulation first

Already enrolled? Log in.