INC-3571 · P1
The database came back two hours ago, but nobody can log in any more: mobile app, branches and partner blocked
Casablanca · Bouskoura data centre + AWS eu-west-3 + GCP europe-west1 · MasterLinux · Tuesday 08:15
Every MasterLinux login goes through the authentication service ml-auth
(auth.masterlinux.lab):
- on site, in Bouskoura: the nginx API gateway
ml-gw-01(active virtual address, keepalived) and its standbyml-gw-02; it relays to AWS over Direct Connect; - in AWS (
eu-west-3): theml-authDeployment on EKS (namespaceauth, autoscaling from 6 to 40 pods) and its RDS PostgreSQL Multi-AZ databaseml-auth-db; - the clients: the students' mobile app, the terminals of the 14 branches (retry settings
distributed by AWS AppConfig) and the partner synchronisation batch, a Cloud Run job
in the GCP project
masterlinux-batch.
At 06:09, an AWS maintenance triggered a Multi-AZ failover of ml-auth-db: 3 minutes without a
database. Since 06:12, RDS has been available. But authentication never recovered: the gateway
returns 502s and 504s, autoscaling went up to 40 pods that keep restarting, incoming traffic is
ten times higher than usual. At 08:00, the night team already “restarted all the pods” and
“added nodes”, with no effect. The operations director suggests “doing another database failover,
just in case”; the application team wants to “go up to 80 pods”.
You are the on-call site reliability engineer (SRE), on ml-bastion (kubectl on the cluster
eks-masterlinux-prod, AWS CLI v2, gcloud, read-only psql), with SSH access to ml-gw-01 and ml-gw-02.
The gateway load-shedding runbook is called RB-AUTH-07.
This simulation is part of a production simulation pack.
Unlock every production simulation
AWS, Azure, Google Cloud, on-premise and multi-cloud, from Beginner to Extreme: the real incidents of IT service companies and large IT organisations, available for a year.
Try the free demo simulation first
Already enrolled? Log in.