INC-3505 · P1
Payment gateway: “password authentication failed” on the database since 07:02, although nobody touched the passwords
Casablanca · Bouskoura datacenter · bastion-np · Monday 07:25
Nakhil Pay (a fictional company, Casablanca) processes card payments for 3,000 merchants. The
paiement-api API has no hard-coded password: Vault Agent obtains dynamic PostgreSQL
credentials for it (6 h lifetime) from the Vault cluster vault-np-01..03 (Raft storage).
At startup, the Vault nodes unseal automatically thanks to the encryption Vault vault-kms-np
(“transit” seal).
Last night, from 01:00 to 01:10, electrical maintenance cut power to rack B (vault-np-02 and -03).
At 06:00, everything worked. Since 07:02, payments fail: the database keeps repeating “password
authentication failed”. The DBA suspects a change to pg_hba.conf, the developer suggests “putting the
old fixed password back in the config, just for today”.
You are a level-3 security/production engineer on bastion-np (root SSH, vault CLI with an operator
token). Mission: restore payments without bypassing Vault and without weakening the firewall.
This simulation is reserved for students enrolled in a course.
Already enrolled? Log in.