Incident simulations
Exit

INC-3505 · P1

Payment gateway: “password authentication failed” on the database since 07:02, although nobody touched the passwords

Casablanca · Bouskoura datacenter · bastion-np · Monday 07:25

Nakhil Pay (a fictional company, Casablanca) processes card payments for 3,000 merchants. The paiement-api API has no hard-coded password: Vault Agent obtains dynamic PostgreSQL credentials for it (6 h lifetime) from the Vault cluster vault-np-01..03 (Raft storage). At startup, the Vault nodes unseal automatically thanks to the encryption Vault vault-kms-np (“transit” seal).

Last night, from 01:00 to 01:10, electrical maintenance cut power to rack B (vault-np-02 and -03). At 06:00, everything worked. Since 07:02, payments fail: the database keeps repeating “password authentication failed”. The DBA suspects a change to pg_hba.conf, the developer suggests “putting the old fixed password back in the config, just for today”.

You are a level-3 security/production engineer on bastion-np (root SSH, vault CLI with an operator token). Mission: restore payments without bypassing Vault and without weakening the firewall.

This simulation is reserved for students enrolled in a course.

Already enrolled? Log in.