Incident simulations
Exit

INC-3521 · P1

After the power cut, power is back, every machine is on… and the laboratory software is stuck on “502”

Fez · Saïs technical platform · Annour Biologie · Monday 06:40

Annour Biologie groups six medical testing laboratories in Fez and Meknes. All results go through the LIS (laboratory information system): https://sil.annour.ma, served by rp-01 (nginx) in front of two application servers app-sil-01/app-sil-02, a PostgreSQL database on db-01 (data on an iSCSI volume provided by stor-01), the PDF reports on an NFS share of stor-01, and the directory + internal DNS annour.lan on dc-01. All these machines are VMs on the Proxmox hypervisor hv-01, in the technical platform's server room.

Last night, mains power cut from 02:50 to 05:10 across the neighbourhood. The UPS lasted 40 minutes, the generator did not start (broken since August). At 05:10, power comes back: hv-01 boots and starts all the VMs at the same time. This morning, every machine answers ping, but the LIS shows “502 Bad Gateway”. The first samples arrive at 07:00, and the emergency department of the nearby hospital is waiting for test results.

You are the on-call systems administrator on poste-astreinte, with root SSH everywhere. Bring the service back cleanly, layer by layer, without putting the database's data at risk.

This simulation is reserved for students enrolled in a course.

Already enrolled? Log in.