At MasterLinux, the training reimbursement databases have been read-only since the “no impact” SAN maintenance

The MasterLinux training reimbursement department serves 210,000 beneficiaries (teachers and their families). The remboursements and adherents databases run on PostgreSQL 16 on ml-db-01 (RHEL 9). Their data lives on a LUN of the ml-baie-01 storage array, seen through two independent Fibre Channel fabrics (ml-sw-fc-a and ml-sw-fc-b) and aggregated by multipath (/dev/mapper/mpatha, ext4, mounted on /srv/pgdata). The reimbursement management application runs on ml-app-01.

Last night (02:40 → 03:10), the integrator upgraded the firmware of ml-sw-fc-b. The change was classified as “no impact”: “every server keeps its paths through fabric A”. At 07:15, branch staff can no longer record reimbursement requests, and today's reimbursement transfers must be sent to the bank at 11:00.

You are the on-call systems and storage administrator, on ml-bastion, with SSH access to the servers, the FC switches and the storage array console. The integrator suggests “rolling back to the old firmware”, the DBA wants to reboot ml-db-01. Bring the databases back without putting their data at risk, and find out why multipath did not do its job.