Claims cannot be filed: the “high availability” PostgreSQL database no longer has a primary

Tamazirt Assurances (a fictional company, Casablanca) manages 210,000 car and home insurance contracts. Claims are filed on espace.tamazirt-assurances.ma. The PostgreSQL 16 database runs in high availability with Patroni on pg-tz-01 and pg-tz-02; the “leader” lock is stored in a three-member etcd cluster. HAProxy sends writes (port 5000) to the node that answers 200 on /primary, and reads (port 5001) to the replicas.

Since 02:14, browsing works but every write fails. The night on-call engineer saw the alert “pg-tz-01: PostgreSQL demoted” and suggests “promoting pg-tz-02 by hand”. The compensation campaign for the Mohammedia floods starts at 09:00.

You are a level-3 database engineer, on the bastion-tz bastion (root SSH to every server). Mission: understand why Patroni refuses to elect a primary and restore it without any risk of a double primary (split-brain) or data loss.