Nakhla Pay: the Kubernetes API has not responded since 03:12, no deployment or scheduled job goes through

Nakhla Pay is a payment institution based in Salé: merchant payments, instant transfers and a mobile wallet for 2,300 merchants. All of production runs on a Kubernetes cluster installed with kubeadm in the company's server room:

• three control planes cp-01, cp-02, cp-03, each with its own etcd (“stacked” topology);
• the API address k8s-api.nakhla.lan:6443 carried by the HAProxy load balancer lb-01;
• 14 worker nodes.

Last night, the datacenter team shut down cp-03 at 01:00 for a firmware update (BIOS, BMC, RAID controller), following the approved change CHG-0918: “cluster tolerant to the loss of one control plane”. Since 03:12, the API has not responded. Pods already running keep running, but nothing moves: the 08:00 release of api-virements is blocked, the 06:00 bank reconciliation job did not start, and the 09:00 peak will not be absorbed.

The production manager, Rachid El Idrissi, wants to “power cp-03 back on right now, never mind the firmware”; a colleague suggests “restoring etcd from the 02:00 backup”.

You are Soufiane, the on-call platform engineer, on poste-astreinte (admin kubeconfig, SSH access to lb-01, cp-01, cp-02, and to cp-03's BMC card via ipmitool). The change schedule is in ~/INC-3515/changements.txt.