SahlPay payments are slowing down and a Kubernetes node has been at 100% CPU since 04:12

SahlPay is a mobile payment wallet from Marrakech (420,000 users, 3,800 merchants). Its payment API runs in the Kubernetes cluster k8s-prod (namespace paiement, deployment api-paiement), with the image registre.sahlpay.ma/paiement/api-paiement:2.14.0, built and signed with cosign by GitLab CI (runner runner-ci) then pushed to the Harbor registry registre. Version 2.14.0 has been in production since Thursday 15/10; nobody has deployed since.

Last night around 04:10, the HPA went from 2 to 4 replicas (refund batch), the autoscaler added the node k8s-node-03. Since then, the API's p99 has climbed to 6 seconds, k8s-node-03 is at 100% CPU, and a SOC network alert reports outbound connections to 198.51.100.23:3333.

The platform team wants to “drain the node and be done with it”, a developer suggests “re-pushing 2.14.0 from CI”. You are the SOC / DevSecOps analyst on duty (Yassine), on poste-soc (cosign, crane, SahlPay's public key cosign.pub). SSH access: bastion-k8s (kubectl admin), k8s-node-03, registre (Harbor) and runner-ci. Goal: understand, contain, restore the legitimate image and prevent it from happening again, without destroying the evidence.