At Tawfir Pay, payments fail in waves: pods are evicted from node to node and one node no longer responds

Tawfir Pay publishes a mobile wallet used by 900,000 customers (payments at merchants, transfers, top-ups). The payment API paiement-api runs on an on-prem Kubernetes cluster (cp-01..03, worker nodes wk-01..04, 64 GB each), behind HAProxy (lb-01). The logs of every pod are collected by a Fluent Bit DaemonSet and sent to OpenSearch on the logs-01 VM (traceability requirement: 90 days of transaction logs).

Recent changes: on Thursday, paiement-api v5.8 (new fee schedule) and the upgrade of the Fluent Bit chart to 3.1 “with no functional change”. Since 03:00 last night, payment pods have been evicted (“Evicted”) in waves; at 07:50, wk-03 went NotReady and the night operator “cordoned” it in the meantime. Merchants report declined payments, especially in hypermarkets, and it is Saturday.

You are the on-call SRE on poste-sre (cluster-admin kubectl, SSH on the machines). The product manager asks to “roll back to v5.7 right now”. Find the real cause and restore the service without losing the traceability of transactions.