Exit Code 137 = 128 + 9: the process received SIGKILL. With Reason: OOMKilled, the kernel
killed it because the container exceeded its memory limit (cgroup). It was not the node, which still
has 70 % of its RAM free. Version 3.2 loads 412,318 invoices into memory at startup (~420 Mi): in preprod
the limit is 1 Gi, in prod it was left at 256Mi. Hence "it works in preprod".
The liveness probe is not involved: when it fails, the event says Liveness probe failed and the
container ends in Error, not OOMKilled.
Incident resolved
Analysis and fix
Immediate fix: raise the resources according to the measured usage, with a margin:
kubectl set resources deployment facturation --requests=memory=512Mi --limits=memory=768Mi, then
follow the rollout with kubectl rollout status deployment facturation.
To prevent it from happening again:
- put the value into the Git manifests (otherwise the next
kubectl applyputs 256Mi back); - keep the same limits in preprod and prod, and test releases under realistic load;
- alert on container restarts (
kube_pod_container_status_restarts_total) and on theOOMKilledreason; - in the code: load the cache in batches or on demand instead of all in memory at startup.