The 60 event collection servers have been restarting in a loop since 09:14

The MasterLinux school collects events from its platforms in real time (student progress, lab validations, clicks) through the evtfront service: a cluster of Java servers on RHEL 9 (ml-evt-01 to ml-evt-60). Each server keeps permanent connections with all the others to share the streams.

This morning at 09:10, for the start of the school year, the team added 6 nodes (ml-evt-55 to ml-evt-60): the cluster went from 54 to 60 servers. Since 09:14, the servers stop one after the other on a Java error, systemd restarts them, they fall again. No student progress is recorded any more and the teaching dashboards are empty. Clients buffer events for 2 hours at most.

You are logged in as ops on ml-evt-03 (one of the old nodes), with sudo. Find out why the cluster no longer holds, validate a fix on this pilot node: the team will then roll it out in waves.