At lunchtime, one connection in four to the MasterLinux site fails at random

The site and labs of the MasterLinux school all go through the ml-lb-01 reverse proxy (RHEL 9, nginx, firewalld), installed in the server room of the Fez branch. It spreads the traffic across three application servers, ml-app-01 to ml-app-03.

Yesterday, the operations team put a new monitoring platform (ml-sonde-01) into service that tests the site's pages. This lunchtime, at peak attendance (lunch break, 2,000 connected students), support receives dozens of tickets: “the page spins then times out”, “it works on the second try”. The failures are random and also affect new lab connections.

The manager suggests increasing worker_connections in nginx; a colleague thinks the application servers are saturated.

You are the on-call system administrator, connected over SSH to ml-lb-01 with the ops account (sudo).