Order platform: one request in two fails with a 500 after 30 seconds, the database is “saturated” according to the team

Dar Tijara Distribution (a fictional company, Rabat) supplies 1,400 pharmacies. Orders go through commandes.dartijara.ma: the HAProxy load balancer lb-dt-01 sends traffic to two Spring Boot servers (app-dt-01, app-dt-02, port 8080), which write to MySQL 8 db-dt-01 (a replica db-dt-02 serves the management reports).

Last night, version 3.8.0 of the API was deployed by the usual script, during a shortened maintenance window. Since 07:30 (when the pharmacies open), one order in two hangs for 30 seconds and then fails with a 500. The on-call DBA has already posted in the channel: “MySQL is saturated, we need to raise max_connections and restart the database”.

You are a level-3 production engineer, logged in to the bastion-dt bastion with SSH (root) access to the servers. Mission: find the real cause, restore the service without any further outage, and do not restart anything “at random”.