Listings portal: publishing impossible, empty statistics and Elasticsearch cluster “RED”

Ribat Immobilier (fictional company, Rabat) publishes 160,000 listings from 900 agencies on ribat-immo.ma. The listings live in the annonces-v8 index of an Elasticsearch 8 cluster (3 master nodes, 3 data nodes, 1 coordinating node). Visits are stored in a daily index stats-visites-YYYY.MM.DD, created every night at midnight.

At 03:12, es-data-03 crashed with an OutOfMemoryError (a monthly report started by hand) and then restarted on its own. Since the agencies opened, every publication fails and the dashboards are empty; the cluster is RED. The application team is asking to “raise the heap to 64 GB”.

You are a level-3 production engineer on bastion-ri (curl to es-coord-01:9200, root SSH to the nodes). Mission: get the cluster green and publishing working again, without losing any index and without disabling the disk protections.