The “disk full” alert fires constantly on a half-empty server

Medina Hosting, a web host in Fez, monitors its 140 servers with Prometheus and Alertmanager. Since Thursday, the critical alert DisqueRempli ("disk full") has fired on web-03 every four hours. Nobody has found a full disk: every morning someone adds a silence "false positive, will look later".

The technical lead wants an answer today: either we fix the cause, or we admit that web-03 is no longer monitored.

You are on mon-01 (Prometheus on localhost:9090, Alertmanager configured for amtool). To run a command on the affected server: ssh web-03 <command>.