The MasterLinux site goes down and comes back every five minutes, on AWS as on Azure: “503 Service Unavailable”

All of MasterLinux's traffic (showcase site, online school, shop) comes in through eight in-house entry proxies ml-proxy: four on AWS (ml-edge-aws-01 to 04, eu-west-3) and four on Azure (ml-edge-az-01 to 04, francecentral). Each proxy embeds an anti-bot module that scores every request using a signature file (the list of features the model knows how to compute).

This file is regenerated every 5 minutes at the Rabat headquarters by the gen-signatures job on ml-gen-01: it queries the ClickHouse analytics database (ml-ana-01 to 03, behind ml-ana-lb), writes a JSON file and distributes it to all the proxies through cfgsync in under 30 seconds.

Since 08:20, the site goes down for everyone (503) for a few minutes, comes back, then goes down again. Both clouds are hit at exactly the same moments. On the incident channel, some are already talking about a DDoS and suggest enabling country blocking; the data team only mentions a “no-impact permissions migration” in progress on the ClickHouse cluster (CHG-5120).

You are the on-call DevOps engineer, logged on to ml-bastion, from where you can reach over SSH ml-edge-aws-01, ml-edge-az-01, ml-gen-01 and ml-ana-01.