Incident simulations
Exit

INC-3367 · P1

The MasterLinux shop returns 504 on every page since 13:44

Casablanca · MasterLinux server room in Sidi Maarouf · MasterLinux shop · 13:58

The MasterLinux shop (boutique.masterlinux.lab) sells the online courses. Traffic follows this path: the HAProxy load balancer ml-lb-01 → two nginx + ModSecurity v3 front ends (ml-waf-01, ml-waf-02), which filter requests and then pass them to the application servers ml-app-01 and ml-app-02 (port 8080).

Since 13:44, every page times out: HAProxy returns 504 Gateway Time-out after 30 seconds. The dashboard shows both front ends at 100% CPU. Back-to-school sales have stopped (about 9,000 MAD of revenue per hour) and customer service is getting the first calls.

The security team maintains its own WAF rules in the Git repository /srv/waf-regles, deployed to both front ends by the waf-deploy script (nginx -t validation, then reload).

You are logged in as ops on ml-waf-01, with sudo. Find the cause, bring the shop back, and say what will prevent this from happening again.

This simulation is part of a production simulation pack.

Unlock every production simulation

AWS, Azure, Google Cloud, on-premise and multi-cloud, from Beginner to Extreme: the real incidents of IT service companies and large IT organisations, available for a year.

Try the free demo simulation first

Already enrolled? Log in.