INC-3363 · P1
MasterLinux shop: “503 Service Unavailable” since last night's migration of the API servers
Rabat · MasterLinux server room in Hay Riad · ml-lb-01 · Tuesday 08:10
The MasterLinux shop (boutique.masterlinux.lab) sells enrolments for courses and
exams. The HAProxy load balancer ml-lb-01 sends /api/ calls to two API servers,
ml-api-01 and ml-api-02 (port 8080).
Yesterday at 22:00, the systems team migrated the API to two new machines (Ubuntu 24.04, more
memory). Procedure followed: install the new machines, test them, then change the DNS
records ml-api-01.masterlinux.lab and ml-api-02.masterlinux.lab to the new addresses, and
shut down the old machines at 22:03. Nobody touched the load balancer: its configuration
“uses names, not addresses”.
Since 22:04, the whole dynamic part of the shop returns 503 Service Unavailable: no more payments, no more enrolments. The back-to-school campaign starts this morning at 09:00.
You are the on-call administrator, logged in as ops (sudo) on ml-lb-01. Restore the service
cleanly and durably: the same migration is planned for the payment servers next week.
This simulation is part of a production simulation pack.
Unlock every production simulation
AWS, Azure, Google Cloud, on-premise and multi-cloud, from Beginner to Extreme: the real incidents of IT service companies and large IT organisations, available for a year.
Try the free demo simulation first
Already enrolled? Log in.