Incident simulations
Exit

INC-3570 · P1

boutique.masterlinux.lab changes address every minute, and the payment API disappears at times

Casablanca · AKS francecentral + GKE europe-west9 · MasterLinux shop · Tuesday 10:20

The MasterLinux shop runs in production on AKS (aks-boutique-prod, francecentral). Its public names are in the Azure DNS zone masterlinux.lab (group rg-dns) and are kept up to date automatically by external-dns from the Ingresses: boutique, api and paiement.

Last week (13/10), the platform team set up a second GKE cluster (gke-boutique-prod, europe-west9) to prepare the move to multi-cloud: the shop is deployed there as a preview (load tests, without the production database), with the same manifests and the same Helm values for external-dns, copied over from AKS. This morning at 09:57, the cloud team finally gave the GKE external-dns service account write permissions on the Azure DNS zone, “so that everything is ready”.

Since 10:00: boutique.masterlinux.lab sometimes answers with the AKS address, sometimes with the GKE one (the preview, with no carts or accounts), sometimes NXDOMAIN; api and paiement disappear then reappear. About one payment in three fails.

You are on the ml-bastion bastion (ops): kubectl (AKS context by default) and az. The GKE control plane is private: it is administered from the ml-admin-gke VM (ssh admin@ml-admin-gke), which has kubectl, helm and gcloud.

This simulation is part of a production simulation pack.

Unlock every production simulation

AWS, Azure, Google Cloud, on-premise and multi-cloud, from Beginner to Extreme: the real incidents of IT service companies and large IT organisations, available for a year.

Try the free demo simulation first

Already enrolled? Log in.