INC-3570 · P1
boutique.masterlinux.lab changes address every minute, and the payment API disappears at times
Casablanca · AKS francecentral + GKE europe-west9 · MasterLinux shop · Tuesday 10:20
The MasterLinux shop runs in production on AKS (aks-boutique-prod, francecentral). Its public
names are in the Azure DNS zone masterlinux.lab (group rg-dns) and are kept up to date
automatically by external-dns from the Ingresses: boutique, api and paiement.
Last week (13/10), the platform team set up a second GKE cluster (gke-boutique-prod,
europe-west9) to prepare the move to multi-cloud: the shop is deployed there as a preview (load
tests, without the production database), with the same manifests and the same Helm values for
external-dns, copied over from AKS. This morning at 09:57, the cloud team finally gave the GKE external-dns
service account write permissions on the Azure DNS zone, “so that everything is ready”.
Since 10:00: boutique.masterlinux.lab sometimes answers with the AKS address, sometimes with the
GKE one (the preview, with no carts or accounts), sometimes NXDOMAIN; api and paiement disappear then
reappear. About one payment in three fails.
You are on the ml-bastion bastion (ops): kubectl (AKS context by default) and az. The GKE control
plane is private: it is administered from the ml-admin-gke VM (ssh admin@ml-admin-gke), which has
kubectl, helm and gcloud.
This simulation is part of a production simulation pack.
Unlock every production simulation
AWS, Azure, Google Cloud, on-premise and multi-cloud, from Beginner to Extreme: the real incidents of IT service companies and large IT organisations, available for a year.
Try the free demo simulation first
Already enrolled? Log in.