Google's load balancer health checks and proxies do not come from the VPC: they arrive from the
ranges 35.191.0.0/16 and 130.211.0.0/22. The allow-lb-health rule, recreated by hand, was given
the internal range 10.20.0.0/16: the probes are silently rejected by the implied firewall rule, both
VMs become UNHEALTHY and the load balancer, with no healthy backend, answers 502
failed_to_pick_backend.
The certificate is ACTIVE (the 502 is an HTTP response, after all, so TLS succeeded), the health
check does target 8080 and /healthz, and the application responds locally while listening on
0.0.0.0:8080: it is neither TLS, nor the application, nor the probe configuration.
Incident resolved
Analysis and fix
Fix:
gcloud compute firewall-rules update allow-lb-health --source-ranges=35.191.0.0/16,130.211.0.0/22
(target web-boutique, tcp:8080), then check with
gcloud compute backend-services get-health bs-boutique --global: the instances become HEALTHY
within about ten seconds and the shop returns 200.
To prevent it from happening again:
- describe the firewall rules in Terraform, peer-reviewed, rather than recreating them by hand;
- enable firewall rules logging to see refused probes immediately;
- before a switchover, check
get-healthand test a complete path (Connectivity Tests); - never open 8080 to 0.0.0.0/0 “to see what happens”: only Google's ranges should reach the backends.