The ALB health check leaves from the ALB's own interfaces (protected by sg-0a71c3e5f9b2d4068)
towards port 8080 of the instances. The app-souklmarsa-v2 group, copied from the old template
from the nginx days, only allows that group on port 80. Packets to 8080 are therefore
silently dropped: hence Target.Timeout (rather than an HTTP error), and the 504 served by the ALB.
The application is fine: curl localhost:8080/health returns 200 on the instance. A 404 on
/health would give Target.ResponseCodeMismatch, not a timeout. The network ACL is the default
one (everything allowed). And Auto Scaling sees the instances as “Healthy” because it uses the EC2
health check type, which only looks at the state of the machine.
Incident resolved
Analysis and fix
Fix: allow the ALB's group on the application port:
aws ec2 authorize-security-group-ingress --group-id sg-0d58e2a9c4f1b7360 --protocol tcp --port 8080 --source-group sg-0a71c3e5f9b2d4068.
Then check: aws elbv2 describe-target-health … must turn healthy (3 successful checks,
about 45 s) and curl -I https://www.souklmarsa.ma must return 200. The port 80 rule, now useless,
can be removed afterwards.
To prevent it from happening again:
- describe security groups as infrastructure as code and derive the rule's port from the target group's port (a single variable, not two copies);
- always reference the ALB's security group as the source, never 0.0.0.0/0;
- check with VPC Reachability Analyzer (ALB → instance:8080) before changing a group in production, and switch a single instance first;
- set a CloudWatch alarm on the target group's
UnHealthyHostCount > 0, and switch the Auto Scaling health check type toELBso that it sees what the load balancer sees.