At peak hours, the registration API fails at random with “Temporary failure in name resolution”

The API api.masterlinux.lab handles MasterLinux registrations and payments. It runs on three instances ml-api-01 to ml-api-03 (c6i.2xlarge, Ubuntu 24.04) behind an ALB, in the VPC 10.40.0.0/16. For each registration, it calls several internal services (paiement.masterlinux.lab, catalogue.masterlinux.lab, Route 53 private zone) and AWS APIs (SQS, S3).

Two changes last week: the servers' new base image (CHG-1984, “removing unneeded services”) and the opening of the back-to-school registrations, which triples traffic between 10:00 and 12:00. Since then, at peak hours, getaddrinfo EAI_AGAIN errors appear at random: students see “Payment unavailable, try again”. Off-peak, nothing.

The application team blames Route 53 DNS; the network team says “everything is green”. You are the on-call cloud engineer, on ml-bastion (AWS CLI v2), with SSH access to the instances.