Nakhla Mutuelle: the members' area returns 500 errors, the database is “inaccessible” and Auto Scaling can no longer launch a single instance

Nakhla Mutuelle is a health mutual insurer from Rabat (62,000 members, employees of SMEs and self-employed professionals). Its members' area adherents.nakhla.ma and the mobile app go through the API api-adherents (Auto Scaling group asg-api-adherents, EC2), which reads:

• the RDS PostgreSQL database db-nakhla-adherents (KMS-encrypted);
• its password in Secrets Manager (prod/api-adherents/db);
• the supporting documents (prescriptions, invoices) in the S3 bucket nakhla-pieces-justificatives (SSE-KMS);
• and its EBS volumes encrypted by the launch template.

All this encryption relies on one customer-managed KMS key, called through the alias alias/nakhla-donnees. The infrastructure is described in Terraform (repository infra-nakhla), applied by GitHub Actions on the self-hosted runner runner-tf-01, with the role gha-terraform-deploy.

Friday 16/10, two changes were merged at the end of the day: an IAM tightening of the application roles (PR #409) and a Terraform cleanup “group the encryption foundation into a module” (PR #412, Driss Amrani). The pipeline was green. Quiet weekend.

This Monday since 07:50: 500 errors on the members' area, the mobile app shows “service unavailable”, Auto Scaling fails every attempt to add instances for the Monday peak, and the reimbursements service can no longer download any document. This week's reimbursements go out on Thursday. The security manager, Karim Lahlou, is sure that “it's the IAM tightening” and wants to attach AdministratorAccess to the API's role. The CIO, Hicham Bennani, suggests “creating a new key and restoring last night's snapshot”.

You are the on-call cloud engineer (Nadia), on poste-sre (AWS CLI v2, profile nakhla-prod by default, region eu-west-3). You can open an SSM session on the API instance, log in to the Terraform runner, and switch to the nakhla-securite account (organization CloudTrail, SCPs) with aws-vault. Restore the service without losing a single piece of encrypted data.