INC-3064 · P1
Since this morning, students' invoices return “NoSuchKey”: half of the history has vanished from the bucket
Agadir · AWS account eu-west-3 · the MasterLinux Agadir branch · Tuesday 08:50
The PDF invoices of MasterLinux students are stored in the S3 bucket masterlinux-factures
(prefix factures/YYYY/MM/), versioned, encrypted with KMS, and served by CloudFront on
factures.masterlinux.lab from the student portal. The same bucket also receives, under logs/, the
distribution's access logs.
This morning, the Agadir branch receives calls: "The link to my September invoice shows an error page."
The team checks: invoices from July, August and early September return 404 (NoSuchKey), October
ones open. Accounting must close the quarter on Friday and send the supporting documents to the
chartered accountant.
Yesterday afternoon, operations added a lifecycle rule (CHG-2240, "expire logs older than 30 days, they are expensive"). The security manager suspects a KMS key change; the front-end developer thinks of a CloudFront cache.
You are the on-call cloud engineer, on ml-poste-01 (AWS CLI v2). The team keeps a runbook in
~/runbooks/.
This simulation is part of a production simulation pack.
Unlock every production simulation
AWS, Azure, Google Cloud, on-premise and multi-cloud, from Beginner to Extreme: the real incidents of IT service companies and large IT organisations, available for a year.
Try the free demo simulation first
Already enrolled? Log in.