Incident simulations
Exit

INC-3064 · P1

Since this morning, students' invoices return “NoSuchKey”: half of the history has vanished from the bucket

Agadir · AWS account eu-west-3 · the MasterLinux Agadir branch · Tuesday 08:50

The PDF invoices of MasterLinux students are stored in the S3 bucket masterlinux-factures (prefix factures/YYYY/MM/), versioned, encrypted with KMS, and served by CloudFront on factures.masterlinux.lab from the student portal. The same bucket also receives, under logs/, the distribution's access logs.

This morning, the Agadir branch receives calls: "The link to my September invoice shows an error page." The team checks: invoices from July, August and early September return 404 (NoSuchKey), October ones open. Accounting must close the quarter on Friday and send the supporting documents to the chartered accountant.

Yesterday afternoon, operations added a lifecycle rule (CHG-2240, "expire logs older than 30 days, they are expensive"). The security manager suspects a KMS key change; the front-end developer thinks of a CloudFront cache.

You are the on-call cloud engineer, on ml-poste-01 (AWS CLI v2). The team keeps a runbook in ~/runbooks/.

This simulation is part of a production simulation pack.

Unlock every production simulation

AWS, Azure, Google Cloud, on-premise and multi-cloud, from Beginner to Extreme: the real incidents of IT service companies and large IT organisations, available for a year.

Try the free demo simulation first

Already enrolled? Log in.