Noujoum Assurances: the mobile API rejects every token, payroll can no longer authenticate and certificates are “not yet valid”

Noujoum Assurances (car and home insurance, 600 employees) runs in hybrid mode:

• at head office, an Active Directory forest corp.noujoum.ma (the PDC emulator is dc-rabat-01), an AD CS PKI (certificate auto-enrollment) and AD FS (sts.noujoum-assurances.ma), which issues the JWT tokens of the policyholders' mobile app;
• two internal time servers: ntp-gps-01 (GPS receiver on the roof, “stratum 1”) and ntp-02 (stratum 2);
• in AWS (eu-west-3): the mobile API (API Gateway + Lambda authorizer authz-jwt-noujoum), the Lambda sync-sinistres that calls the internal API api-interne.corp.noujoum.ma over the VPN, and domain-joined EC2 servers (SSSD), including app-paie-01.

This weekend, a waterproofing company redid the datacenter roof; a scheduled power cut took place at dawn this morning. Since about 06:40:

• the mobile app: 100 % of API calls return 401 (“invalid token”);
• the payroll team: “Clock skew too great” on app-paie-01, impossible to start the monthly calculation;
• the claims integration: certificate is not yet valid towards the internal API.

You are on the on-call workstation (AWS CLI v2, SSH access to app-paie-01, dc-rabat-01 (OpenSSH, PowerShell), ntp-02 and ntp-gps-01). The CTO wants an update before 09:00.