The incident response cycle and preparation

Objective

Know in what order to act when a Linux server is compromised, who does what on the team, and what must be ready before the crisis so nothing is improvised on the day.

The essentials

• Incident response follows a cycle: preparation, detection and analysis, containment, eradication, recovery, lessons learned. Skipping a step is costly (you reinstall before understanding, the attacker comes back through the same door).
• The first rule is to not destroy evidence: do not power off, do not "clean up" by hand, do not run a tool that overwrites the disk.
• Every action is written in a logbook timestamped in UTC, with its author.
• Preparation happens cold: contacts, emergency access rights, centralized logs, a trusted tool kit with hashes, playbooks, drills.
• You answer a business question, not only a technical one: which data, which customers, which legal obligations?

1. The six phases

The model most used in the field breaks response into six steps. Official guides sometimes group them differently (the US NIST guide, in its 2025 revision, maps them onto the Cybersecurity Framework CSF 2.0 functions: govern, identify, protect, detect, respond, recover), but the work on the ground is the same.

It is not a straight line: during eradication you often find a second affected machine and go back to analysis.

Short and long containment

• Short term: cut off the attacker as fast as possible (firewall rule, cloud security group, network cable, disabling an account).
• Long term: keep serving customers during the investigation (a clean replacement server, a maintenance page).

Cutting the power is almost always a mistake: RAM (processes, connections, encryption keys, deleted-but-open files) vanishes. You isolate from the network, keep the machine on, collect (lesson 1.2).

2. The team and roles

Even in a small shop, each role has a name before the crisis:

In Morocco, public administrations and critical infrastructure also have an obligation to notify the DGSSI (lesson 4.1). The contact must be in the sheet before the incident.

3. The logbook

An investigation runs for days; without a logbook, nobody knows who restarted what and when. One line per action:

# Simple format: UTC time | author | action
date -u +%Y-%m-%dT%H:%M:%SZ          # 2026-10-20T08:15:00Z
echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) | ops | ml-web-01 removed from public security group" >> logbook.txt

Rules:

• always UTC (the server, the firewall and the SIEM are not necessarily in the same zone; Morocco has changed its time several times);
• append (>>), never overwrite; the logbook itself is part of the case file;
• record the decisions and their reasons too ("decided not to power off: memory capture planned").

4. The trusted tool kit

On a compromised machine, ls, ps or netstat may have been replaced to hide the attacker. So you prepare clean binaries and their hashes cold:

mkdir -p /media/ir-key/tools && cd /media/ir-key/tools
cp /usr/bin/{ls,find,grep,stat,sha256sum,ps,ss,lsof} .   # from a clean machine of the same version
sha256sum * > SHA256SUMS
# On the day, on the suspect server, run the tools from the key:
/media/ir-key/tools/sha256sum -c /media/ir-key/tools/SHA256SUMS
/media/ir-key/tools/ls -la /tmp

Add: a blank, formatted collection disk, a network cable, printed playbooks, the contacts list, tested collection scripts (lesson 1.2), and emergency access (a break-glass account, access to the cloud provider's console).

5. Preparing, concretely

• Inventory: which machines, which data, who owns them.
• Logs: persistent, time-synced (NTP), copied off the server.
• Backups: offline or immutable, restore tested.
• Playbooks: "compromised account", "defaced web server", "ransomware", "data breach".
• Tabletop drills: two hours per quarter, one scenario, everyone plays their role.

Common pitfalls

• Powering off or rebooting "to be safe": half the evidence leaves with the memory.
• Reinstalling right away without understanding the attacker's entry: they come back through the same flaw next week.
• Alerting the team on the mail system that may be compromised: use a separate channel (phone, off-domain messaging).
• Mixed local times: a timeline off by an hour leads to wrong conclusions.

Lab

You prepare the response to incident INC-2026-0007:

1. open the logbook with a first UTC-timestamped line;
2. add a second line without overwriting the first;
3. build the trusted tool kit and its hash file;
4. sort the actions taken during the incident into the right phase of the cycle.

Quiz

1. List the six phases in order.
2. Why must you not power off a compromised server?
3. What is the difference between containment and eradication?
4. Why record times in UTC in the logbook?
5. What is the SHA256SUMS file of the tool kit for?