Lesson 1 of 12 · SRE culture
The SRE job: toil, SLOs and error budget burn rate
Objective
Understand what companies expect from an SRE (Site Reliability Engineering) engineer, measure toil (the manual work that never ends) and drive release decisions with the error budget burn rate, going beyond the simple budget calculation covered in the Monitoring course.
Key points
- SRE applies software engineering methods to operations: it measures reliability, automates it and negotiates it with product teams.
- Toil is manual, repetitive, automatable work with no lasting value, which grows with traffic. You measure it and cap it (often at 50% of the time).
- An error budget is useless unless it is tied to a written policy: who decides on a freeze, from which threshold, with which exceptions.
- The burn rate tells how fast the budget is being spent: 1 = it will run out exactly at the end of the window; 14.4 = it will all be gone in 50 hours.
- You alert on burn rate with two windows (a long one for certainty, a short one to know it is still happening), not on a simple instantaneous error rate.
1. What an SRE does day to day
In an IT services company or a large tech organisation, the "SRE" title usually covers the same missions:
| Mission | Concrete example |
|---|---|
| Define and track SLOs | "99.9% of payments succeed over 30 days" for the MasterLinux shop API |
| Carry the pager and run incidents | being called at 3 a.m., restoring service, coordinating |
| Write postmortems and follow up on actions | making sure the same outage does not come back |
| Reduce toil | turning a manual disk cleanup into a safe automatic task |
| Diagnose performance problems | finding out why ml-app-02 answers in 4 s instead of 200 ms |
| Plan capacity | provisioning servers before the sales or Ramadan |
| Review production changes | checking that a change has a rollback |
An SRE is not "the admin who does everything": a large part of their time goes into coding (scripts, tools, automation) so that operations work does not grow as fast as the service.
SRE, DevOps, system administrator
- DevOps is a culture: development and operations teams share goals and tools.
- SRE is a concrete way to implement it, with numbers (SLOs, budget, toil).
- The system administrator remains essential; an SRE keeps those skills (Linux, networking, storage) and adds measurement and code.
2. Toil: measure it to get rid of it
A task is toil if it ticks most of these boxes:
- manual: someone types commands;
- repetitive: it was already done last week;
- automatable: a machine could do it;
- reactive: triggered by an alert or a ticket, not by a project;
- no lasting value: once done, the service is no better than before;
- scales with traffic: twice as many customers, twice as much work.
Not toil: writing a postmortem, designing an alert, automating a task, planning meetings (that is overhead, to be watched in a different way).
Measure
Record the time spent for one or two weeks, then count:
# categorie;nature;minutes
awk -F';' 'NR>1 {total+=$3} NR>1 && $2=="toil" {labeur+=$3}
END {printf "Labeur : %d min sur %d (%.0f %%)\n", labeur, total, 100*labeur/total}' semaine.csv
# The most expensive categories: the first ones to automate
awk -F';' '$2=="toil" {m[$1]+=$3} END {for (c in m) print m[c], c}' semaine.csv | sort -rn | head -3
The most widespread rule: no more than 50% toil per person. Beyond that, the team no longer has time to build the tools that would free it, and toil grows even more. When the cap is exceeded, you report it to management as you would report a full disk: it is a risk, not a merit.
Choosing what to automate
The right candidate combines frequency, duration and risk:
gain mensuel ≈ fréquence par mois × minutes par exécution
+ coût d'une erreur humaine × probabilité de se tromper
That is: monthly gain ≈ runs per month × minutes per run + cost of a human error × probability of making it.
Restarting the API by hand 12 times a month (30 minutes each, at 3 a.m.) is well worth two days of development. Copying a certificate once a year, much less so.
3. SLOs: going further than the budget calculation
The Monitoring course laid the foundations: SLI, SLO, SLA, error budget in minutes. The SRE's job is to use them to make decisions.
Choosing a good SLI
An SLI must reflect what the user experiences. A few common choices:
| Type of service | Recommended SLI | Measured where |
|---|---|---|
| API or website | share of non-5xx requests; share served in under 300 ms | load balancer (ml-lb-01) rather than the application |
| Batch processing | share of jobs finished on time | job scheduler |
| Storage | share of successful reads; data still present (durability) | read probes |
| Message queue | age of the oldest message | broker |
Measuring at the load balancer sees errors the application does not even log (crashed process, timeout). Measuring with a synthetic probe (a fake customer placing an order every minute) also sees outages when nobody is buying.
The error budget policy
A budget without a written rule triggers arguments at every incident. A policy fits on one page:
Service : API de paiement SLO : 99,9 % de requêtes non 5xx, 30 jours glissants
Si le budget consommé dépasse 100 % :
- gel des nouvelles fonctionnalités jusqu'à revenir sous le SLO
- seules partent en production les corrections de bugs et de fiabilité
- le prochain sprint comporte au moins un chantier de fiabilité
Si un seul incident consomme plus de 20 % du budget : post-mortem obligatoire
Exception : correctif de sécurité urgent (décision du responsable technique)
Arbitrage en cas de désaccord : direction technique
In plain words: above 100% of budget consumed, new features are frozen until the service is back under its SLO, only bug and reliability fixes ship, and the next sprint includes reliability work; any single incident that eats more than 20% of the budget requires a postmortem; urgent security fixes are the exception; the engineering leadership arbitrates. This policy is signed in calm times by product and engineering, not negotiated in the middle of an outage.
4. The burn rate
The 30-day budget tells how much has been consumed. The burn rate tells how fast.
burn rate = taux d'erreur observé / taux d'erreur autorisé par le SLO
SLO 99,9 % → taux autorisé = 0,1 % = 0,001
Dernière heure : 1 728 erreurs sur 120 000 requêtes = 1,44 %
burn rate = 0,0144 / 0,001 = 14,4
That is: burn rate = observed error rate / error rate allowed by the SLO. With a 99.9% SLO, the allowed rate is 0.001; 1,728 errors out of 120,000 requests is 1.44%, so the burn rate is 14.4.
| Burn rate | Meaning (30-day window = 720 h) |
|---|---|
| 1 | the budget runs out exactly at the end of the 30 days |
| 2 | exhausted in 15 days |
| 6 | exhausted in 5 days |
| 14.4 | exhausted in 50 hours: 2% of the budget goes in one hour |
| 1,000 | total outage with a 99.9% SLO: everything goes in 43 minutes |
Time before a fresh budget is exhausted: 720 h / burn rate.
Alerting on burn rate, with two windows
Alerting on "error rate > 0.1% for 5 minutes" wakes the on-call engineer for meaningless spikes. The proven practice combines a long window (consumption is significant) and a short window (it is still happening, otherwise the alert clears quickly):
| Severity | Burn rate | Long window | Short window | Budget consumed when the alert fires |
|---|---|---|---|---|
| Page | 14.4 | 1 h | 5 min | 2% |
| Page | 6 | 6 h | 30 min | 5% |
| Ticket | 1 | 3 days | 6 h | 10% |
In PromQL (Monitoring course), the first row is written as two conditions joined by and: the error ratio over [1h] and over [5m] both exceed 14.4 * 0.001. What matters here is the reasoning, not the syntax.
Computing by hand during an incident
During an incident you do not always have the dashboard: an access log is enough.
# Load balancer log: field 9 is the HTTP status code
total=$(wc -l < acces.log)
erreurs=$(awk '$9 ~ /^5/' acces.log | wc -l)
awk -v e="$erreurs" -v t="$total" 'BEGIN {printf "taux=%.4f burn=%.1f\n", e/t, (e/t)/0.001}'
5. Real case: the MasterLinux shop during the sales
Friday, 10 a.m. The dashboard shows 60% of the monthly budget consumed. Nobody worries: 40% is left. At 10:30, the one-hour burn rate climbs to 14.4, and the 5-minute one to 16: the alert fires. The on-call engineer finds that a new payment method returns 503 errors for one customer in seventy. At this pace, the remaining 40% will be gone in 20 hours, before the end of the sales. The rollback is done at 10:50. On Monday, the policy applies: 104% of budget consumed, new features frozen, and the week is spent load-testing the payment method.
Without burn rate, the team would have seen "60% consumed, all good" until the SLO was broken.
Common pitfalls
- Estimating toil "by feel": without a time log, everyone underestimates their manual tasks. Measure for two weeks before deciding.
- A 100% SLO: impossible to meet and paralysing (no change is allowed any more). Aim for what users actually notice.
- Confusing budget consumed and burn rate: 60% consumed with a burn rate of 0.5 is calm; 10% consumed with a burn rate of 50 is an outage in progress.
- A freeze policy that is never applied: if the freeze is always lifted "as an exception", the SLO no longer means anything.
Lab
On Ubuntu 24.04, in /root/sre, you will:
- compute the team's share of toil over one week;
- name the toil category to automate first;
- compute the burn rate of the last hour;
- deduce the time before a fresh budget is exhausted;
- apply the error budget policy to the 30-day window.
Quiz
- Name four characteristics of toil.
- Why is toil capped at about 50% of an SRE's time?
- A 99.9% SLO and 0.6% errors over the last hour: what is the burn rate?
- Why does a burn-rate alert use a long window and a short window?
- Who must sign the error budget policy, and when?
Answers
- Manual, repetitive, automatable, reactive, no lasting value, scales with traffic (any four).
- Beyond that, the team no longer has time to build the automation that would free it, and toil grows with traffic.
- 0.006 / 0.001 = 6: the 30-day budget would be exhausted in 5 days.
- The long window guarantees that consumption is significant; the short window guarantees that it is still going on (the alert stops quickly once the problem is fixed).
- Product and engineering together (with a designated arbiter), in calm times, before incidents.