Observability: metrics, logs, traces, SLIs and SLOs

Objective

Understand what you measure when you "monitor" an application, tell apart the three main signals (metrics, logs, traces) and know how to set a quantified service objective, with its error budget. This is the foundation of the whole course.

Key points

• Monitoring answers "is it working?"; observability lets you answer "why is it not working?" without adding code on the fly.
• Three signals complement each other: metrics (numbers over time), logs (events as text), traces (the path of a request through several services).
• An SLI (Service Level Indicator) measures quality as the user sees it: share of successful requests, share of requests served in under 300 ms…
• An SLO (Service Level Objective) is the target for this SLI over a time window: "99.9% of payments successful over 30 days".
• The error budget = 100% − SLO. As long as some is left, the team can ship; when it is used up, reliability comes first.
• You alert on the symptoms the user experiences (errors, slowness), not on every possible cause (CPU at 80%).

Monitoring or observability?

Take an online shop in Casablanca that sells handicraft products. One Friday evening, customers complain: "the payment keeps spinning".

• With classic monitoring, you have a green or red light per server: the web server answers ping, the database is running. Everything is green… and yet customers cannot pay.
• With observability, you see that the error rate of the /api/paiement API went from 0.05% to 4% at 21:12, that the logs show "timeout" errors towards the payment gateway, and that a trace of a slow request shows 9 seconds spent in the call to the banking provider.

Observability does not replace monitoring: it enriches it. The word comes from control theory: a system is observable if you can deduce its internal state from what it emits.

The three signals

The usual path of a diagnosis: a metric triggers the alert, a dashboard places the problem in time, and logs and traces give the details.

Metrics

A metric is a time series: a name, labels and successive values. Prometheus, the central tool of this course, distinguishes four types:

# Counter: only goes up (or resets to zero on restart)
http_requests_total{route="/api/paiement", status="200"}   184532

# Gauge: goes up and down
node_memory_MemAvailable_bytes{instance="srv-web-rabat:9100"}   2147483648

# Histogram: counters per duration range ("bucket")
http_request_duration_seconds_bucket{le="0.3"}   181020
http_request_duration_seconds_bucket{le="+Inf"}  184532

# Summary: quantiles calculated on the application side
rpc_duration_seconds{quantile="0.95"}   0.41

Metrics are cheap: whether there are 10 or 10,000 requests per second, a counter takes up the same space. That is why you keep them for a long time and alert on them.

Be careful with labels: each combination of values creates a new series. A client_id label with 200,000 customers creates 200,000 series: this is the so-called cardinality mistake that blows up the memory of the metrics server.

Logs

Logs tell events one by one. They are essential to understand a specific case ("why did order 88412 fail?"). Good practices:

{"ts":"2026-10-18T21:12:04Z","niveau":"error","service":"paiement","commande":88412,"msg":"timeout passerelle","duree_ms":9001}

• a structured format (JSON) rather than free text: you can filter by field;
• a level (debug, info, warn, error);
• never a card number, a password or a national ID number (CIN) in clear text in a log (law 09-08 on personal data in Morocco).

Traces

A trace follows one request through all services. Each step is a span with a start, a duration and a shared trace identifier. The current open standard is OpenTelemetry (libraries, OTLP protocol), with tools such as Jaeger or Tempo to store and display traces. Prometheus, for its part, does not store traces: it handles metrics.

Methods for choosing what to measure

Two simple frameworks keep you from measuring everything at random:

• RED for a service that receives requests: Rate, Errors, Duration.
• USE for a resource (CPU, disk, network): Utilization, Saturation (queue), Errors.

The "four golden signals" popularized by Google follow the same idea: latency, traffic, errors, saturation.

Service "paiement" (RED)              Server srv-bdd-casa (USE)
  rate     : requests/s                CPU     : % busy, run queue
  errors   : % of 5xx responses        disk    : % full, I/O wait time
  duration : 95th percentile in ms     network : throughput, lost packets

SLI, SLO, SLA

Three acronyms that people often confuse:

The SLA is always less demanding than the SLO: the SLO acts as an internal alarm before the contract is breached.

A good SLI is calculated as a ratio of good events to total events:

Availability SLI = successful requests / total requests
Latency SLI      = requests served in under 300 ms / total requests

The error budget

If the SLO is 99.9% over 30 days, you are allowed 0.1% of failures:

30 days × 24 h × 60 min = 43,200 minutes
43,200 × 0.001          = 43.2 minutes of tolerated downtime per month

Same calculation in requests: with 2 million payments per month,
2,000,000 × 0.001 = 2,000 failed requests tolerated.

Each extra "9" is very expensive (redundancy, on-call, testing). An online school in Fès does not need the same SLO as a bank. Choose an objective that your users really notice.

The error budget becomes a decision tool: is some left? You ship the new version. Is it used up? You freeze new features and fix reliability.

Write your SLOs down

An SLO that only exists in one person's head is useless. Store them in the team's Git repository, in a simple format:

# slo/paiement.yml
service: paiement
sli: requêtes /api/paiement sans réponse 5xx / requêtes totales
objectif: 99.9          # as a percentage
fenetre: 30d            # rolling 30 days
responsable: equipe-paiement
revue: trimestrielle    # the objective is reviewed every three months

(The keys and values of this file stay in French in this course: objectif = objective, fenetre = window, responsable = owner, revue = review.)

Calculate an SLI by hand

Before you have Prometheus, a simple access log is enough to measure. With a log in nginx's combined format, the status code is the 9th field:

# Total number of requests
wc -l < /var/log/nginx/access.log

# Number of 5xx responses (server-side errors)
awk '$9 ~ /^5/ {n++} END {print n+0}' /var/log/nginx/access.log

# Availability as a percentage
awk '{t++} $9 !~ /^5/ {ok++} END {printf "%.2f %%\n", ok*100/t}' /var/log/nginx/access.log

Why only 5xx? A 404 or 401 error is usually the client's fault (wrong address, wrong password): it must not consume the service's error budget. A 503, on the other hand, means that you could not answer.

In the next lessons, Prometheus will do this calculation continuously:

sum(rate(http_requests_total{status!~"5.."}[30d])) / sum(rate(http_requests_total[30d]))

Where each tool fits

  Applications and servers
  │  metrics (/metrics)        logs                   traces (OTLP)
  ▼                            ▼                      ▼
  Prometheus  ──rules──▶  Alertmanager ──▶ e-mail, chat, on-call
  │                            Loki / Elasticsearch   Jaeger / Tempo
  └──────────────┬─────────────┴──────────────────────┘
                 ▼
              Grafana (dashboards, exploration)

Prometheus is very well suited to purely numeric series and remains usable during an outage, because each server is autonomous. It is not suitable when you need 100% accuracy, for example to bill each request: use a database or the logs instead.

Common pitfalls

• Alerting on everything: 200 alerts a day, and nobody reads them. Alert on symptoms (errors, slowness felt by the user) and keep the causes (CPU, memory) for dashboards.
• A 100% SLO: impossible to meet, it forbids any update. Even the big cloud providers aim lower.
• Counting 4xx as outages: a wave of wrong passwords would look like a service outage.
• High-cardinality labels (customer identifier, IP address, order number) in metrics: put them in the logs, not in labels.

Lab

You are the operations engineer of an online shop. The log /root/obs/acces.log contains 1,000 calls to the payment API. You will:

1. describe the service objective in a small YAML file;
2. calculate the corresponding error budget, in minutes;
3. count the 5xx responses in the log with awk;
4. decide whether the objective is met on this sample.

Quiz

1. Name the three signals of observability and one question that each one answers.
2. What is the difference between an SLO and an SLA? Which one is more demanding?
3. A service has an SLO of 99.5% over 30 days. How many minutes of downtime are tolerated?
4. Why should you not put the customer identifier in a metric label?
5. Should you trigger an alert when a server's CPU goes above 80%? Explain why.

Reference sources: Prometheus documentation (Apache 2.0). Original MasterLinux text.