Putting numbers on availability: SLA, MTTR, RTO, RPO and single points of failure

Objective

Before installing HAProxy or Pacemaker, know what you must guarantee: measure the real availability of a service, understand what lowers it, and find the components whose failure stops everything.

Key points

• Availability is computed from real incidents, following the rules of the contract (what is excluded, how it is measured).
• A = MTBF ÷ (MTBF + MTTR): you gain as much by repairing faster (MTTR) as by failing less often (MTBF).
• RTO (acceptable downtime) and RPO (acceptable data loss) translate directly into technologies: automatic or manual failover, synchronous or asynchronous replication, backups.
• A single point of failure (SPOF) is any element present only once on the path of a request: a server, but also a switch, a power supply, a certificate, an administrator.
• Adding redundancy adds complexity: every failover mechanism becomes a source of outages itself if it is not tested.

The basics (table of “nines”, cloud provider SLAs, the 3-2-1 rule) are in the Cloud fundamentals course; SLOs and error budgets in Observability. Here, we move on to calculations for a platform you run yourself.

1. Measuring availability as the contract defines it

A service level agreement (SLA) always states:

The calculation from an incident log:

# date;type;minutes;description
awk -F';' '$2=="imprevu" {s += $3} END {printf "%.2f %%\n", 100 * (1 - s/43200)}' incidents.csv

The “contractual” figure is not what customers experienced: a planned one-hour maintenance on a Saturday evening does not count in the SLA, but the shop was still closed. Track both: contractual availability (for penalties) and real availability (to decide on investments).

2. MTBF, MTTR: repairing fast matters as much as not failing

• MTBF (mean time between failures): average time between two failures.
• MTTR (mean time to repair/restore): average time to restore the service.

A = MTBF / (MTBF + MTTR)

Une panne par mois (MTBF ≈ 720 h) réparée en 4 h  : 720 / 724   = 99,45 %
La même panne réparée en 30 min                    : 720 / 720,5 = 99,93 %

MTTR breaks down into: detecting (monitoring), diagnosing (logs, procedures), repairing (access, spare parts, backups), checking. Automatic failover mostly shortens the repair; a good alert shortens detection. An on-call team in Casablanca that gets the alert in 1 minute instead of 20 gains more than with a badly configured second server.

3. RTO and RPO: from business need to technology

Points to remember:

• RPO = 0 requires synchronous replication (DRBD protocol C, PostgreSQL synchronous_commit): every write waits for the second server, so latency goes up.
• The real RPO of a daily backup is 24 h in the worst case (failure just before the next backup), not “a few hours”.
• The real RTO is measured during a failover test, stopwatch in hand. An RTO that was never tested is an assumption.

4. Inventorying single points of failure

Follow the path of a request, from the client to the disk, and note each step:

client ─► DNS ─► lien Internet ─► pare-feu ─► ml-lb-01 ─► ml-web-0[1-3] ─► ml-app-0[1-2] ─► ml-db-0[1-2]
                                                    │
                                              ml-nfs-01 (fichiers clients)

# Un inventaire simple : composant;rôle;exemplaires
awk -F';' 'NR > 1 && $3 == 1 {print "SPOF :", $1, "(" $2 ")"}' architecture.txt

The SPOFs most often forgotten:

For each SPOF, three possible answers: duplicate it, reduce its MTTR (spare part, written procedure, expiry monitoring), or accept the risk and put it in writing.

5. The cost of redundancy

Duplicating a component does not double reliability if:

• both instances share a common cause (same rack, same buggy software version, same wrong configuration pushed everywhere);
• the failover mechanism is never tested;
• nobody notices that one of the two has already failed (“dormant” redundancy: you discover the second one is broken the day the first one fails).

Operating rule: all redundancy must be monitored. A load balancer serving traffic with only one server out of three alive must raise an alert, even if customers see nothing.

Version differences

Common pitfalls

• Counting planned maintenance in an SLA calculation that excludes it, or the other way round: read the contract before computing.
• Confusing RPO with backup frequency: a backup every 24 h gives an RPO of 24 h, not 12 h “on average”.
• Forgetting dependencies: two redundant web servers mounting the same single NFS server still depend on that NFS server.
• Promising 99.99% (4 min per month) with a manual failover procedure that takes 20 minutes.

Lab

On a Debian 12 machine, you have the September incident log of the MasterLinux shop and an inventory of its architecture. You will:

1. compute availability as the contract defines it;
2. say whether the 99.9% SLA was met;
3. compute the MTTR;
4. extract the list of single points of failure;
5. give the real RPO of a single nightly backup.

Quiz

1. Why can two teams report two different availability figures for the same month?
2. One failure per month, repaired in 30 minutes instead of 4 hours: which indicator improves?
3. Which replication is needed for an RPO of zero, and what does it cost?
4. Name three single points of failure that are not servers.
5. What is “dormant” redundancy and how do you avoid it?