Reliability, incident response, and operational ownership

Reliability is the property of a system that users can depend on it, incident response is what happens when they briefly cannot, and operational ownership is the arrangement under which the people who build a system are the people who live with it. v1.0 named only the middle of the three. The competency covers all of it: the on-call system, the service-level objectives that say how reliable is reliable enough, the incident in progress with its inverted priorities, and the review afterwards that either changes the system or assigns blame. It matters because reliability is where engineering's promises to the business are tested in public, and because the way an organisation treats failure decides whether it learns from it. The examinations test it because the failures are so legible: the engineer root-causing while customers are down, the review that hunts for a culprit, the fragile system nobody owns, and the leader who obeys the error budget until it is inconvenient.

1. When systems fail, priorities invert: restore, communicate, then diagnose. Mitigation before root cause. The cause can wait; the customers cannot.
2. Someone commands, so everyone else can fix. A defined incident-command role owns coordination and communication, decided before the incident. A commander who starts debugging has vacated the role.
3. Incidents are failures of the system, not of people. A blameless review asks how the system allowed the error and what to change. It is what lets problems surface at all, and it is not the absence of accountability; it is what makes real accountability — fixing the system, honestly — possible.
4. A review that changes nothing is a ritual. Postmortem actions are tracked to completion, and the measure of a review is recurrences prevented, not documents produced.
5. The team that ships it runs it. Operational ownership aligns incentives toward reliability; a team that throws code over a wall has no reason to make it reliable. Orphaned systems do not heal, and every incident in one is an ownership decision deferred.
6. Reliability is a stated objective and a spendable budget. A service-level objective says how reliable is enough; the error budget converts the reliability-versus-velocity argument into a pre-agreed rule that is only as good as leadership's willingness to obey it when it is inconvenient.

Reliability engineering has a well-documented practitioner literature and one substantial body of research, and the unit grades accordingly.

Incident command practice

A single person owns coordination and communication during an incident so that responders can focus on the fix; the priorities during the incident are restore service, communicate, diagnose. The structure is adapted from emergency-services practice and set out for software in Betsy Beyer, Chris Jones, Jennifer Petoff and Niall Richard Murphy (editors), Site Reliability Engineering: How Google Runs Production Systems (2016). It is decided before the incident, because during one is too late, and at organisational scale a cross-team incident needs a commander with authority across the teams. It is a coordination structure, not a technical rescue.

Blameless postmortems practice

The review that asks how the system allowed the failure and what to change — process, tooling, guardrails — rather than who to blame, given its influential statement in John Allspaw, Blameless PostMortems and a Just Culture (2012) and its operational form in Betsy Beyer, Chris Jones, Jennifer Petoff and Niall Richard Murphy (editors), Site Reliability Engineering: How Google Runs Production Systems (2016). The mechanism is that honesty about failure becomes safe, and reliability therefore becomes improvable. Blameless is not consequence-free: patterns of recklessness are performance matters handled separately, through PL-4 Performance management. And a review whose actions die in a backlog has produced a document, not a fix.

Service-level objectives and error budgets practice

From Betsy Beyer, Chris Jones, Jennifer Petoff and Niall Richard Murphy (editors), Site Reliability Engineering: How Google Runs Production Systems (2016): define an explicit reliability target, and treat the tolerable shortfall against it — the error budget — as a spendable resource that gates risk. Budget available, ship; budget exhausted, stabilise. It converts a recurring argument into a shared rule agreed in advance. Its limit is entirely cultural: shipping through an exhausted budget in front of the whole organisation teaches everyone the real policy, which is why the model reappears in CC-5 Values, integrity, and ethical judgment under pressure.

DORA stability metrics research

The research programme reported in Nicole Forsgren, Jez Humble and Gene Kim, Accelerate: The Science of Lean Software and DevOps (2018), which found across several years of large surveys that the organisations with the highest delivery throughput also had the highest stability — change failure rate and time to restore service improve together with deployment frequency, not against it. For this unit the finding matters in two ways: reliability and speed are not a trade-off at the level of the delivery system, and time to restore improves precisely where honesty about failure is safe. The metrics are team-level measures of the system, never individual measures and never a leaderboard.

The incident runbook, decided in calm

Before any incident: who commands, how the role is handed over, where communication happens, what the first three actions are, and what "restored" means for each critical service. A team that has to decide these during an outage decides them badly and late.

The first fifteen minutes

Declare the incident, name the commander, mitigate — roll back, fail over, shed load — and post a first customer-facing update, before anyone looks for the cause. The engineer who wrote the change wants to find the bug; the commander's job is to stop them until service is back.

The postmortem, run as designed

Within days, not weeks; blameless in fact and not only in name; focused on the timeline, the contributing conditions, and what would have caught it earlier; ending in actions with owners and dates. The actions go where the team's real work goes and are reviewed until closed. A recurrence is a review of the review.

The ownership map

Every production system has a named owning team, written where anyone can find it, with on-call and operational responsibility attached. After any reorganisation the map is the first thing checked, because the systems that fall between the new boundaries are where the next incident lives.

Reading on-call load

On-call is a workload, measured: pages per shift, hours out of sleep, time spent on toil. A team whose on-call load is rising is a team whose reliability debt is compounding, and the signal is read at team scope by the manager and at organisational scope as a trend someone owns.

At team scope the object is an incident in progress and one service's operational health, and the manager runs the incident, defends the blameless review, and treats reliability as a deliverable rather than an interruption. At organisational scope the object becomes reliability across services, the ownership gaps a reorganisation created, and the on-call system as a whole; the characteristic moves are installing cross-team incident command rather than joining the debugging, holding the blameless norm against a stakeholder who wants names, and closing ownership gaps that make orphaned systems fail. At executive scope reliability is a strategic property — investment, risk appetite, and what the business is implicitly assuming — with the trend owned at organisational level, and the executive's irreplaceable role is what they visibly do after the most public failure. The pattern is in How Judgment Scales; the structural causes of chronic ownership gaps are SV-6 Organisational design and leading change.

• Restores service, communicates, then diagnoses, and stops the author of the change from root-causing while customers are down. team
• Establishes who is commanding before the incident, and gets out of the way of the fix during it. team
• Runs the postmortem blamelessly and tracks its actions to completion. team
• Treats reliability as a first-class deliverable the team owns, not an interruption to real work. team
• Failure mode — root-causing during the outage. team
• Failure mode — the "who broke it?" review. team
• Failure mode — treating recurring incidents as bad luck. team
• Failure mode — building teams that throw code over a wall. team
• Failure mode — a postmortem measured by the document it produced.
• Installs a single incident commander with cross-team authority for a cross-team incident, rather than becoming the de-facto commander for every incident. org
• Holds the blameless review against a senior stakeholder who demands names, and explains that blamelessness is what makes real accountability possible. org
• Assigns explicit ownership and funds stabilisation for a fragile area a reorganisation left orphaned. org
• Designs an on-call system across several teams that is sustainable, and reads reliability as a trend across services. org
• Failure mode — letting a powerful stakeholder turn a review into a trial. org
• Failure mode — leaving fragile systems orphaned because reassignment is awkward. org
• Failure mode — treating operational load as each team's private problem. org
• Makes the organisation's reliability trend visible and owned, with investment behind it rather than exhortation. exec
• Argues reliability at the executive table as a strategic property: investment, risk appetite, the resilience the business is assuming. exec
• Models blamelessness after the most public failure, including owning the executive-level decisions that contributed. exec
• Obeys the error budget when it is inconvenient, because the whole organisation is watching what the real policy is. exec
• Failure mode — discovering reliability decay only when customers do. exec
• Failure mode — letting the worst incident become the moment blamelessness died. exec
• Failure mode — funding features while reliability debt compounds. exec

Restoring versus understanding. Every engineer in the incident wants to know why, and the customers want it back. The order is fixed — restore first — and the tension is that the evidence of the cause is sometimes destroyed by the restoration. The commander decides what to capture before the rollback, and it is a decision measured in minutes.

Blamelessness versus accountability. A review that names nobody can look like a review that holds nobody responsible. The resolution is the distinction between the review, which is about the system, and performance management, which is about patterns of behaviour and happens separately. Conflating them destroys both.

Reliability versus velocity. At the level of a single decision the two compete; at the level of the delivery system the evidence says they improve together. The error budget is the mechanism that keeps the daily argument from being re-fought, and it works only if the people who set it obey it.

Ownership versus load. Owning what you ship aligns incentives and can bury a team under on-call for systems it inherited. The judgment is in reading on-call as a measured workload and treating a rising load as debt to fund, not as the team's private problem.

Structure versus heroics. The manager who knows the system best is the one most tempted to command every incident, and every incident they command is one the team does not learn to run. The structure exists to make the manager unnecessary, and the manager who resists that has made themselves the single point of failure.

A payment service is down. It has been down for eight minutes; checkout is failing for every customer; the on-call engineer, who deployed a change forty minutes ago, has the logs open and is reading stack traces. Three other engineers have joined the channel and are each proposing a theory. The manager arrives. Nobody has posted a customer-facing update, nobody has said who is coordinating, and the product lead has just asked in the channel what happened.

The pull is to join the diagnosis. The manager knows the system and has a theory too. Instead they do three things in the first two minutes: they name the incident commander — themselves, for now, because nobody else has been designated — and say so in the channel; they ask the on-call engineer one question, whether the change deployed forty minutes ago can be rolled back, and when the answer is yes, they say roll it back now; and they post a first customer-facing update that says what is affected and that a fix is in progress, without a cause.

The on-call engineer objects: they are close to finding the bug, and a rollback will lose the evidence. The manager holds the order. Restore first. They ask the engineer to capture what they need — a snapshot of the logs, the failing request — in the next sixty seconds, and then roll back. Service returns at minute twelve. The customer update is revised. Only then does the manager open the diagnosis, and hands the incident-command role to a senior engineer so that they can step back.

Two days later the review. The product lead, who fielded the customer complaints, opens by asking who approved the deployment. The manager redirects, plainly: the review is about how the system let a change reach production without the failing path being tested, why the rollback took twelve minutes when the tooling should allow four, and why the first update took eight minutes. Those questions produce three actions with owners: a contract test for the failing path, a rollback shortcut, and a change to the runbook that names the commander before the manager has to. The deploying engineer's name is in the timeline, because it is a fact, and nowhere else.

What the manager does not do is find the bug themselves in minute nine, which they might have, and which would have taught the team that incidents are run by whoever knows the most rather than by a structure that works when that person is asleep.

• CC-1 Psychological safety and productive conflict — the safety that makes a blameless review real, and the reaction to bad news that keeps incidents reported.
• CC-5 Values, integrity, and ethical judgment under pressure — obeying the error budget when it is inconvenient; the costly right call after a public failure.
• DE-4 Managing scope, risk, and dependencies across workstreams — ownership gaps and systemic risk that surface as incidents.
• TJ-6 Security, privacy, and compliance as engineering responsibilities — security incidents, which follow the same structure with different stakes.
• SV-6 Organisational design and leading change — the structural causes of orphaned systems and chronic boundary failures.

1. Why mitigation before root cause?
   Answer

   Because customers are down and the cause can wait. The commander captures what evidence the restoration will destroy, in seconds, and then restores; diagnosis begins when service is back.
2. What has a commander who starts debugging done?
   Answer

   Vacated the role. Coordination and communication stop, and the incident reverts to everyone debugging and nobody in charge.
3. A senior stakeholder demands that the postmortem name who is at fault. How do you hold the line, and how is blamelessness different from no accountability? org
   Answer

   Run the review on how the system failed and what to change, and say plainly that names are not its purpose. Blamelessness is what makes honesty about failure safe, and therefore what makes the system fixable; patterns of recklessness are a performance matter handled separately, not in the review.
4. After a reorganisation the same fragile area keeps causing incidents and no team owns it. What is the fix? org
   Answer

   Assign explicit ownership and fund the stabilisation. Orphaned systems do not heal; every incident in one is an ownership decision deferred.
5. What does the DORA research say about the relationship between delivery speed and stability?
   Answer

   That they improve together at the level of the delivery system: the organisations with the highest throughput also had the lowest change failure rates and fastest recovery. Speed and reliability are not a trade-off at that level.
6. Why does the executive's behaviour after a public failure matter more than the incident policy? exec
   Answer

   Because everyone watches what happens after the worst incident, and that moment sets the culture. An executive who models blamelessness under maximum pressure, including owning their own contributing decisions, writes the real policy; one who hunts for a culprit erases the written one.
7. What is an error budget only as good as? exec
   Answer

   Leadership's willingness to obey it when it is inconvenient. Shipping through an exhausted budget in front of the organisation teaches everyone what the actual rule is.

• Betsy Beyer, Chris Jones, Jennifer Petoff and Niall Richard Murphy (editors), Site Reliability Engineering: How Google Runs Production Systems (2016) — incident management, service-level objectives, error budgets, postmortem practice.
• John Allspaw, Blameless PostMortems and a Just Culture (2012) — the case for blameless reviews and a just culture.
• Nicole Forsgren, Jez Humble and Gene Kim, Accelerate: The Science of Lean Software and DevOps (2018) — the research on throughput and stability improving together.