Reliability, incident response, and operational ownership

Reliability is the property that lets users depend on a system. Incident response is what happens when, briefly, they cannot. Operational ownership is the arrangement under which the people who build a system are the people who live with it. Version 1.0 named only the middle one of these. The competency covers all of them: the on-call system, the service-level objectives that say how reliable is reliable enough, the incident in progress with its inverted priorities, and the review afterwards, which either changes the system or assigns blame. Reliability matters because it is where engineering's promises to the business are tested in public, and because the way an organisation treats failure decides whether it learns anything from it. The examinations test it because the failures are so easy to read: the engineer root-causing while customers are down, the review that hunts for a culprit, the fragile system nobody owns, and the leader who obeys the error budget until it becomes inconvenient.

1. When systems fail, priorities invert: restore, communicate, then diagnose. Mitigation comes before root cause. The cause can wait and the customers cannot.
2. Someone commands, so everyone else can fix. A defined incident-command role, decided before the incident, owns coordination and communication. A commander who starts debugging has left the role empty.
3. Incidents are failures of the system, not of people. A blameless review asks how the system allowed the error and what to change, and without it problems do not surface at all. Blamelessness does not remove accountability. It is what makes real accountability possible, meaning an honest fix to the system.
4. A review that changes nothing is a ritual. Postmortem actions are tracked until they are done, and a review is measured by the recurrences it prevents. The documents it produces do not count.
5. The team that ships it runs it. Operational ownership lines incentives up behind reliability, whereas a team that throws code over a wall has no reason to make it reliable. Orphaned systems do not heal, and every incident in one is an ownership decision someone has put off.
6. Reliability is a stated objective and a spendable budget. A service-level objective says how reliable is enough. The error budget turns the argument between reliability and velocity into a rule agreed in advance, and the rule is only as good as leadership's willingness to obey it when that is inconvenient.

Reliability engineering has a well-documented practitioner literature and one substantial body of research, and the unit grades its models accordingly.

Incident command practice

One person owns coordination and communication during an incident so that the responders can concentrate on the fix, and the priorities while it lasts are to restore service, communicate and diagnose, in that order. The structure is adapted from emergency-services practice, and Betsy Beyer, Chris Jones, Jennifer Petoff and Niall Richard Murphy (editors), Site Reliability Engineering: How Google Runs Production Systems (2016) sets it out for software. It is decided before the incident, because once one is under way it is too late. At organisational scale, a cross-team incident needs a commander with authority across all the teams involved. Incident command coordinates people. It is no technical rescue.

Blameless postmortems practice

A blameless postmortem asks how the system allowed the failure and what should change in process, tooling or guardrails, and it does not ask whom to blame. Its influential statement is John Allspaw, Blameless PostMortems and a Just Culture (2012), and its operational form is in Betsy Beyer, Chris Jones, Jennifer Petoff and Niall Richard Murphy (editors), Site Reliability Engineering: How Google Runs Production Systems (2016). The mechanism is that honesty about failure becomes safe, which makes reliability something the team can improve. Blameless does not mean free of consequences: patterns of recklessness are performance matters, handled separately through PL-4 Performance management. A review whose actions die in a backlog has produced a document and fixed nothing.

Service-level objectives and error budgets practice

Betsy Beyer, Chris Jones, Jennifer Petoff and Niall Richard Murphy (editors), Site Reliability Engineering: How Google Runs Production Systems (2016) proposes an explicit reliability target, with the tolerable shortfall against it (the error budget) treated as a spendable resource that gates risk. With budget left, the team ships. With the budget exhausted, it stabilises. This turns a recurring argument into a shared rule agreed in advance. The limit of the model is entirely cultural. If leaders ship through an exhausted budget in front of the whole organisation, everyone learns what the real policy is, which is why the model comes up again in CC-5 Values, integrity, and ethical judgment under pressure.

DORA stability metrics research

The research programme reported in Nicole Forsgren, Jez Humble and Gene Kim, Accelerate: The Science of Lean Software and DevOps (2018) ran large surveys over several years and found that the organisations with the highest delivery throughput also had the highest stability. Change failure rate and time to restore service improve together with deployment frequency, and do not pull against it. This finding matters to the unit in two ways. At the level of the delivery system, reliability and speed are not traded against each other, and time to restore improves precisely where it is safe to be honest about failure. The metrics measure the system at team level. They are never individual measures and never a leaderboard.

The incident runbook, decided in calm

Before any incident, the team settles who commands, how the role is handed over, where communication happens, what the first three actions are, and what "restored" means for each critical service. A team that has to settle these during an outage settles them badly and late.

The first fifteen minutes

Declare the incident, name the commander, mitigate (roll back, fail over, shed load) and post a first customer-facing update, all before anyone looks for the cause. The engineer who wrote the change wants to find the bug, and the commander's job is to stop them until service is back.

The postmortem, run as designed

The review happens within days, and weeks is too late. It is blameless in fact as well as in name. It concentrates on the timeline, the contributing conditions, and what would have caught the problem earlier, and it ends in actions with owners and dates. Those actions go wherever the team's real work goes and are reviewed until they are closed. A recurrence means the review itself needs reviewing.

The ownership map

Every production system has a named owning team, recorded where anyone can find it, with on-call and operational responsibility attached. After any reorganisation the map is the first thing to check, because the systems that fall between the new boundaries are where the next incident will happen.

Reading on-call load

On-call is a workload and gets measured: pages per shift, hours lost from sleep, time spent on toil. When a team's on-call load is rising, its reliability debt is compounding. The manager reads that signal at team scope, and at organisational scope it becomes a trend that someone owns.

At team scope the object is an incident in progress and the operational health of one service. The manager runs the incident, defends the blameless review, and treats reliability as a deliverable instead of an interruption. At organisational scope the object becomes reliability across services, the ownership gaps a reorganisation left behind, and the on-call system as a whole. The characteristic moves are to install cross-team incident command instead of joining the debugging, to hold the blameless norm against a stakeholder who wants names, and to close the ownership gaps that make orphaned systems fail. At executive scope reliability is a strategic property, covering investment, risk appetite and what the business is implicitly assuming, with the trend owned at organisational level. The part only the executive can play is what they are seen to do after the most public failure. How Judgment Scales sets out the pattern, and SV-6 Organisational design and leading change covers the structural causes of chronic ownership gaps.

• Restores service, communicates, then diagnoses, and stops the author of the change from root-causing while customers are down. team
• Establishes who is commanding before the incident, and gets out of the way of the fix during it. team
• Runs the postmortem blamelessly and tracks its actions to completion. team
• Treats reliability as a first-class deliverable the team owns, and not as an interruption to real work. team
• Failure mode: root-causing during the outage. team
• Failure mode: the "who broke it?" review. team
• Failure mode: treating recurring incidents as bad luck. team
• Failure mode: building teams that throw code over a wall. team
• Failure mode: a postmortem measured by the document it produced.
• Installs a single incident commander with cross-team authority for a cross-team incident, instead of becoming the de facto commander for every incident. org
• Holds the blameless review against a senior stakeholder who demands names, and explains that blamelessness is what makes real accountability possible. org
• Assigns explicit ownership and funds stabilisation for a fragile area a reorganisation left orphaned. org
• Designs an on-call system across several teams that is sustainable, and reads reliability as a trend across services. org
• Failure mode: letting a powerful stakeholder turn a review into a trial. org
• Failure mode: leaving fragile systems orphaned because reassignment is awkward. org
• Failure mode: treating operational load as each team's private problem. org
• Makes the organisation's reliability trend visible and owned, with investment behind it instead of exhortation. exec
• Argues reliability at the executive table as a strategic property: investment, risk appetite, the resilience the business is assuming. exec
• Models blamelessness after the most public failure, including owning the executive-level decisions that contributed. exec
• Obeys the error budget when it is inconvenient, because the whole organisation is watching what the real policy is. exec
• Failure mode: discovering reliability decay only when customers do. exec
• Failure mode: letting the worst incident become the moment blamelessness died. exec
• Failure mode: funding features while reliability debt compounds. exec

Restoring versus understanding. Every engineer in the incident wants to know why, and the customers want the service back. The order is fixed, restore first, and the difficulty is that restoring sometimes destroys the evidence of the cause. The commander decides what to capture before the rollback, and has minutes to decide it.

Blamelessness versus accountability. A review that names nobody can look like one that holds nobody responsible. The answer is to keep the review, which is about the system, apart from performance management, which deals with patterns of behaviour and happens separately. Mixing the two ruins both.

Reliability versus velocity. In a single decision the two compete. Across the delivery system the evidence says they improve together. The error budget stops the daily argument from being fought again each day, and it works only if the people who set it obey it.

Ownership versus load. Owning what you ship lines up incentives, and it can also bury a team under on-call for systems it inherited. The judgment lies in reading on-call as a measured workload and treating a rising load as debt to fund, not as the team's private problem.

Structure versus heroics. The manager who knows the system best is the one most tempted to command every incident, and each incident they command is one the team does not learn to run. The structure exists to make the manager unnecessary. A manager who resists it has turned themselves into the single point of failure.

A payment service has been down for eight minutes and checkout is failing for every customer. The on-call engineer, who deployed a change forty minutes ago, has the logs open and is reading stack traces. Three other engineers have joined the channel, each proposing a theory. The manager arrives. Nobody has posted a customer-facing update or said who is coordinating, and the product lead has just asked in the channel what happened.

The manager knows the system and has a theory of their own, so the temptation is to join the diagnosis. They do three other things in the first two minutes. They name the incident commander, themselves for now because nobody else has been designated, and say so in the channel. They ask the on-call engineer a single question, whether the change deployed forty minutes ago can be rolled back, and when the answer is yes they say to roll it back now. And they post a first customer-facing update saying what is affected and that a fix is in progress, with no cause given.

The on-call engineer objects that they are close to finding the bug and that a rollback will lose the evidence. The manager holds the order: restore first. They ask the engineer to capture what they need, a snapshot of the logs and the failing request, in the next sixty seconds and then roll back. Service returns at minute twelve and the customer update is revised. Only then does the manager open the diagnosis, handing the incident-command role to a senior engineer so that they can step back.

The review takes place two days later. The product lead, who handled the customer complaints, opens by asking who approved the deployment. The manager redirects plainly. The review is about how the system let a change reach production without the failing path being tested, why the rollback took twelve minutes when the tooling should allow four, and why the first update took eight minutes. Those questions produce three actions with owners: a contract test for the failing path, a rollback shortcut, and a runbook change that names the commander before the manager has to. The deploying engineer's name appears in the timeline, because it is a fact, and nowhere else.

The manager does not find the bug themselves in minute nine, though they might have managed it. Doing so would have taught the team that incidents are run by whoever knows most, when they should be run by a structure that still works while that person is asleep.

• CC-1 Psychological safety and productive conflict: the safety that makes a blameless review real, and the reaction to bad news that keeps incidents reported.
• CC-5 Values, integrity, and ethical judgment under pressure: obeying the error budget when it is inconvenient; the costly right call after a public failure.
• DE-4 Managing scope, risk, and dependencies across workstreams: ownership gaps and systemic risk that surface as incidents.
• TJ-6 Security, privacy, and compliance as engineering responsibilities: security incidents, which follow the same structure with different stakes.
• SV-6 Organisational design and leading change: the structural causes of orphaned systems and chronic boundary failures.

1. Why mitigation before root cause?
   Answer

   Customers are down and the cause can wait. The commander takes a few seconds to capture whatever evidence the restoration will destroy, then restores. Diagnosis begins once service is back.
2. What has a commander who starts debugging done?
   Answer

   Left the role empty. Coordination and communication stop, and the incident goes back to everyone debugging with nobody in charge.
3. A senior stakeholder demands that the postmortem name who is at fault. How do you hold the line, and how is blamelessness different from no accountability? org
   Answer

   Run the review on how the system failed and what to change, and say plainly that naming people is not its purpose. Blamelessness makes honesty about failure safe, and that is what makes the system fixable. Patterns of recklessness are a performance matter, handled separately and outside the review.
4. After a reorganisation the same fragile area keeps causing incidents and no team owns it. What is the fix? org
   Answer

   Assign explicit ownership and fund the stabilisation. Orphaned systems do not heal, and every incident in one is an ownership decision put off.
5. What does the DORA research say about the relationship between delivery speed and stability?
   Answer

   At the level of the delivery system they improve together. The organisations with the highest throughput also had the lowest change failure rates and the fastest recovery, so at that level speed and reliability are not traded against each other.
6. Why does the executive's behaviour after a public failure matter more than the incident policy? exec
   Answer

   Everyone watches what happens after the worst incident, and that moment sets the culture. An executive who models blamelessness under maximum pressure, including owning their own contributing decisions, writes the real policy. One who hunts for a culprit erases the written one.
7. What is an error budget only as good as? exec
   Answer

   Leadership's willingness to obey it when it is inconvenient. Shipping through an exhausted budget in front of the organisation teaches everyone what the rule is in practice.

• Betsy Beyer, Chris Jones, Jennifer Petoff and Niall Richard Murphy (editors), Site Reliability Engineering: How Google Runs Production Systems (2016): incident management, service-level objectives, error budgets, postmortem practice.
• John Allspaw, Blameless PostMortems and a Just Culture (2012): the case for blameless reviews and a just culture.
• Nicole Forsgren, Jez Humble and Gene Kim, Accelerate: The Science of Lean Software and DevOps (2018): the research on throughput and stability improving together.