Engineering metrics and measurement

Metrics make engineering legible and improvable, and the wrong metrics do active harm. v1.0 framed the competency as setting and using meaningful measures. v3.0 adds interpretation and the measurement of AI impact, because the pressure to show a number is now strongest exactly where the numbers are least settled. The competency starts with choosing a small, balanced set of measures that describe the delivery system and leave the people in it alone. Those measures are read as trends that open a conversation, never as verdicts. The manager refuses the uses that corrupt them: individual productivity scores, leaderboards across teams, a single number for the board. At executive scope the competency also means declaring in advance how the organisation will know whether its strategy is working. Every metric becomes a target as soon as it is tied to a reward or a comparison, and growth that is not measured cannot be told apart from bloat. The examinations test this through the productivity score per engineer, the league table of teams, the metric that improved while nothing else did, and the AI investment that has to show a return the honest number does not yet support.

1. Measure the system, not the people. The validated delivery measures describe the team's delivery system, its throughput and stability. They are never individual productivity scores.
2. A measure that becomes a target stops measuring. Activity metrics such as lines of code, commit counts and story points per engineer are the easiest to game and do the most damage when turned into weapons. Work out how a measure will be gamed before attaching a target to it.
3. Small, balanced, and paired with context. Keep a few measures that span throughput and stability, so that improving one cannot quietly wreck another, and put the numbers next to the qualitative signals that explain them.
4. Trend a team against itself; never rank teams against each other. Contexts differ, story points are not a currency, and a ranking teaches every team to optimise the number.
5. Instrument the system between teams. Above team scope, the thing only the manager owns is the set of queues, handoffs and dependencies between teams. Measure those across the whole group instead of measuring each team's internals one by one.
6. Declare the success measure when the strategy is set. A strategy with no falsifiable measure cannot be shown to be failing, and its "success" will be judged by the effort spent on it. Report AI's effect on the same terms: honestly, with the uncertainty stated, including where the effect is negative.

No area of this standard has more research behind it than measurement, or more misuse of that research, and the unit grades both carefully.

DORA metrics research

The research programme in Nicole Forsgren, Jez Humble and Gene Kim, Accelerate: The Science of Lean Software and DevOps (2018) validated four delivery measures. Deployment frequency and lead time for changes measure throughput; change failure rate and time to restore service measure stability. It also found that in high-performing organisations the two move together and do not trade off. They are deliberately team-level measures of the delivery system, meant for trending a team against its own history. Used to measure individuals, or turned into a leaderboard across teams, they become targets and the research no longer applies.

The SPACE framework practice

Nicole Forsgren, Margaret-Anne Storey, Chandra Maddila, Thomas Zimmermann, Brian Houck and Jenna Butler, The SPACE of Developer Productivity: There's More to It Than You Think (2021), written by several of the same researchers, argues that no single metric can capture developer productivity. It proposes five dimensions to be measured in balance: satisfaction and wellbeing, performance, activity, communication and collaboration, and efficiency and flow. In practice it works as a checklist against one-dimensional measurement. Any proposed productivity metric that sits in only one dimension, and above all one that sits in activity alone, is incomplete by construction. The paper proposes a framework and reports no empirical finding, and it is graded on that basis.

Goodhart's law practice

In the form given by Marilyn Strathern, 'Improving Ratings': Audit in the British University System (1997): when a measure becomes a target, it ceases to be a good measure. This is the hazard governing everything in the unit, at every scope. It shows up as the engineer who games a personal metric, the team that optimises a leaderboard, and the organisation whose activity dashboard looks healthy while outcomes stay flat. The law does not say stop measuring. It argues for balanced sets, for measures anchored in outcomes, and for treating metrics as the start of a conversation instead of a verdict.

Measuring AI impact contested

DE-5 Designing how work flows introduces the evidence. In a controlled experiment, Sida Peng, Eirini Kalliamvakou, Peter Cihon and Mert Demirer, The Impact of AI on Developer Productivity: Evidence from GitHub Copilot (2023) found that a defined task was completed substantially faster with an AI assistant. In a randomised trial, Joel Becker, Nate Rush, Elizabeth Barnes and David Rein (METR), Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (2025) found experienced developers working slower in their own repositories while believing they were faster. The lesson for measurement is specific. Self-reported speed-up is not a measure. Adoption and licence counts are not measures either, and pull-request counts measure activity. The effect has to be measured in the organisation's own conditions, against outcomes, with the uncertainty stated. The honest report to a board may be "not yet, and here is why".

The falsifiable scorecard practice

This is a practice, not a named model, and the unit treats it as the executive's answer to "is the strategy working?". It is a small, stable scorecard declared when the strategy is set. It covers the business outcomes the investment was meant to move, leading indicators chosen in advance, trends in the delivery system at organisation level, and organisational health. It is reviewed on a cadence, revised whenever the strategy is, and able to prove the strategy wrong. It fails in two ways: the dashboard swells until nobody reads it, or a board meeting discovers that "working" was never defined.

The team's four numbers

Deployment frequency, lead time, change failure rate and time to restore are trended monthly against the team's own history and read in the retrospective alongside the team's account of what happened. The numbers find the problems, and the conversation explains them. Nobody's name appears next to any of them.

The gaming pre-mortem

Before tying any metric to a target, a reward or a comparison, the manager asks how a reasonable person could hit the number without doing the thing it stands for. If that would be easy, the target is withdrawn, or the measure is paired with another that would expose the gaming.

The between-teams instruments

A group of teams needs measures of the system that connects them: queue times at shared dependencies, handoff latency, and how long cross-team requests wait. These instruments belong to the manager of managers. They show capacity drains and platform decay before delivery does.

Measuring AI in the team's conditions

A rollout of AI assistance is measured against outcomes such as cycle time, change failure rate and rework, in the organisation's own codebases. The period has to be long enough to show effects on understanding as well as on speed. Self-reports are collected, but they are not the measure. The report states the uncertainty.

The scorecard, declared in advance

When a strategy or a major investment is set, the executive writes down how the organisation will know whether it is working. That means the outcomes, the leading indicators, the trends in delivery and organisational health, and the date of the first review. The scorecard fits on one page, and that page is what the board sees.

At team scope the manager looks after one team's metrics and what they hide. They measure the delivery system, pair throughput with stability and refuse to score individuals. At organisational scope a new temptation appears: comparing teams. The manager's task becomes finding measures that aggregate or compare without inviting the gaming that comparison creates. Teams are trended against themselves, the system between them is instrumented, and measured evidence goes into every argument about capacity. At executive scope the question is whether the strategy is working. The executive chooses the measures the business uses to judge engineering, picking ones the organisation can hit without being made worse; declares a falsifiable scorecard in advance; and reports AI's effect honestly to the board. The pattern is in How Judgment Scales, and the flow the measures describe is covered in DE-5 Designing how work flows.

• Measures delivery outcomes at team level and pairs throughput with stability. team
• Uses metrics to find problems, never to rank people. team
• Anticipates gaming before attaching a target, and stays alert to it afterward.
• Declines to use a single productivity number for individuals, and can say why.
• Failure mode: individual output treated as "productivity". team
• Failure mode: chasing one metric until another breaks. team
• Failure mode: metrics as surveillance. team
• Failure mode: numbers without context. team
• Trends each team against itself (cycle time and its direction, predictability, change failure rate) and never puts teams on a leaderboard. org
• Instruments the system between teams: queues, handoffs, dependencies. org
• Brings measured evidence to every argument about capacity, drains and platform decay. org
• Measures the effect of AI assistance honestly across teams, in their own conditions, including where it is not helping. org
• Failure mode: velocity leaderboards across teams. org
• Failure mode: normalising story points into a fake currency. org
• Failure mode: measuring team internals while the queues between teams go dark. org
• Failure mode: an AI rollout where the only number that moved is pull-request count. org
• Declares the strategy's success criteria when the strategy is set, and keeps the scorecard small and stable. exec
• Chooses the measures by which the business judges engineering, and refuses any measure the organisation would be made worse by hitting. exec
• Reports AI's effect to the board with its uncertainty, including a negative or not-yet result. exec
• Treats growth without measured outcomes as indistinguishable from bloat, and fixes the measurement instead of the narrative. exec
• Failure mode: declaring victory on effort expended. exec
• Failure mode: a strategy with no falsifiable success measure. exec
• Failure mode: dashboard sprawl. exec
• Failure mode: a single productivity number for the board. exec

Legibility versus distortion. Every measure that makes engineering legible to the business also invites the organisation to optimise it. A balanced set and the gaming pre-mortem deal with this. Giving up measurement does not.

Comparison versus context. Leadership wants to compare teams, and comparison is what turns a measure into a target. Trending each team against itself tells leadership what it needs to know, whether this team is improving, without a leaderboard.

Evidence versus timing. Measuring an AI investment honestly takes longer than the quarter in which the board wants to see a return. The judgment is to report the uncertainty instead of the convenient number, and to set the review date at the moment the investment is made.

Simplicity versus completeness. A single number is what executives ask for and what corrupts fastest. A full dashboard is complete and goes unread. The one-page scorecard, declared in advance, is the compromise that survives both pressures.

Measuring versus trusting. A manager who instruments everything signals distrust, and one who instruments nothing manages by anecdote. The line falls between the system and the people. Measure the delivery system thoroughly and never measure the individual.

A senior manager leads six teams. Their vice-president asks for a monthly league table ranking the teams by velocity, so that leadership can see which teams are performing. Two of the teams work on a mature product with heavy compliance requirements. Two work on a new product with none. Two are platform teams whose output the other four consume. The vice-president's aim is reasonable, since they cannot tell which teams are in trouble, and the request is specific.

Producing the table is the easy answer, and it would damage all six teams. Story points are not a currency that works across teams. The contexts differ so much that the ranking would measure the product instead of the team. Within a month every team would learn to optimise the number, which is the one thing leadership does not want. Refusing the request is the other wrong answer. It leaves the vice-president unable to see what they need to see, and it looks like engineering resisting accountability.

The senior manager answers the need behind the request. Within a month they propose and build a view that gives leadership what it wants. Each team is trended against itself on cycle time, predictability (committed against delivered) and change failure rate. A team whose cycle time is rising or whose predictability is falling becomes visible without being ranked against a team in a different context. Next to this sit measures of the system between the teams: how long requests to the platform teams wait, and how often cross-team dependencies slip. The trouble the vice-president suspects usually lives there, and that is the part of the system the senior manager alone owns.

They also show the vice-president, using two months of history, what the league table would have done. By velocity the "worst" team was the compliance team, whose cycle time was flat and whose change failure rate was the lowest of the six. The "best" was a new-product team whose predictability had collapsed. The table would have rewarded the team in trouble and punished the one performing well.

The vice-president accepts the trended view. The senior manager neither produced a ranking for the sake of appearing to comply nor refused and left the need unmet.

• DE-5 Designing how work flows: the flow the measures describe, and Goodhart's law in the operating cadence.
• DE-3 Reliability, incident response, and operational ownership: the stability measures, and the research showing throughput and stability move together.
• SV-1 Engineering strategy: the strategy whose success measure is declared when it is set.
• SV-2 Communicating and partnering across functions: bringing measured evidence into the conversation with the board.
• TJ-5 AI-assisted engineering: the AI adoption whose effect is being measured.

1. Why are the DORA metrics team-level rather than individual?
   Answer

   Because they describe the delivery system's throughput and stability, not a person's output, and the research that validates them applies to that use. Used as individual measures they become targets and stop measuring.
2. What question is asked before any metric is attached to a target?
   Answer

   How would a reasonable person hit this number without doing the thing it stands for? If that would be easy, withdraw the target or pair the measure with one that would expose the gaming.
3. Why is trending a team against itself sound where ranking teams against each other is not? org
   Answer

   A trend answers "is this team improving?" within the team's own context. A ranking compares contexts that differ, treats story points as a currency, and teaches every team to optimise the number.
4. What does a manager of managers uniquely own, and therefore measure? org
   Answer

   The system between teams: queues at shared dependencies, handoff latency, and cross-team requests waiting. That is where capacity drains and platform decay show up before delivery suffers.
5. An AI-tooling rollout must show a return and the only number that moved is pull-request count. What is the honest position? org
   Answer

   Pull-request count measures activity, not outcome. Measure against cycle time, change failure rate and rework in the organisation's own conditions, over a period long enough to see effects on understanding. Collect self-reports without treating them as the measure, and report the uncertainty.
6. What does it mean for a strategy to have a falsifiable success measure, and what happens at the board when it does not? exec
   Answer

   It means outcomes and leading indicators declared when the strategy is set, able to show that the strategy is failing. Without them, "working" is judged by effort expended, and the board finds out that success was never defined. At that point growth cannot be told apart from bloat.
7. The board wants a single number for engineering productivity. What is the answer? exec
   Answer

   A one-page scorecard declared in advance, covering strategic outcomes, delivery-system trends and organisational health, in place of a single number. Give the reason: any single number would make the organisation worse to hit.

• Nicole Forsgren, Jez Humble and Gene Kim, Accelerate: The Science of Lean Software and DevOps (2018): the DORA delivery metrics and the research behind them.
• Nicole Forsgren, Margaret-Anne Storey, Chandra Maddila, Thomas Zimmermann, Brian Houck and Jenna Butler, The SPACE of Developer Productivity: There's More to It Than You Think (2021): the SPACE framework; productivity is not one number.
• Marilyn Strathern, 'Improving Ratings': Audit in the British University System (1997): Goodhart's law in its standard form.
• Sida Peng, Eirini Kalliamvakou, Peter Cihon and Mert Demirer, The Impact of AI on Developer Productivity: Evidence from GitHub Copilot (2023): the controlled experiment on AI-assisted task completion.
• Joel Becker, Nate Rush, Elizabeth Barnes and David Rein (METR), Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (2025): the randomised trial on experienced developers with AI tools.