Skip to content
Strategy & Vision

What a Request Changes Count Cannot Tell You

Review tooling will happily count how often each engineer's pull requests get a request for changes. Before that number goes into a spreadsheet, decide what it would have to mean.

Strategy & VisionEngineering Management Academy6 min read

Every code host now ships review analytics. Pull requests per engineer, time to first review, and one number that looks especially telling, the share of each person's pull requests that received a "request changes" review. One engineer sits at 45 per cent while the team averages 12. It feels like you have found something.

We think what you have found is a question rather than an answer. The difference matters because the way you treat that number decides what your engineers do next.

What the number measures

Start with the mechanics. GitHub's documentation on pull request reviews describes three states a reviewer can choose, Comment, Approve and Request changes, the last of which "flags feedback that the author should address before merging." That is all the data records. A reviewer picked one of three buttons.

Which button they pick depends on habit far more than on the code. Some reviewers use Request changes for anything they want answered, including a question about naming. Others never use it, because they were taught that blocking is rude, and leave the same feedback as a comment. A team that follows Google's standard of code review, which asks reviewers to favour approving a change "once it is in a state where it definitely improves the overall code health of the system being worked on, even if the CL isn't perfect", will show very few blocked reviews. A team that uses the red button as a conversation starter will show many. Same code, same quality, different number.

The number also depends on where in the system the work happens. A change to the schema of a table three other teams read from should be sent back more often than a change to an internal admin page. The engineer who owns the migrations will carry a higher ratio for years, and that ratio is evidence of their reviewers doing their job.

Three readings of the same outlier

Suppose the 45 per cent is real and persistent. Open the pull requests before you form a view, because there are at least three stories that produce the same figure.

The first is that this engineer works in the hardest part of the codebase, with the most careful reviewers. The ratio maps risk rather than ability.

The second is that one reviewer is responsible for most of the blocks. Look at who is clicking. A single senior engineer who treats every review as a design debate will inflate the ratio of everyone they review, and the fix belongs with the reviewer.

The third is the one the dashboard wants you to assume, that the engineer is struggling and needs support. It is sometimes true. You will only know by reading the changes and the comments, and by asking the engineer what they think is going on. The metric earns you the conversation. It does not excuse you from having it.

What happens when you reward it

The dangerous move is to treat the ratio as a score. The moment engineers believe a low request-changes rate is good for them, they will get one, and not by writing better code.

They will negotiate the change privately before opening the pull request, so the review that happens on record is a formality. They will split risky work into pieces too small to object to, or avoid the hard parts of the system altogether. Reviewers, who like their colleagues, will quietly stop using the blocking state and bury the same concerns in comments. Your number improves and your review process gets worse, which is the oldest story in measurement.

Nicole Forsgren and her co-authors made the broader point in the SPACE framework, that developer productivity "cannot be measured by a single metric or dimension." The framework's name spells out five dimensions, and a review count belongs to just one of them, activity, which is the easiest to collect and the easiest to game.

The work that leaves no trace

There is a deeper problem with any pull request metric, and Juraj Masar's essay on 100x software engineering puts it well. His argument is that the most effective engineers understand the product, the business model and the customer well enough to "judge the cost/benefit ratio of a feature better than the product manager", and to cut the small slice of scope that makes the implementation dramatically simpler. In his words, the gain "is about everything an engineer doesn't have to do."

None of that appears in review data. The feature that was never built generates no pull request. The migration that was avoided by a twenty-minute conversation gets no approval. If your picture of an engineer is built from what the code host counted, the people doing the most valuable work in the room are invisible to you, and the people generating the most churn look busiest.

What we would measure instead

Measure the process at team level rather than the person. Two things are worth tracking and both come from Google's guidance on the speed of code reviews.

The first is review latency. Google's guide states that "one business day is the maximum time it should take to respond to a code review request", and observes that making the process faster resolves most complaints about it. A pull request waiting two days tells you something true about how work flows. A request-changes ratio tells you something true about button habits only.

The second is change size. Google's guidance on small changes suggests that "100 lines is usually a reasonable size for a CL, and 1000 lines is usually too large", because small changes are "reviewed more quickly" and "reviewed more thoroughly". If your median pull request is 600 lines, the blocked reviews are a symptom of that, and you can fix the cause.

Then decide, explicitly and in writing, that per-person review ratios will not appear in a performance document. Use them to pick which pull requests to read and which questions to ask in a one-to-one. The refusal is the management decision here. Dashboards will keep offering you the number. Your job is to know what it cannot say.

Where this sits in the standard

Engineering metrics and measurement is a competency in the Strategy & Vision domain of the EMA Competency Framework. The metrics unit of the Body of Knowledge covers how to choose measures that survive being watched, and the code review and quality standards unit covers what review is for in the first place.

A scenario question on this might describe a situation in which a director asks for per-engineer review statistics, and ask you to decide what to give them and what to say. See what EMA-I covers.

Filed undermetricscode reviewmeasurementengineering management
This sits under Strategy & Vision in the Competency FrameworkThe Body of Knowledge chapter covers the full set of competencies the examinations assess in this domain.Read it →

Test your judgment, not your recall.

EMA-I is a scenario-based examination in the five domains of the framework. There is no course to buy and no attendance to prove. Sit a sample first, or read what the credential covers.

Keep reading