AI-assisted engineering

AI-assisted engineering is the use of large-model tools in how software is written, reviewed, tested, operated and understood, and the competency is the manager's judgment about it: how a team or an organisation adopts the assistance, governs its use, builds the capability to use it well, and manages the risks it introduces. No v1.0 competency covered it; the previous edition of this standard mentioned AI as an example of a transformative-but-unproven technology and left it there. It has its own unit now because it is no longer optional — engineers use the tools whether or not anyone decided — and because the evidence about what it does is genuinely mixed, which makes it the place in this standard where the difference between a considered judgment and a fashionable one is widest. It matters because the tools change what review must establish, what a hiring interview measures, what a productivity metric means, and what an engineer understands about the system they own; and because the same tools can produce an organisation that ships faster and one that no longer understands its own code, depending on decisions a manager makes or fails to make. The examinations test it through the chief executive who asks whether to adopt everywhere, the engineer who ships code they cannot explain, the rollout measured by licences, and the board that expects the tools to cut headcount.

1. Decide the adoption; do not let it happen. What the tools are used for, what they may not touch, and what changes about review, testing and ownership are decided explicitly, before or during adoption, not discovered after an incident.
2. Ownership does not move. Whoever submits a change owns it and can explain it, whatever produced the text. Verification is the submitter's, and disclosure is required where the organisation says so.
3. Govern proportionately to risk. What must be verified, which data may be sent to which tools, what is permitted on which paths. A documentation change and a payment path are not governed alike, and blanket prohibition is as much a failure as no governance.
4. Build the capability, not just the licence count. Using the assistance well is a skill; the organisation invests in it, shares what works, and treats a team whose engineers can no longer explain their own service as an early warning, not a productivity gain.
5. Measure in your own conditions and report honestly. The evidence is mixed and self-reports are unreliable. The effect is measured against outcomes in the organisation's own codebases, and the answer to a board may be "not yet, and here is why".
6. Size the bet to teach. A transformative but unproven capability gets a bounded bet — real enough to build capability and evidence, with checkpoints to double down or fold — funded by explicit subtraction, not sprinkled on everyone's full plate.

This unit is built around the fact that its evidence disagrees, and it grades that disagreement rather than resolving it by preference.

AI-assisted developer productivity contested

The two kinds of evidence introduced in DE-5 Designing how work flows. The controlled experiment in Sida Peng, Eirini Kalliamvakou, Peter Cihon and Mert Demirer, The Impact of AI on Developer Productivity: Evidence from GitHub Copilot (2023) found a defined task completed substantially faster with an assistant. The randomised trial in Joel Becker, Nate Rush, Elizabeth Barnes and David Rein (METR), Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (2025) found experienced developers working in their own mature repositories slower with the tools, while believing they were faster. The studies measured different things, and the honest management position is that the effect depends on the work, the codebase and the engineer, that perceived speed-up is not a measure, and that the effect must be established locally. A manager who cites only the study that supports their preference has failed this unit's central test.

The ironies of automation practice

The analysis in Lisanne Bainbridge, Ironies of Automation (1983) of what happens when a task is automated: the human is left with the parts the automation cannot do, which are the hardest; the skills needed to intervene when the automation fails decay because they are no longer practised; and the operator is asked to monitor a system they increasingly do not understand. Written about industrial control, it describes the risk this unit is most concerned with — an engineering organisation whose engineers can no longer explain, debug or extend the code the tools produced — and it explains why capability-building and review for understanding are not optional overheads.

Security of AI-generated code research

The finding in Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt and Ramesh Karri, Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions (2022) that a substantial share of assistant-generated code across security-relevant scenarios contained known vulnerability classes. The proportion changes with tools and models; the governance consequence does not: generated code is checked as a class, through scanning and review aimed at the known failure modes, and the paths where a vulnerability is most costly get the most verification. The security treatment is in TJ-6 Security, privacy, and compliance as engineering responsibilities.

Proportionate AI governance practice

Not a named model but the standard this unit proposes: an explicit, short policy stating what the tools may be used for, what data may be sent where, what verification and disclosure each class of change requires, and who owns a change regardless of what produced it — scaled to the risk of the path, reviewed as the tools change, and enforced in the pipeline where it can be. Its two failures are symmetrical: blanket prohibition, which is ignored, and no policy, which is discovered in an incident.

Staged bets and checkpoints practice

The practice from SV-3 Roadmapping, applied to the capability itself: a transformative and unproven technology gets a bounded bet — enough to build real capability and learn fast — with explicit checkpoints to double down or fold, and it is funded by clearing something out rather than adding it to every team's plate. Too small to matter and too big to survive being wrong are both errors.

The adoption decision, written

Before or at adoption, one page: what the tools are for, what they may not touch, what data may be sent to them, what review and testing now require (the checklist in TJ-3 Code review and quality standards), what disclosure is expected, and how the effect will be measured. Revisited quarterly, because the tools change faster than most policies.

The understanding check

At intervals, an engineer is asked to explain a recent change in their own area in their own words — how it works, what it assumes, what would break it. Not as a test of the person but as a measure of the team: a team where the answers are getting thinner is a team whose understanding is eroding, and it is the earliest signal this unit knows.

Capability-building

Time is set aside for engineers to share what works: which tasks the assistance helps with, which it does not, how to verify what it produces, how to keep the system in one's head while using it. The organisation treats fluency with the tools as a skill to be developed, not a licence to be issued.

Measuring in local conditions

The rollout is measured against outcomes — cycle time, change failure rate, rework, incident trends — in the organisation's own codebases, over long enough to see effects on understanding as well as speed. Self-reports are collected and not treated as the measure. The report states the uncertainty, and it is the report that goes upward.

The bounded bet

For a capability that could transform the product and is unproven: a defined scope, a team that owns it, a budget that came from something explicitly stopped, and checkpoints with criteria for doubling down or folding. The bet is reviewed on its evidence, not its excitement.

Governance across teams

At organisational scope, the policy is set once for the seams — data, disclosure, verification on shared or high-risk paths — and teams choose their tools inside it. Adoption is read across teams for unevenness: the team that is dangerous and the team that has not started both need attention, for different reasons.

At team scope the object is how the team uses the assistance and the review it needs, and the manager introduces it with explicit expectations, watches for erosion of understanding, and distinguishes where it genuinely helps from where it is used to look fast. At organisational scope the object becomes adoption and governance across teams — proportionate rules at the seams, capability built deliberately, uneven adoption read and addressed — and the measurement of the effect in the department's own conditions. At executive scope the object is the organisation's strategy for AI-assisted engineering: investment, governance, capability and risk as a strategic bet; what changes about the work and the workforce, stated honestly; the evaluation of an emerging capability as a bounded bet; and the conversation with a board that expects the tools to reduce headcount. The pattern is in How Judgment Scales; the economics of the AI option are SV-4 Engineering economics.

• Introduces AI assistance to the team with explicit expectations for verification, ownership and disclosure. team
• Recognises the risks the assistance introduces — unverified code, leaked data, eroded understanding — and puts proportionate controls in place. team
• Distinguishes where the assistance genuinely helps the team from where it is being used to look fast. team
• Treats an engineer who cannot explain the code they shipped as a signal about the review and the adoption, not as a personal failing.
• Failure mode — letting adoption happen without a decision. team
• Failure mode — blanket prohibition, which is ignored. team
• Failure mode — measuring the adoption by how fast pull requests arrive. team
• Failure mode — citing the study that supports your preference and not the one that does not.
• Governs AI-assisted work across teams proportionately: what is permitted, what must be verified, what data may be used, scaled to the risk of the path. org
• Builds the capability to use the assistance well across the department, and shares what works. org
• Reads uneven adoption across teams and addresses both the dangerous and the absent. org
• Recognises a team whose engineers can no longer explain their own service as an early warning of eroded understanding. org
• Measures the effect in the department's own conditions and reports it honestly, including where it is not helping. org
• Failure mode — a department-wide rollout measured by licences used. org
• Failure mode — governance that prescribes every team's tools instead of the rules at the seams. org
• Failure mode — treating a productivity gain as real because engineers say it is. org
• Sets the organisation's strategy for AI-assisted engineering — investment, governance, capability, risk — as a strategic bet, and says what changes about the work and the workforce. exec
• Evaluates an emerging AI capability that could be transformative and is unproven as a bounded bet with checkpoints, funded by explicit subtraction. exec
• Answers a board that expects the tools to reduce headcount with the evidence, the uncertainty, and what the organisation actually knows. exec
• Reports AI's effect to the board honestly, including a negative or not-yet result. exec
• Failure mode — committing the organisation to a headcount plan the evidence does not support. exec
• Failure mode — sprinkling the existential capability across everyone's full plate. exec
• Failure mode — an AI strategy that is a tools purchase. exec
• Failure mode — stampede or prohibition, in place of a bounded bet. exec

Speed versus understanding. The tools can raise output and lower the team's understanding of its own systems, and the second effect is slower to show than the first. The judgment is to hold review to understanding, measure understanding as well as throughput, and accept that some of the apparent speed was borrowed.

Governance versus adoption. Rules strict enough to eliminate risk are ignored; rules loose enough to be followed leave risk. Proportionality is the resolution — the payment path and the documentation change are not governed alike — and the rules are enforced where they can be, in the pipeline.

Evidence versus urgency. The board wants a return this quarter and the honest measurement takes longer. The judgment is to report the uncertainty rather than the convenient number, and to have set the review date when the investment was made.

Bet versus caution. A capability that could transform the product is also unproven; too small a bet learns nothing and too large a bet cannot survive being wrong. The bounded bet with checkpoints is the resolution, and the discipline is in honouring the checkpoints.

Workforce honesty versus reassurance. What the tools change about the work is uncertain and the people doing the work are anxious about it. Saying what the organisation knows and does not know, including about headcount, is harder than reassurance and is the only version that survives the next quarter.

A head of engineering is asked by the board, at a quarterly meeting, to present a plan for reducing engineering headcount by a third over two years on the strength of AI assistance. The chair cites a vendor's productivity figures and a competitor's public statement. The chief executive is sympathetic to the board and has asked for the plan in a month. The organisation adopted assistants a year ago; adoption is uneven, the measured effects are mixed, and two teams have had incidents traced to generated code nobody understood.

The two easy responses are to produce the plan, because the board asked and the chief executive wants it, or to refuse, because the evidence does not support it. The first commits the organisation to a headcount trajectory built on a vendor's number; the second reads as engineering resisting the future, and it is not even the honest answer, because the honest answer is more specific than no.

The head of engineering brings the evidence, in the board's terms. The vendor's figure comes from a controlled experiment on a defined task; the strongest independent trial found experienced developers slower in real codebases while believing they were faster; the organisation's own measurements over a year show cycle-time gains on some kinds of work, no gain on others, two incidents from unverified generated code, and an erosion of understanding in two teams that took a quarter to repair. The head of engineering says plainly what the organisation knows: the tools help with some work, the effect is not yet a headcount figure, and a plan built on a third is a plan built on a number nobody has measured here.

They then offer what a board can act on. A strategy for AI-assisted engineering as a bounded, staged bet: governance at the seams, capability built deliberately across teams, a defined measurement of effect on outcomes over the next three quarters, and two checkpoints at which the board will see the organisation's own numbers. They say what changes about the work — review, testing, the shape of some roles — and that any change to headcount will follow the measured effect rather than precede it. They also say that if the effect turns out to be large, the organisation should be the first to know and the board will hear it from them.

The board is not fully satisfied, and the chair says so. The chief executive backs the staged plan, because it is one the organisation can be held to. What the head of engineering does not do is put a headcount number in front of the board that the evidence did not support, which would have been the easiest thing to say and the thing the organisation would have spent two years failing to deliver.

• TJ-3 Code review and quality standards — what review must establish for generated code; the standards floor.
• DE-5 Designing how work flows — AI assistance as part of how work flows; the contested evidence introduced.
• SV-5 Engineering metrics and measurement — measuring AI impact honestly and reporting it upward.
• SV-4 Engineering economics — the AI service as a third option beside build and buy.
• TJ-6 Security, privacy, and compliance as engineering responsibilities — the security profile of generated code, and data sent to tools.
• PL-3 Hiring and onboarding — assessing capability rather than tool access in hiring.

1. What is the honest management position on the evidence for AI-assisted productivity?
   Answer

   That it is mixed: a controlled experiment on a defined task found a large speed-up; a randomised trial on experienced developers in their own repositories found a slowdown they did not perceive. The effect depends on the work, the codebase and the engineer, perceived speed-up is not a measure, and the effect must be established locally.
2. What do the ironies of automation predict for an engineering organisation, and what practice answers them?
   Answer

   That engineers will be left with the hardest parts, that the skills to intervene when the automation fails will decay, and that they will be asked to own systems they increasingly do not understand. Review for understanding, the understanding check, and deliberate capability-building answer them.
3. An engineer ships code generated by an assistant that they cannot explain in review. What is the signal, and what is not?
   Answer

   A signal about the review practice and the adoption decision — review is passing changes it should not. Not primarily a personal failing; the system passed it. The fix is review that establishes understanding, and ownership that does not move.
4. What does proportionate governance of AI-assisted work across teams contain, and what are its two failures? org
   Answer

   What is permitted, what must be verified, what data may be sent where, and disclosure — scaled to the risk of the path, set at the seams, with tools left to teams. The failures are blanket prohibition, which is ignored, and no policy, which is discovered in an incident.
5. How is a department-wide rollout measured? org
   Answer

   By what teams can now do, at a quality they can defend, against outcomes in their own conditions — cycle time, change failure rate, rework, understanding — not by licences used or self-reported speed.
6. The board expects AI to reduce engineering headcount and wants a plan. What is the answer? exec
   Answer

   The evidence in the board's terms — including the organisation's own measurements — and a staged, bounded strategy with checkpoints at which the board sees local numbers. Any headcount change follows the measured effect rather than precedes it, and the organisation will be the first to say if the effect is large.
7. How is an emerging AI capability that could transform the product but is unproven evaluated? exec
   Answer

   As a bounded bet sized to teach — enough to build real capability and evidence, with checkpoints to double down or fold — funded by explicitly clearing something out. Neither prohibition nor stampede.

• Sida Peng, Eirini Kalliamvakou, Peter Cihon and Mert Demirer, The Impact of AI on Developer Productivity: Evidence from GitHub Copilot (2023) — the controlled experiment on AI-assisted task completion.
• Joel Becker, Nate Rush, Elizabeth Barnes and David Rein (METR), Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (2025) — the randomised trial on experienced developers with AI tools.
• Lisanne Bainbridge, Ironies of Automation (1983) — the ironies of automation: what automation leaves to the human, and what it erodes.
• Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt and Ramesh Karri, Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions (2022) — the vulnerability profile of AI-generated code.