Capability, roles and measuring impact

This competency is the manager's judgment about people in an organisation that uses AI: how the team builds the skill to use it well, how early-career engineers keep learning when the tools do the work they used to learn from, how candidates and engineers are assessed fairly when assistance is everywhere, how the effect of AI is measured honestly, and how its effect on roles and headcount is discussed without false promises in either direction.

Core competency TJ-5 AI-assisted engineering asks a manager to build capability rather than count licences and to measure in their own conditions. This unit takes those two duties further, into the decisions that shape careers: who learns what, who gets hired on what evidence, what is claimed about productivity, and what people are told about their jobs.

It matters because the effects on people arrive slowly and are hard to reverse. A team that stops practising a skill does not notice until it needs it. A cohort of juniors that never learned to debug is a staffing problem three years later. A productivity claim built on how fast people feel becomes a headcount plan nobody can deliver. The examination tests the competency through a plan to hire a third fewer engineers on the strength of AI, early-career engineers who ship changes they cannot explain, a survey that says engineers feel twice as fast while delivery has not moved, and a team asking whether their jobs are safe.

1. Capability is built, not bought. A licence gives access, not skill. Teams learn to use AI well through practice, shared conventions and time set aside to learn, and the manager plans for that like any other capability.
2. Protect how early careers learn. When assistants do the tasks juniors used to learn from, the learning has to be put back on purpose: work they do unaided, review that asks them to explain, and problems chosen to teach.
3. Assess the judgment, with clear rules. In hiring and in performance, assess what the person can reason through, under rules about assistance that are stated in advance and applied to everyone.
4. Measure outcomes, not feelings. Self-reported speed-ups are not a measure. Measure outcomes in the team's own conditions, over long enough to see beyond novelty, across more than one dimension of productivity.
5. Hold contested claims as contested. The evidence on AI and productivity disagrees with itself. Say so, rather than citing the study that suits the decision.
6. Be honest about roles and headcount. Say what the organisation knows and does not know about AI's effect on jobs, and promise nothing it cannot keep, in either direction.

Gains concentrated among the less experienced research

Brynjolfsson et al. (2025) found that an assistant raised customer-support productivity on average, with the largest gains for the least experienced agents and little for the most experienced. The study is in customer support, and its relevance to engineering is a question to test rather than a given. Its management consequence is double-edged: the people who gain most in output are the people whose learning is most at stake.

Assistance and learning research

Bastani et al. (2025) ran a field experiment in which students practised with an AI assistant. An unrestricted assistant improved their practice performance and harmed their performance once it was taken away; an assistant designed to give hints rather than answers largely avoided the harm. The study is of school students, not engineers, and its lesson transfers as a mechanism rather than a number: when the tool supplies the answer, the learner can perform without learning, and the design of the assistance decides which happens.

Confidence and critical thinking research

Lee et al. (2025) surveyed knowledge workers about tasks they did with generative AI and found that greater confidence in the tool went with less critical thinking, and greater confidence in their own ability went with more. The data are self-reported, so the study shows an association in how people describe their work, not a measured decline. It is still a useful warning about where review effort goes when the tool is trusted.

Measuring developer productivity practice

Forsgren et al. (2021) argued that developer productivity has several dimensions (satisfaction and well-being, performance, activity, communication and collaboration, efficiency and flow) and that no single metric captures it. Becker et al. (2025) adds a specific warning for AI: experienced developers in a randomised trial were slower with the tools while believing they were faster. Together they define an honest measurement of AI's effect: outcomes across several dimensions, against a baseline, with self-report collected and not trusted on its own.

Structured assessment research

Hunter (1998) synthesised decades of research on selection methods and found structured interviews among the strongest predictors of job performance. The finding predates AI and matters more because of it: when candidates can use assistants for take-home work, a structured conversation about how they reasoned, what they would check and what could go wrong is harder to outsource and closer to the judgment the job needs (PL-3 Hiring and onboarding).

Labour-market effects contested

The evidence on AI and jobs disagrees. Eloundou et al. (2024) estimated how many occupations' tasks are exposed to language models, which is a measure of potential, not of jobs lost. Humlum et al. (2025) used administrative data from Denmark and found effects of AI chatbots on earnings and hours close to zero, ruling out effects larger than 2% two years on. Brynjolfsson et al. (2025), a working paper, found early-career workers in the most exposed occupations falling behind their less exposed peers in employment. What is contested is whether, where and how fast AI changes employment. A manager talking to a team about their jobs should know that the honest answer is uncertain, and that the early signs, where there are any, concern the start of careers.

Building capability

The team sets aside regular time for engineers to share what works and what does not: which tasks the tools help with, how to check what they produce, conventions for prompts and for review. Fluency is treated as a skill with a learning curve, so the manager expects the first months to be slower for some, and does not judge the tools or the people on that period alone.

Early-career development

Junior engineers do some work unaided, chosen because it teaches: debugging, reading unfamiliar code, writing a design before asking for one. In review they explain their changes, generated or not, and a senior engineer probes the reasoning, not only the result. Assistance for juniors is set up to explain and hint rather than to hand over answers where learning is the point. The manager watches for the signal that matters most: a junior who ships quickly and cannot explain what shipped.

Assessing people

Hiring states its rules on assistance in advance: what candidates may use, in which parts of the process, and what is assessed. Structured interviews probe reasoning, trade-offs and what the candidate would check. Performance assessment looks at outcomes and judgment, not at volume of output, which assistants inflate. The same rules apply to everyone, so that nobody is advantaged or penalised for using, or declining to use, the tools.

Measuring impact

Before an adoption, the team records a baseline on outcomes: cycle time, change failure rate, rework, incident trends, and a short survey of satisfaction and flow. After, it measures the same things over at least a quarter. Self-reported speed is collected alongside and reported separately. The result is reported upward with its uncertainty, and with the outcome measures leading (SV-5 Engineering metrics and measurement).

Talking about roles and headcount

When leadership raises headcount, the manager brings the organisation's own measured results, the state of the external evidence, and a plan that lets any change follow the evidence instead of running ahead of it. When the team asks whether their jobs are safe, the manager says what is known and what is not, what the organisation has decided and what it has not, and does not offer a guarantee they cannot keep.

At team scope the object is the team's own skill, its juniors, and the honesty of its measurement: time to learn, unaided work that teaches, explanation in review, a baseline before adoption. At organisational scope the object becomes the system across teams: hiring rules on assistance that are consistent, a career framework that still develops the skills AI erodes, a shared method for measuring AI's effect, and a manager-of-managers who notices when one team's juniors are learning and another's are not (PL-5 Career development and growth paths). At executive scope the questions are the workforce plan and the organisation's word: what it commits to on headcount, on what evidence, how it keeps a pipeline of future senior engineers when the entry-level work is changing, and what it says publicly and internally about AI and jobs (AI-1 Adoption strategy and staged bets).

• Builds the team's AI capability through practice, shared conventions and time to learn. team
• Gives early-career engineers unaided work that teaches, and asks them to explain their changes in review. team
• Assesses candidates and engineers on judgment, under assistance rules stated in advance. team
• Measures AI's effect on outcomes against a baseline, and reports self-reported speed separately. team
• Failure mode: counting licences or active users as capability. team
• Failure mode: praising a junior's output without checking they can explain it. team
• Protects the learning of early-career engineers across teams, in the career framework and in how assistance is set up. org
• Keeps hiring rules on assistance consistent across teams. org
• Holds contested productivity evidence as contested in what it reports upward. org
• Failure mode: a productivity claim built on a survey of how fast people feel. org
• Failure mode: take-home tests that the assistant can complete, assessed as if it could not. org
• Failure mode: cutting junior hiring without a plan for where future senior engineers come from. org
• Speaks honestly about AI's effect on roles and headcount, without promising what the organisation cannot keep. exec
• Lets any headcount change follow measured evidence rather than forecasts. exec
• Failure mode: a headcount plan that assumes a productivity gain nobody has measured. exec

Output versus learning. Assistants let juniors produce more and can stop them learning. The judgment is to accept some loss of short-term output for unaided work and explanation that build the skill the organisation will need later.

Measurement versus morale. Engineers who feel faster may resent being told the outcome measures have not moved. The judgment is to share the measures openly, credit the feeling as real, and treat the gap as something to understand together.

Fairness versus realism in assessment. Banning assistance in hiring tests something the job no longer looks like; allowing it unstructured tests the tool. The judgment is to state the rules, allow what the job allows, and assess the reasoning the tool cannot supply.

Reassurance versus honesty. A team anxious about its jobs wants to hear it is safe, and nobody can promise that. The judgment is to say what is known, what has been decided and what has not, and to keep every promise made.

A senior engineering manager leads three teams through their managers. Leadership has asked for next year's hiring plan with "a third fewer engineers, because of AI", and the plan as drafted cuts all graduate hiring. Separately, two of the managers report the same pattern: their newest engineers ship changes quickly with assistants, but in review they cannot say why the code works or what would break it. An internal survey says engineers feel about twice as fast; delivery metrics over the same period are flat.

The easy responses are to submit the plan, because leadership asked and the survey seems to support it, or to refuse any reduction and defend every role. The first builds a headcount commitment on a feeling, and removes the entry point through which the organisation grows its future senior engineers. The second ignores a real question and spends the manager's credibility on a position the evidence does not support either.

The senior manager starts with measurement. With the three managers, they assemble the outcome measures for the year: cycle time slightly better on two teams, change failure rate slightly worse on one, rework unchanged. They put the survey beside it and note the gap, and they bring the external evidence: studies that find large gains on some tasks and none or losses on others, a randomised trial in which experienced developers were slower while feeling faster, and labour-market studies that disagree on whether AI is changing employment at all.

They then deal with the juniors, because that problem is already costing review time. The managers agree a practice: graduates do some work unaided each sprint, chosen to teach; every change they submit is explained in review; and their assistant is configured to explain and suggest rather than complete whole tasks where learning is the point. A quarter later, the managers report that review is slower for graduates and that they can now explain what they ship.

The hiring plan goes to leadership with the measured results, the uncertainty, and a proposal: hold total headcount flat for the year, keep graduate hiring at a reduced but non-zero level with the new development practice, and revisit at two checkpoints with the organisation's own numbers. The senior manager says plainly that if the measures show a lasting gain, the plan will change, and that cutting the graduate intake to zero would save money this year and cost senior engineers in three. Leadership accepts the flat plan and the checkpoints. When one of the teams asks at an all-hands whether their jobs are safe, the senior manager tells them what was decided, what the numbers show, when the next review is, and that they will hear about any change from their manager first.

• TJ-5 AI-assisted engineering: the baseline judgment about building capability and measuring AI's effect.
• PL-3 Hiring and onboarding: hiring on evidence of capability, as opposed to access to tools.
• PL-5 Career development and growth paths: career development and growth paths that still build the skills AI erodes.
• SV-5 Engineering metrics and measurement: measuring and reporting AI's impact honestly.
• AI-1 Adoption strategy and staged bets: the adoption decisions whose effects this unit measures.
• AI-3 AI-assisted software delivery: review for understanding in delivery.

1. Why might the people who gain most in output from AI be the ones most at risk?
   Answer

   Because the largest gains have been found among the least experienced, and their output can rise while their learning stalls if the tool supplies answers they would otherwise have worked out. Output and learning have to be managed separately.
2. What practices protect how early-career engineers learn when assistants are available?
   Answer

   Some unaided work chosen to teach, explanation of every change in review with the reasoning probed, and assistance configured to hint and explain rather than hand over complete answers where learning is the point.
3. A survey says engineers feel twice as fast; delivery metrics are flat. What should be reported upward?
   Answer

   The outcome measures first, with their baseline and uncertainty, and the survey reported separately as perception. Self-reported speed-ups have been shown to diverge from measured results, so they are not the measure of AI's effect.
4. How should hiring adapt when candidates can use assistants?
   Answer

   State the rules on assistance in advance and apply them to everyone, allow what the job allows, and assess reasoning through structured interviews that probe trade-offs and what the candidate would check, which is harder to outsource than a take-home task.
5. What is the honest summary of the evidence on AI and jobs?
   Answer

   That it is contested: exposure estimates measure potential, not job loss; some administrative data show effects close to zero; and some early evidence suggests early-career workers in the most exposed occupations are falling behind. Nobody can promise a team their jobs are safe or unsafe on this evidence.
6. Why is cutting all graduate hiring a risky response to AI?
   Answer

   Because it saves money now and removes the pipeline through which the organisation grows its future senior engineers, on the strength of a productivity gain that may not have been measured. A reduced, non-zero intake with a deliberate development practice keeps the option open.

• Brynjolfsson et al. (2025): productivity gains concentrated among the less experienced.
• Bastani et al. (2025): how the design of assistance decides whether learners learn.
• Lee et al. (2025): confidence in AI and self-reported critical thinking.
• Becker et al. (2025): experienced developers slower with AI tools while believing they were faster.
• Forsgren et al. (2021): the dimensions of developer productivity.
• Hunter (1998): the validity of structured interviews.
• Brynjolfsson et al. (2025): early-career employment in AI-exposed occupations, a working paper.
• Humlum et al. (2025): small labour-market effects in Danish administrative data.
• Eloundou et al. (2024): occupational exposure to language models.