Engineering managers who lead AI work tend to hear one of two pieces of advice. The first says the technology is moving too fast to understand, so focus on people and leave the models to the engineers. The second says a manager must now be hands-on, building agents and tuning prompts like everyone else. Both miss what the job requires.
A manager makes decisions about technical work without doing it. In AI work those decisions turn on a handful of mechanisms that behave differently from ordinary software. Understand the mechanism and the decision is usually clear. Miss it and a reasonable-sounding choice goes wrong in a way nobody sees for weeks. Four of them come up again and again.
One run of an evaluation is a sample
A team compares two versions of a prompt. One scores 81% on the evaluation set, the other 84%. The obvious move is to ship the higher score.
The mechanism a manager needs is non-determinism. Hosted models vary from run to run, even when configured to be as deterministic as they can be, so a single run of an evaluation is one draw from a distribution. Three points between two versions may be noise. The useful questions are how much the score moves when the same version runs again, and which cases changed between versions. If the gain comes from one category of question and costs another, that tells you more than the headline.
The decision follows from the mechanism: run both versions several times on the same cases, read which ones moved, and treat the evaluation set as an instrument whose precision you know. A manager who asks "what is the spread on that number?" is doing technical leadership without opening an editor.
Changing knowledge belongs in retrieval
An internal assistant answers questions about company policy and keeps quoting rules that were retired months ago. Someone proposes fine-tuning the model on the current policies.
The mechanism here is where knowledge lives. Fine-tuning shapes behaviour and style; it bakes in a snapshot that has to be repeated with every change. Retrieval puts the current document in front of the model at the moment of the question, and lets the answer cite it. For knowledge that changes every few weeks, retrieval is the fix, and a policy change goes live the day it is published. The manager's decision is architectural, and it depends only on knowing which approach suits which kind of problem.
A prompt is part of the release
A teammate wants to improve the assistant by editing its prompt in the vendor's dashboard this afternoon. It feels like configuration, the sort of change nobody reviews.
The mechanism is simple once stated: anything that changes what the product says is a production change, wherever it happens to be edited. A prompt edit can change every answer the product gives, for every user, at once. So it gets what any other release gets: review, a run against the evaluation set, a rollout to a share of traffic first, and a way back. The same reasoning covers a provider's model update. Pin the version you tested, and move to a new one deliberately.
Agents raise the stakes because they act. An agent's permissions should be set by the damage a wrong action could do, so the sensible default is a sandboxed branch, no production credentials, and no right to merge. Merging stays with a person, because that is where somebody accepts responsibility for a change.
A feature can be up and wrong
The assistant has been available all week and its latency is normal. A daily sample of graded answers shows quality fell sharply after a change nobody can yet identify.
Conventional monitoring would call this a healthy service. AI features usually fail by answering wrongly while every availability dashboard stays green, which is why quality has to be measured directly, by sampling and grading real outputs. A feature that is up and wrong is worse than one that is plainly down, because users trust it. The decision is to fall back and treat the drop as an incident, before customers find it for you.
Content that the model reads raises a related risk. A system that passes untrusted input to a model and then lets the model act, such as an assistant that reads customer email and can update accounts, can be steered by instructions planted in that input. The design review for a system like that starts by cutting the model's permissions to what a hostile email could safely trigger.
How the extension examines this
These four mechanisms sit inside AI-3, AI-4 and AI-5 of the EMA AI Competency Model, the technical half of the EMA AI Extension, which carries 34 of the examination's 60 questions. The units on building AI-powered products and operating AI systems in production set out the judgment in full, with the sources behind it.
Every question puts a situation in front of you and asks what you would decide. None asks you to write code or a prompt, or to name a product. If you want to check whether you could answer them, the EMX-AI sample questions include most of the situations above, with an explanation for every option.
Comment on version 1.0 of the AI Standard closes on 15 February 2027. If a mechanism a manager needs is missing from it, or one of these judgments reads wrong to you, the consultation page is the place to say so, and every comment gets an answer.