Building AI-powered products

Building AI-powered products is the work of putting a language model inside something customers use: a support assistant, a search that answers in sentences, a feature that drafts, classifies or summarises. The competency is the manager's judgment about that work. It covers which approach to take, how to know whether the product is good enough to ship and whether a change made it better or worse, how to design for a component that is sometimes wrong, what the product will cost at real volume, and how to depend on a model the team does not control.

Core competency TJ-5 AI-assisted engineering covers AI as a tool for building software; this unit covers AI as part of the software shipped. A wrong coding assistant costs a reviewer some time. A wrong product feature is wrong in front of a customer, at scale, in a phrasing nobody chose.

It matters because these products behave unlike most software. The same input can produce different outputs, quality is a distribution, a demonstration convinces long before the product is reliable, and the cost of each answer grows with use. Ordinary delivery instincts produce two characteristic failures: a launch date set by the demo, and a quality claim nobody can check. The examination tests this competency through a fine-tuning proposal for data that changes every week, a four-week deadline set after a board demonstration, a model bill that tripled after launch, and a provider withdrawing the model version a product depends on.

1. Choose the approach by what is wrong. Prompting, retrieval, fine-tuning and tool-using agents fix different problems. Start from the failure you need to remove, then choose the cheapest approach that removes it, and know what each choice makes harder to change later.
2. No evaluation, no change. A model, prompt or retrieval change ships only after it has been measured against a set of cases that represents real use. An opinion formed from a few impressive examples is not a measurement.
3. An evaluation set is a maintained asset. It goes stale as the product, the users and the world change. It has an owner, it is refreshed from real use, and its age is known.
4. Design for the wrong answer. Assume the model will sometimes be confidently wrong. Decide where that is tolerable, where a person must check, and how the product tells users what it does not know.
5. Price it at volume. Latency, unit cost and quality are one trade-off. Work out what the product costs at the usage you hope for before the launch makes that usage real.
6. Treat the model as a dependency you do not control. Pin the version, test before every upgrade, and keep the option to change provider in proportion to what being locked in would cost.

Technical primer

A language model produces text one fragment at a time, each chosen by probability from what came before. Everything the model knows about the task in front of it arrives in its context: the instructions, any documents supplied, the conversation so far. The context has a limit, and models do not use all of it equally well.

The four approaches change different things. Prompting changes the instructions in the context and nothing else. It is cheap to try and easy to undo. Retrieval searches a store of the organisation's own documents for each request and places the most relevant passages in the context, so the model answers from current material it was never trained on. Fine-tuning adjusts the model itself by further training on examples. It can shift tone, format and behaviour on a narrow task, but it is slow to change and must be repeated on each new base model. A tool-using agent lets the model call functions (search, query a database, take an action) and act on the results across several steps. It extends what the product can do, and with it what the product can get wrong.

Because the output is sampled, the same request can produce different answers, so quality is measured over many cases. And because the context is assembled at run time, which passages go into it, in which order, affects the answer as much as the model does.

Retrieval-augmented generation research

Lewis et al. (2020) combined a generative model with a retrieved index of documents and found answers that were more specific and more factual than the model alone, and that could be traced to the passages used. Ovadia et al. (2024) compared retrieval with unsupervised fine-tuning as ways to give a model new knowledge and found retrieval better for both familiar and new facts. That comparison does not cover supervised fine-tuning on worked examples, and it is often cited as if it did. Liu et al. (2024) found that models use information at the start and end of a long context better than information in the middle. Taken together, the management position is that knowledge which changes belongs in retrieval, behaviour and format may belong in fine-tuning, and a retrieval system is only as good as what it retrieves and where it puts it.

LLM-as-a-judge contested

Using one model to grade another makes evaluation cheap enough to run on every change. Zheng et al. (2023) found that a strong model judge agreed with human preferences about as often as humans agreed with each other: 85% against 81%, with ties excluded. The same study found the judge preferred whichever answer it saw first, and could be swayed by longer answers. It could not determine whether a model favours its own answers. What is contested is how far that agreement generalises beyond the tasks studied. A model judge is useful once checked against human grading on the product's own cases, and no substitute for that check.

Criteria drift research

Shankar et al. (2024) studied people building automated evaluations and found that they could not fully define their grading criteria before looking at outputs: grading outputs was how they discovered what they cared about. The first rubric will be wrong in ways only real outputs reveal, so the plan needs time for at least one revision before the numbers mean much.

Non-determinism in hosted models research

Atıl et al. (2025) ran identical requests against hosted models configured to be as deterministic as their settings allow, and found accuracy varying by up to 15% across runs, with no model consistent on every task. A single run of an evaluation is a sample. A difference of a few points between two versions may be noise, and a product that must give the same answer twice has to be designed to do so: by storing answers, by constraining output, or by not using a model for that step.

Model behaviour drift research

Chen et al. (2024) measured the same named model service three months apart and found large changes in behaviour, some tasks better and some much worse; on one task accuracy fell from 84% to 51%. A model reached by name through a provider is a moving dependency unless its version is pinned, and an upgrade changes the product whether or not the team's code changed.

Hidden technical debt in machine-learning systems practice

Sculley et al. (2015) argued that in a production machine-learning system the model is a small part of the whole, and that most of the long-term cost sits around it: data dependencies, entangled components, feedback loops and changes in the outside world. It predates language models but describes their products closely: the prompt, the retrieval index, the evaluation set and the pinned model version all need owners, versions and tests.

The approach decision

Before building, the team writes down the failure to remove, the approaches considered, their cost to build and run, and what each makes harder to change. Most products start with prompting and retrieval, which can be changed in a day. Fine-tuning is chosen for a stated reason (a format the model will not hold, a latency or cost target a smaller tuned model meets), never because it sounds more serious. An agent is chosen when the product must act, and its permissions are agreed with whoever owns the risk (AI-2 Governance, security and compliance).

The evaluation set

The team assembles cases from real use, including the hard and embarrassing ones, each with the answer or criteria a good response meets. It starts small and grows from production, where every complaint and surprising output is a candidate case. The set has an owner and a refresh date, and Huyen (2025) describes building the evaluation before the feature as evaluation-driven development. A model judge is used only for criteria where it has been compared with human grading on these cases.

Shipping a change

Every change to the model, the prompt or the retrieval index runs against the evaluation set, several times where variation matters, and is compared with the current version on the same cases. The decision rule is agreed in advance: what improvement justifies the change, and what regression on which kind of case blocks it. The change then ships the way any risky change does, behind a flag or to a share of traffic first, with a way back. AI-5 Operating AI systems in production covers running it once it is live.

Designing for failure

For each kind of output the team decides what a wrong answer costs. Where the cost is low, the product answers and makes correction easy. Where it is high, a person reviews before the answer takes effect, or the product declines. The interface says what the answer is based on, and what the product cannot do. A feature that always sounds certain trains users to stop checking.

The cost model

Before launch, the team estimates cost per request, requests per user and users at planned volume, compares the result with what the feature earns or saves, and identifies the levers: a smaller model for easy cases, shorter contexts, caching, limits per user.

The model dependency

The product names a specific model version, not a model family. Upgrades are planned work, run through the evaluation set, never automatic. The team knows the provider's deprecation policy, keeps prompts and evaluation sets portable enough that a second provider could be tested in weeks, and decides openly how much of that portability is worth paying for (SV-4 Engineering economics).

At team scope the object is one product: the approach, the evaluation set, the failure design and the cost model for a feature the team owns. The manager makes sure evaluation exists before changes ship, and holds launch dates to the evidence. At organisational scope the object becomes the practice across products: shared evaluation tooling, a common rule that model and prompt changes are production changes, a view of model cost across teams, and one position on providers instead of each team negotiating its own lock-in. The organisational manager also sets delivery expectations upward, because the gap between demonstration and reliable product is where most AI commitments fail (DE-2 Estimation, planning, and honest commitments). At executive scope the object is the portfolio: which products justify their running cost, how dependent the organisation is on one provider, and how the strategy holds up if prices, models or regulation change (AI-1 Adoption strategy and staged bets).

• Chooses between prompting, retrieval, fine-tuning and an agent by the failure to be removed and the cost of each, and can say what each choice makes harder to change. team
• Requires an evaluation set that represents real use before a model or prompt change ships. team
• Keeps the evaluation set refreshed from production and knows how old it is. team
• Reads an evaluation result with its limits in view: run-to-run variation, a model judge that has not been checked against people, a set that no longer matches the users. team
• Designs the product around wrong answers: where to answer, where a person checks, where to decline. team
• Failure mode: proposing fine-tuning to give a model knowledge that changes every week. team
• Failure mode: judging a change from a handful of impressive examples. team
• Failure mode: treating one evaluation run as a precise measurement. team
• Prices the product at the volume the plan expects before launch, and knows which levers move the cost. org
• Manages model choice as a dependency: pinned versions, tested upgrades, portability proportionate to the lock-in risk. org
• Sets delivery expectations that allow for the distance between a demonstration and a reliable product. org
• Establishes across teams that a model, prompt or retrieval change is a production change. org
• Failure mode: a launch date set by the demonstration. org
• Failure mode: a model referenced by family name, upgraded silently by the provider. org
• Failure mode: discovering the unit cost from the first month's invoice. org
• Decides which AI products justify their running cost, and how much dependence on one provider the organisation will accept. exec
• Failure mode: an AI product portfolio whose total cost nobody has added up. exec

Demonstration versus product. A convincing demonstration takes days and a reliable product takes months, and the demonstration is what the stakeholders saw. The judgment is to show the evaluation results beside the demonstration from the first meeting, so that the date is set by evidence, and to say early how the last share of quality will be paid for.

Evaluation versus speed. Building and maintaining an evaluation set feels like delay when the team could be shipping. Without it, every change is a guess and every regression is found by customers. The judgment is to start with a small set immediately and grow it from production, rather than wait for a complete one or skip it.

Automated grading versus human judgment. A model judge makes evaluation cheap and fast, and it has biases. Human grading is trustworthy and slow. The judgment is to check the automated grader against people on the product's own cases, use it where it agrees, and keep people for the criteria where it does not.

Convenience versus lock-in. One provider's full platform is the fastest way to build, and the hardest to leave. The judgment is to decide how much portability is worth paying for, given what a forced migration would cost, and to keep at least the prompts and evaluation sets portable.

An engineering director at a business-software company runs three product teams. Sales has promised a large customer an assistant that answers questions about the customer's contracts, and the chief executive showed a prototype to the board, where it answered five questions well. The board expects launch in six weeks. The team lead proposes fine-tuning a model on the customer's contract archive, which grows by several hundred documents a month. Quality so far has been judged by the product manager trying a dozen questions. Nobody has estimated the cost at the customer's expected usage.

The easy responses are to accept the date and the plan because the board saw it working, or to refuse the date. The first commits the company to unmeasured quality built on an approach that fixes the wrong problem. The second hands the decision to people with less information than the director.

The director starts from what is wrong. The customer's contracts change monthly, so the knowledge belongs in retrieval, where a new document is searchable the day it arrives. Fine-tuning on the archive would bake in a snapshot and need repeating, and the evidence for fine-tuning as a way of adding knowledge is weak. The team keeps fine-tuning in reserve in case the model will not hold the citation format the customer needs. The director then asks for an evaluation set before any further building: two hundred real questions collected with the customer's legal team, with the clauses a correct answer must cite, including questions the contracts do not answer, where the right response is to say so. The product manager's dozen questions become the first twelve cases.

The first evaluation run shows the retrieval prototype answering well on simple lookups and badly when the answer spans two documents, and occasionally inventing a clause. The director takes that to the chief executive with three things: the measured results, a cost per question at the customer's expected volume (which is affordable with a smaller model for simple lookups), and a staged plan. In six weeks, the customer gets the assistant for single-document questions, with every answer linked to the clauses it used and a clear refusal when no clause applies. Questions across documents follow when the evaluation shows they are reliable, with a date set by that evidence.

The chief executive dislikes telling the board the launch is narrower than the demonstration suggested, but accepts the director's case: a narrow product that cites its sources will be trusted, and a broad one that sometimes invents a clause will lose the customer. The model version is pinned, and the evaluation set gates every later change.

• TJ-5 AI-assisted engineering: the baseline judgment about AI assistance that this unit builds on.
• AI-1 Adoption strategy and staged bets: whether an AI product is the right bet at all, and when to stop.
• AI-2 Governance, security and compliance: the security and data risks of what the product sends to a model and lets it do.
• AI-5 Operating AI systems in production: running the product once it is live, including incidents and drift.
• SV-4 Engineering economics: the AI service as a build, buy or rent decision, and the economics of lock-in.
• DE-2 Estimation, planning, and honest commitments: honest commitments when the work is uncertain.

1. A product manager proposes fine-tuning a model so that it can answer questions about prices that change every week. What is the better approach, and why?
   Answer

   Retrieval, which puts current documents in the model's context at request time. Fine-tuning would fix a snapshot that needs repeating with every change, and the evidence for it as a way of adding knowledge is weak. It suits behaviour and format better.
2. Two versions of a prompt score 81% and 83% on a single run of the evaluation set. What should the team conclude?
   Answer

   Nothing yet. Hosted models vary from run to run even with deterministic settings, so a two-point difference from one run may be noise. Run each version several times on the same cases, look at the spread, and look at which kinds of case changed, before deciding.
3. When is it reasonable to use one model to grade another's output?
   Answer

   When the automated judge has been checked against human grading on the product's own cases and agrees on the criteria in question. Model judges can approach human agreement on some tasks but show biases, such as favouring the first answer shown or longer answers, so they are an instrument to calibrate, not a substitute for human judgment.
4. Why should the first version of an evaluation rubric be expected to change?
   Answer

   Because people discover what they care about by grading real outputs (criteria drift). Criteria written before looking at outputs miss failures nobody anticipated, so the plan should include reviewing real outputs and revising the rubric before relying on the numbers.
5. A team references its model by family name and lets the provider update it. What is the risk, and what is the practice?
   Answer

   The behaviour of a named model service can change substantially between versions, so the product can change without any change to the team's code. Pin a specific version, treat upgrades as planned changes run through the evaluation set, and know the provider's deprecation policy.
6. What should be known about cost before an AI feature launches?
   Answer

   The cost per request, the expected requests per user and the number of users at the volume the plan hopes for, compared with what the feature earns or saves, plus the levers that reduce cost: smaller models for easy cases, shorter contexts, caching and usage limits.

• Lewis et al. (2020): retrieval-augmented generation and traceable answers.
• Ovadia et al. (2024): retrieval against unsupervised fine-tuning for adding knowledge.
• Liu et al. (2024): how models use long contexts.
• Zheng et al. (2023): agreement and biases of model judges.
• Shankar et al. (2024): criteria drift in building evaluations.
• Atıl et al. (2025): run-to-run variation in hosted models.
• Chen et al. (2024): behaviour change in a named model service over time.
• Sculley et al. (2015): where the cost of a machine-learning system sits.
• Huyen (2025): evaluation-driven development and choosing the approach.