Operating AI systems in production

Operating AI systems in production is the work of running a product that depends on a language model once real users rely on it: knowing whether it is still behaving well, changing it safely, responding when it goes wrong, keeping its cost under control, and noticing when the world it serves has moved. The competency is the manager's judgment about that operation.

Core competency DE-3 Reliability, incident response, and operational ownership covers reliability and incidents for any service, and this unit inherits all of it: service-level objectives, on-call, blameless review. What changes with AI is the shape of failure. A conventional service fails loudly: it errors, times out, falls over. An AI feature usually fails quietly. It answers, promptly and fluently, and the answer is wrong, or harmful, or quoting a document it should never have seen. Every availability dashboard stays green.

It matters because users and courts treat what the product says as what the company says. In Tribunal (2024), a tribunal held an airline responsible for its website chatbot's misstatement of a fare policy and called the suggestion that the chatbot was responsible for its own words "a remarkable submission". The decision is a small-claims ruling and binds no one, but its reasoning is the one a manager should plan for. The examination tests the competency through a confidently false answer shared publicly, a provider outage that takes the main feature down, a prompt changed in a dashboard without review, and a cost alert fired by an agent calling a model in a loop.

1. Watch quality, not only uptime. An AI feature can be available and wrong. Instrument it for the quality, safety and cost of its answers as well as for whether it responds.
2. A model, prompt or index change is a production change. Anything that changes what the product says goes through testing, staged rollout and a way back, whoever made it and wherever it was made.
3. An AI incident is an incident. A false or harmful output at scale, an injection, a runaway cost: each gets the same discipline as an outage, including containment, communication and a blameless review.
4. Fail safe, not silent. Decide in advance what the product does when the model or provider fails or misbehaves, and make sure someone can switch the feature off in minutes.
5. Cap the cost before it caps you. Budgets, alerts and limits go in before launch, because a model bill can grow faster than anyone reads an invoice.
6. Evaluate again when the world moves. The users, the documents and the model all change. Know what the system was evaluated on, notice when production no longer looks like it, and evaluate again.

Technical primer

Several things can change an AI feature's behaviour without anyone changing its code. The model: a provider may update a model behind a name, or retire a version. The prompt: often edited in a configuration file or a vendor's dashboard, outside the code review that guards everything else. The retrieval index: when documents are added, removed or re-processed, what the model sees changes. The inputs: users ask new things, in new ways, and some of them are adversarial. The provider's infrastructure: latency, rate limits and outages are someone else's operations.

A request that returns a fluent but wrong answer looks, to ordinary monitoring, exactly like a good one: a successful response in normal time. Quality therefore has to be measured directly. The usual signals are a sample of real answers graded by people or by a checked automated judge, user feedback and escalations, the rate at which the product declines or hands over to a person, the documents retrieved for each answer, the number of tool calls or steps an agent takes, and tokens and cost per request. None of these comes free with a standard monitoring stack; each is a decision.

Service-level objectives and error budgets practice

Beyer et al. (2016) describes setting explicit objectives for a service and treating the gap between the objective and perfection as a budget to spend on change. For an AI feature the practice carries over with one extension: the objectives include quality as well as availability and latency, for example the share of sampled answers judged correct and the share that cite a source. When quality burns through its budget, the response is the same as for availability: slow down change and fix the cause.

Production as the place models are understood research

Shankar et al. (2024) interviewed engineers who run machine-learning systems and found that teams do not know how a model will behave until it meets production, and that successful operation is a continual loop of validation, versioning and monitoring rather than a launch followed by maintenance. The finding predates most language-model products, but it describes them well: the evaluation before launch is the start of knowing how the product behaves, not the end.

Distribution shift and model drift research

Huyen (2022) describes how the data a system meets in production moves away from the data it was built and evaluated on, and why monitoring must look for that movement. Language-model products add a second source of drift: the model itself. Chen et al. (2024) found that the behaviour of the same named model service changed substantially within three months. A product therefore drifts from both ends, its users and its model, and needs evaluation that is repeated, not only performed once.

Indirect prompt injection research

Greshake et al. (2023) showed that applications which feed a model content from outside, such as retrieved documents, web pages or emails, blur the line between data and instructions: text planted in that content can steer the model, including towards leaking data or misusing its tools. For operations this means that the retrieval index and every external input are part of the attack surface, and that an injection is an incident with a security owner (AI-2 Governance, security and compliance, TJ-6 Security, privacy, and compliance as engineering responsibilities).

Unbounded consumption practice

Project (2025) names unbounded consumption as a risk: inference that is excessive or uncontrolled, whether from abuse, a loop or a design that never anticipated the volume, leading to degraded service and economic loss. It is the AI form of a capacity problem, with a difference that matters to a manager. The failure shows up first on a bill, not on a latency graph.

Quality monitoring

The team samples real answers every day and grades them, by people or by an automated judge that has been checked against people (AI-4 Building AI-powered products). It tracks feedback, escalations and the rate of declined or handed-over requests, and sets objectives for quality beside availability. A drop in quality pages someone in the same way an outage would.

Changing the system

The model version, the prompt, the retrieval configuration and the index-building pipeline are all versioned and changed through review. A change runs against the evaluation set, then ships to a share of traffic with its quality signals watched, with a way back that has been tested. Prompts are not edited in a dashboard by whoever has access; where a vendor dashboard is the only place to change them, access is restricted and every change is logged and reviewed.

The AI incident

The team defines in advance what counts as an incident: a false or harmful answer seen by users at scale, a confirmed injection, data shown to the wrong person, a cost spike. The first action is containment: switch the feature to its fallback, disable a tool, or turn it off. The second is the record: users who received a wrong answer that could cost them are told and the error is put right, because the company owns what its product said. The review that follows is blameless in the sense of Allspaw (2012): it asks how the system allowed the failure, and accountability attaches to fixing that system.

Fallbacks and the kill switch

Each AI feature has a defined fallback: a simpler rule, a search without generated answers, a hand-off to a person, or a clear message that the feature is unavailable. A flag can switch to it in minutes without a deploy, and the switch is tested. Where possible a second provider or model has been tried against the evaluation set, so that a long outage has an answer beyond waiting.

Cost controls

Before launch the team sets a budget per feature, alerts well below it, limits per user and per tenant, and, for agents, a maximum number of steps and tool calls per task. Cost per request is on the same dashboard as latency. A cost alert is treated as a possible incident, not an accounting matter, because a loop or an abuse pattern looks the same on a bill as success does.

Evaluating again

The team knows what the system was evaluated on and when. It re-runs the evaluation on a schedule, after any model or index change, and whenever the monitoring shows production moving away from the evaluation set: new kinds of questions, new documents, a change in declined requests. New production failures become evaluation cases.

At team scope the object is one feature in production: its quality signals, its change process, its fallback and its budget. The manager makes sure someone is watching quality and can switch the feature off, and runs incidents with the team. At organisational scope the object becomes the operating practice across AI features: one definition of an AI incident, shared tooling for sampling and grading answers, a rule that model and prompt changes are production changes everywhere, a cost view across teams and providers, and on-call that knows what to do when a provider fails. At executive scope the questions are exposure and accountability: what the organisation's AI features are saying to customers, who is answerable for it, how much the organisation depends on any one provider, and whether its incident record would stand up in front of a regulator or a court.

• Instruments an AI feature for quality, safety and cost as well as availability. team
• Puts model, prompt and index changes through review, staged rollout and a tested way back. team
• Runs an AI-specific failure as an incident: contain, correct the record with users, review blamelessly. team
• Designs a fallback and a kill switch that work in minutes without a deploy. team
• Treats a confirmed prompt injection as a security incident with an owner. team
• Failure mode: green availability dashboards taken as evidence the feature is working. team
• Failure mode: prompts edited in a vendor dashboard by anyone with access. team
• Failure mode: treating a wrong answer as a support ticket rather than an incident. team
• Sets budgets, alerts and limits before launch, including step limits for agents. org
• Establishes one definition of an AI incident and one change rule across teams. org
• Re-evaluates when production drifts from what the system was evaluated on. org
• Failure mode: learning the unit cost from the first month's invoice. org
• Failure mode: no plan for a long provider outage beyond waiting. org
• Knows what the organisation's AI features say to customers and who answers for it. exec
• Failure mode: assuming the vendor is responsible for what the product tells customers. exec

Quality monitoring versus cost. Grading real answers takes people's time or a judge's tokens, and it is tempting to sample too little to see anything. The judgment is to sample enough to detect the failures that matter most, weighted towards the answers with the highest stakes.

Change speed versus control. Prompts are easy to change, and a fix is often one sentence away. The same ease lets an untested edit reach every user at once. The judgment is to keep the edit easy and the release controlled: review, evaluation and staged rollout, fast enough that nobody is tempted to bypass them.

Availability versus correctness. Keeping a feature up feels like success, and switching it off feels like failure. A feature that is up and wrong is worse than one that is clearly unavailable. The judgment is to decide in advance at what point the fallback is the better product.

Provider dependence versus resilience. One provider is simpler to build on and to operate. A second is insurance with a running cost. The judgment is to buy as much resilience as the cost of a long outage justifies, and to have tested it.

A team lead runs the team that owns the help assistant on a software company's customer site. It answers questions from the public help centre, which the team re-indexes nightly. On a Monday morning, support reports several customers quoting a refund rule that does not exist: the assistant is telling them that annual plans can be refunded in full within sixty days. The real policy is thirty days. Availability and latency dashboards are normal.

The easy responses are to treat it as a content error, ask the help-centre team to fix the article and move on, or to switch the assistant off indefinitely until someone can promise it will never happen again. The first leaves customers holding a promise the company made and has not corrected, and fixes nothing about how it happened. The second removes a feature most customers use well because of a failure nobody has yet understood.

The team lead declares an incident. Within the hour the team switches the assistant to its fallback, search results without generated answers, and confirms the false answer is no longer being given. Support pulls the conversations from the weekend in which the assistant gave the sixty-day rule, and the company contacts each customer, honours the refund for those who acted on it, and corrects the record for the rest. Nobody suggests that the assistant, not the company, made the promise.

The blameless review finds the cause. A draft policy page, written for a promotion that never launched, was published to the help centre with a hidden flag the indexing pipeline ignored. The nightly index picked it up, and the assistant retrieved it because it matched refund questions better than the real page. Nothing in the pipeline compared what the assistant said before and after an index change, and nothing in monitoring sampled answers about money.

The fixes follow. The indexing pipeline honours the hidden flag and logs every document added or removed. An index change now runs the evaluation set before it goes live, and the set gains cases on refunds, cancellations and pricing, the answers that cost the most when wrong. Daily sampling is weighted towards those topics, with an alert on any answer that cites a page published in the last day. The assistant returns to service on Wednesday. The team lead writes the incident up for the wider organisation, because the help centre is not the only team whose documents feed a model.

• TJ-5 AI-assisted engineering: the baseline judgment about AI assistance that this unit builds on.
• DE-3 Reliability, incident response, and operational ownership: reliability, incident response and operational ownership for any service.
• TJ-6 Security, privacy, and compliance as engineering responsibilities: security and privacy, including the data an AI feature can expose.
• AI-2 Governance, security and compliance: the governance and security policy that injections and data exposure fall under.
• AI-4 Building AI-powered products: the evaluation set and the design decisions that operation depends on.

1. Why are availability and latency dashboards not enough for an AI feature?
   Answer

   Because an AI feature usually fails by answering wrongly, promptly and fluently, which looks like success to availability monitoring. Quality has to be measured directly: sampled and graded answers, feedback and escalations, declined requests, retrieved documents and cost per request.
2. A prompt was changed in a vendor dashboard without review and complaints rose the next day. What practice was missing?
   Answer

   Treating the prompt as part of the production system: versioned, changed through review, run against the evaluation set, rolled out to a share of traffic with its quality watched, and reversible. Where a dashboard is the only place to change it, access is restricted and every change is logged and reviewed.
3. The product gave customers a false statement of policy. What are the first two actions?
   Answer

   Contain it, by switching to the fallback or turning the feature off, and correct the record with the customers who received the answer, because the company is responsible for what its product said. The blameless review follows.
4. What makes an application that retrieves outside content vulnerable to prompt injection, and what follows for operations?
   Answer

   Retrieved content and instructions reach the model together, so text planted in that content can steer it. The index and every external input are part of the attack surface, and a confirmed injection is a security incident with an owner.
5. What cost controls belong in place before launch?
   Answer

   A budget per feature with alerts below it, limits per user and per tenant, step and tool-call limits for agents, and cost per request on the same dashboard as latency, with a cost alert treated as a possible incident.
6. When should a team evaluate a production AI feature again?
   Answer

   On a schedule, after any model, prompt or index change, and whenever monitoring shows production moving away from what the system was evaluated on. New production failures become evaluation cases.

• Beyer et al. (2016): service-level objectives, error budgets and incident management.
• Shankar et al. (2024): how engineers learn model behaviour in production.
• Chen et al. (2024): behaviour change in a named model service.
• Huyen (2022): distribution shift and monitoring.
• Greshake et al. (2023): indirect prompt injection through retrieved content.
• Project (2025): unbounded consumption.
• Allspaw (2012): blameless review and accountability.
• Tribunal (2024): a company held responsible for its chatbot's statement.