Governance, security and compliance

Governance, security and compliance for AI is the set of decisions that keep an organisation's use of AI accountable and safe: what data may go to which tools and providers, what an AI system may be allowed to do, which uses need review and which can be left to teams, when a use falls into a regulated category, and who owns each system once it is running. The competency is the manager's judgment about those decisions. It is not the lawyer's, the security engineer's or the auditor's judgment, though it decides when each of them is brought in.

Core competency TJ-5 AI-assisted engineering asks a manager to govern AI in proportion to risk; TJ-6 Security, privacy, and compliance as engineering responsibilities asks them to own security and privacy as engineering work. This unit applies both to the specific risks AI brings: data that leaves the organisation in a prompt, systems that cannot tell instructions from content, models that can repeat what they were trained on, agents that act, and regulation written specifically for AI.

It matters because the failures are cheap to cause and expensive to undo. An engineer pastes customer records into a public chatbot to draft a reply; a support assistant reads an email that tells it to forward the conversation history; a screening feature quietly becomes a regulated system. Blanket bans are ignored and blanket permission is discovered in an incident. The examination tests the competency through customer data pasted into a public tool, an assistant with system access reading inbound email, and a question from legal about whether a new screening feature is high-risk.

1. Know where the data goes. For every AI tool and provider, know what is sent, where it is processed, whether it is kept or used for training, and who can see it. A decision about a tool is a decision about a data flow.
2. Untrusted content is untrusted instructions. Any system that passes outside content to a model, and lets the model act, can be steered by that content. Design for it before release.
3. Proportionate, not uniform. Make low-risk use easy and high-risk use reviewed. A policy that treats a documentation helper like a hiring system will be ignored where it matters least and followed where it matters most only by accident.
4. Recognise regulated uses early. Some uses, such as screening job applicants, carry legal obligations. Spot them at the idea stage and bring in legal or compliance then, not at launch.
5. Every AI system has a named owner. Keep an inventory of what is in use, what it does, what it touches and who answers for it, so that accountability survives a reorganisation or a change of vendor.
6. Govern with frameworks, not from scratch. Established frameworks give the questions; the organisation supplies the answers in proportion to its risk.

Prompt injection and excessive agency practice

Project (2025) names prompt injection as the first risk for applications built on language models, and lists sensitive information disclosure and excessive agency among the others. Excessive agency has three causes: more functionality than the task needs, more permissions than the functionality needs, and more autonomy than the system's reliability justifies. The list is a shared vocabulary rather than research, and its value to a manager is exactly that: it lets a design review ask named questions that a security team will recognise.

Indirect prompt injection research

Greshake et al. (2023) showed that when an application feeds a model content from outside (web pages, documents, emails), text planted in that content can steer the model, and demonstrated data theft and misuse of tools against real systems. The authors found effective mitigations lacking at the time. The governance consequence is structural: an assistant that reads untrusted input and can take actions needs its permissions limited to what a hostile input could safely trigger, with a person confirming anything consequential.

Training data extraction research

Carlini et al. (2021) recovered verbatim passages of training data, including personal information, from a language model by querying it, and found larger models more vulnerable. For an organisation the lesson runs in two directions. Data sent to a provider that trains on it may surface elsewhere, which is why data-retention and training terms matter when choosing a tool; and a model the organisation trains or fine-tunes on its own data can leak that data to its users.

Risk management frameworks practice

Technology (2023) organises AI risk management into four functions: govern (policies, roles and culture), map (the context and the risks of a given system), measure (assessing those risks) and manage (acting on them). ISO/IEC (2023) specifies requirements for an AI management system, on the same structure as the information-security management standard many organisations already run. Centre et al. (2023) gives security guidance across design, development, deployment and operation. None of them tells an organisation what to decide. They give a manager a defensible checklist of questions and a structure that auditors and customers recognise.

Risk-based regulation practice

Union (2024) regulates AI by risk. It prohibits some practices, imposes obligations on providers and deployers of systems it classes as high-risk, and sets lighter duties elsewhere. Its Annex III lists high-risk uses, and point 4 covers employment: systems for recruitment and selection, including filtering applications and evaluating candidates, and systems used for decisions on promotion, termination, task allocation and performance monitoring. The obligations apply in stages, and the timetable has been amended since publication, so any date must be checked against the current consolidated text before it is relied on. The manager's task is recognition, not interpretation: knowing that a use may fall into a category, and involving legal or compliance before building it.

The data-flow register

For each AI tool and provider in use, the organisation records what data classes may be sent, where processing happens, the retention and training terms, and who approved it. Engineers can see the register and know which tool to use for which data. The register replaces the question "may I use AI for this?" with "which approved tool handles this data?".

Proportionate policy

The policy sorts uses into a few tiers. Low-risk uses (help with documentation, explaining code, drafting internal text with no personal data) are allowed with approved tools and no further review. Medium-risk uses (generated code on production paths, customer-facing text) follow the review and disclosure rules of AI-3 AI-assisted software delivery. High-risk uses (anything that decides about people, touches regulated data, or acts on systems without confirmation) need a design review with security and, where relevant, legal. The policy fits on a page and is revisited as tools change.

Designing for injection and agency

Any system that reads outside content and can take actions gets a review against the named risks before release: what is the most damaging action a hostile input could trigger, and is that action allowed? Permissions are cut to the task, consequential actions need a person's confirmation, and outputs that go elsewhere (to a database, a browser, another system) are handled as untrusted. Incidents of this kind run through AI-5 Operating AI systems in production.

Recognising regulated uses

The intake for any new AI feature asks a few plain questions: does it make or support decisions about people (hiring, promotion, performance, access to services)? Does it process special categories of personal data? Is it sold into a regulated sector? A yes sends the idea to legal or compliance before design starts. The manager does not decide whether the use is high-risk under a regulation; they make sure someone qualified decides early.

The AI inventory

The organisation keeps a list of every AI system in use, built or bought: what it does, what data and systems it touches, its risk tier, its owner, and when it was last reviewed. The inventory is the first thing an auditor, a customer's security questionnaire or an incident responder will ask for, and the thing most organisations discover they do not have.

Choosing a framework

The organisation adopts one framework as its structure, usually the one its customers or auditors already expect, and maps its policy and inventory onto it. It does not run three frameworks in parallel, and it does not treat certification as a substitute for the decisions the framework asks it to make.

At team scope the object is the team's own use: which approved tools it uses for which data, how it reviews the AI systems it builds against injection and agency, and when it raises a use for wider review. The manager makes sure the team knows the policy and follows it without friction. At organisational scope the object becomes the policy itself, the data-flow register, the AI inventory and the intake that catches regulated uses, set once across teams and owned by named people. At executive scope the questions are accountability and exposure: who answers for the organisation's AI systems, how its governance would look to a regulator, a customer or a court, and whether the framework it has adopted is being lived or only certified (AI-1 Adoption strategy and staged bets).

• Traces where data goes when the team uses an AI tool, and decides what may be sent. team
• Reviews any system that reads untrusted content and can act, against the named injection and agency risks, before release. team
• Treats provenance and licensing questions about generated code as engineering risks with owners. team
• Failure mode: assuming a tool's data terms without reading them. team
• Failure mode: giving an assistant that reads email the ability to send it. team
• Sets AI policy in tiers, so low-risk use is easy and high-risk use is reviewed. org
• Recognises when a use may be regulated, such as screening job applicants, and involves legal or compliance at the idea stage. org
• Keeps a data-flow register and an AI inventory with named owners. org
• Adopts one recognised framework and maps policy onto it. org
• Failure mode: a blanket ban that engineers route around with personal accounts. org
• Failure mode: discovering at launch that a feature falls into a regulated category. org
• Failure mode: an inventory that lists the bought tools and forgets the ones the teams built. org
• Keeps accountability for AI systems with named people as teams and vendors change. exec
• Failure mode: treating certification as proof that governance is working. exec

Control versus adoption. Rules strict enough to remove the risk get worked around; rules loose enough to be followed leave risk behind. The judgment is proportion: easy paths for low-risk use, real review for high-risk use, and an approved tool for every common need, so that the workaround is never the easy option.

Speed to market versus regulatory review. Involving legal early feels like delay; involving them late can mean rebuilding. The judgment is to ask the regulated-use questions at the idea stage, when the answer costs a conversation rather than a redesign.

Agent capability versus attack surface. Every tool an assistant can use is a tool a hostile input can try to use. The judgment is to grant capability for the task and confirm consequential actions, accepting that some convenience is the price of not being steered.

Framework compliance versus real governance. A certificate is evidence a structure exists, not that its decisions are good. The judgment is to use the framework for its questions and to keep asking whether the answers are still true.

A director of engineering runs four product teams at a company that sells software to human-resources departments. A product manager proposes a feature that ranks job applicants by how well their CVs match a role, using a language model, and a large customer wants it within the quarter. At the same time, a security engineer reports that a support team has been pasting customer tickets, which include employee names and salary details, into a public chatbot to draft replies.

The easy responses are to build the ranking feature quickly, because the customer wants it, and to ban public chatbots outright. The first risks building a system that falls into a regulated category without anyone having asked, on data the company may not be allowed to use that way. The second will be obeyed for a week and then routed around, because the support team still has to answer tickets and now has no tool to help.

The director deals with the data flow first, because it is happening now. The security engineer confirms which provider the support team used and what its terms say about retention and training. The director asks for an approved assistant with enterprise terms (no training on inputs, a defined retention period, processing in the right region) to be made available to support within days, adds it to the data-flow register, and has the policy say plainly that customer data goes only to approved tools. The support team is told what happened and why, without blame; the gap was the absence of an approved option.

The ranking feature goes through intake. It ranks people for hiring, so the director asks legal before any design work. Legal's view is that a system that filters or evaluates job applicants is very likely high-risk under the EU regulation for the company's European customers, with obligations on the company as provider and on customers as deployers. The director takes that to the product manager and the customer with three options: build the feature with the controls and documentation the obligations require, on a longer timeline; build a narrower feature that helps recruiters search and read CVs without ranking or filtering people; or decline. The customer chooses the narrower feature for this quarter and the full feature on the longer timeline.

The narrower feature still reads documents uploaded by outsiders, so the design review covers injection: the model's output is shown to a recruiter, never acted on, and it has no tools. The feature, the support assistant and the provider are added to the AI inventory with owners. When the customer's procurement team later sends a security questionnaire about AI, the director answers it from the register and the inventory in an afternoon.

• TJ-5 AI-assisted engineering: the baseline judgment about governing AI assistance in proportion to risk.
• TJ-6 Security, privacy, and compliance as engineering responsibilities: security and privacy as engineering work, including data sent to tools.
• AI-1 Adoption strategy and staged bets: the adoption decisions that governance has to keep pace with.
• AI-3 AI-assisted software delivery: agent permissions and review rules in delivery.
• AI-5 Operating AI systems in production: running injections and data exposures as incidents.

1. An engineer asks whether they may use AI to draft replies to customer tickets. What question should the policy answer instead?
   Answer

   Which approved tool handles this data. The data-flow register records what data classes each approved tool may receive and on what terms, so the decision is about the data flow, not about AI in general.
2. An assistant reads inbound customer email and can look up and send account details. What is the risk, and what is the design response?
   Answer

   Indirect prompt injection: instructions planted in an email can steer the assistant into misusing its tools, for example sending data to an attacker. Cut its permissions to what a hostile input could safely trigger, require a person to confirm consequential actions, and treat its output as untrusted.
3. A product manager proposes using a model to rank job applicants. What should the manager do first?
   Answer

   Send it to legal or compliance before design starts. Systems that filter or evaluate job applicants appear in the EU AI Act's list of high-risk uses, and the manager's job is to recognise that and get a qualified view early, not to interpret the regulation.
4. Why does a blanket ban on public AI tools usually fail?
   Answer

   Because the work it was helping with still has to be done, so people route around the ban with personal accounts, and the risk moves out of sight. The effective response is an approved tool with acceptable terms for each common need, and a clear rule about which data goes where.
5. What does an AI inventory contain, and who needs it?
   Answer

   Every AI system in use, built or bought: what it does, what data and systems it touches, its risk tier, its owner and its last review. Auditors, customers' security questionnaires and incident responders all ask for it first.
6. What does a risk management framework give a manager, and what does it not?
   Answer

   A structured, recognised set of questions (for example govern, map, measure, manage) and a shape that auditors and customers understand. It does not make the decisions, and certification against it is not proof that the decisions are good.

• Project (2025): named risks, including prompt injection and excessive agency.
• Greshake et al. (2023): indirect prompt injection against real applications.
• Carlini et al. (2021): extracting training data from a language model.
• Technology (2023): the govern, map, measure and manage functions.
• ISO/IEC (2023): requirements for an AI management system.
• Centre et al. (2023): secure development of AI systems across the life cycle.
• Union (2024): risk-based regulation and its list of high-risk uses.