AI agents change the security boundary

The boundary changes when the model receives authority

A chatbot that drafts text has a limited action surface. An agent that can search internal data, read files, call APIs, change records, run code, send messages, approve workflows, or purchase services has been given operational authority. The security boundary now includes every system and identity through which that authority can flow.

The model remains important, but it is not the whole system. Instructions, conversation state, retrieved content, memory, tools, credentials, orchestration, approval logic, output handling, logs, and downstream services all influence what the agent can do.

The first security question is therefore not how intelligent the model is. It is what authority the system has, where that authority comes from, and which controls remain outside the model when its reasoning is wrong or manipulated.

Map the complete action surface

Create an inventory of models, system instructions, data sources, retrieval indexes, memory stores, tools, connectors, service accounts, human users, approval gates, destinations, and owners. For each tool, record the actions it exposes, data it can reach, identity it uses, environment it affects, and whether the action is reversible.

Follow indirect paths. A read-only document tool may return content that influences a write-capable ticketing tool. An email connector may provide untrusted instructions and also expose a send function. A browser may reach authenticated applications whose permissions are broader than the agent's stated role.

A diagram that stops at model input and output cannot support a meaningful review. The system map needs to show where data becomes instructions, where model output becomes an action, and where another control can permit, constrain, observe, or stop that action.

Untrusted data can become an instruction

Prompt injection is not limited to a user typing a malicious request. Instructions can arrive through retrieved documents, websites, emails, tickets, code repositories, images, tool responses, memory, or data supplied by another model. If the agent treats that content as authority, untrusted data can change its objective.

This is why prompt injection is also an authorization problem. The dangerous result is not that the model produced unusual text. It is that manipulated reasoning caused the system to access data, invoke a tool, disclose a secret, or complete an action that the originating user or content should not have been able to authorize.

Input filtering and model instructions can reduce some risk, but they are not a complete security boundary. The durable controls are narrow permissions, trusted context separation, deterministic validation, action constraints, and approvals that cannot be rewritten by the same content they are meant to govern.

Delegate identity deliberately

An agent should not inherit a broad service account merely because integration is easier. Decide whether each action runs as the end user, a constrained workload identity, a task-specific delegated token, or a separately approved privileged identity. The choice affects accountability, least privilege, revocation, logging, and incident scope.

Permissions should be limited by tool, action, resource, tenant, environment, sensitivity, amount, frequency, and duration where practical. Separate read and write capabilities. Avoid giving a planning component credentials it does not need. Use short-lived authorization and explicit delegation for higher-risk operations.

Identity context must survive the full chain. If a user asks an agent to update a record, the downstream service should be able to evaluate who requested the action, what the agent is permitted to do on that user's behalf, and whether policy allows that exact change.

Constrain tools outside the model

Tool descriptions are instructions to the model, not enforcement controls. The tool boundary should validate arguments, resource scope, allowed destinations, data classification, rate, size, and state transitions before execution. Deny by default when required context is missing or ambiguous.

Prefer narrow functions such as create a draft ticket for this project over general functions such as call any endpoint with these credentials. Parameterized queries, destination allowlists, schema validation, transaction limits, isolated execution, and safe defaults reduce the damage from reasoning errors and manipulated input.

Return values need the same treatment. Tool output can contain secrets, personal data, hidden instructions, or content from another tenant. Minimize the result, label its trust level, and prevent the agent from forwarding sensitive content to an unrelated tool or user.

Human approval must be specific and informed

A generic confirmation button is weak when the reviewer cannot see the exact action, target, data, consequence, and reason. Approval should be tied to a concrete operation after arguments have been resolved and validated. Material changes after approval should invalidate it.

Use risk-based gates. Reading public information may not need approval. Sending an external message, changing access, moving money, deleting data, executing code, or acting across tenants may require a named approver, stronger authentication, separation of duties, or a second control.

Approval fatigue is a security condition. If routine activity produces constant prompts, users will approve without examining them. Reduce authority, group low-risk actions safely, and reserve deliberate approval for decisions that genuinely need human judgment.

Observability must reconstruct authority and action

Traditional application logs may show that an API call occurred without showing the model context, originating user, retrieved source, policy decision, approval, or tool arguments that produced it. Agentic systems need a trace that connects these events while respecting data-minimization and confidentiality requirements.

For material actions, defenders may need the user and agent identity, model and policy version, input and retrieved-source references, selected tool, validated arguments, authorization result, approval record, execution outcome, and downstream changes. Raw prompts are not automatically the right answer because they can contain sensitive data.

Monitoring should detect unusual tool sequences, cross-tenant access attempts, repeated denials, new destinations, privilege changes, sensitive-data movement, high-risk actions, and divergence from the expected task. Teams also need a fast way to revoke tokens, disable tools, isolate memory, suspend automation, and preserve evidence.

Test the system, not a collection of clever prompts

AI security testing should begin with misuse cases linked to assets and consequences. Test direct and indirect prompt injection, unauthorized tool use, identity confusion, cross-tenant access, retrieval poisoning, memory contamination, sensitive-output handling, approval bypass, unsafe retries, and chained actions across tools.

Each case needs a starting state, trusted and untrusted inputs, expected control, success criteria, evidence plan, and cleanup method. Repeat tests across relevant models, configurations, roles, tools, and workflow states. Record variability instead of selecting only the most dramatic result.

A successful prompt is not yet a finding. Confirm whether the system crossed a security boundary, whether an external control stopped the action, what conditions were required, what business consequence was possible, and whether the evidence is reproducible enough to support remediation.

Govern the agent as an operational system

NIST's Generative AI Profile and the NCSC-led secure-development guidance both reinforce lifecycle risk management. For an agent, that means named ownership, approved purpose, documented authority, data and model provenance, change control, evaluation, monitoring, incident response, supplier dependencies, and retirement procedures.

Changes to a model, system instruction, retrieval source, tool, permission, orchestration rule, or provider can alter the effective security boundary. Material changes should trigger impact assessment and targeted regression testing rather than waiting for the next annual review.

The governance record should connect the use case to the controls that make it acceptable, the evidence that those controls operate, the exceptions that weaken them, and the person who owns the residual risk.

A practical minimum baseline

Before an agent reaches sensitive data or a consequential tool, establish an inventory, owner, purpose, data classification, threat model, identity design, least-privilege permissions, tool validation, approval rules, logging, monitoring, incident playbook, evaluation cases, and a tested way to stop activity.

Then challenge the complete path with realistic untrusted content and misuse scenarios. Fix the system boundary, not only the prompt that revealed the weakness. The security objective is not to make the model perfectly obedient. It is to ensure that uncertain model behaviour cannot silently become unauthorized organisational action.