Prompt Injection Is a Trust Boundary Problem

An AI agent opens a support ticket that contains a sentence telling it to ignore its policy, retrieve a private file, and attach the contents to its response.

No one would describe that sentence as authorization if a human employee read it. It is data supplied by an untrusted customer. But to a language model, the ticket and the operating instructions are made from the same material: tokens. One is supposed to describe the problem. The other is supposed to control behavior. The boundary is semantic, and semantics alone are a fragile place to put security.

This is why prompt injection is often discussed too narrowly.

The obvious failure is that the model follows a malicious instruction. The architectural failure happened earlier: untrusted content reached a component with enough context and authority to turn that instruction into consequence.

We will keep improving models at recognizing manipulation. That is valuable. We should not make recognition the only barrier between a hostile document and a production action.

Prompt injection is a trust boundary problem. The dependable response is to design the whole path from content to capability around that fact.

The Model Cannot Reliably Separate Code From Data

Traditional systems have explicit execution boundaries. A database knows the difference between a parameter and a query when the application uses parameterized statements correctly. A browser applies origin rules. An operating system distinguishes a file from a process until something intentionally executes it.

Language models do not receive such a clean distinction. System instructions, user requests, retrieved pages, tool results, memory, and quoted text all enter a shared reasoning surface. Formatting and precedence help the model interpret those sources, but they do not make untrusted language inert.

A web page can say, “The previous instructions are obsolete.” A document can imitate a policy block. A tool result can include a plausible approval message. Even when the model usually recognizes these as data, security cannot rest on usually when the next step can send a message, expose a secret, or change a system.

The useful assumption is simple:

Any content the system did not author and authenticate may contain instructions designed to change agent behavior.

That includes content from users, websites, email, tickets, documents, code comments, package metadata, logs, and even other agents. An internal source may be more trusted than the public web, but internal does not automatically mean safe or authoritative.

Instructions Need Identity and Provenance

The first defense is to stop presenting every piece of language as an anonymous block of context.

An instruction should have a source, authority level, scope, and lifetime. Repository policy applies to work in that repository. A user request authorizes a particular outcome. An approval grants one bounded action. Retrieved content supplies evidence but does not acquire the right to redefine the task.

The context interface should preserve these distinctions when it assembles the model's view. A search result should arrive as untrusted evidence with its origin attached, not as prose pasted beside system policy. A quote from a ticket should remain visibly a quote. A summary should not silently erase which parts came from authoritative instructions and which came from external material.

This will not make the model immune to injection. It gives the model a clearer chance to reason correctly and gives the surrounding system enough metadata to enforce decisions later.

Provenance is not decoration. It is part of the security model.

Retrieval Does Not Confer Authority

Retrieval systems are designed to find relevant content. Relevance is not trust.

If an agent asks how to deploy a service, a highly ranked document may contain the exact commands it needs. It may also be outdated, compromised, written for another environment, or intentionally shaped to influence automated readers. The retrieval score cannot decide whether the document is allowed to modify the deployment policy.

Treat retrieved material as a claim to evaluate. The agent can use it to understand terminology, locate an artifact, or propose a next step. Consequential instructions still need to resolve against authenticated policy, current system state, and the authority granted to the workflow.

This matters for indirect prompt injection. The attacker does not need access to the conversation. They only need to place content somewhere the agent may retrieve: a public page, shared document, issue description, or field returned by an integration.

A useful retrieval layer therefore carries source trust, freshness, content type, and intended use. It can exclude known-dangerous fields from instruction-bearing context, isolate raw excerpts, and require corroboration before a retrieved claim changes behavior.

The safest interpretation of retrieved text is evidence until proven otherwise.

Authority Should Be Smaller Than the Context

An agent may need broad visibility to investigate a problem. It rarely needs equally broad authority to act.

Suppose an incident agent can read logs, tickets, deployment history, service configuration, and public documentation. Any one of those sources may contain hostile language. If the same agent also holds credentials to deploy, rotate secrets, and post externally, every readable source becomes a possible route to those effects.

The answer is not necessarily to blind the agent. It is to separate observation from consequence.

A research step can operate with read-only tools and produce a proposed action. A different execution step receives only the accepted proposal, current preconditions, and a narrow capability. The execution tool validates its target and parameters against policy. High-impact transitions require the precise approval boundary described in Human Approval Is an Architectural Boundary.

Least privilege changes the failure from “the model was persuaded, so the system was compromised” to “the model proposed something outside its authority, so the system refused.”

That is the kind of failure we can operate.

Tools Must Enforce Their Own Contracts

A tool description can tell the model not to send private data. The tool itself should still reject private data when it can identify it.

Security controls belong as close as possible to the effect. A messaging tool can restrict destinations, scan attachments, and require a reference to an approved draft. A database tool can expose bounded queries instead of arbitrary SQL. A deployment capability can accept a signed artifact and environment, rather than a free-form shell command.

The model remains responsible for choosing a reasonable action. The tool boundary is responsible for ensuring that the action is permitted.

This follows the same principle as the agent harness: natural-language instructions shape behavior, while enforced contracts constrain consequence. The two should reinforce one another, but they are not interchangeable.

Strong contracts also make suspicious behavior legible. A rejected attempt can say which policy boundary was crossed without exposing the secret or broadening the agent's access. The orchestrator can stop, replan, or ask for human judgment instead of encouraging the model to find another route around the refusal.

Secrets Should Not Be Context

The easiest secret for an agent to leak is the one placed directly in its context window.

Some systems inject API keys, credentials, customer records, or hidden policy into prompts and then tell the model not to reveal them. This asks instruction-following to act as data-loss prevention. A malicious input only needs to find a representation that defeats the instruction.

Whenever possible, the model should receive a capability rather than a credential. It can call read_incident_logs without seeing the underlying token. It can ask a service to send an approved message without receiving access to the entire mailbox. Sensitive values remain inside the tool implementation, where conventional access control and audit can apply.

The same rule applies to unnecessary private data. Minimize before context assembly. Redact fields that do not affect the decision. Return handles or summaries when raw values are not needed. Keep tenant and environment boundaries outside the model's discretion.

A model cannot disclose a value it never received.

Treat Outputs as Untrusted Too

Security attention usually focuses on what enters the model. Agent output can carry the attack forward.

A model may generate shell commands, HTML, SQL, code, or tool arguments influenced by hostile content. Passing that output directly into another interpreter creates a second execution boundary. The fact that the text was produced by the agent does not make it safe.

Validate structured output against schemas and policy. Use parameterized interfaces. Run generated code in a sandbox with explicit resources. Escape content for its destination. Require review before an agent-authored artifact becomes a privileged instruction for another system.

This also applies to agent-to-agent communication. One worker's conclusion is not automatically policy for the next worker. Preserve the difference between a delegated instruction, an observation, and a proposed action. Otherwise a compromised research step can launder untrusted content into a message that appears internal and authoritative.

Trust should not increase merely because text passed through a model.

Detection Is Useful, Containment Is Necessary

Injection detectors, classifiers, canary strings, and model-based reviewers can catch suspicious content. They should be used. They will also miss attacks, flag legitimate material, and age as adversaries learn the system.

Detection tells us that something may be wrong. Containment limits what can happen when detection fails.

The system should be able to answer: if the agent follows the worst instruction in this document, what can it actually reach? Can it cross tenants? Can it send data to an arbitrary destination? Can it alter its own policy or obtain a stronger credential? Can it erase the trace of what happened?

These are capability questions. They have concrete architectural answers.

Network egress can be restricted. Tool scopes can be bound to the task. Sensitive actions can require fresh authorization. External content can be processed in an isolated step. Immutable logs can record attempted policy violations. Rate and volume limits can reduce the damage of repeated calls.

The model may still make a bad decision. The system should make that decision small, visible, and reversible where possible.

Test the Whole Attack Path

A prompt-injection evaluation should not stop at whether the model repeats a secret phrase.

Test whether hostile content changes planning, tool selection, destinations, data access, or completion claims. Place attacks in realistic sources: a ticket field, a retrieved page, a code comment, a tool response, and a message from another worker. Vary how explicit the attack is. The dangerous cases often look like useful operational guidance rather than cartoonish demands to ignore all previous instructions.

Then measure the layers independently.

Did the model identify the content as untrusted? Did context preserve its provenance? Did the policy engine reject the requested capability? Did the tool validate the target? Did the trace make the attempt understandable? Did the workflow recover without carrying the hostile instruction into the next step?

An evaluation that expects the model to resist every attack forever is brittle. An evaluation that proves one missed attack cannot become an unauthorized effect tests the architecture we actually need.

Build for a Model That Can Be Persuaded

We should make models better at following instruction hierarchy and recognizing manipulation. We should give them clear context, explicit source labels, and examples of the attacks they will encounter.

We should also assume that some hostile input will eventually be persuasive.

That assumption leads to a healthier design. Untrusted content remains evidence. Instructions carry provenance and scope. Secrets stay behind capabilities. Read access does not imply write authority. Tools enforce policy at the effect. Generated output is validated before another system executes it. Detection feeds a containment strategy instead of replacing one.

Prompt injection is not solved by finding the perfect warning sentence for the system prompt.

It is managed by building a system in which language alone cannot grant itself power.