AI Policy Wiki
Dashboard
⚲
AI Policy Wiki › Concepts & Frameworks

Prompt Injection

high confidence · updated 2026-08-10

The security vulnerability in which instructions embedded in untrusted content hijack an LLM's behavior — the defining vulnerability class for agentic AI systems.

Contents

Prompt injection is a security vulnerability in which an attacker embeds instructions into content the LLM processes, causing the model to treat those instructions as though they came from the legitimate operator or user. It is the defining vulnerability class for agentic AI systems and is widely considered unsolved.

Definition

  • Direct prompt injection. A user deliberately includes attack text in their own prompt to a chatbot (e.g., "ignore previous instructions and..."). The older, less serious variant.
  • Indirect prompt injection. Attack text is placed in external content (a web page, an email, a document) that the LLM subsequently reads. The LLM treats the embedded instructions as real and acts on them — despite the operator/user never having intended this. Formalized by Greshake et al., 2023.
  • Universal. In the agentic era, any content the model ingests — tool outputs, file contents, search results, emails, image EXIF, even pixels in OCR — is a potential injection surface.

Mechanism

LLMs have no intrinsic mechanism to distinguish "instructions from the principal" from "content that happens to contain instruction-like text." All tokens are in-distribution text. Training teaches a model to follow instructions; indirect injection exploits this disposition across trust boundaries.

Attack patterns include:

  • Instruction override. "Ignore previous instructions and do X."
  • Persona hijack. "You are now DAN (Do Anything Now)..."
  • Tool abuse. Instructions in web content tell the agent to exfiltrate user data via a tool call.
  • Confused-deputy. The agent has higher privileges than the content author; the content author uses the agent to act beyond their own authority.
  • Multimodal injection. Instructions hidden in images (visible or steganographic) processed by a vision-language model.

History / Origin Work

  • Simon Willison, Sep 2022 — coined the term "prompt injection" and has documented the problem class continuously; his blog remains the canonical tracker. See Simon Willison.
  • Greshake et al., "Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection," 2023 — formalized indirect injection as a distinct threat.
  • NIST AI 100-2 (Adversarial Machine Learning), 2024 — catalogued prompt injection as a canonical LLM attack category.
  • OWASP LLM Top 10, 2023–present — prompt injection consistently ranks #1.
  • Anthropic Computer Use / agent red-team reports, 2024–2025 — documented real injection vectors in browsing and desktop-control agents; see (Source: anthropic.com).

Policy and Safety Relevance

  • Agentic deployment risk. Agentic AI agents with tool access, long-horizon tasks, and exposure to untrusted content compound injection risk. Anthropic, OpenAI, and Google all publish agent deployment guidance that treats prompt injection as the dominant deployment hazard.
  • Regulatory attention. NIST AI 100-2 and the EU AI Act (Regulation 2024/1689) treat prompt injection as a security risk for which GPAI providers must have mitigations.
  • Liability. Who is liable when an injected email causes an agent to wire money or leak data? See AI liability discussions — the principal-vs-deployer-vs-content-author allocation is unresolved.
  • Safety evaluations. AISI and AI safety institutes include prompt-injection robustness in frontier evaluations.

Production Infrastructure Defense (from Managed Agents)

Anthropic's Managed Agents architecture documents a structural injection defense beyond training-based approaches: credential isolation via vault pattern.

The key insight: if credentials are never reachable from the sandbox where agent-generated code runs, prompt injection cannot be exploited for credential theft. Implementation:

  • For Git: access token used at clone time, wired into local git remote; agent pushes/pulls without ever handling the token
  • For MCP tools: OAuth tokens stored in a secure vault; Claude calls tools via a proxy that fetches credentials; agent never sees them
  • Harness is never credential-aware

This is a structural fix, not a training or filtering fix — it removes the reachability problem rather than trying to limit what injected code can do with reachable tokens. Anthropic explicitly notes this is superior to narrow token scoping, because scoping "encodes an assumption about what Claude can't do with a limited token — and Claude is getting increasingly smart."

The four-component agent model (from Trustworthy Agents in Practice (Anthropic, April 2026)) reinforces this: "The more open an agent's environment, the more entry points exist. The more tools it can use, the more an attacker can do once they gain access." Injection defense must be built into harness, tool, and environment design — not just model training.

Defense Approaches

No defense is considered robust in isolation; production systems layer them.

  • Delimiters / structured prompts. Instructing the model to treat certain regions as data, not instructions. Easily defeated by sophisticated attacks.
  • Fine-tuning for instruction hierarchy. OpenAI's "instruction hierarchy" (2024) trains models to privilege system > developer > user > tool-output instructions. Partially effective.
  • Input filtering. Detect and strip injection patterns before the LLM sees them. Arms-race dynamic; attackers evolve phrasings.
  • Output verification. Before tool calls execute, verify they match user intent (often via a second model or rule-based checker).
  • Capability restriction. Limit what tools an agent can call; require human-in-the-loop approval for dangerous actions.
  • Sandboxing. Execute agent actions in isolated environments where injection-induced actions have limited blast radius.
  • Provenance and signing. Track which content regions came from which trust domain; down-weight instructions from low-trust regions.
  • Dual-LLM patterns. A "privileged" LLM controls tools; an "unprivileged" LLM processes untrusted content and can only return data.
  • Pre-installation skill scanning. Tooling that inspects installable agent "skills" for injection and other malicious behavior before they are added to an agent. On June 20, 2026 NVIDIA published SkillSpector, an open-source scanner that inspects agent skills for prompt injection, data exfiltration, and malicious code prior to installation, scoring each skill from 0 to 100. It was released as a response to the security surface created by installable agent skills (Source: github.com).

Current State

  • Prompt injection is the single most commonly cited reason frontier agentic systems are gated or limited in production.
  • No lab claims to have "solved" it; mitigations are partial and composable.
  • Anthropic Computer Use, OpenAI Operator, and Google Agent Space all include injection-specific safeguards and warnings.

The August 2026 default change in Claude Code is the first case of a lab citing injection-resistance evidence as grounds for removing a human-approval gate. Anthropic said on August 8, 2026 that auto mode would become the default for new Claude Code sessions on Pro, Max and Team plans from August 14, and published an evaluation it commissioned from Trajectory Labs: 72 indirect prompt-injection scenarios, held out from Anthropic, run against the publicly available versions of Claude Code and Codex as of July 17, 2026, in which none of the 720 attack attempts succeeded against Claude Fable 5, Claude Opus 5 or Claude Sonnet 5 in auto mode. A separate commissioned study of 1,053 paid testers substituted a clearly dangerous command for one permission prompt per session, displayed as text with nothing executed; 13.6 percent of human testers refused it, against Anthropic's report that auto mode blocked 89 percent of the same actions.

Simon Willison, who coined the term, accepted the premise that per-action human approval is unworkable because of confirmation fatigue, but wrote that the unblocked 11 percent still matters and that he wants independent confirmation before treating prompt injection as solved (Source: simonwillison.net). Both evaluations were commissioned by the lab whose default they justify; the 720-attempt result is scoped to the scenarios tested and to model and harness versions as of mid-July 2026, and a zero-success rate on a held-out scenario set is not a general claim about the vulnerability class.

Real-World Cases

  • Meta AI support-chatbot account takeover (June 1, 2026). Attackers seized control of high-profile Instagram accounts — including those of the Obama White House and the Chief Master Sergeant of the U.S. Space Force, which were briefly defaced with pro-Iranian messaging — by socially engineering Meta's own AI support chatbot into granting account access. The case illustrates the confused-deputy pattern on a customer-support surface: the support agent held privileges higher than the untrusted party it was conversing with, and the attackers manipulated its conversational logic into approving access it should not have granted. It is a concrete instance of an AI support agent acting as an authorization / account-takeover vector, rather than a software-bug exploit. Subsequent reporting added the retailer Sephora to the list of compromised accounts and quoted a Sophos researcher characterizing the technique as a prompt-injection attack likely to become more common as services deploy chatbots without adequate protections. (Sources: krebsonsecurity.com; techcrunch.com; theguardian.com; 404media.co)

A related but distinct attack targets what an agent retrieves rather than what it is instructed. The WARP attack (Deep-Research Agents Can Be Poisoned via User-Generated Content (Zhang, Triedman & Shmatikov, 2026)) exploits the fact that deep-research agents repeatedly retrieve the same user-generated pages — Wikipedia, Reddit, community forums — across the many related queries of a single session. Appending short crafted text to one frequently retrieved page can therefore steer citations "across many related queries, regardless of the user's specific query." The threat model is weak by design: no control over retrieval, no knowledge of the agent's model, prompts, or the user's query, and no ability to inject new documents — only the ability to edit existing ones.

A third variant places the payload where neither the user nor an external monitor can read it. Stealing Reasoning Traces from Proprietary LLM APIs (arXiv:2608.09867, 10 August 2026) shows that the encrypted chain-of-thought blocks Anthropic, OpenAI and Google return to clients are replayable across sessions, users and models, so an adversary can plant an opaque block that has internalized a malicious instruction into a published agent trace and have a victim resume the run. In one demonstration the authors captured a thought from o4-mini internalizing an instruction to upload PowerPoint files to an attacker domain, ported the block to GPT-5.6 Sol, and gave it an unrelated slide-editing request; the model produced a script that both made the edit and uploaded the deck to the attacker's server. In a second, data-exfiltration instructions injected into PostTrainBench long-horizon traces via a Haiku 4.5 and Opus 4.7 pair caused Opus 4.7 to upload a file after every change when the trace was continued. The distinguishing feature is location rather than technique: because the payload sits in hidden reasoning rather than visible text, it is inaccessible both to the user and to monitors that inspect only the observable conversation. Providers patched the replay behaviour after disclosure.

The term's originator, Simon Willison, coined it "to describe this key issue of mixing together trusted and untrusted content in the same context," naming it after SQL injection, and maintains a distinction now often collapsed: attacks in which a user directly tricks a model into embarrassing output are jailbreaking, "a different issue than prompt injection." His lethal trifecta formulation names the conjunction that makes exfiltration possible — access to private data, exposure to untrusted content, and the ability to communicate externally — and argues the root cause is architectural: "LLMs are unable to reliably distinguish the importance of instructions based on where they came from. Everything eventually gets glued together into a sequence of tokens and fed to the model." He notes that vendors have historically fixed reported instances by closing the exfiltration path rather than the first two legs, and that MCP makes the combination easy to assemble accidentally, leaving users who mix their own tools unprotected: "there's nothing those vendors can do to protect you."

Relationships

Categories: Concepts & Frameworks | technical | safety | agentic | security
What links here (36)