SafetyPublished ExplainedAll news from that day
How prompt injection works and how to limit the damage
OWASP, Anthropic and OpenAI describe prompt injection as untrusted text that a model may follow as an instruction, and the guidance centres on limiting what a fooled model can do.
Check our sources · 25 facts from 4 sources
Illustration made with AI for ai notis
Key points
- OWASP says a prompt injection occurs when prompts alter a model's behaviour in unintended ways, either directly from the user or indirectly through external sources such as websites or files.
- OWASP says it is unclear whether fool-proof prevention exists, and that retrieval augmented generation and fine-tuning "do not fully mitigate" prompt injection vulnerabilities.
- OWASP lists least privilege, human approval for high-risk actions and separating external content; Anthropic adds running tools in sandboxed environments, and OpenAI says agents can still be tricked.
What happened
OWASP, the open security project, defines a prompt injection vulnerability as one that occurs when user prompts alter the behaviour or output of a language model in unintended ways. The inputs do not need to be readable by a person, as long as the model parses the content.
OWASP separates two kinds. In a direct injection, the user's own prompt changes the model's behaviour. In an indirect injection, the model accepts input from an external source such as a website or a file, and that content changes its behaviour.
Anthropic's documentation draws the same line by who the adversary is: in the first case the user of the application is hostile, and in the second the user is trusted but Claude reads third-party content, such as web pages, emails, documents and tool results, that contains adversarial instructions.
OpenAI's guidance for agent builders describes the goals of an attacker as exfiltrating private data through downstream tool calls, taking misaligned actions, or otherwise changing the model's behaviour. OWASP says the severity depends on the business context and on how much agency the model has been given.
It lists disclosure of sensitive information, unauthorised access to functions available to the model, and execution of arbitrary commands in connected systems among the possible results.
OWASP says it is unclear whether any prevention is fool-proof, given how models work, and that retrieval-augmented generation and fine-tuning do not fully mitigate the problem. Its measures therefore limit the impact. They include giving the model the least privilege it needs, requiring human approval for high-risk actions, and separating and marking untrusted external content.
Anthropic's documentation applies the same ideas. It says to put third-party content only in tool results, to tell Claude what the content is and where it came from, and to state in the system prompt that returned content is untrusted data and must never override the user's request.
It also says to wrap untrusted strings in JSON, to limit Claude's access to sensitive data and actions, and to run tools in sandboxed environments.
OpenAI's guidance adds that untrusted input should not be placed in developer messages, which take precedence over user messages, and that structured outputs between steps remove free-text channels an attacker could use.
Claude Code, Anthropic's coding agent, applies some of this in the product. Its documentation says commands that fetch content from the web, such as curl and wget, are not auto-approved by default, and that sandboxing can add filesystem and network isolation for shell commands. OpenAI's guidance states that even with its mitigations agents will not be perfect and can still be tricked.
Underlined sentences link to the facts we checked. Dotted words open a short explainer.
Fact 13 of 25 · from Anthropic (Claude Platform Docs)
where the user is trusted but Claude processes third-party content (web pages, emails, documents, tool results) that contains adversarial instructions
What it means for you
Our viewThe vendors agree a model cannot be relied on to refuse every injected instruction, so the design question becomes what a fooled model can reach. OWASP ties the severity to the agency the model has been given, and OpenAI says that even with mitigations agents can still make mistakes or be tricked.
For an agent that reads email, web pages or documents, that points to reviewing its permissions as well as its prompt. Ask your team or vendor which tools the agent can call, which need human approval, which run in a sandbox, and whether untrusted content is labelled as data. Start with the one tool that could send data out.
It adds no new facts.
Your reaction
We count reactions per story and day, never who reacted. The counts help us choose what goes in the monthly issue. If you are signed in, your own page shows yours too.
We count reactions per story and day, never who reacted. They help choose what goes in the monthly issue.
Check our sources
Every sentence above is checked against these 4 sources.
1 LLM01:2025 Prompt Injection
Open the source archived copy-
OWASP's LLM01:2025 page says: "A Prompt Injection Vulnerability occurs when user prompts alter the LLM’s behavior or output in unintended ways."
citeLLM01:2025 Prompt Injection A Prompt Injection Vulnerability occurs when user prompts alter the LLM's behavior or output in unintended ways.
-
OWASP says these inputs "can affect the model even if they are imperceptible to humans" and that prompt injections "do not need to be human-visible/readable, as long as the content is parsed by the model".
citeThese inputs can affect the model even if they are imperceptible to humans, therefore prompt injections do not need to be human-visible/readable, as long as the content is parsed by the model.
-
OWASP says direct prompt injections occur "when a user’s prompt input directly alters the behavior of the model in unintended or unexpected ways".
citeDirect prompt injections occur when a user's prompt input directly alters the behavior of the model in unintended or unexpected ways.
-
OWASP says indirect prompt injections occur "when an LLM accepts input from external sources, such as websites or files" and that the content, when interpreted by the model, alters its behavior in unintended or unexpected ways.
citeIndirect prompt injections occur when an LLM accepts input from external sources, such as websites or files. The content may have in the external content data that when interpreted by the model, alters the behavior of the model in unintended or unexpected ways.
-
OWASP says the severity and nature of a successful prompt injection "are largely dependent on both the business context the model operates in, and the agency with which the model is architected".
citeare largely dependent on both the business context the model operates in, and the agency with which the model is architected
-
OWASP lists "Providing unauthorized access to functions available to the LLM" and "Executing arbitrary commands in connected systems" among the unintended outcomes of prompt injection.
citeprompt injection can lead to unintended outcomes, including but not limited to: Disclosure of sensitive information ... Providing unauthorized access to functions available to the LLM ... Executing arbitrary commands in connected systems
-
OWASP says: "it is unclear if there are fool-proof methods of prevention for prompt injection".
citeit is unclear if there are fool-proof methods of prevention for prompt injection
-
OWASP says that techniques like Retrieval Augmented Generation (RAG) and fine-tuning aim to make outputs more relevant and accurate, but "research shows that they do not fully mitigate prompt injection vulnerabilities".
citeresearch shows that they do not fully mitigate prompt injection vulnerabilities
-
OWASP's listed mitigations include "Enforce privilege control and least privilege access", "Require human approval for high-risk actions", and "Segregate and identify external content".
citeEnforce privilege control and least privilege access ... Require human approval for high-risk actions ... Segregate and identify external content
-
OWASP says to "Separate and clearly denote untrusted content to limit its influence on user prompts".
citeSeparate and clearly denote untrusted content to limit its influence on user prompts.
-
OWASP's LLM01:2025 page lists "Disclosure of sensitive information" first among the unintended outcomes prompt injection can lead to.
citeprompt injection can lead to unintended outcomes, including but not limited to: Disclosure of sensitive information
2 Mitigate jailbreaks and prompt injections
Open the source archived copy-
Anthropic's documentation says jailbreaks and direct prompt injection are the case "where the user of your application is the adversary and crafts inputs intended to bypass your guardrails".
citewhere the user of your application is the adversary and crafts inputs intended to bypass your guardrails
-
Anthropic's documentation says indirect prompt injection is the case "where the user is trusted but Claude processes third-party content (web pages, emails, documents, tool results) that contains adversarial instructions".
citewhere the user is trusted but Claude processes third-party content (web pages, emails, documents, tool results) that contains adversarial instructions
-
Anthropic's documentation says: "Put untrusted content only in tool results." and says Claude is trained to treat instructions that appear inside tool results with appropriate skepticism.
citePut untrusted content only in tool results. Deliver third-party content to Claude inside tool_result blocks, never in system prompts or plain user text blocks. Claude is trained to treat instructions that appear inside tool results with appropriate skepticism.
-
Anthropic's documentation says to tell Claude what the content is and where it came from, for example that it is the body of an inbound email from an unknown sender.
citeTell Claude what the content is and where it came from. In the tool's description, or in the structure of the result itself, make the nature and source of the content explicit: for example, that it is the body of an inbound email from an unknown sender
-
Anthropic's documentation says to tell Claude explicitly that content returned from tools, documents, or searches is untrusted data and must never override the system prompt or the user's original request.
citeState the policy in your system prompt. Tell Claude explicitly that content returned from tools, documents, or searches is untrusted data and must never override the system prompt or the user's original request.
-
Anthropic's documentation says: "JSON-encode untrusted content."
citeJSON-encode untrusted content. Where possible, wrap third-party strings in a JSON object rather than concatenating them into free-form text.
-
Anthropic's documentation says to apply the principle of least privilege, including "run tools in sandboxed environments".
citeApply the principle of least privilege so that a successful injection can do minimal damage: don't give Claude access to secrets it doesn't need, run tools in sandboxed environments, and scope permissions as narrowly as possible.
3 Safety in building agents
Open the source archived copy-
OpenAI's guide on safety in building agents says: "A prompt injection happens when untrusted text or data enters an AI system, and malicious contents in that text or data attempt to override instructions to the AI."
citeA prompt injection happens when untrusted text or data enters an AI system, and malicious contents in that text or data attempt to override instructions to the AI.
-
OpenAI's guide says the end goals of prompt injections "can include exfiltrating private data via downstream tool calls, taking misaligned actions, or otherwise changing model behavior in an unintended way".
citecan include exfiltrating private data via downstream tool calls, taking misaligned actions, or otherwise changing model behavior in an unintended way
-
OpenAI's guide says: "Because developer messages take precedence over user and assistant messages, injecting untrusted input directly into developer messages gives attackers the highest degree of control."
citeBecause developer messages take precedence over user and assistant messages, injecting untrusted input directly into developer messages gives attackers the highest degree of control.
-
OpenAI's guide says that by defining structured outputs between nodes, such as enums and fixed schemas, "you eliminate freeform channels that attackers can exploit to smuggle instructions or data".
citeyou eliminate freeform channels that attackers can exploit to smuggle instructions or data
-
OpenAI's guide says: "even with these mitigations, agents won’t be perfect and can still make mistakes or be tricked".
citeHowever, even with these mitigations, agents won't be perfect and can still make mistakes or be tricked
4 Security - Claude Code Docs
Open the source archived copy-
Claude Code's security documentation says: "Commands that fetch content from the web such as curl and wget are not auto-approved by default."
citeCommands that fetch content from the web such as curl and wget are not auto-approved by default.
-
Claude Code's security documentation lists a sandboxed bash tool that provides "filesystem and network isolation" for bash commands.
citeSandboxed bash tool: Sandbox bash commands with filesystem and network isolation, reducing permission prompts while maintaining security.
Every sentence above is checked against these 4 sources. Each fact has its own address you can cite.
Source 1 · OWASP Gen AI Security Project
LLM01:2025 Prompt Injection
OWASP's LLM01:2025 page says: "A Prompt Injection Vulnerability occurs when user prompts alter the LLM’s behavior or output in unintended ways."
LLM01:2025 Prompt Injection A Prompt Injection Vulnerability occurs when user prompts alter the LLM's behavior or output in unintended ways.
OWASP says these inputs "can affect the model even if they are imperceptible to humans" and that prompt injections "do not need to be human-visible/readable, as long as the content is parsed by the model".
These inputs can affect the model even if they are imperceptible to humans, therefore prompt injections do not need to be human-visible/readable, as long as the content is parsed by the model.
OWASP says direct prompt injections occur "when a user’s prompt input directly alters the behavior of the model in unintended or unexpected ways".
Direct prompt injections occur when a user's prompt input directly alters the behavior of the model in unintended or unexpected ways.
Show all 11 factsShow fewer facts
OWASP says indirect prompt injections occur "when an LLM accepts input from external sources, such as websites or files" and that the content, when interpreted by the model, alters its behavior in unintended or unexpected ways.
Indirect prompt injections occur when an LLM accepts input from external sources, such as websites or files. The content may have in the external content data that when interpreted by the model, alters the behavior of the model in unintended or unexpected ways.
OWASP says the severity and nature of a successful prompt injection "are largely dependent on both the business context the model operates in, and the agency with which the model is architected".
are largely dependent on both the business context the model operates in, and the agency with which the model is architected
OWASP lists "Providing unauthorized access to functions available to the LLM" and "Executing arbitrary commands in connected systems" among the unintended outcomes of prompt injection.
prompt injection can lead to unintended outcomes, including but not limited to: Disclosure of sensitive information ... Providing unauthorized access to functions available to the LLM ... Executing arbitrary commands in connected systems
OWASP says: "it is unclear if there are fool-proof methods of prevention for prompt injection".
it is unclear if there are fool-proof methods of prevention for prompt injection
OWASP says that techniques like Retrieval Augmented Generation (RAG) and fine-tuning aim to make outputs more relevant and accurate, but "research shows that they do not fully mitigate prompt injection vulnerabilities".
research shows that they do not fully mitigate prompt injection vulnerabilities
OWASP's listed mitigations include "Enforce privilege control and least privilege access", "Require human approval for high-risk actions", and "Segregate and identify external content".
Enforce privilege control and least privilege access ... Require human approval for high-risk actions ... Segregate and identify external content
OWASP says to "Separate and clearly denote untrusted content to limit its influence on user prompts".
Separate and clearly denote untrusted content to limit its influence on user prompts.
OWASP's LLM01:2025 page lists "Disclosure of sensitive information" first among the unintended outcomes prompt injection can lead to.
prompt injection can lead to unintended outcomes, including but not limited to: Disclosure of sensitive information
Source 2 · Anthropic (Claude Platform Docs)
Mitigate jailbreaks and prompt injections
Anthropic's documentation says jailbreaks and direct prompt injection are the case "where the user of your application is the adversary and crafts inputs intended to bypass your guardrails".
where the user of your application is the adversary and crafts inputs intended to bypass your guardrails
Anthropic's documentation says indirect prompt injection is the case "where the user is trusted but Claude processes third-party content (web pages, emails, documents, tool results) that contains adversarial instructions".
where the user is trusted but Claude processes third-party content (web pages, emails, documents, tool results) that contains adversarial instructions
Anthropic's documentation says: "Put untrusted content only in tool results." and says Claude is trained to treat instructions that appear inside tool results with appropriate skepticism.
Put untrusted content only in tool results. Deliver third-party content to Claude inside tool_result blocks, never in system prompts or plain user text blocks. Claude is trained to treat instructions that appear inside tool results with appropriate skepticism.
Show all 7 factsShow fewer facts
Anthropic's documentation says to tell Claude what the content is and where it came from, for example that it is the body of an inbound email from an unknown sender.
Tell Claude what the content is and where it came from. In the tool's description, or in the structure of the result itself, make the nature and source of the content explicit: for example, that it is the body of an inbound email from an unknown sender
Anthropic's documentation says to tell Claude explicitly that content returned from tools, documents, or searches is untrusted data and must never override the system prompt or the user's original request.
State the policy in your system prompt. Tell Claude explicitly that content returned from tools, documents, or searches is untrusted data and must never override the system prompt or the user's original request.
Anthropic's documentation says: "JSON-encode untrusted content."
JSON-encode untrusted content. Where possible, wrap third-party strings in a JSON object rather than concatenating them into free-form text.
Anthropic's documentation says to apply the principle of least privilege, including "run tools in sandboxed environments".
Apply the principle of least privilege so that a successful injection can do minimal damage: don't give Claude access to secrets it doesn't need, run tools in sandboxed environments, and scope permissions as narrowly as possible.
Source 3 · OpenAI (API docs)
Safety in building agents
OpenAI's guide on safety in building agents says: "A prompt injection happens when untrusted text or data enters an AI system, and malicious contents in that text or data attempt to override instructions to the AI."
A prompt injection happens when untrusted text or data enters an AI system, and malicious contents in that text or data attempt to override instructions to the AI.
OpenAI's guide says the end goals of prompt injections "can include exfiltrating private data via downstream tool calls, taking misaligned actions, or otherwise changing model behavior in an unintended way".
can include exfiltrating private data via downstream tool calls, taking misaligned actions, or otherwise changing model behavior in an unintended way
OpenAI's guide says: "Because developer messages take precedence over user and assistant messages, injecting untrusted input directly into developer messages gives attackers the highest degree of control."
Because developer messages take precedence over user and assistant messages, injecting untrusted input directly into developer messages gives attackers the highest degree of control.
Show all 5 factsShow fewer facts
OpenAI's guide says that by defining structured outputs between nodes, such as enums and fixed schemas, "you eliminate freeform channels that attackers can exploit to smuggle instructions or data".
you eliminate freeform channels that attackers can exploit to smuggle instructions or data
OpenAI's guide says: "even with these mitigations, agents won’t be perfect and can still make mistakes or be tricked".
However, even with these mitigations, agents won't be perfect and can still make mistakes or be tricked
Source 4 · Anthropic (Claude Code Docs)
Security - Claude Code Docs
Claude Code's security documentation says: "Commands that fetch content from the web such as curl and wget are not auto-approved by default."
Commands that fetch content from the web such as curl and wget are not auto-approved by default.
Claude Code's security documentation lists a sandboxed bash tool that provides "filesystem and network isolation" for bash commands.
Sandboxed bash tool: Sandbox bash commands with filesystem and network isolation, reducing permission prompts while maintaining security.
Topics
The morning email
On the mornings we publish: the three top stories and up to four short ones. Free.
We email you a link to confirm. An issue may include one sponsor, always labelled Sponsored · Advertisement. Our emails count opens and clicks, not who made them. Unsubscribe in one click. What we keep