Prompt injection involves manipulating model responses through specific inputs to alter its behavior, which can include bypassing safety measures. The EU AI Act specifically requires high-risk AI systems to be resilient against input manipulation. What is the difference between direct and indirect prompt injection? Prompt injection is a security vulnerability where attackers craft malicious inputs that trick AI language models into ignoring their original instructions and following attacker commands instead.
Data poisoning requires access to the training pipeline; indirect injection only requires the ability to place content where the model will retrieve it. The model may follow the injected instructions instead of answering the user’s actual question. Invest in fundamental research on reliable instruction following that maintains the separation between data and instructions. Build automated testing pipelines that evaluate each model update against indirect injection benchmarks like BIPIA.
Even if indirect injection manipulates the model’s intent, human review can catch unauthorized actions before they execute. While not a robust defense on its own (attackers can include fake delimiters in their payloads), it provides additional signal that helps the model distinguish instruction sources. OpenAI’s instruction hierarchy research demonstrated that training models to respect this priority ordering significantly reduces indirect injection success rates. Studies have shown that virtually all current LLMs are vulnerable to indirect injection to some degree, with attack success rates ranging from 20% to over 90% depending on the model, attack technique, and context. The BIPIA (Benchmark for Indirect Prompt Injection Attacks) dataset provides standardized evaluation of indirect injection defenses. While this attack is more subtle (influencing suggestions rather than hijacking responses), it can introduce vulnerabilities at scale.
- The attacker instead had Copilot include an innocuous-looking web link in its answer, pointing to an attacker-controlled URL with the sensitive data as a URL parameter.
- A follow-up finding through Gemini Enterprise’s Jira integration, which silently wiped victim memory via an assigned task description, earned $15,000.
- If you want to test specific facts for a RAG architecture or fine-tuned model, we recommend using evals.
- Open guardrail models include Llama Guard, ShieldGemma, IBM Granite Guardian, and Prompt Guard.
- Layer 3 (Agent Frameworks) addresses the multi-agent propagation risks documented in the ARGUS and cross-agent provenance research cited here.
What are prompt injection attacks on AI agents?
Understanding the mechanics of both is a prerequisite for building defenses that actually cover your attack surface. A comprehensive guide to prompt injection attacks — how they work, the different types, real-world examples, and defense strategies for securing LLM applications. Meta’s AI research division publishing open-source safety tools including LlamaGuard and LlamaFirewall. Systems that only process direct user input without external data retrieval are not vulnerable to indirect injection (though they remain vulnerable to direct injection).
- Promptfoo includes a RAG poisoning utility that tests your system’s resilience against adversarial attacks on the document retrieval process.
- System prompts frequently contain confidential business logic, API configuration, internal instructions, and data that organizations treat as proprietary.
- Since prompts are expressed in free-form natural language, they cannot be sanitized as strictly as structured inputs, creating a challenging attack surface (OWASP Foundation 2025a).
- The NIST AI Risk Management Framework (AI RMF 1.0) addresses both classes under its Govern and Measure functions, specifically calling for adversarial testing that covers both user-turn and retrieved-content attack surfaces.
- As part of the first campaign, the threat actor has been using SEO poisoning to target AI agents searching for the Python library requests-secure-v2.
Types and taxonomy of prompt injection
Anything that treats the LLM as the trust boundary is shipping a credential thief with a friendly interface. The operating assumption for any enterprise deploying AI today has to be that the model will follow injected instructions some fraction of the time. The first asks about detection cadence, specifically which classifiers the vendor runs against prompt injection and how often it retrains them.
Google Bard and Gemini have also been shown vulnerable to indirect injection through Google Docs, Gmail, and web content. The model would follow the web page’s embedded instructions, potentially spreading misinformation or performing actions not intended by the user. Bing Chat (now Microsoft Copilot) was one of the first production systems shown to be vulnerable to indirect injection. Web-browsing agents that can read web pages are exposed to indirect injection through any web content they access. Indirect injection is considered higher severity because it bypasses input defenses, can be scaled (one poisoned document can affect many users), and is harder to detect in real time.
A Simple Defense in Depth Model for Indirect Prompt Injections
- All three require documented controls at the model, application, and context layers.
- Prompt injection is a security vulnerability where attackers craft inputs that trick AI language models into ignoring their intended instructions and following attacker commands instead.
- Over the next several months, organizations should implement architectural controls that reduce the exploitability of image-based injection even before robust detection is available.
- Provide specific instructions about the model’s role, capabilities, and limitations within the system prompt.
- That lets a malicious page slip in commands dressed up as ordinary content or game rules, and the agent cannot reliably tell the difference.
- A privileged LLM plans actions based only on the user’s original request.
URL allow/deny lists, strict content security policies (CSPs), and signed media proxies collectively block network-based leakage paths by denying or tightly brokering outbound requests. Prompt partitioning and provenance gating constrain the model’s scope by isolating untrusted inputs, reducing https://lievell.com/top-11-software-development-trends-2024-2025.html instruction following and limiting classifier or redaction bypasses. CSPs break the exfiltration path by preventing the client from making unintended network requests or executing active content derived from model output. Treat model outputs as untrusted until vetted by a policy gate. Enforce access scoping and add anomaly controls (rate limits, flags on unusual external requests) to trigger temporary lock-down. Microsoft’s patches reportedly introduced options to restrict Copilot from using external communications in certain contexts.
Indirect Prompt Injections
The Cline/OpenClaw supply chain attack and PromptPwnd CI/CD pipeline attacks further illustrate agentic injection at scale. Vectra AI’s Moltbook analysis documented the security implications in detail. Analysis of the Moltbook AI agent network found that 2.6% of agent posts contained hidden prompt injection payloads — the first large-scale demonstration of bot-to-bot injection in a production environment. This includes bot-to-bot injection, where malicious agents inject payloads designed to manipulate peer agents’ behavior.
The hidden metadata falsely claimed that a missing license key exception could only be resolved by purchasing a $3.00 developer API license. To manipulate the AI, the attackers abused JSON-LD, a structured metadata format that search engines and AI agents rely on to interpret page content. Attackers used SEO poisoning to push a fraudulent website to the top of search results for a fake Python library called requests-secure-v2. Recent findings from Zscaler ThreatLabz highlight this growing threat, exposing malicious websites that use IPI to manipulate AI-driven workflows. When an AI agent scrapes this content, it unwittingly processes the hidden directives, effectively hijacking the model’s reasoning and execution. As artificial intelligence agents become deeply integrated into everyday web browsing and automated tasks, threat actors are adapting their strategies to exploit this new attack surface.
AI Analysis
Because LLMs treat all prompt content as potential instructions, malicious text embedded in retrieved or external data can bypass traditional trust boundaries (OWASP Foundation 2025a). This AI-native attack paradigm underscores the need to treat AI integration points as part of the threat surface, since an LLM can be coerced into betraying its own context if not properly safeguarded. In January 2025, Aim Labs created a working proof of concept and privately reported the issue to Microsoft’s Security Response Center. EchoLeak represents the first known case of a prompt injection being weaponized to cause concrete data exfiltration in a production AI system. The findings show that exposing AI chatbots to external tools and systems, a key requirement for building AI agents, expands the attack surface by presenting more avenues for threat actors to conceal malicious prompts that end up being parsed by models. The disclosure comes close on the heels of research demonstrating various kinds of prompt injection attacks against AI tools that are capable of bypassing safety and security guardrails –
Rank One, whose board includes a former CIA deputy director and a former FBI science chief, supplied face recognition to Meta for internal development of its smart glasses app. Anthropic is releasing Claude Mythos 5 to trusted organizations and Claude Fable 5 to the public, a version it says can’t be used for cyberattacks. Google’s Bolina adds that when connecting systems to LLMs, people should also follow the cybersecurity principle of least privileges, giving the system the minimum access to data it needs and the lowest ability to make changes required. And the National Cybersecurity Center, a branch of GCHQ, the UK’s intelligence https://dragonsupport-number.com/telos-crypto-innovating-for-financial-accessibility/ agency, has even called attention to the risk of prompt injection attacks, saying there have been hundreds of examples so far. A website the LLM can read, or a PDF that’s being analyzed, could, for example, contain hidden instructions for the AI system to follow. Agents also need to show what a setup command actually will run, including the contents of scripts it invokes and anything that the script fetches at runtime, they wrote.