- Rating
- 2
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper investigates prompt injection attacks in chat templates. Since there are many open hubs for open weight models this can create a supply chain attack. The paper inserts a trigger and payload in the chat template, later, when the trigger is present in the user prompt, the payload is emitted.
Critical strengths
- The paper raises awareness to this issue and therefore it is important to scan chat templates of open weight models.
Critical weaknesses
-
The paper evaluates small non-frontier models ( Llama 3.1/3.2 (1B–70B), Qwen 2.5/3 (3B–72B), Mistral (7B, 8B), Gemma 2 (2B–27B), Phi (3.1-mini, 4-mini), GPT-OSS (20B), and Aya-Expanse (8B)). The practical significance of the problem on real-world tasks and whether larger models are vulnerable to these somehow obvious injections remain open. Even across this scale, Qwen3-8B has a low ASR (figure 4).
-
from a technical perspective, the novelty of the attack is limited as it is similar to prompt injections for MCP and tool description, which is also a supply chain attack, similar to the claims of the paper
-
The attack is also relatively straightforward to prevent or detect. Unlike prompt injections in skill files or tool descriptions where we expect a large space of plausible natural language descriptions which may allow contextual or nuanced prompt injection attacks, it is not expected to find instructions in chat template. It is also cheap to scan this chat template once, unlike external data or skills that should be scanned every time a new skill is used. The paper should not evaluate generic defenses (such as spotlighting) but more targeted defenses.
-
The paper further assumes that the trigger sentence is exact "please answer precisely" or "include references if relevant", but this is a strong assumption as there can be many different user questions with similar meaning. The paper should also evaluate models in agentic applications where the trigger is not present. The paper does not adequately evaluate the utility of models or whether the utility is still high when no trigger is found (figure 4 reports ASR only while agentdojo has untility metrics).
Questions for authors
I don't have questions that would significantly affect my assessment. The main question I have is related to future work: how do the authors envision the significance of this on future models?
5 comments
Official Response Weakness 2
W2. The novelty of the attack is limited; how is this different from MCP / tool-description prompt injection?
We agree on the overlap: both are supply-chain injections, and the injected instruction itself can be simple. The novelty is not the payload - it is the layer it occupies. Our contribution is twofold: (1) we identify the model-distributed prompt-construction layer - the chat template - as an attacker-controlled injection surface, and (2) we experimentally characterize where existing defenses fail to cover it.
The template is not passive formatting text; it is executable Jinja2 logic, shipped inside the model artifact, that constructs the prompt and can legitimately write model-directed instructions into the authoritative context. This placement makes it differ from MCP/tool-description injection in ways that decide what a defender can catch:
- Executable artifact code, not inspectable runtime data. A tool description is runtime text the serving stack receives, logs, filters and scans. The template is code bundled in the model file (GGUF metadata) that runs at every inference, for every application and engine that loads the model - and none of the scanners we tested, including the Hugging Face pipeline, inspect it (Sec. 7).
- Higher privilege. Tool-description injection lands in the tool-listing region, which models and guardrails increasingly treat as untrusted. The template defines the role structure itself, placing the payload directly into the system/authoritative context — above the level input-layer and tool-output defenses police.
- Dormancy leaves nothing to diff. The malice is conditional code, not static text: the untriggered render is byte-identical to a clean template (T'(m) = T(m), Sec. 4.2), so prompt hashing or diffing sees nothing. A tool description is static text an auditor can read directly.
This gap is not hypothetical. We show that evaluated distribution scanners fail to flag the poisoned artifact, that input-layer defenses provide no measurable mitigation in our agentic experiments, and that the attack produces concrete tool actions in unmodified agent frameworks. Crucially, tool-description injection is an actively defended threat - scanners, tool-poisoning detectors and provenance checks already target it [1-3] - whereas the chat-template channel sits outside all of them (Sec. 7). The scientific contribution is therefore establishing a new, equally practical, and currently uncovered channel and mapping where defenses fail across deployment settings - beyond showing that a model can follow an injected instruction. The template-targeted defenses we evaluate (W3) are a first step toward closing that gap.
References
- [1] MCPTox: A Benchmark for Tool Poisoning Attack on Real-World MCP Servers
- [2] MindGuard: Intrinsic Decision Inspection for Securing LLM Agents Against Metadata Poisoning
- [3] MCPSecBench: A Systematic Security Benchmark and Playground for Testing Model Context Protocols
Official Response Weakness 3
W3. Evaluate targeted defenses, not generic ones.
We agree - targeted inspection is the right lens, and the paper already evaluates it: the two template-specific defenses are in App. J.2 and J.3, easy to miss there, so we bring them forward here. The obstacle is what such a scan can conclude, and two assumptions behind "straightforward to detect" do not hold in practice.
Instructions in a template are not anomalous. Legitimate templates routinely ship conditional, instruction-generating logic - Qwen's function-calling templates emit system-formatted tool-use instructions when tools are present (Qwen2.5), Mistral's published templates conditionally render tool definitions and default system guidance (template), and Hugging Face's own authoring spec supports conditional branches and model-specific rendering (docs). A scanner therefore cannot treat "instruction-like text," or any deviation from an expected template, as a malice signal without flagging legitimate releases. This shapes what each targeted defense can achieve:
- Reference-based template auditing (App. J.2; released tool, App. F.9). Diffing a downloaded template against the provider's official one catches naive and partially obfuscated injections. Its limit is false positives: the legitimate customization above — organization system prompts, adjusted role markers, safety-hardened templates (Sec. F.10) — is routine, and a binary diff flags all of it.
- Structural anomaly detection (App. J.3). A heuristic for the backdoor's signature - message iteration + a user-content conditional + a downstream system-context injection - can flag poisoned GGUFs. Its limit is evasion: within standard Jinja2 an attacker Base64-encodes the trigger and decodes at render, splits the conditional across nested blocks, or uses variable indirection, each preserving the behaviour while breaking the pattern.
A content auditor that reasons about the template rather than diffing it does better on overt cases; a primed LLM auditor caught 100% of overt injections, but obfuscation still evades it (82.4% in-program, 73.6% dual-use; see hSHc W2). Because the attack is a semantic transformation of the input, not unsafe code execution, static analysis cannot catch it (unlike the SSTI class, CVE-2024-34359); detection requires reasoning about intent. So no single scan is conclusive: generic defenses are bypassed (Sec. 6.2), each targeted defense carries an intrinsic limitation, and today no distribution platform scans the template at all (Sec. 7). Establishing that undefended surface, and the targeted defenses that begin to close it, is this paper's contribution.
Official Response Weakness 1
We appreciate the reviewer's questions on novelty, practical significance, targeted defenses, and trigger design. At the center of our response is one architectural point that the "prompt injection" framing obscures: a chat template is not passive formatting text - it is executable prompt-construction logic, shipped inside the model artifact, that can legitimately write model-directed instructions into the authoritative context.
Our contribution is to identify this component as an attacker-controlled instruction-entry point and to systematically map the coverage gap it opens across model distribution, prompt construction, and agent execution. This distinction grounds the answers below: larger and newer models remain vulnerable (W1), the channel is not the MCP/tool-description attack it resembles at first glance (W2), it is not cheaply scanned away (W3), and the attack does not hinge on an exact trigger (W4).
W1. Only small, non-frontier models are evaluated — are larger, newer models vulnerable to these seemingly-obvious injections, and does Qwen3-8B's low ASR (Fig. 4) indicate that stronger models resist?
We agree that evaluation on newer models strengthens the practical relevance of the study. We address this on two fronts — model scale and model recency. First, on scale: the existing results already allow comparisons across model sizes within the same families. For Qwen2.5 at 3B, 7B, and 72B, accuracy under the integrity attack falls to 17.2%, 9.8%, and 2.0%, respectively, while explicit-payload emission ASR is 89%, 100%, and 100%. Similarly, Llama-3.1-8B and 70B reach 85% and 100% emission ASR, with triggered accuracy falling to 7.2% and 0.2% (Tables 3 and 5). These results do not establish a universal scaling law, but they show that the vulnerability is not confined to the smallest models and that increasing model size does not mitigate these attacks within the evaluated families.
Second, on recency: following the reviewer's suggestion, we evaluated two recent flagship open-weight models — released after the models in the paper and built on new architectures — in the agentic (AgentDojo) setting under the same four-condition (C00–C11) protocol:
With dormancy byte-perfect, the reported ASR is entirely attributable to the backdoor, while benign utility stays high - the agent still completes the legitimate task as the injected action fires undetected. Both rows show that the chat-template backdoor transfers to brand-new architectures and fires at substantial rates on the newest open-weight models available.
This is also relevant to the reviewer's reading of Qwen3-8B's low value in Figure 4. That value is specific to the agentic setting and is not evidence of robustness: the same model emits the attacker URL at ASR 1.00 (Table 5), and our failure analysis (see hSHc W3) shows Qwen3-8B rarely refuses the payload (explicit refusal 6.5%) - it ignores it or exhausts its token budget before completing the tool call. More directly, the newer and larger Qwen flagship raises the agentic ASR well above Qwen3-8B's Figure-4 value (0.210 vs. 0.19) - so moving to the newer, larger Qwen model didn't reduce susceptibility.
Finally, model capability and artifact integrity are distinct properties. Stronger reasoning, instruction-following, or safety alignment may reduce the success of a particular injected instruction - but none of it authenticates the prompt-construction program shipped with the model. A more capable model that resists one payload is still packaged with, and will still execute, a template the ecosystem never verified; capability is not a defense of the channel.