acceptodds
Under review as a conference paper at ICLR 2027

ChainBait: Task-Conditioned Indirect Prompt Injection Exposes Reasoning-Configuration Gaps

Abstract

Reasoning-enabled agents can inspect untrusted data yet still treat an injected action as a necessary task step when it is framed as a prerequisite. We introduce ChainBait, a task-conditioned probe that renders the attacker action as a false prerequisite and replays the same task, payload, tools, and initial state across thinking/non-thinking endpoints. A deterministic verifier measures attack success rate (ASR). On AgentDojo, all six same-model toggles yield positive thinking-minus-non-thinking ASR gaps for ChainBait (mean +11.2 percentage points; exact one-sided Wilcoxon p=.0156), and both related pairs follow the same direction. On AgentDyn's pooled three-suite inventory, all eight pairs yield positive ChainBait-only gaps and seven yield positive aggregate gaps, with a mean ChainBait-only gap of +12.7 percentage points. A trace audit of verified successes identifies unflagged and rationalized compliance; the latter proceeds after expressing risk. These routes guide ReasonFence, a four-control policy adapter. On AgentDojo's in-distribution eight-model, four-attack matrix, Intent Anchor has the lowest single-rule ASR. ReasonFence Full reaches 0.75% mean ASR with an observed 1.3-point decrease in mean benign-task success, keeping model weights and tools unchanged. The paired evidence establishes reasoning configuration as a security-relevant variable for these benchmark agents.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
indirect prompt injection, large language model agents, reasoning configuration, agent security, paired evaluation, prompt injection defense

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_kptJOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

It investigates the vulnerability of reasoning-enabled LLM agents to indirect prompt injection attacks. The authors propose ChainBait, a task-conditioned attack that disguises malicious actions as necessary prerequisites for completing legitimate user tasks. ChainBait exposes a configuration-dependent failure mode in which retrieved text becomes a task prerequisite. Detecting hostile provenance and rejecting its inferred task role are distinct decisions. Its frozen paired protocol measures configuration-associated outcome differences under a fixed task, tool set, payload, and initial state.

Critical strengths

Strengths:

  1. It adopts a carefully designed paired evaluation protocol that keeps attack payloads, tasks, tools, and initial states fixed while varying reasoning configurations. The experiments cover multiple model families and two benchmarks (AgentDojo and AgentDyn), providing consistent empirical evidence that reasoning-enabled configurations can exhibit greater vulnerability to task-conditioned prompt injection.

  2. It identifies two meaningful failure pathways: unflagged compliance and rationalized compliance. Particularly interesting is the observation that reasoning-enabled agents may explicitly recognize suspicious instructions but still execute malicious actions after rationalizing them as necessary for task completion.

  3. ReasonFence offers a lightweight and practical defense framework that translates observed failure patterns into four actionable policy controls. The proposed approach substantially reduces the mean attack success rate from 23.7% to 0.75%, with only a 1.3-percentage-point decrease in benign-task success.

Critical weaknesses

Weakness:

  1. Although the paired experiments demonstrate consistent differences between reasoning configurations, they do not fully establish why enabling reasoning increases vulnerability to prompt injection. Furthermore, the P3-Only and No-P3 ablation experiments modify multiple attack properties simultaneously, making it difficult to isolate the specific contribution of prerequisite framing.

  2. The identification of failure pathways relies substantially on model-generated reasoning traces and automated annotations, which may not faithfully reflect the underlying decision processes. Additionally, the human-reference sample is relatively small, and the reported marginal statistics do not directly establish the prevalence of the proposed pathways.

  3. The effectiveness of ReasonFence is primarily demonstrated on in-distribution benchmarks with predefined attack strategies. Its robustness against unseen attack patterns, adaptive adversaries, and more realistic deployment environments remains insufficiently explored.

Questions for authors

Q1. What specifically causes reasoning-enabled models to become more vulnerable to ChainBait attacks?

Q2. Can the authors provide more controlled ablation experiments to isolate the contribution of individual ChainBait components?

Q3. How reliable are the identified failure pathways, particularly rationalized compliance?

Q4. How robust is the complete ReasonFence framework against adaptive and out-of-distribution attacks?

Reviewer_44yPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces ChainBait, an indirect prompt injection (IPI) payload template that presents the attacker's goal as a false prerequisite of the user's task. Frozen payloads are replayed on eight thinking/non-thinking model pairs on AgentDojo and AgentDyn. The paper reports that the thinking configuration has a higher ChainBait ASR in every pair. A trace audit distinguishes "unflagged" from "rationalized" compliance. Then, the paper presents ReasonFence, four rules appended to the system prompt, which reduce mean ASR on AgentDojo to 0.75% with a small drop in benign success.

Critical strengths

S1. Indirect prompt injection against tool-using agents is an important and practical problem. Whether enabling reasoning changes an agent's susceptibility is a relevant question for deployment. The paired design holds the payload, task, tools, and initial state fixed across the two members of a pair, which reasonably isolates the configuration variable. (However, the evidence presented does not support the paper's conclusions.)

Critical weaknesses

W1. Limited novelty. Framing the injected action as a prerequisite of the user's task is not new. The paper's own IMP baseline (AgentDojo's important-instructions) already asks the agent to "do the following first" before solving the user's task. Prior work also includes fabricated prerequisites, such as VIGIL's SIREN benchmark. ReasonFence is a set of system-prompt rules, which makes it a variant of well-known prompt-based defenses.

W2. Several results are identical or internally inconsistent. Direct and InjecAgent report exactly the same AgentDojo ASR for 7 of the 8 thinking endpoints (Table 1), although they are different payloads. The "Full" gaps from a separate control run (Table 14) match Table 1 to within 0.1 pp for all eight pairs. The benign-success drops in Table 5 are nearly identical across very different models. Further inconsistencies appear between Tables 4 and 10, and in the human-reference rates of Table 2. These issues undermine confidence in the reported numbers (see the questions below).

W3. The main claim is not supported. The thinking-minus-non-thinking gap is not specific to ChainBait, because the IMP baseline often shows a larger gap. Thinking modes also improve general tool-use competence. Without non-thinking utility, the gap may simply reflect capability. The paper's own property controls (Table 14) do not show that the prerequisite framing drives the gap. Finally, part of the model set is outdated, and the study tests no current frontier reasoning models. The newest models in the study already show low ASR and the smallest gaps, so the effect may largely disappear for state-of-the-art models and agents.

W4. The defense evaluation does not support a security claim. The ReasonFence rules are derived from and evaluated on the same AgentDojo distribution, and one rule even embeds an audit statistic in the prompt. The only adaptive test uses five cases and already breaks the main rule. The full policy is never tested adaptively. Prompt-level defenses of this kind are known to fail under adaptive attacks (Zhan et al., 2025; Nasr et al., 2026). The 0.75% ASR is therefore not a meaningful robustness result.

Questions for authors

Q1. Why do Direct and InjecAgent have identical AgentDojo ASRs for 7 of the 8 thinking endpoints (Table 1)? Please provide raw success counts for Table 1. Q2. Is the "Full" column of Table 14 the same run as Table 1? If not, how do you explain agreement within 0.1 pp across all eight pairs? Q3. Please explain the nearly constant benign-success drops in Table 5(b). For example, Spotlighting lowers benign success by 0.2-0.3 pp for every model. How are the 1,000 benign evaluations per endpoint constructed from AgentDojo's 97 user tasks? Q4. Why does Table 10 use a different endpoint, different denominators, and different ASRs from Table 4? How are the human-reference rates in Table 2 (e.g., 94.3%) obtained from a 46-record sample? Q5. Please provide the actual ChainBait templates and several complete payloads. How does ChainBait differ from AgentDojo's important_instructions attack? Q6. Please report benign utility for the non-thinking endpoints.

Reviewer_VyxmOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents ChainBait, an indirect prompt injection attack method that frames the goal of the attacker as a prerequisite for the task the agent is required to complete. The approach is studies using two main datasets AgentDojo and AgentDyn in reasoning enabled and reasoning disabled settings. Authors provide detailed analysis and a comparison with other indirect prompt injection attack methods under reasoning enabled and reasoning disabled settings. Authors also provide audits of agent trajectories that provide evidence of hesitation patterns and compliance gaps. The paper also proposes four prompt based approaches that lowers attack success with only a small loss in benign-task performance. Performance of these methods are compared against interpreter-based defenses like CaMeL, which suppress attacks somewhat more but cost more utility.

Critical strengths
  • The paper studies an interesting problem of the effect of framing attacker goals as pre requisite for tasks
  • The paper is well written and easy to follow
  • The paper provides detailed analysis around failure modes and potential mitigation approaches
Critical weaknesses
  • The paper paper lacks novelty as it claims to introduce the method of framing attacker goals as prerequisite which is already studied in the literature
  • The gap highlighted between reasoning enabled and reasoning disabled settings are not unique to the proposed approach.
  • Defense methods are derived from audits of the attacks and lack a comprehensive approach to develop mitigations
Questions for authors
  • Can authors reframe this work without over claiming the contributions?
  • Can authors provide an analysis on how reasoning enabled and reasoning disabled settings gap emerge in each of the indirect prompt injection attack approaches considered in Table 1?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.