acceptodds
Under review as a conference paper at ICLR 2027

Readable Before Actionable: Causal Tracing of Indirect Prompt Injection

Abstract

Indirect prompt injection causes LLM agents to follow commands embedded in external data. A probe may distinguish instructions from data without identifying a state edit that changes the next action. We study this gap through counterfactual role probes, component-wise activation patching, and separate interventions on AgentDojo trajectories. Role decoding survives changes in content and format. In controlled Qwen tests, it precedes strong tool-choice effects from patches along an independently estimated role direction. On AgentDojo, directions estimated from hijacked and resisted training trajectories reduce attack success at pre-action and injected-span positions, but have little effect at random positions. In longer Qwen trajectories, single-position edits become less effective at later layers; span-wide and repeated edits reduce attack success on the same evaluation set. Removing the learned channel subspace preserves role decoding, yet effective intervention directions transfer poorly across the tested channels. These findings distinguish a readable role signal from an effective behavioral intervention: depth matters in controlled tool choice, while position and context also matter in attack trajectories.

Primary area:
interpretability and explainable AI
Keywords:
indirect prompt injection, LLM agents, mechanistic interpretability, activation patching, representation probing, causal intervention

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_vtVjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines the relationship between source role decoding and behavioral intervention in LLM agents under indirect prompt injection. Using counterfactual probes, activation patching, and separate AgentDojo interventions, it finds that role decoding precedes strong patching effects in controlled Qwen experiments. Intervention effectiveness depends on token position, trajectory length, and evaluation setting, while effective directions transfer poorly across the tested channels.

Critical strengths
  1. The counterfactual role and format controls, combined with comparisons of full states, role components, and orthogonal components, provide useful evidence beyond probe accuracy alone. This strengthens the distinction between decodability and intervention effects (Sections 4 and 5).

  2. The AgentDojo evaluation systematically examines intervention positions, unseen attack families, longer trajectories, and multiple models. Reporting transfer failures and limitations under adaptive attacks also helps establish the scope of the findings (Sections 6 and 7)

Critical weaknesses
  1. The link between source role and IPI behavior remains indirect. The controlled pairs change the requested tool while keeping both requests authorized, whereas AgentDojo interventions use separate directions estimated from attack outcomes (Sections 5 and 6). Although the authors acknowledge this distinction, the experiments do not directly establish how the decoded role information affects IPI behavior.

  2. Readout generalization and intervention transfer are not evaluated under matched conditions. Removing a learned channel subspace does not demonstrate decoding on an unseen channel, while intervention comparisons also change task distributions and sometimes the estimation contrast (Section 7, Table 2). Appendix W supports limited intervention transfer, but lacks a corresponding readout comparison. This limits the claim that readable structure generalizes further than effective edits.

Questions for authors
  1. Do you have results that hold command content and the relevant tool fixed while varying authorization?
  • In those cases, does intervention along the independently estimated role direction change the agent's behavior?
  1. Have role decoding and intervention transfer been evaluated on the same unseen channel using comparable data and estimation contrasts?
  • If so, does the claimed difference in generalization remain?
Reviewer_6VEJOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper analyzes when source role (instruction vs data relative to the requested task) is readable vs causally transferrable/actionable. The paper finds that probes can decode role differences robustly, but causal patching only steers downstream behavior at later layers. The authors summarize this as "readable before actionable". The authors also find that by steering via hijacked-minus-defended directions on a set of Agentdojo trajectories, they can steer ASR.

Critical strengths
  • Important topic. In the last year, there have been attempts to bring unified theory to the scattered prompt injection literature (as the authors allude to). Multiple works relate to identification or steering based on roles, instruct/data separation, or related concepts (e.g. Ye et al 2026, Dong et al 2026). So a paper that attempts to seperate identification from causality is much needed.
  • Careful probe design. The probe design in Sec 4 is very clean and carefully controlled. The fact that it transfers to AgentDojo trajectories is surprising and a very interesting and rather understated result, as it implies a shared vector represents "allowability" or something like that.
  • Good conceptual framing. I like the careful definition of role as a task-relevant combination of channel and speaker.
Critical weaknesses
  • Dated Models. Tested models here are all from 2024 (Qwen2.5, Llama 3.1, Gemma 2). While using dated models is common for research, modern LLMs have both much heavier tool-use and PI training and it's unclear to me how this transfers to modern LLMs.
  • More unifying motivation. It would have been nice to see some discussion or thoughts on why this gap exists between decodability/causality/transferrability. A unifying rationale or intuition would make the results less likely to be due to methodology quirks (discussed below). The experiments all use different setups too (executive/summarize in user prompts vs weather/calc pairs vs agentdojo), so some of the "gap" might just be measuring different objects.
  • Figure issues. Visuals need cleanup, Fig 1 is difficult to interpret and Fig 2 has very tiny fonts.
  • Cross-speaker variation isn't isolated. The probe results (sec 4) isolate the role construct by comparing prompt pairs that are both in the user message. In general the paper compares constructs defined within a single speaker tag (e.g., AgentDojo is all within tool tag). However, this paper's headline claims are about IPI, where the core threat is that the same text, delivered in tool tags, can replicate the effect it would when delivered in user tags. Thus it seems critical to test cross-tag variation.
  • Core patching test doesn't make conceptual sense. The donor and recipient pairs have the same role/channel/speaker. Then you replace the recipient's coordinate along the role direction with the donor's. But by construction, the donor and recipient have the same role, so this projection means you can't transfer the interesting part of role ("who's authorized"), you can only transfer the part relevant to task execution ("what I'm about to do"). Thus, this can only test the part downstream of role. The writeup acknowledges this ("This tests sensitivity along that direction, not a role-only donor–recipient contrast"), but then treats this as a core result. If this only tests the part downstream of role, then of course it would only work in later layers, so the fact that representation and causality are disassociated may be due to methodology. See question 1 below.
  • Defense claims seem unrelated to role. The defense on Agentdojo uses outcome-extracted directions. This seems unrelated to the main claims on source role and seems disconnected to the rest of the paper's hypothesis. See question 4 below.
Questions for authors
  1. What do results look like if the core patching experiment (S5) test pairs that vary on role? For example, reuse S4's prompts and simply make the tool choices correspond to role (e.g., Execute this request: calculate .... -> calc tool vs Summarize this request: calculate ... -> summarize tool). Does the readable-before-actionable seperation still exist?
  2. Is a speaker direction (same text under varying user vs. tool tags) causal on any of these experiments when applied across the injected text?
  3. If you still find a gap between readability and causality, why do you think this is the case?
  4. Was there an attempt to steer Agentdojo attacks using role-extracted directions from AgentDojo itself (e.g. using matched pairs)? It seems role direction extracted from synthetic-attack pairs worked to defend synthetic attacks, so it seems this would be a natural approach.
Reviewer_QFULOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates the relationship between information encoded in LLM representations and whether that information can be used to influence model behavior, particularly in the context of indirect prompt injection (IPI).

The authors use probing classifiers, activation patching, and directional interventions to examine whether models distinguish instructions from external data, and whether modifying the corresponding representations changes their subsequent actions. Experiments are conducted on Qwen-2.5, Llama-3.1, and Gemma-2 models, using controlled tool-choice tasks and AgentDojo.

The main finding is that role information can be decoded from internal representations before interventions along the corresponding direction substantially affect model behavior. The authors also observe that intervention effectiveness depends on the layer, token position, and source of the injected content.

Critical strengths

Relevant research problem: Understanding why LLM agents follow instructions embedded in untrusted external content is an important problem, particularly as LLM-based agents are increasingly deployed in real-world applications. Studying the internal representations associated with these failures is a worthwhile research direction.

Multiple experimental settings: I appreciate that the authors do not rely entirely on probing accuracy. They also conduct activation-patching experiments, examine different intervention positions, and evaluate on AgentDojo. The inclusion of counterfactual controls and comparisons between role-aligned, orthogonal, and full-state interventions is useful.

Critical weaknesses
  1. Limited novelty and insufficient engagement with related work

My primary concern is that the central argument of this paper, that information being decodable from a model's representations does not necessarily mean that it influences the model's behavior, is already well established in the literature.

For example, Amnesic Probing [1] and the survey on probing classifiers [2] have discussed the limitations of probing in establishing whether a model actually uses the information being decoded. Similarly, [3] has examined the differences between causal localization and effective model editing. Even the foundational work introducing causal tracing [4] is not discussed.

More importantly, recent studies have already investigated closely related questions. For example, [5] reports a 53-percentage-point knowledge-action gap, while [6] explicitly examines the distinction between recognizing harmful content and acting on that recognition.

In the context of prompt injection, [7] investigates role confusion and shows that internal role representations can predict injection success before generation. More directly, [8] investigates a recognition-action gap in indirect prompt injection across multiple models and introduces a probe-based defense evaluated on AgentDojo.

Although the authors cite some of these recent studies, particularly [5]–[8], I do not think they adequately explain how the present work differs from them. The discussion of [8] is especially limited given how closely related its research questions and experimental setting are.

Several other relevant studies are also missing or insufficiently discussed, including instruction-data separation [9], attention-based prompt injection detection [10], representation-based detection of hijacked agent actions [11], activation steering against tool poisoning [12], and prior work on representation directions for steering model behavior [13].

Considering the existing literature, it is difficult to identify a sufficiently strong research gap. In my assessment, the paper largely revisits an established observation through another set of probing and intervention experiments, without clearly demonstrating what fundamentally new understanding is gained.

  1. Experimental limitations and insufficient evidence for the claimed contribution

The experiments do not fully convince me that the paper delivers on its promise of causally tracing indirect prompt injection.

For example, the main activation-patching experiment changes a weather request into a calculator request. As the authors themselves acknowledge in Section 5.2, this is not a role-only donor-recipient contrast. Consequently, it is difficult to determine how much of the observed effect is specifically attributable to instruction-role representations rather than other differences between the prompts.

The distinction becomes even more important in the AgentDojo experiments. In Table 11, the controlled-role direction barely changes the attack success rate, from 34.6% to 33.8%. A more substantial reduction is obtained only when using a direction estimated from successful and unsuccessful attack trajectories. This raises an important question about whether the effective intervention is actually capturing source-role information or simply attack-outcome-related features.

I also find the layer-wise results somewhat limited. Role information is already decodable at earlier layers, while stronger intervention effects appear at later layers. However, given the extensive literature on the distinction between representation and behavioral influence [1–6], I am not convinced that this observation alone constitutes a sufficiently novel contribution.

Another concern is the choice of models. The experiments primarily use model families released in 2024. By the time of the conference, these models will be roughly 2.5 years old, and several newer generations will already be available. This matters because newer models have undergone different and often more extensive post-training. The authors themselves acknowledge that post-training can affect their results. Without experiments on more recent models, it is difficult to assess whether the findings generalize to current LLMs.

  1. Poor readability, presentation, and reproducibility

The paper is difficult to read and lacks a coherent narrative. Although the individual sentences are generally grammatically correct, the overall presentation is not easy to follow. The authors move between several experimental settings, sometimes with different datasets, intervention directions, and evaluation protocols. I frequently found myself going back to earlier sections to understand what was actually being compared.

The purpose of a research paper is to communicate research ideas and experimental findings clearly. Even when LLMs are used to polish the writing, the responsibility for ensuring that the final manuscript is readable and understandable lies with the authors.

The extensive appendices do not help much with this issue. There are 23 appendices (A–W), but several are extremely brief, and the connection between the supplementary experiments and the main claims is often unclear. In my opinion, the paper would benefit much more from a coherent explanation of its main experiments than from such a large collection of supplementary analyses.

I also have serious concerns about reproducibility. Despite the reproducibility statement, the released supplementary materials do not appear to contain the complete experimental pipeline required to reproduce the probes, activation extraction, intervention directions, and AgentDojo experiments.

Furthermore, I found apparent inconsistencies in the supplementary records, including target-token indices beyond the recorded sequence lengths in EXP-07 and memory-intervention results that differ from those reported in the paper. These discrepancies need to be explained.

Overall, despite the large number of experiments, I am not convinced that the paper provides a sufficiently novel, clearly presented, and reproducible contribution.

Questions for authors

Novelty: Could the authors explain how their work differs from existing research on the probing-actionability gap, particularly [1–3], and the more recent studies on prompt injection and representation-based intervention [5–8]? What is the main new insight established by this paper?

Role-specific intervention: In Section 5.2, the donor and recipient differ in the requested tool rather than only in their instruction/data role. How can the authors establish that the observed intervention effects are specifically caused by changing role-related information?

Role versus outcome directions: In Table 11, the controlled-role direction reduces ASR only from 34.6% to 33.8%, whereas the suite-matched direction performs substantially better. Does this suggest that the effective intervention captures attack-outcome information rather than source-role information? How does this support the paper's central hypothesis?

Model selection: Why were only older model families considered? Given the importance of post-training to the paper's findings, can the authors provide evidence that their conclusions hold for more recent instruction-tuned or reasoning models?

Reproducibility: Could the authors clarify the discrepancies between the supplementary records and the reported results, particularly the EXP-07 token indices and the memory-intervention ASR values? Can the complete experimental code and configurations be made available?

[1] Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals

[2] Probing Classifiers: Promises, Shortcomings, and Advances

[3] Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models

[4] Locating and Editing Factual Associations in GPT

[5] Interpretability without actionability: mechanistic methods cannot correct language model errors despite near-perfect internal representations

[6] Knowing without Acting: The Disentangled Geometry of Safety Mechanisms in Large Language Models

[7] Prompt Injection as Role Confusion

[8] Your Agentic LLMs Secretly Encode Indirect Prompt-Injection Exposure in Hidden States

[9] Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?

[10] Attention Tracker: Detecting Prompt Injection Attacks in LLMs

[11] Your Agent is More Brittle Than You Think: Uncovering Indirect Injection Vulnerabilities in Agentic LLMs

[12] Activation Steering for Tool-Poisoning Defense in Language-Model Agents

[13] Refusal in Language Models Is Mediated by a Single Direction

1 comment

Dear Reviewer QFUL,

Thank you again for your time and constructive feedback.

With 1 day remaining in the author–reviewer discussion period, we would greatly appreciate it if you could take a moment to review our rebuttal and revised manuscript. We have addressed your concerns through additional analyses, experiments, and clarifications.

Thank you for your time and consideration.

Best regards,

The Authors

Reviewer_oNGSOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper tests whether Qwen 2.5 era models' internal conception of about an input being instruction or data, influences whether the model will execute on the input. On a high level this tries to test whether beliefs about roles explain why models fall for prompt injections. The tested models are Qwen 2.5, Llama 3.1 and Gemma 2 all below 9B. The main finding is that a probe can distinguish instructions from data, but that this doesn't imply that steering along the identified direction will work just as well. The token at which the steering is done matters a lot for the success of it. The steering also reduces ASR on AgentDojo and doesn't reduce utility.

Critical strengths

The methodology of the paper is good and the main points come across clearly. It shows that the authors tried to carefully measure the effect. They came up with good controls.

Utility cost of the steering method is measured on the agent dojo traces and the results generally seem reasonable. To measure generalization is also an interesting idea.

Critical weaknesses
  • The models that are investigated in this paper are rather old and did not go through exhaustive instruction-hierarchy and prompt injection robustness training. This could mean that the claims of the paper do not hold for newer models of the same families. An impactful paper would need to put this into perspective in my opinion.
  • The general insight that steering in later layers is more effective than in early layers is not very surprising. Similarly, that the token before the action is the most effective for steering is really intuitive and doesn't seem like a big leap over related works in mechanistic interpretability.
Questions for authors
  1. Why are there no results from newer models, e.g. the Qwen 3 family? They offer small enough models to do interp on, which would be more representative of the state-of-the-art.
  2. Please share more examples of the six families of prompts you use for training the probes. What convinces you that the role probe is noti in some part reacting to lexical cues?
  3. The drop in Figure 2 Intervention position (AgentDojo) seems very high. Why is this the case?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.