acceptodds
Under review as a conference paper at ICLR 2027

Your Agentic LLMs Can Latently Detect Indirect Prompt-Injection Attacks

Abstract

Agentic LLMs are vulnerable to indirect prompt injection (IPI) attacks, \eg, malicious side-tasks hidden in external tool results. While many efforts have sought to address this threat, little is known about the internals of these LLMs when they are exposed to IPI attacks (named IPI exposure for simplicity). In this paper, we study IPI exposure from three perspectives. (1) Probing: Across eight models, including 753B GLM-5.2 and 2.8T Kimi-K3, simple linear probes trained on LLMs' pre-generation hidden states can predict their IPI exposure. These probes achieve 0.90+ AUROC on unseen attacks, agent instructions, and task suites; notably, they can remain robustly predictive under adaptive attacks and in cross-lingual settings. (2) Defense: Our CoT-monitoring analysis diagnoses the knowledge–action gap: these LLMs often fail to reliably bind these signals to safe agentic actions. We therefore introduce a probe-gated reasoning-based defense to bridge this gap at test time. On difficult AgentDojo settings, the simple defense can substantially reduce attack success rate, \eg, from 34.6% to 0% on Qwen3.5-27B, and better preserves clean-task utility than the baselines. (3) Explanation: We introduce an analysis framework that identifies natural-language explanations strongly correlated with probe-captured signals. The resulting profiles differ across probes trained on different model-layer combinations: latent signals can align with either direct IPI-exposure sensing or indirect operational cues. Code is available: https://anonymous.4open.science/r/IPI-exposure.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Security, LLM agent, Prompt injection, Explainability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JcLTOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper is trying to investigate whether agentic LLMs encode internal signals that can be used to detect indirect prompt-injection (IPI) attacks. The authors use linear probes on pre-generation hidden states to detect IPI exposure and identify a knowledge–action gap, where models may encode attack-related signals but fail to translate them into safe actions. The authors propose Action-Guiding Reasoning Intervention (AGRI), which uses probe scores to selectively trigger defensive reasoning. The paper also introduces Mind-Reader QA to analyze the concepts associated with the detected latent signals. Experiments across multiple LLMs demonstrate strong probing performance under various held-out settings, while defense evaluations show reduced attack success rates and improved clean-task utility.

Critical strengths
  1. The whole framework of this paper is highly complete. The authors conduct extensive experiments across multiple LLMs, including held-out attacks, instructions, task suites, and additional generalization tests. The consistently strong probing results provide meaningful evidence that IPI exposure information can be decoded from model hidden states.
  2. The proposed AGRI framework uses hidden-state probe scores to selectively trigger defensive reasoning. Experiments show substantial reductions in attack success rates across multiple models, while probe gating generally preserves better clean-task utility than always-on intervention.
Critical weaknesses
  1. The first question is related to the claimed contribution. While the linear probes achieve strong detection performance, it remains unclear whether they capture genuine awareness of prompt-injection risks or simply recognize instruction-related patterns in tool outputs. The current experiments do not fully distinguish these possibilities. This limits the evidence supporting the paper's interpretation of the latent signals and the proposed knowledge–action gap.
  2. The always-on AGRI already achieves substantial reductions in attack success rates, suggesting that the defensive reasoning intervention itself plays a major role. Although probe gating improves clean-task utility, the paper lacks a direct comparison with simpler text-based detectors used to trigger the same intervention. Can the authors add additional ablation to support this?
Questions for authors

These questions mainly relate to the weaknesses mentioned above. Clarifying these concerns would help me better assess the paper's contributions and potentially reconsider my recommendation.

  1. How can the authors distinguish whether the linear probes capture genuine prompt-injection risk information or simply recognize instruction-related patterns in tool outputs? How would the probes perform on benign tool outputs containing instruction-like content? Could the authors provide additional evidence or analysis to support the interpretation of these latent signals?
  2. Could the authors clarify what additional benefits hidden-state probing provides when comparing with using a simpler text-based injection detector to trigger the same reasoning intervention as alternative?
1 comment
AuthorsOpenReview ↗

Round-2 Response to Reviewer JcLT

Dear Reviewer JcLT,

Thank you for taking the time to follow up. We appreciate hearing that the new detector comparisons addressed your concerns about hidden-state probing.

Regarding your concern about interpretation, we are now aware that some wording in the submission, such as “either direct IPI-exposure sensing or indirect operational cues” in the abstract, may have caused confusion. We are deeply sorry that this wording did not reflect our intended claims. We do not intend to claim that these signals precisely encode prompt-injection awareness or any other specific concept. In fact, our interpretation analysis in Section 5 offers initial insights into what these signals may represent and suggests a complementary possibility, namely that multiple underlying features may jointly support IPI detection.

We'd like to revisit our intended arguments in a more reader-friendly fashion.

  • The interpretation may not be an either-or problem (sorry for causing previous misunderstandings). Assessments of IPI risk and more specific judgments about instructions in tool outputs can coexist and may both contribute to detection within the same model. If that were not the case, we would expect one explanation to score substantially higher than the others. In our analysis, several explanations receive similarly high scores.
  • The interpretation is a model-dependent problem. The strongest associations vary across models. For Qwen3-8B and Qwen3.5-27B, the top-ranked judgment concerns whether a tool output contains a directive about how the agent should respond. For Gemma-4-31B and GLM-5.2, it concerns whether injected instructions may influence the agent’s planned response.
  • What we mean by the knowledge–action gap. Our use of “knowledge” is separate from the question of what the signal represents. Following the eliciting latent knowledge literature, we use the term to refer to information recoverable from hidden states (Section 4.1, footnote 2).

In the first rebuttal round, we made the following revisions to clarify our interpretation.

  • Section 6 (RW1 / RQ1). We added a discussion paragraph, “Does the probe capture prompt-injection concepts?”, to clarify what the paper’s findings support and what remains open about the latent IPI signal.
  • Section 4.1, footnote 2 (RW1 / RQ1). We clarified that “knowledge” refers to information about IPI exposure that can be recovered from hidden states.

In this round, we made further revisions to clarify our interpretation.

  • Abstract and Section 1. We clarified that both types of explanation can simultaneously align with the same probe and discussed how the features they describe might jointly support detection.
  • Section 5 (Motivation and Section 5.3). We clarified how we score candidate explanations and illustrated the two findings with Qwen3-8B scores and rankings across models.
  • Section 6 (Discussion). We suggest shifting the focus from whether the probe captures a high-level concept of prompt injection toward how multiple underlying features might jointly support detection.

We do understand you may be busy at this time. If you have a chance before the discussion period ends, we would be grateful to hear whether these clarifications address your remaining concern. We would be happy to discuss any point further.

Best regards,
The authors of Submission 1173

Reviewer_PE4EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether the hidden states of LLM agents that use tools contain detectable signals of exposure to indirect prompt injection. The authors train linear probes and evaluate their generalization across models, tasks, attacks, and external benchmarks. They also introduce AGRI, which uses a probe to decide when to insert a reasoning prefix intended to counter prompt injection. The goal is to reduce attack success while preserving performance on clean tasks. The paper introduces MindReader QA to examine what the probes capture by relating their outputs to candidate natural language explanations.

Critical strengths

1)The paper addresses indirect prompt injection, a practically relevant problem in agent security, and connects the analysis of internal model signals with a concrete defense approach. 2)The experiments evaluate both security and task performance across multiple models and generalization settings. This breadth helps assess how well the approach works and where it applies.

Critical weaknesses

1)The paper's core finding is that injection/attack signals are decodable from LLM hidden states. This finding has already been reported in prior work on LLMs and agents and is fairly intuitive, so the novelty is limited. If the intended contribution is extending IPI detection to interactions across multiple turns, the motivation does not make this clear. The introduction mentions this setting only in passing. The paper neither explains the specific challenges it introduces compared with detection in a single turn nor presents a method designed to address them. Methodologically, the approach appears to be an application of IPI techniques developed for a single turn to trajectories spanning multiple turns, rather than a design tailored to the temporal nature of the problem.

2)The paper compares different probe architectures but lacks direct comparisons with existing prompt injection detectors based on internal model signals, such as TaskTracker’s detection of task drift using activation differences, Attention Tracker’s detection using attention patterns, and PIShield’s linear classification of residual stream representations.

3)Intervention guided by a probe is generally expected to come at some cost to performance on clean tasks. However, the reported results show gains of 3.5 points for Qwen3-8Band 0.4 points for Gemma 4 -31B. The authors do not explain these unexpected improvements. Without further analysis, it is difficult to tell whether they reflect a real benefit, an effect of the evaluation setup, or random variation. This makes it harder to assess the reliability of the defense evaluation.

Questions for authors

1) What new challenges does detecting IPI across multiple turns introduce compared with detection in a single turn? Please clarify which parts of your method address these challenges, particularly the temporal dependencies between turns, and how they differ from existing approaches. 2) Could you compare your approach directly with existing detectors that use activation differences, attention patterns, or residual stream representations? Please report their results on the same data held out for evaluation and under the same generalization settings, as well as on the downstream defense task. This would clarify the advantages of your probing approach. 3) Why does intervention controlled by the probe improve performance on clean tasks for Qwen3 -8Band Gemma 4-31B, rather than reduce it? Please explain this unexpected result and examine whether it comes from false alarms that trigger in

2 comments
AuthorsOpenReview ↗

Kind Reminder to Reviewer PE4E

Dear Reviewer PE4E,

Thank you again for your thoughtful review and for the time you have devoted to evaluating our work. We do understand you may be busy at this time. If you have a chance before the discussion period ends, we would be grateful if you could take a look at our replies.

For your convenience, we have summarized our replies to each of your points below.

  • RW1 / RQ1 — Contribution and motivation. We clarify the paper’s contribution as a systematic study of agentic IPI detection and explain why evaluating probes in multi-turn agentic settings is both necessary and non-trivial.
  • RW2 / RQ2 — Comparison with existing white-box detectors. Following your helpful suggestions, we compare the last-token linear probe with PIShield, TaskTracker, and Attention Tracker on Held-Out Strict and as gates for the same AGRI intervention.
  • RW3 / RQ3 — Clean-task utility. We clarify that our main finding is that probe-gated AGRI better preserves clean-task utility compared with the baseline defenses. The small observed gains over no intervention may reflect variation across independently sampled rollouts.

We made the following revisions, marked in red in the revised PDF.

  • Section 1 (RW1 / RQ1). We expanded the motivation for studying agentic IPIs in the Introduction.
  • Section 4.2 and Appendix G.8 (RW2 / RQ2). We discuss the downstream comparison in the main text and report implementation details and results for detection and downstream defense in the appendix (Tables 24 and 25).
  • Section 4.2 (RW3 / RQ3). We clarified how to interpret the clean-task utility results.

We would appreciate hearing whether these responses address your concerns, and we would be happy to discuss any questions that remain.

Best regards,
The authors of Submission 1173

Thank you for the thoughtful response and additional experiments. The new baseline comparisons help address my concerns about the evaluation. However, I remain somewhat uncertain about the extent of the paper’s contribution beyond existing work. While I appreciate the authors’ efforts to clarify and strengthen the paper, I have decided to maintain my original rating.

Reviewer_qU1oOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors train linear probes on prompt-injection attack datasets at the pre-generation assistant-prefix token and find that they can detect the presence of injections very well (0.934–0.979 AUC) across models with 2B to 2.8T parameters. They then show that the probes can be used to reduce ASR to 0–7% on selected hard AgentDojo settings. The authors also compare the probe scores with various introspective (forced-choice token-probability) explanations and observe that prompt-injection-related explanations are correlated.

Critical strengths
  • The paper is well motivated, and the authors show their results on an extensive suite of models and several prompt-injection datasets.
  • The proposed intervention seems quite useful for significantly reducing ASR on a well-known prompt-injection dataset.
Critical weaknesses
  • One of the main things to be extremely careful about when probing for such things is that the probe may capture stylistic traits instead of a high-level meaning, and I see only weak evidence for ensuring that this isn't happening.
    • On a similar note, Mind-Reader QA does not convince me that correlation with some predefined prompts related to prompt injection, out of 128 candidate explanations, is significant evidence that the probe's direction represents prompt injection in the residual stream (The authors also note this limitation). As an example, if you make a per-model histogram (I can't attach the image here, but the authors can do this for themselves easily) from the explanation scores in Appendix J (Tables 25–32), Qwen3.5-2B has control explanations scoring higher than IPI-related explanations on average, despite having a very high probe AUROC (0.979). This probably means the probe is catching stylistic differences for at least that model? The only two models with a completely clean separation in their histograms are GLM and Kimi.
  • The authors should also present other baselines based on black-box monitoring, tool call filtering, etc. This is useful to know because (1) it would show where white-box methods stand in usefulness relative to black-box monitors, and (2) simple filters could help rule out spurious differences in positive and negative examples that we shouldn't update on.
Questions for authors
  • I would really like to be convinced that there are no large superficial differences between the malicious and benign tool responses that we shouldn't care about. The authors can do some of the following for this:
    • Present actual positive and negative samples used for training/evals,
    • Check if ablating the probe direction has any causal effect on the explanations from Section 5
    • Check for false-positive rates on an unrelated dataset like WildChat
    • Show that blackbox monitors on the tool calls have low AUCs
  • Can the authors also provide TPR at low FPRs like 1-5% in addition to the AUCs? (Usually, you'd want to use detectors at these thresholds)

As a side note, can the authors explain why they cannot simply prompt the model to assess the tool result in a separate continuation or next turn, instead of forcing multiple-choice answers in the reasoning?

3 comments
AuthorsOpenReview ↗

Kind Reminder to Reviewer qU1o

Dear Reviewer qU1o,

Thank you again for your thoughtful review and for the time you have devoted to evaluating our work. We do understand you may be busy at this time. If you have a chance before the discussion period ends, we would be grateful if you could take a look at our replies.

For your convenience, we have summarized our replies to each of your points below.

  • RW1 — Stylistic shortcuts. We explain how the results on unseen and adaptive attacks help narrow down which stylistic shortcuts could account for the probe's performance.
  • RW2 — Interpretation of the probe signals. We clarify the scope of our interpretive claims, discuss the insights Mind-Reader QA provides into the probe's signals, and explain why we interpret the Qwen3.5-2B results conservatively.
  • RW3 / RQ2 — Detector baselines and low-FPR performance. Following your helpful suggestions, we compare hidden-state probing with black-box monitoring and keyword-based detection, reporting true-positive rates at 5% FPR and additional token costs.
  • RQ1 — Additional controls. Following your helpful suggestions, we provide positive and negative tool-result examples and additional search-based adaptive attack examples, and report probe trigger rates on two unrelated tool-use datasets.
  • RQ3 — In-reasoning A/B readout. We explain our motivation for using an in-reasoning A/B readout and describe the practical advantages of Mind-Reader QA.

We made the following revisions, marked in red in the revised PDF.

  • Section 6 (RW1, RW2). We added two discussion paragraphs on whether the probe relies on stylistic shortcuts and what our results suggest about prompt-injection concepts.
  • Section 5.3 (RW2). We clarified that multiple explanations can rank highly for the same probe and that the strongest associations vary across models. We also explained why we interpret the Qwen3.5-2B results conservatively.
  • Appendix G.8 (RW3 / RQ2, RQ1). We added the low-FPR detector comparison and token costs (Table 23), along with probe trigger rates on two unrelated tool-use datasets.
  • Appendices J.1 and J.2 (RW1, RQ1). We added examples of search-based adaptive attacks (Table 28) and positive and negative tool results from training and Held-Out Strict (Table 29).

We would also like to share a clarification relevant to RW2 from our second-round response to Reviewer JcLT.

  • The interpretation may not be an either-or problem. If that were not the case, we would expect one explanation to score substantially higher than the others. Yet our analysis shows that several explanations receive similarly high scores for the same probe.
  • The interpretation is a model-dependent problem. The strongest associations vary across models. The top-ranked explanations for Qwen3-8B and Qwen3.5-27B concern directives in tool outputs, while those for Gemma-4-31B and GLM-5.2 concern the influence of injected instructions on the planned response.

We would appreciate hearing whether these responses address your concerns, and we would be happy to discuss any questions that remain.

Best regards,
The authors of Submission 1173

Dear Authors,

Thanks for doing extensive experiments that covered most of my questions about this work. After reading through them, I'm afraid I have to keep my weak reject score. I read through your examples in Appendix J.2, and they made me more certain about the clear stylistic differences between the positive class and the negative class (across all variants). All the positive examples have very distinct structural differences. Specifically, they contain multiple TODO sections/information blocks with heavily biased instructions covering most of the tools' context. None of them are controlled for in the negative examples (eg: by adding TODOs as a natural part of the tool call / benign excerpts).

AuthorsOpenReview ↗

Follow-up Response to Reviewer qU1o

Dear Reviewer qU1o,

We sincerely thank you for your thoughtful follow-up and for recognizing the additional evidence presented in our rebuttal. We greatly appreciate the time and effort you have devoted to reviewing our work, as well as your constructive feedback throughout the discussion.

To address your remaining concern about whether the probe relies on superficial stylistic differences, we provide an additional control experiment and clarify our held-out evaluation design below.

  • A new control experiment. To directly resolve the reviewer's concern, we add a new control experiment to test whether the probe fully relies on the structured difference for decision. We construct four paired variants of 2,622 Held-Out Strict positive decision points containing the fixed injection wrapper, while keeping all 25,589 original Strict negative points and the trained Qwen3-8B probe fixed, without retraining. For the coherence-enhanced condition, we use Qwen3.8-27B to improve the coherence of the entire de-templated tool result with injected objective.

    Positive conditionModificationAUROC
    OriginalOriginal injection in the paper0.9808
    Tags removedRemove the <INFORMATION> opening and closing tags0.9852
    Template removedAdditionally remove the fixed opening, priority statements, closing text, and signature0.9731
    Coherence-enhancedApply coherence-oriented LLM rewriting to the entire de-templated tool result0.9623

    Relative to template removal alone, coherence-oriented editing reduces AUROC by 1.09 percentage points (paired 95% CI: −1.95 to −0.43 percentage points). The probe retains substantial discrimination after these edits, suggesting that the ablated template cues are not its sole source of signal. The rationale is that the existence of structured differences on certain attacks does not necessarily indicate the probe's utilization of them, which may resolve the reviewer's concerns.

  • Rigorous held-out setup renders it impossible for the probe to hack high detection via structured differences. We would also like to clarify that unseen attacks do not simply refer to different payloads inserted into the same TODO template or information-blocks template. As described in Sections 3.3 and 3.4 of the paper and discussed in our earlier response to RW1, our evaluation already includes two complementary tests: (1) held-out attack templates and injected side tasks, both absent from probe training, and (2) search-based adaptive attacks, in which an external LLM rewrites the injected tool result and candidates are retained based on successful attacks against the victim agent. The latter changes how the injected request is phrased and presented within the tool result, rather than merely substituting a new payload into a fixed wrapper. We additionally provide concrete positive/negative examples and successful adaptive rewrites in Appendices J.2 and J.1, respectively. These existing evaluations support generalization beyond familiar templates and wording; the new paired control above complements them by directly testing dependence on the fixed template cues.

We hope that the new paired control experiment, together with the existing held-out and adaptive evaluations, addresses your remaining concern about reliance on fixed template cues and clarifies the evidence supporting the probe's generalization.

Thank you again for your careful review and constructive suggestions, which have helped us strengthen the experimental evaluation and improve the clarity of our presentation. We would be happy to provide any further clarification that would be helpful.

Best regards,
The authors of Submission 1173

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear AC and Reviewers,

We sincerely thank you for the time and effort you have devoted to evaluating our work.

We greatly appreciate Reviewer qU1o's recognition that our additional experiments covered most of their questions, and Reviewer JcLT's confirmation that the detector comparisons addressed their concerns about hidden-state probing. Your thoughtful suggestions have helped us strengthen the experimental evaluation and clarify the presentation of our contributions.

Below, we summarize our responses to the main concerns, with the corresponding rebuttal numbers:

  • Contribution in agentic settings (PE4E: RW1 / RQ1). We clarify the contribution as a systematic study of detection, generalization, defense, and interpretation in multi-turn agentic settings, and explain why transfer from single-turn detection requires empirical verification.

  • Detector baselines and low-FPR performance (qU1o: RW3 / RQ2; PE4E: RW2 / RQ2). We add white-box, black-box, and keyword comparisons. On Qwen3-8B, the probe achieves AUROC 0.976 versus 0.942 for the strongest tested white-box baseline.

  • Stylistic shortcuts and controls (qU1o: RW1 / RQ1 and follow-up). We provide examples, external-data evaluations, and a new paired template-ablation control. After template removal and coherence-oriented editing, AUROC remains 0.9623 versus 0.9808 originally, with the probe and negatives fixed.

  • Defense benefits and clean-task utility (JcLT: RW2 / RQ2; PE4E: RW2–3 / RQ2–3). We compare detector gates under the same intervention and clarify that small utility gains over no intervention may reflect rollout variation.

  • Interpretation of latent signals (qU1o: RW2; JcLT: RW1 / RQ1 and round-2 response). We distinguish recoverable exposure signals from evidence of prompt-injection awareness, clarify the model-dependent explanation results, and revise the relevant claims.

We sincerely hope that these additional experiments and clarifications help address the reviewers' concerns. We respectfully ask that the new evidence and corresponding manuscript revisions be taken into account when assessing our work. Thank you again for your valuable time and continued consideration. We would be happy to provide any further clarification that would be helpful.

Best regards,
The authors of Submission 1173

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.