acceptodds
Under review as a conference paper at ICLR 2027

Reasoning Self-Intervention for Prompt Injection Robustness

Abstract

Prompt injection remains an unresolved security challenge for large reasoning models (LRMs) that interact with external content. Existing defenses perform well on static benchmarks yet remain vulnerable to stronger, optimization-based adaptive attacks. We introduce self-intervention, a plug-in defense strategy based on iterative textual interventions. An add-on monitor head detects goal drift from the model’s hidden states during generation and triggers reminder text in its ongoing reasoning. It requires neither an additional language model to guide interventions nor updates to the underlying reasoning model’s weights. Experiments show that self-intervention features improved robustness against strong adaptive attacks, including only 1% attack success rate against a state-of-the-art RL-attack, while preserving benign utility.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
prompt injection, language model security, reasoning intervention, adaptive robustness, goal drift

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3neiOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Self-Intervention, a prompt-injection defense for large reasoning models (LRMs). Instead of directly classifying external content as malicious, the method continuously monitors goal drift, i.e., whether the model’s reasoning or actions begin to deviate from the original trusted user request. The authors attach lightweight monitor heads to the frozen reasoning model and use hidden states to detect token-level drift during generation. Once persistent drift is detected, the method rolls back the recent reasoning tokens, inserts a textual reminder, and resumes generation, allowing multiple interventions when necessary. The approach requires neither updates to the base model nor an additional LLM during inference. Experiments on InjectAgent and AgentDojo, including strong adaptive attacks such as MAPO, PAIR, and PISmith, show that Self-Intervention substantially improves robustness while largely preserving benign task performance. Overall, the paper reframes prompt-injection defense from input-level attack detection to reasoning-level goal-drift monitoring and online intervention.

Critical strengths
  1. Self-Intervention adds lightweight monitor heads to the frozen reasoning model to detect goal drift during generation. Once drift is detected, it rolls back recent reasoning tokens, inserts a reminder, and resumes generation. The method does not require updating the base model or invoking an additional LLM during inference, making it relatively lightweight and practical.
  2. The paper evaluates the defense against several optimization-based adaptive attacks, including MAPO, PAIR, and the RL-based PISmith. Self-Intervention consistently achieves substantially lower attack success rates than the baselines, demonstrating strong robustness beyond fixed or static prompt-injection attacks.
Critical weaknesses
  1. The experiments are mainly conducted on Qwen3.6-27B, leaving it unclear whether the proposed monitor generalizes across different model families and scales. The effectiveness of Self-Intervention may depend on model-specific hidden representations.
  2. Although the additional parameters are relatively small, the method continuously evaluates eight monitor heads at every generated token and may require rollback and regeneration after an intervention. The paper does not provide a systematic analysis of latency, throughput, or computational overhead in practical deployment.
  3. Self-Intervention relies on the final-layer hidden states of the underlying reasoning model. Therefore, the method is not directly applicable to black-box or API-only models where internal representations are inaccessible, which limits its deployment scope.
  4. The paper reports cases where the reasoning appears to return to the original task, but the final tool call still follows the injected instruction. This suggests that safe-looking reasoning does not necessarily ensure safe downstream actions, particularly for high-risk agent operations.
  5. The monitor is trained by pairing two normal requests and treating pursuit of the second request as goal drift when it is hidden from the monitor. While elegant, it remains unclear how well this synthetic formulation captures subtle and complex goal manipulation in real-world prompt-injection attacks. The use of an external LLM judge for labeling may also introduce supervision noise.
Questions for authors

Please see weakness

3 comments

We would like to thank you for your detailed comments and feedback. We address the specified weaknesses below.

W1. Generalization across models

Our method generalizes to other model families; we report below the effectiveness of our defense with Muse Glimmer 30B and Gemma 4 26B-A4B. ASR@1 and ASR@5 evaluate MAPO attacks on InjecAgent (Table 1) and the last three columns evaluate on AgentDojo (Table 2).

ModelMethodMAPO ASR@1 ↓MAPO ASR@5 ↓AgentDojo ASR ↓AgentDojo Utility under attack ↑AgentDojo Benign utility ↑
Qwen3.6-27BUndefended85%95%14.44%83.88%95.88%
Qwen3.6-27BOurs0%1%1.05%86.62%94.85%
Muse Glimmer 30BUndefended74%86%2.32%86.72%91.75%
Muse Glimmer 30BOurs5%17%0.21%85.88%89.69%
Gemma 4 26B-A4BUndefended78%92%8.64%87.57%89.69%
Gemma 4 26B-A4BOurs2%3%2.42%83.88%88.66%

W2. Runtime overhead

On AgentDojo benign tasks, the full eight-head defense adds 9.06% end-to-end wall-clock overhead, including monitoring, hint prefill, stopping, and regeneration, and generates only 2.71% more tokens. We attribute this efficiency to infrequent intervention on benign tasks and reuse of the retained KV cache when generation resumes. Specifically, only 3/360 turns (across 3 out of the 97 tasks) triggered intervention; this is less than 1% of total turns. One turn ended in blocking (0.28% of all turns).

W3. Hidden-state access

Our defense is best characterized as a model-based method. Thus, we envision that our method will be most relevant for anyone hosting models with access to their hidden states. It might be possible to apply a text-based monitor to protect models that expose reasoning traces but where the internal representation is not accessible; we consider this out-of-scope and do not evaluate it in our work.

W4. Reasoning and action mismatch

We agree; we will mention this limitation in our revised manuscript.

W5. Training data and attack coverage

We appreciate the concern. We believe that our scheme’s success against three different adaptive optimization-based attacks, along with the fact that it substantially outperforms all prior methods (Table 1), is strong evidence for our method’s effectiveness. But we appreciate that it is hard to be sure we have covered all possible attack strategies.

Reviewer_3neiOpenReview ↗

response to rebuttal

The rebuttal partially solved my concern. Especially on weakness 1,2.

Thank you for your follow-up! We’re glad that our rebuttal has helped address some of your concerns. Please let us know if there are any remaining concerns we could further clarify.

Reviewer_tZcsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Self-Intervention, a defense against prompt injection attacks for large reasoning models. The method trains lightweight monitor heads to detect goal drift from hidden states during generation, then intervenes by rolling back tokens and inserting reminders into the reasoning process.

Critical strengths

The idea is interesting. The paper proposes a promising approach to improving prompt injection robustness through monitoring and intervening in the reasoning process.

The method significantly reduces attack success rates against several adaptive attacks.

The defense maintains performance close to the original model on benign reasoning and agentic tasks.

Critical weaknesses

The experiments only consider Qwen3.6-27B, making it unclear whether the method generalizes to other reasoning models.

The method uses eight monitor heads and continuously evaluates hidden states during generation, but does not report the additional inference latency or computational cost.

Questions for authors

See weakness.

3 comments

We would like to thank you for your feedback. We address the specified weaknesses below.

W1. Generalization across models

Our method generalizes to other model families; we report below the effectiveness of our defense with Muse Glimmer 30B and Gemma 4 26B-A4B. ASR@1 and ASR@5 evaluate MAPO attacks on InjecAgent (Table 1) and the last three columns evaluate on AgentDojo (Table 2).

ModelMethodMAPO ASR@1 ↓MAPO ASR@5 ↓AgentDojo ASR ↓AgentDojo Utility under attack ↑AgentDojo Benign utility ↑
Qwen3.6-27BUndefended85%95%14.44%83.88%95.88%
Qwen3.6-27BOurs0%1%1.05%86.62%94.85%
Muse Glimmer 30BUndefended74%86%2.32%86.72%91.75%
Muse Glimmer 30BOurs5%17%0.21%85.88%89.69%
Gemma 4 26B-A4BUndefended78%92%8.64%87.57%89.69%
Gemma 4 26B-A4BOurs2%3%2.42%83.88%88.66%

W2. Runtime overhead

On AgentDojo benign tasks, the full eight-head defense adds 9.06% end-to-end wall-clock overhead, including monitoring, hint prefill, stopping, and regeneration, and generates only 2.71% more tokens. We attribute this efficiency to infrequent intervention on benign tasks and reuse of the retained KV cache when generation resumes. Specifically, only 3/360 turns (across 3 out of the 97 tasks) triggered intervention; this is less than 1% of total turns. One turn ended in blocking (0.28% of all turns).

Thanks for the additional results. These partially solved my concern.

Thank you for your response! We're glad the additional results were helpful. Could you please let us know what concerns remain? We'd be happy to clarify further.

Reviewer_WjHQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes self-intervention, an inference-time prompt-injection defense for LRMs. Small transformer heads (one cross-attention block over trusted-request hidden states, then three causal self-attention blocks; ~18M parameters each) read the frozen model's final-layer hidden states and predict token-level goal drift. Untrusted tool output is excluded from the heads' inputs. If any of 8 heads flags at least 10 of the last 20 tokens, the controller rolls back ~20 tokens, inserts a fixed first-person reminder into the reasoning trace, and resumes decoding. The 5th trigger blocks the turn. Heads are trained with token-level BCE on labels from an external LLM judge. Training data comes from paired APIGen requests (U1, U2): the same trajectory is labeled differently depending on whether U2 is visible to the monitor. Post-intervention continuations and response-replacement augmentations are added. All experiments use Qwen3.6-27B. On InjecAgent, ASR is 0%/1% under MAPO (ASR@1/@5), 10% under PAIR (20 iterations × 5 seeds), and 1% under PISmith (ASR@10). Under PAIR, sandwich, reasoning-prefix, and SecOPD reach 31%, 88%, and 99%. On AgentDojo, ASR drops from 14.4% to 1.1%, with benign utility at 94.8% (undefended: 95.9%).

Critical strengths
  1. The detector triggers a rollback-and-hint, not a block/allow decision. A false positive therefore costs some tokens rather than a failed task, so the threshold can be set aggressively. Repeated interventions (up to 4) cover the case where the model drifts back to the injected goal after the first hint.
  2. Emperical experiments demonstrate the proposed defense works against strong optimization-based adaptive attacks
Critical weaknesses
  1. MAPO is optimized on the undefended model, and its payloads are reused across all defenses. PAIR gets a text description of the defense and a binary success signal. The PISmith section does not say which defender the attacker was trained against. The heads are differentiable functions of hidden states, so a white-box attacker can minimize drift probability directly (e.g., GCG on the injected span with a monitor-score term). A gray-box attacker can use trigger/rollback events as a dense reward. The conclusion already names the failure mode such an attacker would exploit: injections framed as necessary to finish the user's task. Without a monitor-aware attack, the 1% PISmith number measures robustness to an attacker tuned for a different target. None of the attacks optimize against the monitor.
  2. The ablations vary head count and head architecture, but not the method's components. Four are missing. (a) Detect-and-block vs. detect-and-intervene, which tests whether rollback and hint add anything beyond the detector. (b) A linear or MLP probe on the same hidden states, to justify the 4-block head. (c) Hints inserted at random positions at matched frequency, to separate trigger timing from hint content. (d) The LLM judge used directly as the monitor (the Section 3.1 oracle), as an upper bound. The architecture comparison in Figure 4 uses a single seed and reports only validation loss and controller accuracy, with no ASR or utility.
  3. The controller allows up to 4 rollbacks and hard-blocks on the 5th trigger. The paper therefore needs four numbers: the benign trigger rate, the share of BLOCKED turns on benign and attacked inputs, interventions per turn, and wall-clock and token overhead. A BLOCKED turn is a utility failure. An attacker who can reliably cause blocks has a denial-of-service attack. Tables 2 and 3 do not say how blocked turns are scored.
Questions for authors
  1. Which defender was PISmith trained against? On the same undefended model, ASR@1 is 20% for PISmith and 85% for MAPO. Why is the RL attacker this much weaker on the undefended baseline?
  2. What fraction of benign turns trigger at least one intervention, and what fraction end BLOCKED? Is BLOCKED scored as a utility failure in Tables 2–3 and as a security success in Table 1?
  3. Are the 100 InjecAgent cases in Section 5.1 drawn from the Table 1 evaluation set? If so, the 8-head setting was selected on test data. How were τ, w, q, B, the rollback length, and the hint text chosen?
3 comments

We would like to thank you for your detailed comments and feedback. We begin by addressing each of the specified weaknesses, followed by answers to your questions.

Response to weaknesses

W1. Clarification on Adaptive Attacks

MAPO iteratively searches for universal attack strategies. We use this to construct a shared strong baseline attack against all methods. Optimizing against the undefended model yields a better attack, as the optimizer receives dense rewards. We tried optimizing MAPO directly against the full defense, but the resulting attack is not any stronger. We thus use PAIR as our evaluation against strong adaptive attacks; the PAIR attack uses the MAPO-generated strategy as a starting point.

PAIR and PISmith optimize against each of the defenses individually. Our PAIR attack is monitor-aware: it can observe both the full reasoning trace and response/tool call from the defended model (i.e., not just a binary success signal). Therefore when tested against self-intervention, PAIR observes how many hints were applied and how they changed the defended model’s behavior, and can adapt to the monitor’s interventions across iterations. We will clarify this in the revised version of the paper, and we hope that this addresses your concern regarding the adaptive nature of our attacks.

W2. Additional ablations on method components

We have evaluated the requested ablations. ASR@1 and ASR@5 use the same cached MAPO500 trials on InjecAgent. Utility metrics use AgentDojo.

(a) Detect-then-block versus intervention

We analyze detect-then-block using the saved evaluation trajectories and find that intervention preserves more utility, particularly under attack.

MetricCurrent policyDetect-then-block
Benign utility92/97 (94.85%)91/97 (93.81%)
Utility under attack822/949 (86.62%)642/949 (67.65%)

(b) Linear and MLP probes, and existing architectural ablations

Our four-block attention head outperforms linear and MLP probes trained on the same hidden-state corpus, in both single-head and eight-head settings. Validation accuracy is measured on the held-out validation split used during training.

Single headValidation accuracyASR@1ASR@5Benign utility
Linear89.38%44%56%94.85%
MLP96.65%21%34%94.85%
Cross-attention only97.34%17%26%90.72%
Self-attention only89.68%73%87%95.88%
Ours99.09%7%29%95.88%
Ensemble of 8ASR@1ASR@5Benign utility
Linear31%45%93.81%
MLP21%34%94.85%
Ours0%1%94.85%

(c) Random intervention at matched frequency

As requested, we used our defense's mean intervention count to set the random baseline's overall intervention budget. We then inserted the same hint texts at randomly selected positions.

MethodASR@1ASR@5
Random intervention48%92%
Ours, eight heads0%1%

(d) External LLM monitor

We also evaluated the external LLM judge directly as the monitor. It observes the complete candidate reasoning and proposed action with full context, while our heads operate on streaming hidden states. This provides an empirical upper-bound at additional deployment cost, since each turn waits for an external LLM evaluation.

MethodMAPO ASR@1MAPO ASR@5AgentDojo ASRUtility under attackBenign utility
External LLM monitor0%0%0%90.52%95.88%
Ours, eight heads0%1%1.05%86.62%94.85%

W3. Additional statistics

a. Intervention and blocking statistics

For Table 2, on AgentDojo benign tasks, only 3/360 turns (across 3 out of the 97 tasks) triggered intervention; this is less than 1% of total turns. One turn ended in blocking (0.28% of all turns). Under attack, 263/3,243 turns (8.11%) triggered intervention, of which 44 were blocked (1.36% of all turns). For Table 1, PAIR generated 9,145 defender trials across 100 InjecAgent instances, covering diverse, often case-specific attack strategies. Of these, 2,036 trials (22.26%) triggered intervention. The distribution among triggered trials is shown below. 79.22% had only one or two triggers, while 10.56% reached the fifth trigger and were blocked.

Trigger countTrialsShare
11,32965.28%
228413.95%
31437.02%
4653.19%
5 → BLOCKED21510.56%

b. System overhead

On AgentDojo benign, the full eight-head defense adds 9.06% end-to-end wall-clock overhead, including monitoring, hint prefill, stopping, and regeneration, and generates only 2.71% more tokens. We attribute this efficiency to infrequent intervention on benign tasks and reuse of the retained KV cache when generation resumes.

Answer to questions

Q1. PISmith training

The PISmith attacker was trained against each of the defenses separately. We are not sure why MAPO is stronger than PISmith. One possible explanation is that we use Gemini 3.1 Pro and Gemini 3 Flash Preview for MAPO, while PISmith uses Qwen 3 4B (a much smaller model) due to the high cost of RL.

Q2. Intervention and blocking outcomes

Please see the rates in response to weaknesses above, for the hint counts and BLOCKED counts. In Tables 1–3, the benchmark grading functions determine attack success and utility, including when a turn ends in BLOCKED. A block stops model inference and generally results in either a utility failure (for benign instances) or a successful defense (for attacked instances).

Q3. Head count, controller settings, and hints

We picked the number of heads during the design phase on a separate held-out set, before evaluation for table 1-3. In general, increasing the number of heads improves the security of our defense but also results in a higher number of false positives. The 1/4/8-head ablation in our manuscript was performed after all other evaluations were completed, and is meant to empirically demonstrate the security benefits. Section 5.1 uses the same 100 InjecAgent cases as Table 1.

Other hyperparameters were established at the beginning of development and held fixed throughout the reported Qwen experiments, without a hyperparameter search. We also retained the same controller settings for the ablations presented in this rebuttal. These are practical heuristic choices; we do not claim they are optimal.

Our hints were generated with the assistance of an LLM. The LLM was asked to generate hints in a neutral tone, and to provide a brief alert as to the possibility of an external goal. We did not optimize the hint texts for evaluation performance.

AuthorsOpenReview ↗

Follow-up from Authors

We appreciate your time and consideration. We would be grateful if you could take a look at our rebuttal and share any remaining concerns. Thank you again for your thoughtful feedback!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.