acceptodds
Under review as a conference paper at ICLR 2027

A Two-Neuron Edit To Mitigate Prompt Injections with Minimal Model Drift

Abstract

Indirect prompt injection causes language agents to follow instructions embedded in untrusted tool outputs instead of the user's request. Its success across several language models raises a fundamental question: does a prompt injection succeed because the model cannot identify where the instruction came from? We find that this is not necessarily true. Across Qwen models from 4B–27B, we find that successfully injected instructions remain strongly identifiable as tool-originating in internal representations, even when the model follows them. We call this dissociation the source–behavior gap: source is represented, but behavior does not reliably respect it. This gap suggests a localized defense: reuse the model's existing tool-source information and learn the behavioral correction needed to make it actually be used by the model. Our method compiles this behavioral correction into just two existing SwiGLU neurons of the model: an internal tool-content signal determines when the edit should be applied, while a learned residual-stream correction determines what should be written to restore clean behavior under attack. This approach improves robustness while mostly preserving original model behavior. On Qwen3.6-27B, our method reduces AgentDojo attack success from to while maintaining clean utility at . On ordinary prompts, of outputs remain exactly unchanged from the base model with next-token KL divergence at only from it, compared with - for other baseline defenses. This robustness also persists under adaptive attacks like AutoDojo, where attack success remains compared with for the undefended model.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Prompt Injections, Language Models, Alignment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mzKDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates indirect prompt injection attacks against LLM agents and proposes a defense method based on dual-neuron editing. The authors find that models can internally represent and distinguish the sources of external instructions, but this source information does not necessarily ensure that their behavior respects the corresponding authority boundaries. Based on this finding, the authors train a residual vector to bridge the gap between model outputs in benign and indirect prompt injection settings. They further determine where to apply this vector based on the tool context. By applying these two techniques to SwiGLU neurons in LLMs, the authors achieve prompt injection defense while preserving performance on benign tasks.

Critical strengths

1.The problem is clearly defined and practically meaningful. The proposed technique has potential for practical applications in real-world engineering scenarios. 2.The overall research workflow is comprehensive, with well-structured experimental procedures and a coherent narrative. The paper is logically organized, it's easy to follow and understand.

Critical weaknesses

1.All experiments are conducted on models from the Qwen family, with many experiments relying solely on Qwen3.6-27B. This is insufficient to demonstrate the applicability and reliability of the proposed technique across a broader range of LLMs. In particular, the key innovation of this work appears to rely heavily on the SwiGLU activation function, which is not adopted by all LLMs. 2.The core idea is relatively simple, and its description remains somewhat vague. For example, the specific procedure for identifying the residual vector is not clearly explained. As currently presented, the approach appears to involve a considerable degree of randomness and opportunism, raising concerns about its generalizability. A particular residual vector may work on Qwen and in a few indirect prompt-injection cases, which does not demonstrate its reliability across a broader range of experimental settings. The paper lacks sufficient mathematical justification or empirical evidence to substantiate this claim.

Questions for authors

Could the authors conduct additional experiments or provide further evidence to demonstrate that the proposed method generalizes effectively to larger-scale datasets?

Reviewer_FCWMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a weight-space defense against indirect prompt injections (IPI) that modifies only two model neurons, yet achieves good robustness against IPIs on the AgentDojo benchmark. The authors show that attack instructions remain strongly identifiable in the representation space, and evaluate their proposed method against multiple baselines on AgentDojo. On both static jailbreaks and a prompt-level adversarial attack method (AutoDojo), the proposed method achieves low ASRs while retaining benign capabilities. The authors also show their method generalizes to the Qwen model family.

Critical strengths
  • The research topic is important and the paper is of sufficient clarity
  • Showing that changing only two neurons can lead to a meaningful safety increase is useful for real adoption, where minimally changing the model is important.
Critical weaknesses
  • The evaluation setup is weak: Only three simple baselines are proposed, out of which only one shares the same training data as the proposed method. For IPI in particular, a large amount of defenses have been proposed: PromptGuard, PIGuard, ProtectAI, DRIFT, CaMeL, ROPE, MELON... In particular, inner defenses like ASIDE or Meta Sec Align also make the model more robust to prompt injections with minimal to none benign capability loss. At least a few of these methods should be used and compared to the proposed defense to show its strengh. Using the same training data and a similar hyperparameter search budget is also be important to remove possible confounds. Finally, it is also not clear whether the observed improvement over the proposed baselines is statistically significant due to the lack of confidence intervals.

  • Lack of ablations and missing details. Section 3 describes the two-neuron edit defense, where many choices (such as the number of neurons to edit, the norm penalty, the score function...) are not well motivated and ablated. Including additional details and ablation studies to motivate these choices would be important. For instance, does the defense work better if we increase the number of neurons modified? In addition, Appendix D goes over the training data, but some key details are missing about how it was created (which model generated the data, how was the data generated, how to reproduce it...).

Questions for authors

Besides the weaknesses raised above on their evaluation setup and the lack of ablations, I would also like to ask to the authors:

  • Appendix H and I show the performance of other models beyond the main model. What is the ASR before and after applying the edit on these models? The appendix and tables such as Table 14 seem to be missing those columns, which makes the performance of the proposed method on other models not clear. Reporting the same metrics as Tables 1 and 2 on these other models would be much more clear.

  • The paper deviates from AutoDojo, using Qwen3-14B as an attacker, whereas the original AutoDojo paper uses a much more capable model (Gemini-3.1-Pro) and a higher budget. Would the ASR of the proposed defense be higher with the original AutoDojo configuration?

  • How were the synthetic lookup worlds generated and selected?

  • (Clarification request) An "adaptive" attack has generally a different meaning in the AI safety field compared to how it is being used in this paper. (Tramer et al, 2020) define it as an attack that was specifically designed to target a given defense. But AutoDojo is a general attack, and it was not designed to bypass the proposed defense specifically, so by this definition it is not an adaptive attack. A clarification on how the word "adaptive" is used should be added, or another term should be used instead (for instance "automatic jailbreak attack", or "prompt-based adversarial attack").

  • (Minor concern) Apart from AgentDojo, it could also be good to compare against another benchmark, for instance either AgentDyn or InjecAgent, to show generalization to other tool call formats.

  • (Minor concern) Would the findings hold on models like Gemma, which use a different activation (GeGLU)?

(Tramer et al, 2020): On Adaptive Attacks to Adversarial Example Defenses

Reviewer_PvUoOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a lightweight model-editing method to defend LLM agents against indirect prompt injection attacks. The authors first identify a source–behavior gap, showing that models can recognize instructions from untrusted tool outputs but still follow them. Based on this observation, the proposed method learns a behavioral correction and a tool-content detection signal, then integrates them into two existing SwiGLU neurons. Experiments demonstrate that the method significantly reduces attack success rates while largely preserving the original model's behavior.

Critical strengths
  1. The source–behavior gap provides an interesting perspective on why prompt injection attacks can succeed. This observation also offers a clear motivation for the proposed defense.

  2. The idea of integrating a behavioral correction into only two existing neurons is interesting. Unlike conventional fine-tuning methods, it requires no additional model parameters or inference-time modules.

  3. The paper connects its observations about internal representations with the proposed defense, making the overall methodology reasonably well motivated.

Critical weaknesses
  1. Although the authors show that tool-source information can be identified from hidden representations, this does not fully explain why models follow malicious instructions. The relationship between the source–behavior gap and the effectiveness of the proposed correction could be explored more deeply.

  2. The method relies on selecting a suitable intervention layer and two existing neurons. It is unclear how sensitive the results are to these choices and whether the same level of robustness can be achieved consistently with different selections.

  3. The appendix shows that white-box optimization can still find successful adversarial examples against the edited model. Although these attacks fall outside the primary threat model, this suggests that the defense does not completely eliminate the underlying vulnerability.

Questions for authors

see the weakness

Comments on the paper

1 comment

Dear Reviewers,

The authors have submitted their rebuttals. I would be grateful if you could take a moment to review them carefully and respond as appropriate. Thank you for your ongoing engagement and support.

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.