- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Heads of Security (HoS), a lightweight defense against prompt injection based on instruction–data separation. Instead of broadly fine-tuning the model, HoS assigns tokens to trusted, untrusted, and template channels and learns a small set of per-head attention gates that control how each attention head attends to different trust regions. This allows the model to still read and reason over untrusted external content while reducing the chance of following malicious instructions embedded in it. Experiments on Llama-3.3-70B and GPT-OSS-20B show that HoS substantially improves robustness against both black-box RL-based and white-box adaptive attacks, while largely preserving the base model’s utility on standard and agentic benchmarks. Mechanistic analysis further suggests that different heads learn complementary roles, with most suppressing untrusted instructions and some helping detect injections.
Critical strengths
- The method is simple yet highly targeted, directly addressing prompt injection through per-head attention gating for instruction–data separation.
- The method is also extremely lightweight, requiring only a very small number of trainable gate parameters compared with conventional fine-tuning-based defenses.
Critical weaknesses
- The method assumes that the system can correctly label which tokens or sources should be treated as trusted and which should be treated as untrusted. However, determining this boundary can itself be difficult in realistic agentic scenarios, especially when external content may legitimately contain instructions that need to be followed. As a result, HoS should be viewed more as a component for enforcing instruction–data separation than as a complete end-to-end prompt injection defense.
- The proposed gating mechanism operates by modifying pairwise attention logits before the softmax operation, which makes it naturally compatible with standard dense-attention Transformers. However, the method cannot be directly applied to architectures such as linear attention, where explicit token-to-token attention coefficients are not available. This architectural dependency may limit the general applicability of HoS as model designs continue to evolve.
- The main evaluation is conducted on only two open-weight models, GPT-OSS-20B and Llama-3.3-70B. Although these models differ in architecture and scale, it remains unclear whether the same attention-gating behavior and robustness gains would consistently generalize to a broader range of model families, sizes, and training paradigms. Evaluating HoS on more diverse models would provide stronger evidence for the generality of the proposed approach.
Questions for authors
Please see weakness
2 comments
response to rebuttal
Thank you for the detailed response. I appreciate the authors' clarification and the discussion from the other reviewer.
-
Trust boundary assumption: I understand that treating system/user instructions as trusted and tool outputs as untrusted is a common setting in prior work. However, I remain concerned about how realistic this assumption is in practical agent scenarios, where trusted and untrusted content may be mixed and the trust boundary may not be clearly defined. The fact that this assumption is widely adopted does not fully address this concern. The authors' response does not provide additional evidence on how HoS performs when trust labels are imperfect or ambiguous.
-
Linear attention: The authors acknowledge this limitation and discuss possible extensions. However, these remain conceptual, without implementation or experimental validation. Therefore, my concern about the architectural generalizability of HoS remains.
-
Evaluation across models: I appreciate the authors' explanation of their model selection and the extensive adaptive attack evaluations. I agree that the two selected models provide meaningful architectural diversity, and I consider this concern adequately addressed.
We thank reviewer ZF2B for their time and for their assessment. We are also grateful to reviewer BbmP for contributing to the discussion and for raising many excellent points, which we fully agree with.
We agree with reviewer ZF2B that the paper has limitations, including some agentic scenarios and linear attention architectures, and we discuss those limitations in Section 6 of the paper. We also believe that the setup we focus on -- instruction-data separation for models with standard softmax attention -- is an important one and is widely adopted in the field. We show a substantial progress over the current state-of-the-art for this setup, and we hope that the reviewer will agree that it deserves the attention in the community.
We elaborate on the individual weaknesses below.
We assume all user and system instructions to be trusted, and external tool outputs to be untrusted. This follows the role hierarchy widely used by both industry and academic research, for example:
The problem is still largely unsolved [8]: models are still vulnerable to prompt injection attacks, even when all external data and tool outputs are considered untrusted (see, e.g. our results on AgentDojo, AgentDyn, WASP and LLMail).
We believe that enforcing strict instruction--data separation is a necessary step towards defending against prompt injection attacks, including in more complicated agentic scenarios. Establishing the trust boundary then becomes a separate problem, but we still need to be able to make the model read the untrusted text and not follow unwanted instructions.
We agree this is a limitation of the method, but this is intrinsic to any research on model architecture improvements.
The type of attention required by our method is still a popular option among model developers. Our method is compatible with many of the recent releases: Muse Spark Open and Muse Glimmer by Meta, Inkling family by Thinking Machines, Mistral Large 4, GLM-5.3, Flint by Recursive.
We have highlighted incompatibility with linear attention in Section 6 mostly because Qwen and Kimi families both use linear attention, and we want to encourage future research to adapt our method to that architecture as well.
To that end, we have some preliminary ideas on how to adapt our method to linear attention. The simplest option would be to scale each token's contribution to the attention state based on a per-head per-channel trained coefficient. This would be similar to how we scale pre-softmax coefficients, but without the softmax normalization.
We believe the two models we have chosen offer a good diversity:
Considering this, we have decided to spend more compute on extensive evaluations for the two models we have (a range of white-box and black-box adaptive attacks, including autoresearch loop and RL training), instead of adding another model to the mix. We hope the reviewer agrees this was a reasonable decision to make.
References
[1] Wallace, Eric, et al. "The instruction hierarchy: Training llms to prioritize privileged instructions." arXiv preprint arXiv:2404.13208 (2024).
[2] Guo, Chuan, et al. "IH-Challenge: A Training Dataset to Improve Instruction Hierarchy on Frontier LLMs." arXiv preprint arXiv:2603.10521 (2026).
[3] Chen, Sizhe, et al. "Meta secalign: A secure foundation llm against prompt injection attacks." arXiv preprint arXiv:2507.02735 (2025).
[4] Debenedetti, Edoardo, et al. "Defeating prompt injections by design." 2026 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML). IEEE, 2026.
[5] Abdelnabi, Sahar, et al. "Get my drift? catching llm task drift with activation deltas." 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML). IEEE, 2025.
[6] Wu, Tong, et al. "Instructional segment embedding: Improving llm safety with instruction hierarchy." International Conference on Learning Representations. Vol. 2025. 2025.
[7] Zverev, Egor, et al. "Aside: Architectural separation of instructions and data in language models." International Conference on Learning Representations. Vol. 2026. 2026.
[8] Kurakin, Alexey, et al. "Scaling Adaptive Prompt Injection Attacks with Agent Swarms." arXiv preprint arXiv:2611.11861 (2026).