acceptodds
Under review as a conference paper at ICLR 2027

Heads of Security: Learned Attention Gates Improve Prompt Injection Robustness

15 views

Abstract

We introduce Heads of Security (HoS), a prompt injection defense that enforces instruction–data separation via a small set of learned per-head attention gates. HoS learns how each head should redistribute attention between trusted and untrusted tokens, enabling models to reason over untrusted content without following injected instructions. We evaluate HoS on Llama-3.3-70B-Instruct and GPT-OSS-20B. We compare it against SecAlign++, a state-of-the-art finetuning-based defense, and ASIDE, a defense that, like ours, builds instruction–data separation into the model architecture. We show that HoS offers greater robustness to adaptive attacks, better performance on standard utility benchmarks, and matches top performance on standard agentic prompt injection benchmarks. More specifically, the black-box RL-Hammer attack reaches 97% ASR against GPT-OSS-20B and 85% against SecAlign, but only 10% against HoS. Claudini, a state-of-the-art white-box attack, strengthened against HoS through autoresearch, requires more FLOPs to reach the same ASR on HoS as it does on GPT-OSS-20B or SecAlign. HoS preserves base-model utility across standard multiple-choice benchmarks and prompting setups, whereas SecAlign and ASIDE degrade in few-shot settings. On agentic benchmarks with static attacks, HoS and SecAlign perform similarly and substantially improve the security–utility trade-off over the base model. ASIDE struggles with tool use, achieving only 10% utility. These results show that prompt injection robustness can be achieved and improved through targeted attention interventions.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Prompt injection, Prompt Injection Defence, Adversarial attacks, Model internals, Attention Heads

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZF2BOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Heads of Security (HoS), a lightweight defense against prompt injection based on instruction–data separation. Instead of broadly fine-tuning the model, HoS assigns tokens to trusted, untrusted, and template channels and learns a small set of per-head attention gates that control how each attention head attends to different trust regions. This allows the model to still read and reason over untrusted external content while reducing the chance of following malicious instructions embedded in it. Experiments on Llama-3.3-70B and GPT-OSS-20B show that HoS substantially improves robustness against both black-box RL-based and white-box adaptive attacks, while largely preserving the base model’s utility on standard and agentic benchmarks. Mechanistic analysis further suggests that different heads learn complementary roles, with most suppressing untrusted instructions and some helping detect injections.

Critical strengths
  1. The method is simple yet highly targeted, directly addressing prompt injection through per-head attention gating for instruction–data separation.
  2. The method is also extremely lightweight, requiring only a very small number of trainable gate parameters compared with conventional fine-tuning-based defenses.
Critical weaknesses
  1. The method assumes that the system can correctly label which tokens or sources should be treated as trusted and which should be treated as untrusted. However, determining this boundary can itself be difficult in realistic agentic scenarios, especially when external content may legitimately contain instructions that need to be followed. As a result, HoS should be viewed more as a component for enforcing instruction–data separation than as a complete end-to-end prompt injection defense.
  2. The proposed gating mechanism operates by modifying pairwise attention logits before the softmax operation, which makes it naturally compatible with standard dense-attention Transformers. However, the method cannot be directly applied to architectures such as linear attention, where explicit token-to-token attention coefficients are not available. This architectural dependency may limit the general applicability of HoS as model designs continue to evolve.
  3. The main evaluation is conducted on only two open-weight models, GPT-OSS-20B and Llama-3.3-70B. Although these models differ in architecture and scale, it remains unclear whether the same attention-gating behavior and robustness gains would consistently generalize to a broader range of model families, sizes, and training paradigms. Evaluating HoS on more diverse models would provide stronger evidence for the generality of the proposed approach.
Questions for authors

Please see weakness

2 comments

We thank reviewer ZF2B for their time and for their assessment. We are also grateful to reviewer BbmP for contributing to the discussion and for raising many excellent points, which we fully agree with.

We agree with reviewer ZF2B that the paper has limitations, including some agentic scenarios and linear attention architectures, and we discuss those limitations in Section 6 of the paper. We also believe that the setup we focus on -- instruction-data separation for models with standard softmax attention -- is an important one and is widely adopted in the field. We show a substantial progress over the current state-of-the-art for this setup, and we hope that the reviewer will agree that it deserves the attention in the community.

We elaborate on the individual weaknesses below.

The method assumes that the system can correctly label <...> tokens

We assume all user and system instructions to be trusted, and external tool outputs to be untrusted. This follows the role hierarchy widely used by both industry and academic research, for example:

  • Instruction Hierarchy [1,2] by OpenAI
  • Meta-SecAlign [3] by Meta
  • CaMeL [4] by Google DeepMind
  • TaskTracker [5] by Microsoft
  • ISE (ICLR 2025) [6], ASIDE (ICLR 2026) [7]

The problem is still largely unsolved [8]: models are still vulnerable to prompt injection attacks, even when all external data and tool outputs are considered untrusted (see, e.g. our results on AgentDojo, AgentDyn, WASP and LLMail).

We believe that enforcing strict instruction--data separation is a necessary step towards defending against prompt injection attacks, including in more complicated agentic scenarios. Establishing the trust boundary then becomes a separate problem, but we still need to be able to make the model read the untrusted text and not follow unwanted instructions.

the method cannot be <...> applied to <...> linear attention

We agree this is a limitation of the method, but this is intrinsic to any research on model architecture improvements.

The type of attention required by our method is still a popular option among model developers. Our method is compatible with many of the recent releases: Muse Spark Open and Muse Glimmer by Meta, Inkling family by Thinking Machines, Mistral Large 4, GLM-5.3, Flint by Recursive.

We have highlighted incompatibility with linear attention in Section 6 mostly because Qwen and Kimi families both use linear attention, and we want to encourage future research to adapt our method to that architecture as well.

To that end, we have some preliminary ideas on how to adapt our method to linear attention. The simplest option would be to scale each token's contribution to the attention state based on a per-head per-channel trained coefficient. This would be similar to how we scale pre-softmax coefficients, but without the softmax normalization.

The main evaluation is conducted on only two open-weight models.

We believe the two models we have chosen offer a good diversity:

  • GPT-OSS is a reasoning model, Llama-3.3 is not
  • GPT-OSS is Mixture-of-Experts, Llama-3.3 is dense
  • GPT-OSS uses hybrid attention, mixing full attention with sliding window attention layers. Llama-3.3 uses full attention in all its layers
  • Between GPT-OSS-20B and Llama-3.3-70B we cover the usual size range for popular open-weight models, balancing advanced capabilities with inference cost.

Considering this, we have decided to spend more compute on extensive evaluations for the two models we have (a range of white-box and black-box adaptive attacks, including autoresearch loop and RL training), instead of adding another model to the mix. We hope the reviewer agrees this was a reasonable decision to make.

References

[1] Wallace, Eric, et al. "The instruction hierarchy: Training llms to prioritize privileged instructions." arXiv preprint arXiv:2404.13208 (2024).

[2] Guo, Chuan, et al. "IH-Challenge: A Training Dataset to Improve Instruction Hierarchy on Frontier LLMs." arXiv preprint arXiv:2603.10521 (2026).

[3] Chen, Sizhe, et al. "Meta secalign: A secure foundation llm against prompt injection attacks." arXiv preprint arXiv:2507.02735 (2025).

[4] Debenedetti, Edoardo, et al. "Defeating prompt injections by design." 2026 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML). IEEE, 2026.

[5] Abdelnabi, Sahar, et al. "Get my drift? catching llm task drift with activation deltas." 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML). IEEE, 2025.

[6] Wu, Tong, et al. "Instructional segment embedding: Improving llm safety with instruction hierarchy." International Conference on Learning Representations. Vol. 2025. 2025.

[7] Zverev, Egor, et al. "Aside: Architectural separation of instructions and data in language models." International Conference on Learning Representations. Vol. 2026. 2026.

[8] Kurakin, Alexey, et al. "Scaling Adaptive Prompt Injection Attacks with Agent Swarms." arXiv preprint arXiv:2611.11861 (2026).

Reviewer_ZF2BOpenReview ↗

response to rebuttal

Thank you for the detailed response. I appreciate the authors' clarification and the discussion from the other reviewer.

  1. Trust boundary assumption: I understand that treating system/user instructions as trusted and tool outputs as untrusted is a common setting in prior work. However, I remain concerned about how realistic this assumption is in practical agent scenarios, where trusted and untrusted content may be mixed and the trust boundary may not be clearly defined. The fact that this assumption is widely adopted does not fully address this concern. The authors' response does not provide additional evidence on how HoS performs when trust labels are imperfect or ambiguous.

  2. Linear attention: The authors acknowledge this limitation and discuss possible extensions. However, these remain conceptual, without implementation or experimental validation. Therefore, my concern about the architectural generalizability of HoS remains.

  3. Evaluation across models: I appreciate the authors' explanation of their model selection and the extensive adaptive attack evaluations. I agree that the two selected models provide meaningful architectural diversity, and I consider this concern adequately addressed.

Reviewer_Ug86OpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Heads of Security (HoS) is a prompt-injection defense that learns how individual attention heads redistribute attention across externally assigned trusted, untrusted, and template channels. It adds channel-dependent biases to pre-softmax attention logits, freezes the original model weights, and trains only the gates using the SecAlign++ DPO recipe. The method requires 18,432 trainable parameters for GPT-OSS-20B and 46,080 for Llama-3.3-70B-Instruct. It assumes that the application supplies the trust boundary; it does not infer whether external instructions are authorized. The paper evaluates these two backbones against SecAlign and, for GPT-OSS, ASIDE, using adaptive black-box and white-box attacks, benign capability tests, and agentic security–utility benchmarks. Under the headline RL-Hammer configuration, the highest attack success rate falls to 10% for GPT-OSS at medium reasoning effort and 1.2% for Llama. White-box attacks require approximately three times more compute to reach 50% ASR on GPT-OSS. Additional analyses examine learned gates, uniform attention suppression, and sensitivity to demonstration formatting.

Critical strengths
  1. A simple, targeted intervention produces substantial security improvements with very few trainable parameters. HoS uses an externally available trust boundary to control attention without updating the original model weights. The improvements occur on both a dense model and an MoE reasoning model. Appendix D further shows that uniformly suppressing untrusted content does not match the learned gates' security–utility trade-off, providing evidence that the proposed intervention offers more than indiscriminate reduction of attention to external data. This combination of simplicity and empirical effectiveness is a substantial contribution.

  2. The evaluation meaningfully engages with adaptive attacks and practical utility. The paper goes beyond static injection benchmarks by training RL-Hammer attackers against defended models and developing a white-box attack specifically against HoS before evaluating it on separate cases. It reports attack-compute curves, cross-target transfer, and repeated-run uncertainty for key black-box results. The utility evaluations cover conventional benchmarks, conversation behavior, and agentic tasks. Appendix I's demonstration-format comparison is particularly useful: it identifies a concrete source of SecAlign failures rather than merely reporting low aggregate scores.

Critical weaknesses
  1. The utility-preservation claim has consequential exceptions, and its architectural explanation is not fully isolated. On AgentDyn, Llama's clean utility falls from 33.7% to 18.3% with HoS, compared with 41.7% for SecAlign (Table 3). For GPT-OSS, the headline HoS learning rate of 0.1 reduces AgentDyn clean utility from 58.33% to 41.67% (Table 13). Thus, preserving performance on all-trusted academic prompts does not establish that useful processing of external content is preserved. The comparison also leaves alternative explanations for the utility gap: GPT-OSS SecAlign uses three training epochs versus one for HoS, and the untrusted span is rendered differently. Independent learning-rate sweeps are valuable, but do not isolate these factors. Moreover, SecAlign's Llama MMLU-Pro 5-shot score recovers from 16.1% to 54.0% under multi-turn demonstrations (Table 14), so the extreme failures should be characterized as format sensitivity rather than general loss of task knowledge. A more controlled comparison would strengthen the claim that restricted gate parameterization itself explains the utility advantage.

  2. The robustness gains are conditional on the evaluated attack budget and reasoning setting, with limited evidence for broader generalization. Appendix F does evaluate both low and medium effort, but HoS's highest RL-Hammer ASR rises from 10% at medium to 36% at low (Table 8); high effort is not reported. The white-box result primarily demonstrates increased attack cost: at the full budget of 101710^{17} FLOPs on InjecAgent, HoS reaches 90% cumulative ASR and 84% delivered ASR (Appendix G). On Llama, the white-box evaluation contains only 20 examples and cannot distinguish HoS from SecAlign reliably. Finally, HoS is trained on only two backbones, without within-family scaling experiments. These limitations do not negate the large black-box improvements, and several are explicitly acknowledged, but they restrict conclusions about consistently strong protection across deployment settings.

Questions for authors
  1. What failure modes account for the AgentDyn utility reductions in Tables 3 and 13? In particular, does HoS prevent the model from using task-relevant external information, disrupt tool use, or cause another failure? Can a less aggressive gate configuration recover Llama's utility, and what adaptive-attack robustness would that configuration retain?

  2. Can you compare HoS with SecAlign using matched preference examples and training duration, with separately tuned learning rates and a control for the placement of untrusted content? An early-stopped or more strongly regularized SecAlign baseline would help determine how much of the utility advantage comes from attention gating rather than the amount or format of adaptation. The existing learning-rate sweeps and released Llama checkpoint are useful, but do not settle this attribution.

  3. Which reasoning-effort settings are used for GPT-OSS in the white-box, benign-utility, and agentic evaluations? For the checkpoints used in adaptive-attack testing, can you report a joint comparison of ASR, benign task success, and inference cost across low, medium, and high effort? This would clarify how the security–utility benefit varies with deployment configuration and whether the headline medium-effort result is representative.

3 comments

We thank reviewer Ug86 for their time and assessment. We are glad that the reviewer finds our adaptive attack evaluations convincing.

We address questions and weaknesses below. We have also updated the paper: new content is highlighted blue in the pdf.

AgentDyn utility (W1, Q2)

On AgentDyn <...> clean utility falls

preserving performance on all-trusted prompts does not establish that useful processing of external content is preserved.

We evaluate utility on four prompt injection benchmarks with untrusted input: AgentDojo, AgentDyn, WASP, and LLMail.

For GPT-OSS the results are shown in Figure 4 (utility under attack) and the new Figure 10 (clean utility). HoS largely preserves clean utility on AgentDojo and offers different security–utility trade-offs on AgentDyn and WASP, all with substantially lower ASR than the base model. Due to compute budget, we only ran adaptive attacks against one model per method, choosing the most defended checkpoint with a reasonable AgentDojo utility; we can't predict how other checkpoints would perform under adaptive evaluations.

For Llama, HoS reduces utility on AgentDyn, but largely preserves it on AgentDojo, WASP, and LLMail. We discuss the AgentDyn failures below. For GPT-OSS, HoS offers substantial security gains over undefended models and improves on or matches SecAlign’s performance across all benchmarks.

What failure modes account for the AgentDyn utility reductions?

Analyzing the transcripts shows that HoS models tend to ignore instructions in tool outputs that are necessary to complete the task. E.g., completing a refund requires following an instruction returned by the tool: retrieve a one-time password from an email and verify the account.

For Llama-3.3-70B, this explains failures on 40% of tasks with HoS, and 17% with the base model. For GPT-OSS, ignoring necessary instructions causes failures on 25–37% of tasks with HoS models vs 12% with the base model. See Appendix M for more details.

As we discuss in Section 6, supporting workflows that require following untrusted instructions is an important direction for future work.

Can a less aggressive gate configuration recover Llama's utility

We trained additional HoS variants of Llama-3.3-70B at different learning rates and it does recover better utility, up to base model’s performance. The results are reported in the new Appendix K.6.

BenchmarkMetriclr=1e-3lr=3e-3lr=5e-3lr=8e-3
AgentDojoClean utility41.3%55.4%48.9%58.5%
Attacked utility38.5%49.9%52.7%52.4%
ASR12.9%4.3%1.6%0.9%
AgentDynClean utility16.7%30.0%15.0%18.3%
Attacked utility27.5%27.1%20.0%18.4%
ASR10.5%5.0%3.2%1.4%

Due to computational constraints, we were only able to test the new checkpoints on static attacks with only one run. Results for our main checkpoint (8e-3) are averaged over five runs.

On AgentDyn, lower learning rates increases ASR but can recover utility, with the best clean utility at lr=3e-3 -- HoS can provide different balance between security and utility, depending on the setup. On AgentDojo, the pattern is different: both SecAlign and HoS improve utility over the base model, and lower HoS learning rates generally reduce utility while increasing ASR.

SecAlign trainig setup (W1, Q2)

GPT-OSS SecAlign uses 3 epochs vs 1 for HoS.

As suggested by the reviewer, we trained SecAlign for GPT-OSS-20B for 1 epoch, to match the training duration of HoS. Previously, as reported in the paper, we trained SecAlign models following the training recipe from Chen et al. (2025) with the same data and the same number of epochs (3).

Training SecAlign for 1 epoch instead of 3 does not fix SecAlign's few-shot utility issue: the performance drop compared to the base model is reduced slightly, but remains high.

ARC-CGSM8KMMLUMMLU-Pro
0-shot0-shot1-shot8-shot0-shot5-shot0-shot5-shot
GPT-OSS-20B96.1%91.4%93.6%95.6%83.0%82.3%72.0%68.3%
w/ SecAlign 3-epoch (paper)94.8%93.3%70.9%91.8%82.8%54.4%71.6%59.4%
w/ SecAlign 1-epoch (new)95.4%92.7%72.7%88.9%82.8%60.8%72.1%61.2%

We report additional results in the new Appendix K.5.

Can you compare HoS with SecAlign using matched preference examples <...> and a control for the placement of untrusted content?

Rendering untrusted content as "input" is an integral part of SecAlign's design: it is the intended way to render untrusted content, and is presented as a key improvement over prior work. Our design does not require a separate chat role. We believe that our current training setup is the best possible option to compare the two methods, given their inherent design differences.

On the data side, we followed the exact training data generation recipe from Chen et al. (2025). For GPT-OSS-20B, we extended it to include reasoning traces, what we believe is a natural way to extend the recipe to a reasoning model.

Few-shot utility failures (W1)

the extreme failures should be characterized as format sensitivity rather than general loss of task knowledge.

We agree that this demonstrates SecAlign's sensitivity to prompt format, but we see it as a critical weakness for the method. A successful defense should preserve the model’s ability to apply its knowledge across prompt formats. The base model and HoS do not show such sensitivity to prompt formats. SecAlign’s sensitivity therefore limits the reliability of its task performance, even when the underlying knowledge is preserved.

Reasoning effort for GPT-OSS-20B (W2, Q3)

Which reasoning-effort settings are used <...>?

Unless specified otherwise, we use medium reasoning effort in both training and evaluation.

can you report a joint comparison <...> across low, medium, and high effort?

We note that both defenses were trained with medium reasoning effort only, so low and high effort are out-of-distribution settings for both defenses. Even so, HoS keeps its advantage across all three settings. On agentic benchmarks, HoS Pareto-dominates SecAlign on AgentDojo and matches it on AgentDyn at every effort level (see newly added Figures 11 and 12). Under adaptive attacks at low effort, the highest RL-Hammer ASR against HoS rises from 10% to 36%, but it remains well below SecAlign (82%) and the undefended model (100%) (Table 8). We expect that training with mixed reasoning efforts would further reduce this gap.

Due to a limited compute we were only able to run varying resoning effort experiments on static benchmarks. Checkpoints used for adaptive attacks are highlighted with a black border on Figures 11 and 12.

White-box adaptive attacks (W2)

The white-box result primarily demonstrates increased attack cost

Our white-box evaluations are consistent with adopted practices both in academia[3,4,5] and in the industry (see Anthropic blogpost: they compare attack success under given a compute budget. We view this as a meaningful measure of robustness: an adversary with unbounded compute is not a realistic assumption, and required compute directly affect attack's feasibility.

Previous theoretical work shows that for undesirable behaviours retained by the model with nonzero probability, there always exist prompts that can elicit them [1,2]. We therefore do not expect our learned defense to prevent attacks from a motivated white-box attacker with a sufficiently large budget.

On Llama, the white-box evaluation <...> cannot distinguish HoS from SecAlign reliably.

We agree that the 20-example Llama evaluation is inconclusive. We acknowledge this in Appendix G.2 and report the results for completeness. White-box attacks against Llama-3.3-70B are very compute intensive -- they require 50×50\times more FLOPs per sample compared to GPT-OSS. We hope that results from our broader adaptive attack evaluations sufficiently demonstrate HoS's consistent improvement in security.

Models evaluated (W2)

HoS is trained on only two backbones, without within-family scaling experiments.

We believe the two models we have chosen offer a good diversity:

  • GPT-OSS is a reasoning model, Llama-3.3 is not
  • GPT-OSS is Mixture-of-Experts, Llama-3.3 is dense
  • GPT-OSS uses hybrid attention, mixing full attention with sliding window attention layers. Llama-3.3 uses full attention in all its layers
  • Between GPT-OSS-20B and Llama-3.3-70B, we cover the usual size range for popular open-weight models, balancing advanced capabilities with inference cost.

Considering this, we have decided to spend more compute on extensive evaluations for the two models we have (a range of white-box and black-box adaptive attacks, including autoresearch loop and RL training), instead of adding another model to the mix. We hope the reviewer agrees this was a reasonable decision to make.

References

[1] Wolf, Yotam, et al. "Fundamental limitations of alignment in large language models." Proceedings of the 41st International Conference on Machine Learning. Vol. 235. 2024.

[2] Su, Jingtong, Julia Kempe, and Karen Ullrich. "Mission impossible: A statistical perspective on jailbreaking llms." Advances in neural information processing systems 37 (2024): 38267-38306.

[3] Panfilov, Alexander, et al. "Claudini: Autoresearch discovers state-of-the-art adversarial attack algorithms for llms." arXiv preprint arXiv:2603.24511 (2026).

[4] Wang, Zijun, et al. “AttnGCG: Enhancing Jailbreaking Attacks on LLMs with Attention Manipulation.” Transactions on Machine Learning Research, 2025, openreview.net/forum?id=prVLANCshF.

[5] Li, Qizhang, et al. "Improved generation of adversarial examples against safety-aligned llms." Advances in Neural Information Processing Systems 37 (2024): 96367-96386.

Thank you for the detailed response. My concerns have been addressed. I remain positive about this paper.

Reviewer_BbmPOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a post-training method to improve the robustness of LLMs against strong prompt injection attacks. It uses a thoughtfully designed architectural refinement to bias the architecture towards being able to reason about messages roles/types and resist attacks.

Critical strengths
  • Practically relevant problem
  • SOTA results on defense against strong attacks
  • Significant improvement over previously published methods
  • Interesting, simple, and principled tweak to architecture
  • Thorough evaluation
  • Extensive experiments
  • Newly identifies some shortcomings of past work that I wasn't aware of
  • Better maintains utility
  • Better security than GPT 5.6 Luna, which I find impressive
  • Paper is well written and easy to read
Critical weaknesses
  • Nothing rising to the level deserving rejection
Questions for authors

None

1 comment

We sincerely thank reviewer BbmP for their supportive assessment of the paper, and for contributing to the discussion with other reviewers. We are especially pleased to see that the reviewer highlighted our results on the issues with existing defenses and comparison with GPT-5.6-Luna, those were some of our favourite results as well.

Please let us know if you have any further questions!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.