acceptodds
Under review as a conference paper at ICLR 2027

Safety Fragility Is a Routing Problem: The Attention Bottleneck Between Harm Detection and Refusal

Abstract

Aligned language models refuse harmful requests most of the time, and that refusal is remarkably fragile. Two lines of research explain the fragility in ways that do not fit together: one localizes safety in a single residual-stream direction or a handful of attention heads, the other finds it confined to a narrow basin in weight space that fine-tuning drifts out of. We show that they describe different components of one mechanism. To unify these findings, we decompose the safety into three components: detection of harm, routing of that signal through attention, and emission of a refusal. Both detection and action are stable under weight perturbation, and what breaks is the routing: a small, non-redundant set of attention heads, which we call the gate, connecting detection to action. Standard feature attribution, the method the field uses to identify which heads matter for a behavior, ranks this set incorrectly in all three model families we test, because it scores how much a head writes toward refusal rather than whether the output depends on it. The routing bottleneck also explains the basin geometry that prior work observes without an underlying cause: safety is narrow along the gate direction because small perturbations there sever the connection between two otherwise stable representations. If fragility comes from depending on a few heads, then training the model to distribute that dependence should make it robust. We test this with noise-injected fine-tuning on the bottleneck heads. The model's dependence on them disappears, and the basin widens along that direction. Applying the same training to the top-attribution head changes nothing, because its contribution is redundant and removing it leaves the circuit intact.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
mechanistic interpretability, refusal behavior, attention heads, safety alignment robustness

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_TzmkOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper asks why the safety alignment of instruction-tuned language models is fragile. A handful of fine-tuning examples or a targeted perturbation can break safety behavior, yet the underlying safety-relevant representations are robust. How do these two facts coexist? The authors decompose safety refusal into three components: detection, a linear feature at the instruction token position that separates harmful from benign inputs; action, a refusal direction in the residual stream at the generation position; and routing, attention heads that carry the detection signal to the action site. The core finding is that detection and action are stable under random perturbation, with cosine stability above 0.96, while routing depends on a handful of irreplaceable attention heads — sometimes just one — that form a fragile bottleneck.

The main methodological contribution is showing that standard direct feature attribution selects the wrong heads. The highest-scoring heads are highly redundant: removing them barely changes the attack success rate. The truly indispensable heads score modestly on attribution but are irreplaceable. This mismatch between attribution and causal necessity holds across SmolLM2-1.7B, Llama-3.1-8B, and Qwen-2.5-7B. The authors also introduce basin anisotropy to characterize the directional sensitivity of the gate: perturbing along the gate direction raises ASR substantially, while perturbing along random directions does not.

A training intervention using a LoRA adapter with self-distillation, a KL anchor, and noise injection into the gate heads successfully redistributes routing dependence, collapsing the original gate's causal effect and basin anisotropy. A decoy control targeting the top attribution head has no effect. The trained adapter causes over-refusal, which the paper analyzes as a symmetric routing failure: the adapter learns a discriminating direction that separates harmful from benign inputs, and also learns to write toward the refusal axis, but these two signals live in orthogonal subspaces, so detection cannot modulate action. A self-gating fix that projects the adapter output onto the discriminating direction and scales accordingly achieves a 14× improvement in the safety–utility trade ratio on Llama-3.1.

The framework fails on 3 of 7 tested configurations — Qwen-32B, Qwen-14B, and Llama-2-7b-chat — which the authors attribute to already-distributed routing. The appendices provide activation patching validation, a no-prefill replication, and analysis of safety-pretrained models.

Critical strengths

1. The mismatch between attribution and causal necessity is a valuable finding. Ranking attention heads by their projection onto a direction of interest and treating the top-ranked heads as the important ones is standard practice in interpretability. This paper gives clear evidence that doing so for safety-critical behavior selects the wrong heads. The heads it selects are highly redundant — removing them has little effect. The heads that actually matter rank lower on attribution but are irreplaceable. The mismatch is sizable: on SmolLM2, the top attribution head has Δabl≈0\Delta_\text{abl} \approx 0, while the gate head produces over 15× the ablation effect. Results are consistent across all three model families. The decoy control in Section 4, which applies the same training intervention to the attribution-selected head and sees no effect, adds causal confirmation. Whether or not one accepts the "routing" narrative, this finding on its own is worth knowing for anyone doing safety-related interpretability work.

2. The experimental work is thorough, honestly reported, and highly reproducible. Seven model configurations are tested, and the three failures are presented upfront rather than buried. Perturbation experiments use 30 seeds, basin measurements use 15. The safety judge is an ensemble of keyword matching and WildGuard/Llama Guard, with κ=0.67\kappa = 0.67 reported. The prefill setup is clearly stated, and Appendix F provides a no-prefill replication. The reproducibility package includes a machine-readable manifest, a numerical consistency checker, and the ability to regenerate all figures from a single script — well above the norm for this venue. The component ablation in Table 2 is set up as a controlled experiment rather than a post hoc narrative, distinguishing noise injection as the only indispensable ingredient and showing that the KL anchor controls over-refusal but does not drive routing redistribution.

Critical weaknesses

1. The paper claims to have identified a "routing" mechanism, but the evidence stops at "certain heads are necessary" and does not show how these heads actually work. The paper's narrative is a three-component pipeline where detection travels through gate heads to action. Supporting this narrative requires two layers of evidence: first, that the detection and action features are really what the paper claims they are; second, that the gate head is actually performing signal transmission. But the evidence provided is only that the upstream and downstream features are robust, that removing the head breaks refusal, and that the gate direction has high basin anisotropy. This establishes that the gate head is necessary, not what it is doing.

Specifically, the paper extracts a harmfulness direction and a refusal direction using mean-difference and shows they are stable across seeds. But mean-difference will always find a separating direction between two groups, and stability shows the direction exists and is consistent — not that the model actually uses it. The paper also does not examine whether the gate head's attention pattern differs between harmful and benign inputs, does not analyze whether its query/key space aligns with the harmfulness direction, and does not check whether its OV projection aligns with the refusal direction.

More importantly, the paper does not use its framework to explain actual safety phenomena. What happens at the gate head when an attack succeeds? What is the head doing during normal refusal? What about during over-refusal? These comparisons are what would show that "routing" is an explanatory framework with real analytical power. The paper proposes a detection-routing-action decomposition but never applies it to analyze any concrete scenario. The framework stays at the level of naming rather than demonstrating analytical capability.

An alternative explanation is available. The head may sit at a bottleneck on the refusal computation path, performing some generic operation that happens to be critical. Remove it and refusal breaks, but it need not be "transporting" a detection signal. The paper's data is equally compatible with a pure bottleneck interpretation.

2. When the framework applies is unpredictable, and some of the finer claims lack statistical validation. The method fails on 3 of 7 configurations, and the paper provides no criterion that would let a practitioner know, before running the full experimental pipeline, whether a given model has a concentrated gate. The post hoc explanation that these models already have distributed routing has no predictive value. The gate's identity is also unstable across evaluation conditions: on Llama-3.1, the gate is L12H6 under prefill and shifts to L10H27 without prefill. The mismatch phenomenon replicates, but the fact that which head serves as the gate changes with the input distribution raises the question of whether the gate is a stable architectural feature or an artifact of the evaluation protocol.

The discriminating direction in Section 4.3 is especially concerning. This direction is the mean difference between adapter outputs on harmful and benign inputs, and the cosine similarity between the two classes is 0.83–0.98 — a small difference that at first glance looks like it could be noise. In high-dimensional space, a nonzero mean difference is expected even under the null hypothesis, and the paper provides no permutation test or bootstrap confidence interval to rule this out. What this direction represents semantically is also left entirely unanalyzed. What is its relationship to the base model's harmfulness direction? Did the adapter rediscover the model's native detection feature, or did it learn an entirely different discriminator? These questions are not addressed. The self-gate uses this direction for inference-time correction, with the sigmoid calibrated on 64+64 prompts and no held-out evaluation. The 14× trade ratio is suggestive but falls short of conclusive without basic cross-validation.

Questions for authors
  1. (Clarification, may affect my assessment) Have you looked at the gate head's attention pattern on harmful vs. benign inputs? If the gate head attends noticeably more to instruction-position tokens on harmful inputs, that would be direct evidence for the routing interpretation. If the attention pattern does not differ, the bottleneck interpretation becomes more plausible. I think this is the single experiment that would most strengthen or weaken the central claim.

  2. (Clarification, may affect my assessment) What is the cosine similarity between the discriminating direction dd and the base model's harmfulness direction? If they align, it would mean the adapter rediscovered the model's native detection feature in a disconnected subspace. If they are orthogonal, the adapter learned an independent discriminator. Either answer would help clarify how over-refusal actually arises.

  3. (Clarification) Can you provide a permutation test for the discriminating direction? Shuffling harmful/benign labels across 1000 permutations and recomputing ∥d∥\|d\| would establish whether the observed direction is statistically distinguishable from chance. Also, does the self-gate's trade ratio hold on a held-out prompt set not used for sigmoid calibration?

  4. (Suggestion) Is there a lightweight diagnostic that could predict whether a model has a concentrated gate structure without running the full ablation sweep? For instance, the entropy of the ablation-effect distribution across heads, or the rank of the attention output matrix at the generation position.

6 comments
AuthorsOpenReview ↗

Response to Reviewer Tzmk (1/3)

Thank you for a precise review. We ran the attention experiment you proposed in Q1 and a second test that separates routing from a generic bottleneck. Both support routing, and one result narrows our claim. We answer W1 in four parts.

Q1. Does the gate attend more to the instruction tokens on harmful inputs than on benign inputs? A difference would support routing, and no difference would favour a bottleneck.

Yes, the gate's attention is harm-selective. The extra attention goes to the template tokens that follow the instruction. We measured the gate's attention from the last prompt token on harmful prompts (AdvBench 0–199) and on benign prompts (Alpaca 0–199), with the prefill for Llama-3.1. The template tokens are the chat-template tokens between the user's message and the reply. The gate places far more attention on them when the prompt is harmful.

Share of the gate's attention on one template tokenHarmfulBenign
SmolLM2 L21H23, template token istant0.610.28
Llama-3.1 L12H6, \n\n0.140.02
Llama-3.1 L12H6, end-of-turn token0.080.01

The harm signal can already be read out almost perfectly at these tokens, against Alpaca and against XSTest, so the gate picks it up after the model has re-encoded it there. This corrects the submission, which said that the gate reads the instruction tokens.

An attention pattern is correlational, so we also tested whether this attention is needed. We left the gate's output in place and blocked its attention to one group of tokens. Blocking its attention to the template tokens breaks refusal in both models, while in SmolLM2 blocking its attention to the instruction tokens has no significant effect.

ΔASR when the gate's attention is blocked (AdvBench 0–199)SmolLM2 (L21H23)Llama-3.1, "Sure" (L12H6)
to the template tokens+0.175 (p = 1e-8)+0.085 (p = 0.014)
to the template tokens, also during generationnot run+0.115 (p = 1e-3)
to the instruction tokens+0.015 (n.s.)not run

W1, alternative reading. The gate may sit at a bottleneck and perform a generic operation, without transporting the detection signal.

The content of the gate's output decides whether the model refuses. A generic bottleneck should keep working with any typical output. We replaced only the gate's output with its output on another prompt. Its output from a benign prompt breaks refusal, while its output from a different harmful prompt works as well as its own. The contrast is significant in SmolLM2 and in Llama-3.1 without the prefill. The table and the prefill case are in General response (1/2), because all three reviewers asked for this test.

In the revision: we will add both tests as a new §3.4 in the main text, with the full tables in a new App. J. The remaining two parts of W1 follow in Tzmk 2/3.

AuthorsOpenReview ↗

Response to Reviewer Tzmk (2/3)

W1, features and weights. A difference of means always finds a direction, so stability does not show that the model uses it. The paper checks neither the gate's QK against the harm direction nor its OV against the refusal direction.

We now test each step causally, and the OV result narrows our claim. For detection, the signal that the gate reads is causal. The harm-detecting neuron of Kazemi et al., 2026 [1] sits at the token the gate reads, and pinning it brings the gate's attention on harmful prompts down close to its benign level (NjoS 4/4, W4). The gate does not compute this signal itself. We fit a harm direction on separate prompts (AdvBench 400–519 against Alpaca 1000–1119) and measure the harm signal as how well this fixed direction separates held-out harmful and benign prompts (Cohen's d). After we ablate the gate, the harm signal is as strong as before, both at the last instruction token and at the template token the gate reads. A head that computed the signal would lower it, as some early heads do. For action, we rely on the established causal role of the refusal direction (Arditi et al., 2024 [2]).

For the gate's weights, the QK evidence is the attention pattern in Tzmk 1/3, where the gate selects the template tokens more strongly on harmful prompts. The OV evidence is negative for the two gates. In the gate's group, only L10H27 maps the harm direction onto the refusal direction, and inserting the gate's output from a harmful prompt into benign prompts does not cause refusal. The gate therefore does not write the refusal itself. The revision will narrow the claim accordingly: the gate relays a harm-specific signal that later layers need in order to refuse.

TestResult
Harm signal after ablating the gate, relative to the intact model (Cohen's d on AdvBench 200–399 against Alpaca 0–199, read at the last layer; L21H23, L12H6 and L10H27)0.99–1.01
The same measure after ablating single layer-0 heads in Qwen-2.5-7B, for contrast0.11–0.60
Does the head's OV circuit map the harm direction onto the refusal direction?L10H27: yes (99th percentile). L12H6 and L21H23: no
Gate output from a harmful prompt inserted into benign prompts (Alpaca)no refusal induced

W1, concrete cases. The framework is never applied to a concrete scenario: normal refusal, a successful attack, or over-refusal.

We applied it to four cases. The settings and numbers for the two attacks are in NjoS 3/4 (W3).

CaseWhat happens at the gate
Normal refusalIt attends more to the template tokens on harmful prompts, and blocking that attention breaks refusal
Adversarial suffixIts attention to those tokens drops (0.71 → 0.31), and restoring its output undoes 8 of 11 jailbreaks
Harmful fine-tuningUnchanged; the attack breaks refusal later in the network
Over-refusalRare in base models (JBB ≤ 0.09); after training it comes from the adapter (Tzmk 3/3, W2c)

W2a. On Llama-3.1 the gate is L12H6 with the prefill and L10H27 without. Is it a stable feature of the model or an artifact of the protocol?

The stable unit is a small group of heads in layers 10–13, and the prefill changes which of them has the largest effect. The full head screen places Llama-3.1's necessary heads in this group, all ranked low by attribution. With the "Sure" prefill on AdvBench 0–199, L12H6 has the largest effect. Without the prefill on HarmBench 0–199, L10H27 has the largest effect, and L12H6 keeps a smaller but significant effect (+0.040, p = 0.02, exact paired test). The prefill therefore shifts the load within the group, and the group itself stays the same.

References

[1] H. Kazemi, A. Chegini, M. Safi. A Single Neuron Is Sufficient to Bypass Safety Alignment in Large Language Models. arXiv:2605.08513, 2026.

[2] A. Arditi et al. Refusal in Language Models Is Mediated by a Single Direction. NeurIPS, 2024.

AuthorsOpenReview ↗

Response to Reviewer Tzmk (3/3)

W2b + Q4. The method fails on 3 of 7 configurations, and nothing tells a practitioner in advance whether a model has a concentrated gate. Is there a lightweight diagnostic, such as the entropy of the ablation effects?

We do not yet have a lightweight diagnostic. We report one quantity from the head screen that separates the outcomes we have, and one pre-registered test that it passed. We first tried your entropy suggestion and related concentration statistics (top-1 share, Gini, number of significant heads). On the keyword-judged screens, none of them separated the models where gate training succeeds from those where it fails. One quantity does separate them once we use the ensemble judge and held-out prompts: the largest ΔASR of any single routing head. By a routing head we mean a head whose ablation leaves at least 90% of the harm signal intact at the template token the gate reads (measured as in Tzmk 2/3), so it passes the signal on without computing it. Gate training succeeds where one routing head has a large effect, and it fails where the effect is spread thinly over many heads. The Qwen-2.5 models are out for a different reason: neither has a routing head to measure, because in both the non-redundant heads sit in layer 0 and their ablation removes the harm signal (General response 2/2, correction 3).

Llama-3.1SmolLM2Llama-2-7b-chat
Largest ΔASR of a single routing head+0.200+0.180+0.085
Gate training succeeds?yesyesno

Because this quantity was identified after the fact, we tested it on a second gate-training run on Llama-2, aimed at its strongest heads. We recorded the predictions of both rules before seeing the result.

Pre-registered test: Llama-2 (held out)PredictionOutcome
Candidate rule (ensemble judge)failurefailure (ASR 0.44 → 0.48)
Earlier rule (keyword screen)successwrong

This is an observation on three models, and the pre-registered test is a second training run on Llama-2, which is one of the three. We will therefore present it as a candidate rule that still needs a test on a new model. Gemma-2-9B-it, with a largest effect of +0.105 (General response 2/2), lies between the two groups, so the candidate rule makes no prediction for it yet. In the revision: we will add a paragraph to §4.2 and replace the old head-level diagnostic in App. C with this candidate rule and the pre-registered test.

W2c / Q2 / Q3. The discriminating direction d may be noise; please run a permutation test (Q3). Does d align with the base model's harm direction (Q2)? Does the self-gate's 14× trade ratio hold on prompts not used to calibrate it?

d is a real direction, it is unrelated to the model's harm direction, and the self-gate holds on held-out prompts. While running your permutation test, we found two errors in the submission. It read the adapter's output at the wrong token under left padding, and it calibrated the self-gate on its own test prompts. The numbers below are recomputed with both fixed. d and the sigmoid are fitted on AdvBench 400–463 and JBB 0–49, and everything else is measured on held-out prompts.

Llama-3.1SmolLM2
‖d‖ vs. 95th pct of 1,000 label shuffles (Q3)2.21 vs. 0.50, p = 0.001350 vs. 86, p = 0.001
cos(d, base harm direction) vs. random 95th pct (Q2)0.036 vs. 0.0320.005 vs. 0.046
Separation with prefill status reversed (AUROC)0.70n/a (no prefill)
JBB 50–99 refusal: untrained / trained / + self-gate0.10 / 0.64 / 0.120.02 / 0.56 / 0.06
Trade ratio in the submission14×1.3×
Corrected trade ratio, 95% bootstrap CI17.3× [8.7, 58.0]6.7× [4.0, 12.9]

For Q3, d is far longer than any direction obtained by shuffling the labels. For Q2, d aligns with the base model's harm direction only at chance level, so the adapter learned a separate discriminator. For the self-gate, held-out refusal on borderline benign prompts returns close to the untrained level, and the corrected trade ratio is higher than the submitted one for both models. On Llama-3.1, part of the separation comes from the prefill, because harmful-side evaluation uses the prefill and benign-side evaluation does not. SmolLM2 uses no prefill, so its 6.7× is the cleaner estimate.

In the revision: we will rewrite §4.3 and App. B with these numbers.

Thank you again; this review shaped much of the new work. If these results resolve your soundness concerns, we would be grateful if you would consider raising your score.

Reviewer_NjoSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies refusal meachinsm in LLMs, and proposes a decomposition of it into three stages with corresponding attention heads: harm detection, routing through a small set of non-redundant attention heads (the “gate”), and refusal action. It argues that safety fragility comes from the routing heads, and that what makes it fragile in some weight directions. Paper shows that single-head ablation identifies "gate" heads that direct feature attribution misranks, and that perturbing the "gate"’s weights breaks safety while perturbing the top-attribution head does not. Finally, it shows that noise-injected fine-tuning on the "gate" removes the model’s dependence on it at a cost in over-refusal, while the same training on the top-attribution head has no effect.

Critical strengths
  • The paper presents an interesting idea and analysis on refusal mechanism and its brittleness, attempts to reconcile ideas from the prior literature (single refusal direction, sparse safety heads, and the narrow safety basin).

  • The paper provides interesting insights into the mechanism of refusals, namely detect -> gate -> refuse. From my experience working in a similar direction with undersatanding prompt injection mechanisms, I can confirm similar observations regarding the existence of heads that facilitate exclusively detection, and others facilitating exclusively "action".

  • Overall the conclusions seem meaningful, and I am positive about the idea and direction of the work.

Critical weaknesses
  • The paper is hard to read, and it reads like AI slop. Many parts of the paper (e.g., Figure 1, where captions leave their boxes, or captions overlapping with the parts of the plots) look very raw and, in my opinion, AI generated. Certain phrasings (e.g., regarding metrics) are never introduced, explained, defined, or explicitly stated how they were calculated. For example, "binomial floor" appears only in the caption of Figure 1(c); the GSM8K evaluation set size it depends on is never reported, and to be honest I had to ask an LLM to infer possible meanings of this. "Cosine vs ASR" (Figure 1(d)) is never explained in the text and only appears as "mean cosine to baseline" in Section 2.2, from which I can only vaguely infer what it refers to. I would appreciate it if the authors defined all the metrics they use in a dedicated section. In particular, I would appreciate an explainer on basin anisotropy, which is reported in Table 1 and Figure 3(b) but never formally defined (I assume it is the ASR difference between gate-direction and random-direction perturbation at a given σ, but this is not stated). Despite me being positive of the paper idea and its finding, I dont believe that the mnuscript should be accepted in its current state and requires substantial HUMAN revision.

  • Small sample size. Sample sizes in this paper are 30–60 prompts for all training and diagnostic evaluations (Table 1, Table 4, Appendix C) and 50 prompts for the perturbation experiments of Section 2.2; only single-head ablation uses 200 prompts. This is surprising given the very small model sizes (1.7B, 7B and 8B in the main text), and since the experiments are inference only, without any adversarial attacks in input space. The authors report that each experiment takes 1–3 hours on a single H100, so larger evaluations seem feasible.

  • The proposed explanation is not tested under adversarial attack. While the paper motivates the fragility with fine-tuning attacks and jailbreaks, the authors only test it on direct harmful requests (AdvBench, HarmBench) under random weight perturbation and head ablation, with a fixed "Sure" prefill. There is no real adversarial pressuer and attacks where refusals are actually subverted (e.g., GCG, random search), and they don't attempt to analyse attention patterns there. This is quite surpsising since matters the claim is that the gate routes the detection signal to the refusal position, which should imply that this routing is what jailbreaking attacks break. Additionally authors themselves acknowledge (Section 3.3) that output-zeroing ablation does not show this. Checking whether successful jailbreaks or harmful fine-tuning disrupt the gate heads' attention from the decision position to the instruction tokens would be a direct test of the routing hypothesis.

  • Some missing related work. The paper does not discuss neuron-level work on safety localization. Most notably, [1] shows that pinning a single MLP neuron is sufficient to bypass refusal across seven models (including Llama-3.1-8B, one of the models studied here), and that the same neuron acts as a harmfulness detector at the post-instruction position. This raises the question of whether the gate heads carry the routing themselves or simply supply inputs to such MLP units, which the attention-only analysis cannot distinguish. I would like the masucript to reconcile these observations.

References

[1] H. Kazemi, A. Chegini, M. Safi. A Single Neuron Is Sufficient to Bypass Safety Alignment in Large Language Models. arXiv:2605.08513, 2026.

Questions for authors

See weaknesses.

8 comments
AuthorsOpenReview ↗

Response to Reviewer NjoS (1/4)

Thank you for a careful review. It was encouraging that your own work on prompt injection found the same split between detection-only and action-only heads. We address your concerns below.

W1. The paper is hard to read and looks unedited in places, for example Figure 1's layout. Several metrics are used without definition (binomial floor, cosine vs ASR, mean cosine to baseline, basin anisotropy); please define them in one place.

We agree, and we took this concern most seriously. The idea is simple, but supporting it takes a chain of analyses: §2 shows that detection and action are stable, §3 finds the routing heads, and §4 tests the account. Each step relies on the one before, and the submission never told the reader where the argument was heading. It also used several quantities before defining them. Figure 1's panels (a) and (b) were made separately and combined late, which broke the layout, and we should have caught it.

The revision will add a roadmap at the end of §1 and will open each section with the result it takes from the previous one. We will rebuild Figure 1 with every label inside its box and clear of the plots, and we will revise the prose of the main text (§1–§5) ourselves, passage by passage. All metrics will be defined in one paragraph in §2.1, which every caption will refer to. If particular passages read badly to you, we would be glad to know which, and we will revise those too.

The metrics paragraph we will add to §2.1:

  • ASR: as defined in the Setting paragraph of General response (1/2). The revision will report the ensemble judge throughout.
  • Binomial floor (Fig. 1c): a rate measured on n items varies between runs even without perturbation, with standard deviation (SD) √(p(1−p)/n). Variation across perturbation seeds above this floor is caused by the perturbation. The revision will measure ASR on 200 AdvBench prompts (the submission used 50) and GSM8K accuracy on 200 problems. The values are in NjoS 2/4.
  • Cosine to baseline (Fig. 1d): for each perturbation seed we re-extract the detection direction (difference of means at the last instruction token, layers 12/14/16) and the refusal direction (post-instruction position, layers 20/22) from 128 harmful and 10 harmless prompts. The submission said 128 pairs, which was wrong. We then take the cosine with the same direction in the unperturbed model, averaged over seeds.
  • Cosine vs ASR (Fig. 1d): the Pearson correlation, across the 30 seeds, between a seed's cosine and its ASR.
  • Δ_abl (Eq. 3): ASR with one head's output zeroed at every position, minus the intact model's ASR, on the same prompts.
  • Basin anisotropy (Table 1, Fig. 3b): as you inferred. At noise scale σ we add Gaussian noise to the gate's rows of W_Q and columns of W_O, repeat with the same seeds on random heads from the same layers, and report the mean paired difference ASR(gate) − ASR(random head), with a 95% interval over 15 seeds. A positive value means the model is more fragile along the gate than along an arbitrary head.
  • Trade ratio (self-gate): the drop in benign refusal divided by the rise in ASR when the self-gate is switched on.
AuthorsOpenReview ↗

Response to Reviewer NjoS (2/4)

W2. Most training and diagnostic evaluations use 30–60 prompts, and the §2.2 sweep uses 50, although larger evaluations are feasible.

Agreed. All new results use 200 prompts with exact paired tests (the suffix attack uses 100), and the conclusions hold. The gate itself is now selected by a screen of every head and confirmed on held-out prompts (dcZc 1/3 and 3/3). Below are the re-runs of the submission's Tables 1 and 2, the basin measurement and the §2.2 sweep.

Tables 1 and 2. The submission's Table 1 was scored with the keyword judge, although its caption said ensemble, on 30–60 prompts that overlapped the training prompts. We re-ran it on 200 held-out prompts with the ensemble judge. Gate training still removes the model's dependence on the gate, while the same training aimed at the top-attribution head (decoy training) leaves it in place.

Table 1, held outASR before → afterGate's ΔASR before → afterJBB refusal before → afterWith self-gate: ASR / JBB refusal
SmolLM2, gate training0.355 → 0.015+0.180 → +0.0100.02 → 0.560.085 / 0.06
Llama-3.1, gate training0.485 → 0.015+0.225 → +0.0150.09 → 0.630.050 / 0.12
Llama-3.1, decoy training0.485 → 0.305+0.225 → +0.1650.09 → 0.380.335 / 0.10

Gate training raises benign refusal, and we treat this as a real cost. The self-gate scales the adapter down on inputs that look benign. It brings JBB refusal back close to the untrained level while ASR stays far below the untrained model's. The self-gate values use JBB 50–99, the half not used to fit it (Tzmk 3/3).

The re-run of Table 2 on Llama-3.1 confirms that the noise injection is what removes the dependence. Without the KL term, the gate's ΔASR after training is +0.005. Without the noise, it stays at +0.205.

Basin anisotropy. We re-measured it with the paper's probe (σ = 0.08, 15 seeds) on held-out AdvBench 200–399 with the ensemble judge. The narrow direction along the gate reproduces, at a lower value than the submission's keyword-judged, in-sample estimate. Gate training removes it, and decoy training leaves it.

Anisotropy along L12H6, Llama-3.1Mean [95% CI]
Untrained, submission (keyword, in-sample)+0.278 [+0.154, +0.402]
Untrained, re-run+0.171 [+0.109, +0.236]
After gate training+0.024 [−0.043, +0.096]
After decoy training+0.160 [+0.072, +0.261]

One concern with this result is a floor effect. After gate training the model's ASR is close to zero, so weak noise might fail to raise ASR for any head, and the gate would then look no different from random heads for a trivial reason. We checked this in two ways. First, we increased the noise to σ = 0.12 (30 seeds), where noise raises the trained model's ASR to about 0.3. Noise on the gate still does no more harm than noise on random heads (−0.016 [−0.108, +0.071]). Second, we applied the same probe to the model trained without noise injection (the no-noise arm of Table 2), which still depends on the gate. There the gate remains a narrow direction (+0.229 [+0.136, +0.321]). The narrow direction therefore disappears when training removes the dependence on the gate, and it stays when the dependence stays.

§2.2. We re-ran the SmolLM2 perturbation sweep on AdvBench 0–199 with the same 30 seeds, scored with the ensemble judge. Its first 50 prompts reproduce the submitted values exactly. Safety varies across seeds well above its sampling floor, capability stays within its floor, and drift in neither direction tracks ASR (|r| ≤ 0.32; with 30 seeds, |r| ≥ 0.35 would have been detected).

§2.2, SmolLM2, σ = 0.005, 30 seedsSubmission (50 prompts, keyword)Revision (200 prompts, ensemble)
ASR: standard deviation (SD) across seeds vs. binomial floor0.113 vs. 0.0700.090 vs. 0.035
GSM8K (200 problems): SD vs. floor0.030 vs. 0.0340.030 vs. 0.034
Direction drift vs. ASR (Pearson r, both directions)|r| ≤ 0.28|r| ≤ 0.32
Mean cosine to baseline (both directions)> 0.96> 0.96

In the revision: we will replace Tables 1–2 in §4.2 and update §2.2 and Fig. 1.

AuthorsOpenReview ↗

Response to Reviewer NjoS (3/4)

W3. The account is tested only on direct requests under weight perturbation and ablation, with a fixed prefill. Do real jailbreaks or harmful fine-tuning disrupt the gate's attention from the decision position?

A jailbreak suffix disrupts the gate, and harmful fine-tuning breaks refusal further downstream. We ran both attacks on Llama-3.1-8B without the prefill and scored them on held-out prompts. The suffix is found by random search (Andriushchenko et al., 2024 [1]) for each of AdvBench 200–299. Fine-tuning uses LoRA on about 30 of the model's own compliant answers to AdvBench 0–199 and is evaluated on AdvBench 200–399. For each attack we measured the share of the gate's attention that falls on the template tokens after the instruction, which are the chat-template tokens between the user's message and the reply. We then restored only the gate's clean output to see whether refusal returns.

ASR before → afterGate's attention to the template tokens (jailbroken prompts)Restoring only the gate's clean output
Adversarial suffix (100 prompts)0.030 → 0.1500.71 → 0.31refusal returns on 8 of the 11 jailbreaks the ensemble confirms
Harmful fine-tuning (200 prompts)0.045 → 0.425unchangedno effect

The suffix pulls the gate's attention away from the template tokens. Restoring the gate's clean output, with the suffix still present, brings refusal back on 8 of the 11 jailbreaks, while restoring the top-attribution head in the same way brings it back on 1. The gate alone also recovers more than every random set of six same-layer heads we tried (2–7), and the six-head group in layers 10–13 recovers 10. With 11 cases, we treat this as indicative. Fine-tuning leaves the harm signal and the gate intact, in line with Zhao et al., 2025 [2], so it breaks refusal later in the network. The revision will state that gate training leaves the model open to fine-tuning attacks.

In the revision: we will add both attacks as a new App. L, and the fine-tuning result as a limitation in §5 and App. H.

References

[1] M. Andriushchenko, F. Croce, N. Flammarion. Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks. arXiv:2404.02151, 2024.

[2] J. Zhao et al. LLMs Encode Harmfulness and Refusal Separately. arXiv:2507.11878, 2025.

Reviewer_dcZcOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focus on encoding at refusal suppression as a re-routing problem, proposing a decomposition of the refusal behaviour into three stages (detection, routing and action), and showing that the routing constitutes the fragile component. In particular, the paper claims that a small set of attention heads (i.e., the gate) carries the harm signal from the instruction (i.e., where the model detects harm) to the position where it decides to refuse. Experiments are performed on one small and two medium models showing that the causally relevant heads are not those ranked highest by attribution and that noise-injected fine-tuning on them removes this dependence at the cost of increased over-refusal.

Critical strengths
  • The paper’s main idea is interesting and relevant, as it decomposes refusal behaviour into three components detection, routing, and action), identifying routing as the weak one.

  • The work proposes an analysis based on a (known) noise-injected fine-tuning procedure, in which noise is added to the identified heads (the gate) while the surrounding layers are fine-tuned. This analysis shows that after this fine-tuning, removing the gate no longer increases harmful answers in three of the six models tested.

  • The paper shows that direct feature attribution is not sufficient to identify the heads that have the biggest impact on the refusal. Indeed, in all three models tested, the head ranked first (from the direct feature attribution) by this method can be removed with no measurable effect on refusal. While, in Llama-3.1, the head whose removal most increases the number of harmful answers (with a +0.18 in attack success rate) is not even among the top 50 heads of the ranking. This highlights how selecting safety-relevant heads by attribution alone may not be sufficient.

Critical weaknesses
  • Although the work proposes the decomposition into three components, the link between the gate and the routing component is not clear and is not directly supported by experimental results, since experiments only show that there are a few heads that are more relevant than others for refusal, as the authors themselves acknowledge (Sec. 3.3).

  • To identify the gate, they compute Δabl\Delta_{abl} on a candidate pool made of the ten top-attribution heads and an unspecified number of randomly drawn heads. However, neither the number of random heads in the pool nor the size of the control sets is justified, so I would expect a more rigorous justification of the chosen sizes.

  • [Minor] The paper evaluates the effectiveness of the direct feature attributions but do not mention the original papers that proposed it (e.g., [1])

[1] Kissane, Connor, et al. "Interpreting attention layer outputs with sparse autoencoders." arXiv preprint arXiv:2406.17759 (2024).

Questions for authors

Q1. How do you think the redistribution of the load of the removed gate would impact on a new research of gate after the fine tuning? Will you expect that delta ablation ranking would select the same gate previously identified before the training?

Q2. Is it possible to provide more insight into how the gate heads carry the signal from the instruction to the point where refusal is decided? For example, ablating a head only at the final position, versus only at the instruction tokens, would indicate where it acts.

Q3. Does the gate’s effect hold on prompts not used to select it? The head is currently selected and evaluated on the same 200 AdvBench prompts, and Sec. 3.3 notes that this biases its effect upward. An evaluation on held-out prompts would clarify its size.

Q4. Is it possible to report the refusal rate on JBB before training, for each model? Table 1 gives the post-training values (e.g., 66% on Llama), and the pre-training values would show how much refusal on borderline benign requests increased.

5 comments
AuthorsOpenReview ↗

Response to Reviewer dcZc (1/3)

Thank you for a clear review with specific, answerable questions. Q2 proposed exactly the experiment we needed, and thank you for pointing out the missing citation.

W1. The experiments show that a few heads matter for refusal, but the link between the gate and routing is not directly tested (as §3.3 acknowledges).

We tested the link directly. If the gate routes, the content it carries must matter. We replaced only the gate's output with its output on another prompt. An output from a benign prompt breaks refusal, while an output from another harmful prompt works as well as the gate's own. The contrast holds in SmolLM2 and in Llama-3.1 without the prefill, and the table is in General response (1/2). The gate also acts only after the instruction (Q2, dcZc 3/3). A head that merely mattered, carrying no harm-specific content, would show neither pattern. In the revision: we will add these tests as a new §3.4.

W2. The gate is chosen from a pool with an unspecified number of random heads, and neither the pool nor the control sizes are justified.

We replaced the pool with a screen of every head. We ablated each head on the 200 selection prompts (keyword judge, as in the paper), then measured the selected gate and random control heads on held-out AdvBench 200–399, also with the keyword judge, so the columns are comparable. The gate comes out at or near the top, and random heads have small effects. The top-attribution heads rank 616th (L30H20, Llama-3.1) and 184th (L23H9, SmolLM2) in the same screen. The gate's effect under the ensemble judge is in Q3 (dcZc 3/3).

Full head screenGate: rankGate: held-out ΔASR (keyword)Random heads: held-out ΔASR (keyword)
Llama-3.1 (1,024 heads)3rd (L12H6)+0.170≤ +0.045
SmolLM2 (768 heads)1st (L21H23)+0.315≤ +0.045

One correction relevant to your summary: the submission placed L12H6 outside the attribution top 50. It ranks 49th, still far below the heads that attribution puts first.

In the revision: we will replace the candidate pool in §3.3 and Fig. 2 with the full screen, correct the rank, and give details in a new App. K.

Minor. Kissane et al. (2024), who introduced direct feature attribution for attention outputs, is not cited.

Thank you; we will add it. The revision will cite Kissane et al. (2024) [1] in §3.3 and App. A.

References

[1] C. Kissane et al. Interpreting Attention Layer Outputs with Sparse Autoencoders. arXiv:2406.17759, 2024.

AuthorsOpenReview ↗

Response to Reviewer dcZc (2/3)

Q1. After training, would a new Δ_abl search select the same gate?

No. The search selects a different head, L13H12. We re-ran the full screen on the trained Llama-3.1 with the same protocol (selection prompts, keyword judge) and checked the top heads on held-out prompts with the ensemble judge. The gate loses its effect entirely. L13H12 sits in the next layer, inside the trained layers. It was weak before training and becomes as necessary as the gate was. Training therefore removes the dependence on the targeted head, and routing re-forms through a neighbour that training strengthens.

This matters for the paper's account of fragility. The gate was a narrow direction of the safety basin. Weight noise on L13H12 does no more harm than noise on a random head of its layer, also at a larger noise level where ASR under noise is far from zero. So training removes the narrow direction along the gate, and no new narrow direction appears. L13H12 is therefore necessary but robust to weight noise. Being necessary does not by itself make a head a narrow direction. The revision will narrow the claim of the basin section (§3.4 in the submission, §3.5 in the revision): in the untrained models, the narrow direction coincides with the necessary head. We do not yet have an explanation for why L13H12 is robust to weight noise when the original gate was not. Scaling down either head's output weakens refusal gradually and in the same way, so output scale does not explain it.

Llama-3.1Before trainingAfter gate training
L12H6: rank in the full screen3rd85th
L12H6: held-out ΔASR (ensemble)+0.200+0.000 (n.s.)
L13H12: rank in the full screen64th1st
L13H12: held-out ΔASR (ensemble)+0.050 (n.s.)+0.195 (p = 4e-12)
Basin anisotropy, σ = 0.08along L12H6: +0.171along L12H6: +0.024; along L13H12: −0.021
Basin anisotropy, σ = 0.12, 30 seeds (ASR under noise 0.3–0.5)not runalong L12H6: −0.016; along L13H12: −0.044

The L12H6 values in this table (+0.200 before training, +0.000 after) come from the held-out check that follows the head screen. Table 1 in NjoS 2/4 reports the same quantity as +0.225 and +0.015. It was measured in a separate run, as part of the gate-training pipeline, on the same prompts. The two runs differ by at most 0.025.

Preventing the relocation. A likely cause of the relocation is that noise reaches only the gate during training, so the adapter learns to route through a head that never receives noise. We therefore added head dropout to the same recipe: in each training step, 12 heads drawn at random from layers 9–16 are switched off. We chose this region after seeing the relocation, to cover the original group of necessary heads (layers 10–13) and the heads that took over (layers 9, 13 and 15). We fixed two dropout strengths before training, 12 and 32 heads per step, and report both. As a control, the same dropout with decoy training leaves the model dependent on the gate, so the spreading requires training against the gate (held out, ensemble judge):

Llama-3.1, after trainingASRGate's ΔASRLargest single-head ΔASR
Gate training0.015+0.000+0.195 (L13H12)
Gate training + dropout, 12 heads/step0.005−0.005+0.015
Gate training + dropout, 32 heads/step0.215+0.025+0.145 (L13H12)
Decoy training + dropout, 12 heads/step0.275+0.145+0.175 (L11H3)

The 12-head dropout model also over-refuses less than the original recipe: its JBB refusal is 0.42 against 0.64. With the self-gate, its JBB refusal returns to the untrained level (0.10) while ASR stays at 0.010. For comparison, decoy training with the same dropout ends at ASR 0.260.

One concern is that a model which refuses very strongly could hide a necessary head, because removing a single head would no longer be enough to change its answers. The gate-trained model without dropout refuses as strongly (ASR 0.015), and the screen still finds its +0.195 head, L13H12. The screen can therefore detect a necessary head at this level of refusal, and in the 12-head dropout model it finds none above +0.015.

This is a single run on one model. In SmolLM2, ASR after training is too low for the screen to tell whether another head took over.

In the revision: we will add the re-screening, the basin measurement and the head-dropout result to §4.2, with details in App. I, and describe the original recipe as removing dependence on the gate and the narrow direction along it.

AuthorsOpenReview ↗

Response to Reviewer dcZc (3/3)

Q2. Where does the gate act? Ablating it only at the instruction tokens or only at the final position would show this.

The gate acts after the instruction. We split each prompt into two parts: the instruction tokens (the user's message) and the positions after the instruction, which start with the chat-template tokens. We then ran your test on the SmolLM2 gate (L21H23, selection prompts) in two forms.

The first form asks where the gate's output matters. We ablated (zeroed) the gate's output in one part only. Ablating it at the instruction tokens has no significant effect, while ablating it after the instruction gives most of the full effect.

The second form asks what the gate reads. We left its output in place and blocked its attention to one part, so the gate could not read those tokens. Blocking its attention to the instruction tokens again has no significant effect, while blocking its attention to the template tokens after the instruction breaks refusal.

For the Llama-3.1 gate (L12H6), ablating its output only after the instruction gives +0.225, as large as ablating it everywhere (+0.200). These results correct the submission, which said the gate reads the instruction tokens. The gate reads the template tokens after the instruction, where the harm signal has already arrived.

ΔASR, SmolLM2 gateAt instruction tokens onlyAfter the instruction onlyEverywhere
Ablate the gate's output+0.015 (n.s.)+0.200 (p = 3e-7)+0.235
Block the gate's attention to these tokens+0.015 (n.s.)+0.175 (p = 1e-8)not run

In the revision: we will add this test to the new §3.4, with the full table in App. J.

Q3. The gate is selected and evaluated on the same 200 prompts. Does its effect hold on prompts not used to select it?

Yes. For both models the effect stays large and significant on held-out prompts. We kept each gate fixed and ablated it on AdvBench 200–399 (same benchmark, unseen prompts) and on HarmBench 0–199 (a different benchmark). On unseen AdvBench prompts the effect is close to the in-sample value. On HarmBench it is smaller, consistent with a modest selection effect plus the change of benchmark. The first column is the paper's own run, so the three keyword columns are directly comparable.

Gate's ΔASRSelection, AdvBench 0–199 (keyword)Held out: AdvBench 200–399 (keyword)Held out: HarmBench 0–199 (keyword)Held out: AdvBench 200–399 (ensemble)
Llama-3.1 L12H6+0.175+0.170+0.110+0.200
SmolLM2 L21H23+0.350+0.315+0.180+0.165

For SmolLM2 the keyword judge overstates the held-out effect. Without the gate, many of its responses stop refusing without becoming harmful, and the keyword judge counts those as attacks. Under the ensemble judge the effect remains large and significant: 40 prompts switch to unsafe and 7 the other way (p = 1e-6).

In the revision: §3.3 will report both the in-sample and the held-out effects and note the selection effect, with details in App. K.

Q4. Please report JBB refusal before training for each model.

Benign refusal was low before training and rises with the adapter. The self-gate brings it back close to the untrained level.

JBB refusalBefore training (all 100)After gate training (all 100)With self-gate (JBB 50–99)
SmolLM20.020.560.06
Llama-3.10.090.630.12

The self-gate scales the adapter down on inputs that look benign. It is fitted on JBB 0–49 and evaluated on JBB 50–99. Under it, ASR is 0.085 for SmolLM2 and 0.050 for Llama-3.1, against 0.355 and 0.485 for the untrained models. At the same level of benign refusal, decoy training on Llama-3.1 reaches an ASR of only 0.335, so the lower ASR after gate training does not come from refusing more.

In the revision: we will add both columns to Table 1 (§4.2).

Thank you again. If these answers and the full head screen address your concerns about the gate–routing link and how the gate is selected, we would be grateful if you would consider raising your score. We are glad to run further checks during the discussion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised PDF uploaded

Dear Reviewers,

The revised PDF is now uploaded, with every change marked in blue. It contains all experiments and changes listed in the table of our General response (2/2), in the sections and appendices named there.

We would like to point you to the main new parts of the paper. The new Section 3.4 tests what the gate does, which all three of you asked about. Two new paragraphs in Section 4.2 report where the routing goes after training and when the method applies. Figures 1 to 3 are redrawn from the new data, in a cleaner layout.

One note on numbering, since our replies were written before the revision: the routing tests are the new Table 1, so the tables our replies call Table 1 and Table 2 are now Tables 2 and 3. For readers of the appendix, a short guide at its start maps each section of the main text to the appendices that support it.

We thank the reviewers again and are glad to answer further questions during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.