- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper asks why the safety alignment of instruction-tuned language models is fragile. A handful of fine-tuning examples or a targeted perturbation can break safety behavior, yet the underlying safety-relevant representations are robust. How do these two facts coexist? The authors decompose safety refusal into three components: detection, a linear feature at the instruction token position that separates harmful from benign inputs; action, a refusal direction in the residual stream at the generation position; and routing, attention heads that carry the detection signal to the action site. The core finding is that detection and action are stable under random perturbation, with cosine stability above 0.96, while routing depends on a handful of irreplaceable attention heads — sometimes just one — that form a fragile bottleneck.
The main methodological contribution is showing that standard direct feature attribution selects the wrong heads. The highest-scoring heads are highly redundant: removing them barely changes the attack success rate. The truly indispensable heads score modestly on attribution but are irreplaceable. This mismatch between attribution and causal necessity holds across SmolLM2-1.7B, Llama-3.1-8B, and Qwen-2.5-7B. The authors also introduce basin anisotropy to characterize the directional sensitivity of the gate: perturbing along the gate direction raises ASR substantially, while perturbing along random directions does not.
A training intervention using a LoRA adapter with self-distillation, a KL anchor, and noise injection into the gate heads successfully redistributes routing dependence, collapsing the original gate's causal effect and basin anisotropy. A decoy control targeting the top attribution head has no effect. The trained adapter causes over-refusal, which the paper analyzes as a symmetric routing failure: the adapter learns a discriminating direction that separates harmful from benign inputs, and also learns to write toward the refusal axis, but these two signals live in orthogonal subspaces, so detection cannot modulate action. A self-gating fix that projects the adapter output onto the discriminating direction and scales accordingly achieves a 14× improvement in the safety–utility trade ratio on Llama-3.1.
The framework fails on 3 of 7 tested configurations — Qwen-32B, Qwen-14B, and Llama-2-7b-chat — which the authors attribute to already-distributed routing. The appendices provide activation patching validation, a no-prefill replication, and analysis of safety-pretrained models.
Critical strengths
1. The mismatch between attribution and causal necessity is a valuable finding. Ranking attention heads by their projection onto a direction of interest and treating the top-ranked heads as the important ones is standard practice in interpretability. This paper gives clear evidence that doing so for safety-critical behavior selects the wrong heads. The heads it selects are highly redundant — removing them has little effect. The heads that actually matter rank lower on attribution but are irreplaceable. The mismatch is sizable: on SmolLM2, the top attribution head has , while the gate head produces over 15× the ablation effect. Results are consistent across all three model families. The decoy control in Section 4, which applies the same training intervention to the attribution-selected head and sees no effect, adds causal confirmation. Whether or not one accepts the "routing" narrative, this finding on its own is worth knowing for anyone doing safety-related interpretability work.
2. The experimental work is thorough, honestly reported, and highly reproducible. Seven model configurations are tested, and the three failures are presented upfront rather than buried. Perturbation experiments use 30 seeds, basin measurements use 15. The safety judge is an ensemble of keyword matching and WildGuard/Llama Guard, with reported. The prefill setup is clearly stated, and Appendix F provides a no-prefill replication. The reproducibility package includes a machine-readable manifest, a numerical consistency checker, and the ability to regenerate all figures from a single script — well above the norm for this venue. The component ablation in Table 2 is set up as a controlled experiment rather than a post hoc narrative, distinguishing noise injection as the only indispensable ingredient and showing that the KL anchor controls over-refusal but does not drive routing redistribution.
Critical weaknesses
1. The paper claims to have identified a "routing" mechanism, but the evidence stops at "certain heads are necessary" and does not show how these heads actually work. The paper's narrative is a three-component pipeline where detection travels through gate heads to action. Supporting this narrative requires two layers of evidence: first, that the detection and action features are really what the paper claims they are; second, that the gate head is actually performing signal transmission. But the evidence provided is only that the upstream and downstream features are robust, that removing the head breaks refusal, and that the gate direction has high basin anisotropy. This establishes that the gate head is necessary, not what it is doing.
Specifically, the paper extracts a harmfulness direction and a refusal direction using mean-difference and shows they are stable across seeds. But mean-difference will always find a separating direction between two groups, and stability shows the direction exists and is consistent — not that the model actually uses it. The paper also does not examine whether the gate head's attention pattern differs between harmful and benign inputs, does not analyze whether its query/key space aligns with the harmfulness direction, and does not check whether its OV projection aligns with the refusal direction.
More importantly, the paper does not use its framework to explain actual safety phenomena. What happens at the gate head when an attack succeeds? What is the head doing during normal refusal? What about during over-refusal? These comparisons are what would show that "routing" is an explanatory framework with real analytical power. The paper proposes a detection-routing-action decomposition but never applies it to analyze any concrete scenario. The framework stays at the level of naming rather than demonstrating analytical capability.
An alternative explanation is available. The head may sit at a bottleneck on the refusal computation path, performing some generic operation that happens to be critical. Remove it and refusal breaks, but it need not be "transporting" a detection signal. The paper's data is equally compatible with a pure bottleneck interpretation.
2. When the framework applies is unpredictable, and some of the finer claims lack statistical validation. The method fails on 3 of 7 configurations, and the paper provides no criterion that would let a practitioner know, before running the full experimental pipeline, whether a given model has a concentrated gate. The post hoc explanation that these models already have distributed routing has no predictive value. The gate's identity is also unstable across evaluation conditions: on Llama-3.1, the gate is L12H6 under prefill and shifts to L10H27 without prefill. The mismatch phenomenon replicates, but the fact that which head serves as the gate changes with the input distribution raises the question of whether the gate is a stable architectural feature or an artifact of the evaluation protocol.
The discriminating direction in Section 4.3 is especially concerning. This direction is the mean difference between adapter outputs on harmful and benign inputs, and the cosine similarity between the two classes is 0.83–0.98 — a small difference that at first glance looks like it could be noise. In high-dimensional space, a nonzero mean difference is expected even under the null hypothesis, and the paper provides no permutation test or bootstrap confidence interval to rule this out. What this direction represents semantically is also left entirely unanalyzed. What is its relationship to the base model's harmfulness direction? Did the adapter rediscover the model's native detection feature, or did it learn an entirely different discriminator? These questions are not addressed. The self-gate uses this direction for inference-time correction, with the sigmoid calibrated on 64+64 prompts and no held-out evaluation. The 14× trade ratio is suggestive but falls short of conclusive without basic cross-validation.
Questions for authors
-
(Clarification, may affect my assessment) Have you looked at the gate head's attention pattern on harmful vs. benign inputs? If the gate head attends noticeably more to instruction-position tokens on harmful inputs, that would be direct evidence for the routing interpretation. If the attention pattern does not differ, the bottleneck interpretation becomes more plausible. I think this is the single experiment that would most strengthen or weaken the central claim.
-
(Clarification, may affect my assessment) What is the cosine similarity between the discriminating direction and the base model's harmfulness direction? If they align, it would mean the adapter rediscovered the model's native detection feature in a disconnected subspace. If they are orthogonal, the adapter learned an independent discriminator. Either answer would help clarify how over-refusal actually arises.
-
(Clarification) Can you provide a permutation test for the discriminating direction? Shuffling harmful/benign labels across 1000 permutations and recomputing would establish whether the observed direction is statistically distinguishable from chance. Also, does the self-gate's trade ratio hold on a held-out prompt set not used for sigmoid calibration?
-
(Suggestion) Is there a lightweight diagnostic that could predict whether a model has a concentrated gate structure without running the full ablation sweep? For instance, the entropy of the ablation-effect distribution across heads, or the rank of the attention output matrix at the generation position.
6 comments
Response to Reviewer Tzmk (2/3)
W1, features and weights. A difference of means always finds a direction, so stability does not show that the model uses it. The paper checks neither the gate's QK against the harm direction nor its OV against the refusal direction.
We now test each step causally, and the OV result narrows our claim. For detection, the signal that the gate reads is causal. The harm-detecting neuron of Kazemi et al., 2026 [1] sits at the token the gate reads, and pinning it brings the gate's attention on harmful prompts down close to its benign level (NjoS 4/4, W4). The gate does not compute this signal itself. We fit a harm direction on separate prompts (AdvBench 400–519 against Alpaca 1000–1119) and measure the harm signal as how well this fixed direction separates held-out harmful and benign prompts (Cohen's d). After we ablate the gate, the harm signal is as strong as before, both at the last instruction token and at the template token the gate reads. A head that computed the signal would lower it, as some early heads do. For action, we rely on the established causal role of the refusal direction (Arditi et al., 2024 [2]).
For the gate's weights, the QK evidence is the attention pattern in Tzmk 1/3, where the gate selects the template tokens more strongly on harmful prompts. The OV evidence is negative for the two gates. In the gate's group, only L10H27 maps the harm direction onto the refusal direction, and inserting the gate's output from a harmful prompt into benign prompts does not cause refusal. The gate therefore does not write the refusal itself. The revision will narrow the claim accordingly: the gate relays a harm-specific signal that later layers need in order to refuse.
| Test | Result |
|---|---|
| Harm signal after ablating the gate, relative to the intact model (Cohen's d on AdvBench 200–399 against Alpaca 0–199, read at the last layer; L21H23, L12H6 and L10H27) | 0.99–1.01 |
| The same measure after ablating single layer-0 heads in Qwen-2.5-7B, for contrast | 0.11–0.60 |
| Does the head's OV circuit map the harm direction onto the refusal direction? | L10H27: yes (99th percentile). L12H6 and L21H23: no |
| Gate output from a harmful prompt inserted into benign prompts (Alpaca) | no refusal induced |
W1, concrete cases. The framework is never applied to a concrete scenario: normal refusal, a successful attack, or over-refusal.
We applied it to four cases. The settings and numbers for the two attacks are in NjoS 3/4 (W3).
| Case | What happens at the gate |
|---|---|
| Normal refusal | It attends more to the template tokens on harmful prompts, and blocking that attention breaks refusal |
| Adversarial suffix | Its attention to those tokens drops (0.71 → 0.31), and restoring its output undoes 8 of 11 jailbreaks |
| Harmful fine-tuning | Unchanged; the attack breaks refusal later in the network |
| Over-refusal | Rare in base models (JBB ≤ 0.09); after training it comes from the adapter (Tzmk 3/3, W2c) |
W2a. On Llama-3.1 the gate is L12H6 with the prefill and L10H27 without. Is it a stable feature of the model or an artifact of the protocol?
The stable unit is a small group of heads in layers 10–13, and the prefill changes which of them has the largest effect. The full head screen places Llama-3.1's necessary heads in this group, all ranked low by attribution. With the "Sure" prefill on AdvBench 0–199, L12H6 has the largest effect. Without the prefill on HarmBench 0–199, L10H27 has the largest effect, and L12H6 keeps a smaller but significant effect (+0.040, p = 0.02, exact paired test). The prefill therefore shifts the load within the group, and the group itself stays the same.
References
[1] H. Kazemi, A. Chegini, M. Safi. A Single Neuron Is Sufficient to Bypass Safety Alignment in Large Language Models. arXiv:2605.08513, 2026.
[2] A. Arditi et al. Refusal in Language Models Is Mediated by a Single Direction. NeurIPS, 2024.
Response to Reviewer Tzmk (3/3)
W2b + Q4. The method fails on 3 of 7 configurations, and nothing tells a practitioner in advance whether a model has a concentrated gate. Is there a lightweight diagnostic, such as the entropy of the ablation effects?
We do not yet have a lightweight diagnostic. We report one quantity from the head screen that separates the outcomes we have, and one pre-registered test that it passed. We first tried your entropy suggestion and related concentration statistics (top-1 share, Gini, number of significant heads). On the keyword-judged screens, none of them separated the models where gate training succeeds from those where it fails. One quantity does separate them once we use the ensemble judge and held-out prompts: the largest ΔASR of any single routing head. By a routing head we mean a head whose ablation leaves at least 90% of the harm signal intact at the template token the gate reads (measured as in Tzmk 2/3), so it passes the signal on without computing it. Gate training succeeds where one routing head has a large effect, and it fails where the effect is spread thinly over many heads. The Qwen-2.5 models are out for a different reason: neither has a routing head to measure, because in both the non-redundant heads sit in layer 0 and their ablation removes the harm signal (General response 2/2, correction 3).
| Llama-3.1 | SmolLM2 | Llama-2-7b-chat | |
|---|---|---|---|
| Largest ΔASR of a single routing head | +0.200 | +0.180 | +0.085 |
| Gate training succeeds? | yes | yes | no |
Because this quantity was identified after the fact, we tested it on a second gate-training run on Llama-2, aimed at its strongest heads. We recorded the predictions of both rules before seeing the result.
| Pre-registered test: Llama-2 (held out) | Prediction | Outcome |
|---|---|---|
| Candidate rule (ensemble judge) | failure | failure (ASR 0.44 → 0.48) |
| Earlier rule (keyword screen) | success | wrong |
This is an observation on three models, and the pre-registered test is a second training run on Llama-2, which is one of the three. We will therefore present it as a candidate rule that still needs a test on a new model. Gemma-2-9B-it, with a largest effect of +0.105 (General response 2/2), lies between the two groups, so the candidate rule makes no prediction for it yet. In the revision: we will add a paragraph to §4.2 and replace the old head-level diagnostic in App. C with this candidate rule and the pre-registered test.
W2c / Q2 / Q3. The discriminating direction d may be noise; please run a permutation test (Q3). Does d align with the base model's harm direction (Q2)? Does the self-gate's 14× trade ratio hold on prompts not used to calibrate it?
d is a real direction, it is unrelated to the model's harm direction, and the self-gate holds on held-out prompts. While running your permutation test, we found two errors in the submission. It read the adapter's output at the wrong token under left padding, and it calibrated the self-gate on its own test prompts. The numbers below are recomputed with both fixed. d and the sigmoid are fitted on AdvBench 400–463 and JBB 0–49, and everything else is measured on held-out prompts.
| Llama-3.1 | SmolLM2 | |
|---|---|---|
| ‖d‖ vs. 95th pct of 1,000 label shuffles (Q3) | 2.21 vs. 0.50, p = 0.001 | 350 vs. 86, p = 0.001 |
| cos(d, base harm direction) vs. random 95th pct (Q2) | 0.036 vs. 0.032 | 0.005 vs. 0.046 |
| Separation with prefill status reversed (AUROC) | 0.70 | n/a (no prefill) |
| JBB 50–99 refusal: untrained / trained / + self-gate | 0.10 / 0.64 / 0.12 | 0.02 / 0.56 / 0.06 |
| Trade ratio in the submission | 14× | 1.3× |
| Corrected trade ratio, 95% bootstrap CI | 17.3× [8.7, 58.0] | 6.7× [4.0, 12.9] |
For Q3, d is far longer than any direction obtained by shuffling the labels. For Q2, d aligns with the base model's harm direction only at chance level, so the adapter learned a separate discriminator. For the self-gate, held-out refusal on borderline benign prompts returns close to the untrained level, and the corrected trade ratio is higher than the submitted one for both models. On Llama-3.1, part of the separation comes from the prefill, because harmful-side evaluation uses the prefill and benign-side evaluation does not. SmolLM2 uses no prefill, so its 6.7× is the cleaner estimate.
In the revision: we will rewrite §4.3 and App. B with these numbers.
Thank you again; this review shaped much of the new work. If these results resolve your soundness concerns, we would be grateful if you would consider raising your score.
Response to Reviewer Tzmk (1/3)
Thank you for a precise review. We ran the attention experiment you proposed in Q1 and a second test that separates routing from a generic bottleneck. Both support routing, and one result narrows our claim. We answer W1 in four parts.
Yes, the gate's attention is harm-selective. The extra attention goes to the template tokens that follow the instruction. We measured the gate's attention from the last prompt token on harmful prompts (AdvBench 0–199) and on benign prompts (Alpaca 0–199), with the prefill for Llama-3.1. The template tokens are the chat-template tokens between the user's message and the reply. The gate places far more attention on them when the prompt is harmful.
istant\n\nThe harm signal can already be read out almost perfectly at these tokens, against Alpaca and against XSTest, so the gate picks it up after the model has re-encoded it there. This corrects the submission, which said that the gate reads the instruction tokens.
An attention pattern is correlational, so we also tested whether this attention is needed. We left the gate's output in place and blocked its attention to one group of tokens. Blocking its attention to the template tokens breaks refusal in both models, while in SmolLM2 blocking its attention to the instruction tokens has no significant effect.
The content of the gate's output decides whether the model refuses. A generic bottleneck should keep working with any typical output. We replaced only the gate's output with its output on another prompt. Its output from a benign prompt breaks refusal, while its output from a different harmful prompt works as well as its own. The contrast is significant in SmolLM2 and in Llama-3.1 without the prefill. The table and the prefill case are in General response (1/2), because all three reviewers asked for this test.
In the revision: we will add both tests as a new §3.4 in the main text, with the full tables in a new App. J. The remaining two parts of W1 follow in Tzmk 2/3.