- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper asks which attention heads detect harmful prompts in aligned language models and whether safety fine-tuning built these heads or found them in the base model. The authors define a Safety Specificity Score (S³), which divides the effect of ablating a head on harmful prompts by its effect on benign prompts, and select the heads whose score is far above the typical score of each benchmark. They then score the same heads on three checkpoints of six model families: the base model, the instruct model, and an abliterated model whose refusal direction is removed from the weights. Each head is classified by whether fine-tuning added it (recruited) and whether abliteration removed it (erased). The paper makes three claims. First, most detection heads are recruited and erased. Second, zeroing these heads leaves refusal unchanged, so detection is separate from execution. Third, a seeded NMF of the head outputs groups prompts by writing style and source dataset rather than by harm category.
Critical strengths
Comparing the base, instruct and abliterated checkpoints on identical prompts is a clean design, because it follows the same head before and after each training stage. The statistical reporting is also careful, with Holm correction, equivalence tests and paired baselines, and the authors report several negative results in the appendix.
Critical weaknesses
-
The findings have little practical value. The paper explains where detection heads come from, but the deployment tests in Appendix G show that these heads are hard to use. As a guardrail on XSTest, the head probe reaches AUROC 0.73 on Qwen3-8B and 0.82 on Gemma-3-4B, while Llama-Guard-3-8B and WildGuard reach 0.975 and 0.990. Zeroing the top heads of each NMF component is no better than zeroing random heads, and the claim about residual-stream defenses in the Discussion is never tested.
-
The headline numbers are not robust. The abstract states that fine-tuning recruited 16 of the 19 heads that abliteration erases on Qwen3-8B, but the re-scoring in Appendix G gives only 3 such heads. Moreover, only 15 of the 1152 base heads have a positive S³, so almost every head selected in the instruct model counts as recruited by construction.
-
The erased set depends on how the model is abliterated. All main results use abliterated models from one community source (huihui-ai). In Table 9, the Heretic abliteration method erases only 5 heads of Gemma-3-4B, and the test of whether abliteration lowers S³ gives p = 1.0.
-
The separation of detection from execution may come from the selection rule. S³ divides by the effect on benign prompts, so it removes heads that respond to every input. Appendix G shows that zeroing the 19 heads removed by this division cuts the first-token refusal probability of Gemma-3-4B from 0.50 to 0.05. Therefore S³ may exclude the heads that execute refusal, and a null effect from zeroing the remaining heads is expected. The control in Section 4.3 only checks the cosine with the refusal direction and does not cover this case.
-
The novelty is limited. Zhao et al. (2025) already show that harmfulness and refusal are encoded separately, Frank (2026a) argues that detecting harm is easy while deciding to refuse is learned, and Prakash et al. (2024) show that fine-tuning enhances existing mechanisms. The new part is the three-checkpoint comparison at the head level.
-
The paper is hard to read. Almost every sentence contains several statistics, and terms such as abliterated, presence rule, scaffold position and register are used before they are defined. The results that weaken the main claims appear only in Appendix G, so the main text gives a stronger impression than the evidence supports. Some numbers also differ without explanation. For example, the base probe AUROC is 0.74 to 0.80 in the abstract and 0.78 to 0.79 in the introduction.
Questions for authors
Please see weakness.
3 comments
Response to Reviewer r3Rk (2/3): abliteration recipe and the selection rule
W3. Does the erased set depend on which abliteration recipe is used?
The erased counts do depend on the recipe. Table 9 labels them accordingly. The direction holds under five of six recipes on Gemma-3-4B and under all five on Qwen3-8B.
- Correct as printed: the Heretic recipe erases 5 Gemma-3-4B heads. The direction test asks whether abliteration lowers S³; for Heretic it gives p = 1.0, and under the other five recipes it is significant.
- Two results hold under all six recipes. Fine-tuning recruited the erased heads at rates of 89 to 100 %, and the both set's probe AUROC falls under every recipe (Heretic: 0.837, from 0.965).
- New in this discussion. Since submission, we scored Qwen3-8B under five recipes, two of them built with Heretic. Instruct sits above abliterated under all five (Holm p ≤ 0.033).
W4. Could the S³ selection rule itself be hiding the heads that execute refusal?
On Gemma-3-4B, the division does remove heads that carry refusal. It removes none on Qwen3-8B. Zeroing the whole selection moves refusal no more than random heads do.
- New in this discussion. In a new experiment we ran the test your point calls for: we zeroed the whole S³ selection and read first-token refusal probability on harmful prompts.
- On Gemma-3-4B it fell from 0.50 to 0.34, but 69 of 200 random sets of the same size cut it at least as much. Qwen3-8B shows the same: 0.40 to 0.38, with 65 of 200 random sets doing at least as much.
Changed in the revised PDF: Appendix G adds one sentence on this run.
Response to Reviewer r3Rk (3/3): novelty, readability and summary
W5. What is new beyond prior work separating harm from refusal?
We agree that the new part is the three-checkpoint comparison at the head level. That is the contribution. The prior designs cannot provide it.
- Zhao et al. (2025) separate harm from refusal as two directions on one instruct checkpoint. Frank (2026a) also works on one checkpoint, and Prakash et al. (2024) study entity tracking.
- None of these designs can say whether fine-tuning built a given head or which heads abliteration lowers. Pairing one head across three checkpoints is what lets the paper read recruited and erased on the same head.
W6. Is the paper hard to read, with terms used before they are defined?
We agree on all three points. The revised PDF defines the terms, labels the two probe contrasts, and points from the headline count to the re-score.
- The revised PDF now defines presence in the Figure 1 caption and register in the introduction.
- The introduction now labels the two probe contrasts: against benign Alpaca prompts, 0.78 to 0.79; against harm-matched XSTest prompts, 0.74 to 0.80.
- Section 4.2 now points from the 16-of-19 count to the Appendix G re-score.
Summary for Reviewer r3Rk
New in this discussion:
- Zeroing the whole S³ selection moves refusal no more than random heads do (W4).
- On Qwen3-8B, instruct sits above abliterated under five abliteration recipes (W3).
Changed in the revised PDF:
- Figure 1 caption: presence defined.
- Introduction: register defined; the two probe contrasts labeled.
- Section 4.2: a pointer from the 16-of-19 count to the Appendix G re-score.
- Discussion: the residual-stream prediction marked untested.
- Appendix G: one sentence on zeroing the whole selection.
Not changed, and why:
- The 16-of-19 count stays as the paper's count, because the 16 named heads replicate on the re-score (W2).
- The guardrail comparison stays in Appendix G as a limitation. The paper proposes no guardrail (W1).
We hope these answers address your concerns, and we would be grateful if you would reconsider your rating.
Response to Reviewer r3Rk (1/3): practical value and the 16-of-19 count
Thank you for the careful review and for crediting the three-checkpoint design and the statistical reporting.
This reply comes in three comments: practical value and the 16-of-19 count; the abliteration recipe and the selection rule; novelty, readability and a summary.
W1. Do the findings have practical value, given the weak guardrail numbers?
The head read-off is not a competitive guardrail. The paper proposes none. Its value is the provenance answer, which has one practical consequence for monitoring.
Changed in the revised PDF: the Discussion marks the residual-stream prediction as untested.
W2. Does the 16-of-19 count hold up when the re-score finds only 3 such heads?
The two counts come from different selections. The claim rests on the 16 named heads, which replicate on the re-score.
Changed in the revised PDF: Section 4.2 now points from the 16-of-19 count to the Appendix G re-score.