acceptodds
Under review as a conference paper at ICLR 2027

Recruited and Erased: Head-Level Provenance of Safety Detection in Language Models

Abstract

Aligned language models refuse harmful requests, yet the dominant mechanistic account collapses harm detection and refusal execution onto one residual-stream direction. It cannot say which heads detect harm, or whether safety fine-tuning built or found them. We introduce a Safety Specificity Score () that selects heads whose output shifts more on harmful than benign prompts. To trace their origin, we score each selected head at three checkpoints, base, instruct, and abliterated, whose refusal direction is removed, across six checkpoint sets. Fine-tuning raises on the heads abliteration pushes back down. A Holm-corrected Wilcoxon places instruct above abliterated on Qwen3-8B and Gemma-3-4B, and the direction replicates on two of four further sets. On Qwen3-8B, fine-tuning recruited 16 of the 19 heads abliteration erases. Detection heads are recruited, not built: harm-matched, base checkpoints already reach 0.74 to 0.80 AUROC through the instruct-trained probe, below the 0.852 of an input-only classifier, so what the recruited heads read out is bounded by the prompt text. Zeroing them never reproduces abliteration's refusal collapse: detection is separable from execution, and head-level provenance measures a stage that direction-editing defenses act beside rather than measure.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, attention heads, safety fine-tuning, refusal, abliteration, harm detection, LLM safety, AI alignment, provenance, representation engineering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_r3RkOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper asks which attention heads detect harmful prompts in aligned language models and whether safety fine-tuning built these heads or found them in the base model. The authors define a Safety Specificity Score (S³), which divides the effect of ablating a head on harmful prompts by its effect on benign prompts, and select the heads whose score is far above the typical score of each benchmark. They then score the same heads on three checkpoints of six model families: the base model, the instruct model, and an abliterated model whose refusal direction is removed from the weights. Each head is classified by whether fine-tuning added it (recruited) and whether abliteration removed it (erased). The paper makes three claims. First, most detection heads are recruited and erased. Second, zeroing these heads leaves refusal unchanged, so detection is separate from execution. Third, a seeded NMF of the head outputs groups prompts by writing style and source dataset rather than by harm category.

Critical strengths

Comparing the base, instruct and abliterated checkpoints on identical prompts is a clean design, because it follows the same head before and after each training stage. The statistical reporting is also careful, with Holm correction, equivalence tests and paired baselines, and the authors report several negative results in the appendix.

Critical weaknesses
  1. The findings have little practical value. The paper explains where detection heads come from, but the deployment tests in Appendix G show that these heads are hard to use. As a guardrail on XSTest, the head probe reaches AUROC 0.73 on Qwen3-8B and 0.82 on Gemma-3-4B, while Llama-Guard-3-8B and WildGuard reach 0.975 and 0.990. Zeroing the top heads of each NMF component is no better than zeroing random heads, and the claim about residual-stream defenses in the Discussion is never tested.

  2. The headline numbers are not robust. The abstract states that fine-tuning recruited 16 of the 19 heads that abliteration erases on Qwen3-8B, but the re-scoring in Appendix G gives only 3 such heads. Moreover, only 15 of the 1152 base heads have a positive S³, so almost every head selected in the instruct model counts as recruited by construction.

  3. The erased set depends on how the model is abliterated. All main results use abliterated models from one community source (huihui-ai). In Table 9, the Heretic abliteration method erases only 5 heads of Gemma-3-4B, and the test of whether abliteration lowers S³ gives p = 1.0.

  4. The separation of detection from execution may come from the selection rule. S³ divides by the effect on benign prompts, so it removes heads that respond to every input. Appendix G shows that zeroing the 19 heads removed by this division cuts the first-token refusal probability of Gemma-3-4B from 0.50 to 0.05. Therefore S³ may exclude the heads that execute refusal, and a null effect from zeroing the remaining heads is expected. The control in Section 4.3 only checks the cosine with the refusal direction and does not cover this case.

  5. The novelty is limited. Zhao et al. (2025) already show that harmfulness and refusal are encoded separately, Frank (2026a) argues that detecting harm is easy while deciding to refuse is learned, and Prakash et al. (2024) show that fine-tuning enhances existing mechanisms. The new part is the three-checkpoint comparison at the head level.

  6. The paper is hard to read. Almost every sentence contains several statistics, and terms such as abliterated, presence rule, scaffold position and register are used before they are defined. The results that weaken the main claims appear only in Appendix G, so the main text gives a stronger impression than the evidence supports. Some numbers also differ without explanation. For example, the base probe AUROC is 0.74 to 0.80 in the abstract and 0.78 to 0.79 in the introduction.

Questions for authors

Please see weakness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer r3Rk (1/3): practical value and the 16-of-19 count

Thank you for the careful review and for crediting the three-checkpoint design and the statistical reporting.

This reply comes in three comments: practical value and the 16-of-19 count; the abliteration recipe and the selection rule; novelty, readability and a summary.

W1. Do the findings have practical value, given the weak guardrail numbers?

The head read-off is not a competitive guardrail. The paper proposes none. Its value is the provenance answer, which has one practical consequence for monitoring.

  • Appendix G reports the comparison you quote as a limitation.
  • The contribution is provenance. Fine-tuning raises the safety specificity of heads whose base outputs already separate harm.
  • This has one consequence for monitoring. On Qwen3-8B, abliteration removes refusal. Yet a probe on the 16 heads that fine-tuning recruited and abliteration erased still separates harmful from benign prompts: AUROC 0.875 before abliteration, 0.886 after. A test of refusal alone would miss that.
  • You are right that the residual-stream prediction is untested. The revised PDF now says so in the Discussion.
  • The component-targeted null is likewise our own negative result. We do not claim the decomposition localizes causal heads.

Changed in the revised PDF: the Discussion marks the residual-stream prediction as untested.

W2. Does the 16-of-19 count hold up when the re-score finds only 3 such heads?

The two counts come from different selections. The claim rests on the 16 named heads, which replicate on the re-score.

  • A both head is one that fine-tuning recruited and abliteration erased.
  • The 3 come from the re-score in Appendix G, a second scoring run of the same checkpoints on different hardware. The selection re-derived there has 15 heads, 3 of them both.
  • The 16 named heads replicate on that re-score. Almost all (15 of 16) score higher at instruct than at base. Thirteen of the 16 fall under abliteration (Wilcoxon p = 2.1e-4).
  • A threshold-free read agrees. On the re-score, the paper's top-K heads sit higher at instruct than abliterated at every K from 10 to 200 (largest p = 4.9e-3).
  • On 15 of 1152 base heads. The finding is that the base has almost no safety-specific heads, and "recruited" names exactly that. The paper claims no enrichment.

Changed in the revised PDF: Section 4.2 now points from the 16-of-19 count to the Appendix G re-score.

AuthorsOpenReview ↗

Response to Reviewer r3Rk (2/3): abliteration recipe and the selection rule

W3. Does the erased set depend on which abliteration recipe is used?

The erased counts do depend on the recipe. Table 9 labels them accordingly. The direction holds under five of six recipes on Gemma-3-4B and under all five on Qwen3-8B.

  • Correct as printed: the Heretic recipe erases 5 Gemma-3-4B heads. The direction test asks whether abliteration lowers S³; for Heretic it gives p = 1.0, and under the other five recipes it is significant.
  • Two results hold under all six recipes. Fine-tuning recruited the erased heads at rates of 89 to 100 %, and the both set's probe AUROC falls under every recipe (Heretic: 0.837, from 0.965).
  • New in this discussion. Since submission, we scored Qwen3-8B under five recipes, two of them built with Heretic. Instruct sits above abliterated under all five (Holm p ≤ 0.033).

W4. Could the S³ selection rule itself be hiding the heads that execute refusal?

On Gemma-3-4B, the division does remove heads that carry refusal. It removes none on Qwen3-8B. Zeroing the whole selection moves refusal no more than random heads do.

  • New in this discussion. In a new experiment we ran the test your point calls for: we zeroed the whole S³ selection and read first-token refusal probability on harmful prompts.
  • On Gemma-3-4B it fell from 0.50 to 0.34, but 69 of 200 random sets of the same size cut it at least as much. Qwen3-8B shows the same: 0.40 to 0.38, with 65 of 200 random sets doing at least as much.

Changed in the revised PDF: Appendix G adds one sentence on this run.

AuthorsOpenReview ↗

Response to Reviewer r3Rk (3/3): novelty, readability and summary

W5. What is new beyond prior work separating harm from refusal?

We agree that the new part is the three-checkpoint comparison at the head level. That is the contribution. The prior designs cannot provide it.

  • Zhao et al. (2025) separate harm from refusal as two directions on one instruct checkpoint. Frank (2026a) also works on one checkpoint, and Prakash et al. (2024) study entity tracking.
  • None of these designs can say whether fine-tuning built a given head or which heads abliteration lowers. Pairing one head across three checkpoints is what lets the paper read recruited and erased on the same head.

W6. Is the paper hard to read, with terms used before they are defined?

We agree on all three points. The revised PDF defines the terms, labels the two probe contrasts, and points from the headline count to the re-score.

  • The revised PDF now defines presence in the Figure 1 caption and register in the introduction.
  • The introduction now labels the two probe contrasts: against benign Alpaca prompts, 0.78 to 0.79; against harm-matched XSTest prompts, 0.74 to 0.80.
  • Section 4.2 now points from the 16-of-19 count to the Appendix G re-score.

Summary for Reviewer r3Rk

New in this discussion:

  • Zeroing the whole S³ selection moves refusal no more than random heads do (W4).
  • On Qwen3-8B, instruct sits above abliterated under five abliteration recipes (W3).

Changed in the revised PDF:

  • Figure 1 caption: presence defined.
  • Introduction: register defined; the two probe contrasts labeled.
  • Section 4.2: a pointer from the 16-of-19 count to the Appendix G re-score.
  • Discussion: the residual-stream prediction marked untested.
  • Appendix G: one sentence on zeroing the whole selection.

Not changed, and why:

  • The 16-of-19 count stays as the paper's count, because the 16 named heads replicate on the re-score (W2).
  • The guardrail comparison stays in Appendix G as a limitation. The paper proposes no guardrail (W1).

We hope these answers address your concerns, and we would be grateful if you would reconsider your rating.

Reviewer_hDV2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the origins and roles of safety-related attention heads in LLMs. The authors propose a Safety Specificity Score (S3) to identify safety-related heads and compare their behavior across pretrained, instruction-tuned, and abliterated models. Through head-ablation experiments and NMF-based representation analysis, the paper claims that safety detection mechanisms are recruited rather than newly constructed during fine-tuning, that detection is separable from refusal execution, and that the selected heads primarily encode prompt register rather than harm categories.

Critical strengths
  • The paper addresses an important question about the internal mechanisms of safety alignment in LLMs.
  • The proposed S3 metric and provenance triangle provide a potentially useful approach to studying safety-related attention heads across checkpoints.
  • The authors evaluate multiple model families and include probing, ablation, and representation analyses.
Critical weaknesses
  • The proposed metric measures differential representational sensitivity to harmful and benign prompts, but does not necessarily identify attention heads that functionally perform safety detection.
  • The absence of substantial behavioral changes after head ablation does not rule out redundant detection pathways or contributions from other modules. Therefore, the experiments do not fully establish the claimed mechanistic separation.
  • The main statistical effect is not significant for Qwen3.5-4B or Gemma-4-31B, raising concerns about the robustness and generalizability of the findings.
  • The technical depth and rigor of the current submission fall short of the standards expected at ICLR.
Questions for authors
  1. Based on the efficacy of the proposed method, how would you justify the robustness and generalizability?
  2. The experiments do not fully establish the claimed mechanistic separation.
4 comments
AuthorsOpenReview ↗

Response to Reviewer hDV2 (1/3): what S³ measures and the redundancy test

Thank you for the careful review. We are glad you see S³ and the three-checkpoint design as a potentially useful approach.

This reply comes in three comments: what S³ measures and the redundancy test; the two non-significant sets and rigor; robustness, generalizability and a summary.

W1. Does S³ find heads that perform detection, or only heads that are sensitive?

S³ alone measures sensitivity. The paper tests the selected heads in three further ways. By "detection," it means only that a probe can read harm from them.

  • You are right that S³ alone measures sensitivity, not function. It selects heads whose output shifts more on harmful than on benign prompts.
  • First, abliteration lowers their S³. Abliteration removes refusal from the weights. Instruct S³ stays above abliterated S³ on the same heads (Holm-corrected p = 4.7e-4 on Qwen3-8B, 3.2e-7 on Gemma-3-4B; Table 1).
  • Second, a probe reads harm from them. The both set contains selected heads absent from the base selection that drop out after abliteration. A linear probe on this set separates harmful from benign prompts at AUROC 0.875 on Qwen3-8B and 0.965 on Gemma-3-4B (Section 4.3).
  • Third, zeroing them does not remove refusal (Section 4.3).
  • We do not claim they are the model's only detector.

W2. Could a null ablation hide a redundant detection pathway?

A null alone cannot rule that out. We tested redundancy among the selected heads directly: zeroing the whole S³ selection at once moves refusal no more than random heads do.

  • The separation also appears from the other side. Abliteration removes refusal, yet the Qwen3-8B both-set probe still reads harm (Section 4.3): AUROC 0.875 before and 0.886 after. On Gemma-3-4B it falls from 0.965 to 0.875.
  • New in this discussion. In a new experiment we zeroed the whole S³ selection at once, not only the both set. We measured refusal mass: the probability that the first output token opens a refusal.
  • On Gemma-3-4B, refusal mass went from 0.50 to 0.34, but 69 of 200 random sets of the same size cut it at least as much. Qwen3-8B shows the same: 0.40 to 0.38, with 65 of 200 random sets doing at least as much.
  • The test can detect execution. In the same run, zeroing a different group of Gemma-3-4B heads cut refusal mass from 0.50 to 0.05. None of the 200 random sets did.
  • Those heads also shift on benign prompts, so they are not harm-specific and S³ leaves them out (Appendix G). In this test, refusal depends on these non-specific heads and not on the harm-specific heads S³ selects.

Changed in the revised PDF: Appendix G adds one sentence on zeroing the whole selection.

AuthorsOpenReview ↗

Response to Reviewer hDV2 (2/3): the two non-significant sets and rigor

W3. Do the nulls on Qwen3.5-4B and Gemma-4-31B undermine the main effect?

Both sets were scored with fewer prompts. Instruct S³ is significantly above abliterated S³ on both at the primary budget.

  • Instruct S³ is not significantly above abliterated S³ for these two sets in Table 1 (p = 0.12 and 0.70).
  • Both were scored at 200 prompts per dataset, compared with 500 for the primary sets (Table 10).
  • At 500 prompts, both are significant. Appendix G reports a second scoring run on different hardware at 500 prompts per dataset. The direction holds on both in that run: p = 7.0e-3 on Qwen3.5-4B and p = 2.3e-16 on Gemma-4-31B.
  • The prompt budget explains the Gemma-4-31B null. The second run also scored Gemma-4-31B at 200 prompts, as in Table 1, and found the same null.

W4. Does the submission meet the bar for technical depth and rigor?

We would rather answer with what the paper does than with adjectives.

  • Every claim rests on a paired test of named heads across three checkpoints, with Holm correction within each family of tests.
  • Each ablation has a control. It is compared with size-matched random head sets (Appendix B), and each null is tested for equivalence within a ±5 pp margin (Section 4.3).
  • The result is re-derived under six abliteration recipes on Gemma-3-4B, and every checkpoint is re-scored in a second run on different hardware (Appendix G).
  • Negative results are reported where they occur.
  • New in this discussion. The joint ablation of the whole selection, described under W2.
AuthorsOpenReview ↗

Response to Reviewer hDV2 (3/3): robustness, generalizability and summary

Q1. How robust and how general are the findings?

The direction survives changes in hardware and abliteration recipe. Our general claim concerns the direction and the separation, not the per-model head counts.

  • Different hardware. The second scoring run holds every head set fixed. In that run, 21 of 26 Qwen3-8B heads stay above abliterated (p = 9.5e-5), and on Gemma-3-4B 57 of 62 do (p = 1.7e-9; Appendix G).
  • Different recipes. The Gemma-3-4B direction also holds under five of six abliteration recipes (Table 9).
  • Generality. The six sets span two model families and one hybrid-attention architecture, from 4B to 31B parameters. On all three models tested in Section 4.3, zeroing the both set leaves compliance within ±5 pp.

Q2. Is the mechanistic separation established?

This repeats W2, and the answer there covers it. The new evidence is the joint ablation of the whole selection.

Summary for Reviewer hDV2

New in this discussion:

  • Zeroing the whole S³ selection at once moves refusal no more than random heads do (W2).

Changed in the revised PDF:

  • Appendix G: one sentence on zeroing the whole selection.

Not changed, and why:

  • Table 1 keeps the Qwen3.5-4B and Gemma-4-31B nulls as printed. Appendix G gives the re-scores at the primary budget (W3).
  • The claim stays the direction and the separation, not the per-model head counts (Q1).

We hope these answers address your concerns, and we would be grateful if you would reconsider your rating.

The safety detection relates to the attention heads is not clear analyzed, details are expected. The head ablation is not enough. To this end, most of my concerns still hold. I'll maintain my evaluation.

Reviewer_ExAuOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary
  • This paper studies the provenance of safety detection at the attention-head level by comparing base, instruct, and abliterated checkpoints.
  • It introduces the Safety Specificity Score (S³) to identify heads that respond more specifically to harmful than benign prompts, and argues that safety fine-tuning mainly recruits existing harm-separable representations rather than building detectors from scratch.
  • It further finds that the selected detection heads are largely separable from refusal execution, while unsupervised decomposition suggests that their activations capture more dataset/register/jailbreak style structure than a clean harm taxonomy.
Critical strengths
  • Finer-grained mechanistic framing. The base–instruct–abliterated comparison provides a useful way to separate where safety-related features. This is more informative than analyzing only a single refusal direction or a single checkpoint.
  • Interesting empirical observations. The NMF/HDBSCAN analysis suggests that the selected safety-related representations organize more strongly by dataset, register, and jailbreak style than by harm category, which provides a more nuanced view of what safety-related activations may encode.
  • Reasonable validation effort. The paper evaluates multiple model families and checkpoint sets, includes probe, ablation, circularity, and alternative abliteration analyses, and explicitly reports cases where the main direction does not replicate.
Critical weaknesses
  • The central notion of “detection” is not fully causally validated. S³ identifies heads whose activations are more harm-specific, but this does not necessarily mean that these heads are central to the model’s actual harm recognition process. Random head sets can also carry substantial harm-separable information, and the NMF results suggest that some selected features may reflect surface properties such as dataset or style.
  • The detection–execution separation may be over-interpreted. The near-null refusal effect after ablating S³-selected heads could mean that detection and refusal are genuinely separate, but it could also mean that the selected heads are only weakly causally related to safety decisions. The current experiments do not fully distinguish these explanations.
Questions for authors
  1. How do you rule out the possibility that S³ mainly selects correlates of harmful prompts, such as writing style, benchmark source, or jailbreak format, rather than heads that are functionally important for harm detection?
  2. Given that ablating the selected detection heads has little effect on refusal, what additional evidence supports interpreting them as a core detection mechanism rather than a probe-readable but behaviorally redundant representation?
  3. For the “recruited, not built” conclusion, does linear decodability from base representations suffice to establish recruitment? Could the results instead indicate that generic semantic representations already make harmfulness linearly separable, while safety fine-tuning only makes some heads more safety-specific?
2 comments
AuthorsOpenReview ↗

Response to Reviewer ExAu (1/2): causal validation and separation

Thank you for a careful review and for crediting the three-checkpoint design and the cases we report that do not replicate.

This reply comes in two comments: causal validation and separation; correlates, mechanism, base decodability and a summary.

W1. Does S³ show that these heads are central to how the model recognizes harm?

S³ alone does not show that, and the paper does not claim it. The paper's claim is provenance, measured on the same named heads across three checkpoints of one model.

  • What S³ measures is specificity: it divides a head's shift on harmful prompts by its shift on benign ones. We use "detection" for that harm-specific response.
  • Safety fine-tuning raises these heads' S³, and abliteration lowers it again (Holm p = 4.7e-4 on Qwen3-8B, 3.2e-7 on Gemma-3-4B; Table 1).
  • We agree that random heads also carry harm information. For a harm probe read at the base checkpoint, the selected sets fall between the 27th and 96.5th percentile of matched random-head sets, across two models and two contrasts (Section 4.2). So the probe shows what the heads' outputs make readable, not that the heads are the detector.
  • The register structure in the decomposition is the paper's third result (Section 4.4). Q1 gives evidence that S³ does not select on style alone.

W2. Is this separation, or do the selected heads just carry little causal weight?

Our refusal readout does catch heads that carry refusal. The S³ heads track safety training without carrying refusal. The paper calls that pairing separation.

  • We read refusal as first-token refusal mass: the probability the model puts on refusal openings at its first output token.
  • On Gemma-3-4B, dividing by the benign shift excludes 19 heads that a harmful-only score would select.
  • Zeroing them cut refusal mass on harmful prompts from 0.50 to 0.05. None of 200 random sets of the same size cut it as far (p = 0.005). These numbers come from the new run described below. Appendix G reports the same contrast from a separate, earlier run.
  • New in this discussion. In a new experiment, we zeroed the entire S³ selection with the same readout.
  • On Gemma-3-4B, refusal mass went from 0.50 to 0.34, but 69 of 200 random sets of the same size cut it at least as much. Qwen3-8B shows the same: 0.40 to 0.38, with 65 of 200 random sets doing at least as much.

Changed in the revised PDF: Appendix G adds one sentence on the whole-selection result.

AuthorsOpenReview ↗

Response to Reviewer ExAu (2/2): correlates, mechanism, base decodability and summary

Q1. Could S³ mainly select correlates such as style, source or jailbreak format?

Three results speak against style, source or format as the main driver.

  • First, the prompts are identical at all three checkpoints. Style, source and format stay fixed while the weights change. A head tracking only a benchmark's format would have no reason to gain specificity under safety tuning and lose it under abliteration, yet the selected heads do both (W1).
  • Second, normalization. Dividing by the benign shift lowers the scores of heads that respond to any input.
  • Third, harmful versus harmful-sounding. At instruct, they still separate harmful prompts from XSTest's harmful-sounding safe prompts (AUROC 0.913 on Gemma-3-4B, 0.739 on Qwen3-8B).
  • We do not claim the heads carry no style. Their joint activations organize by register (Section 4.4).

Q2. Core detection mechanism, or a probe-readable but redundant representation?

For refusal, your second description fits what we report. The paper does not claim a core detection mechanism.

  • Safety fine-tuning recruits these heads, and abliteration lowers their specificity. Zeroing them does not reproduce abliteration's refusal collapse.
  • What ties the heads to safety. The evidence is W1's paired movement and one further read. On Gemma-3-4B, their harm separation falls under all six abliterations we tested (AUROC 0.965 at instruct, 0.81 to 0.91 after; Appendix G, Table 9).

Q3. Does base decodability establish recruitment, or only generic separability?

Your second reading is the paper's conclusion. The base probe only rules out "built"; the paired S³ movement provides the evidence of recruitment.

  • Section 4.2 calls a head recruited if fine-tuning raised its specificity while its output already separates harm at base. It calls a head built if the separation appears only after fine-tuning.
  • An input-only baseline. A classifier on sentence embeddings of the prompt text alone reaches AUROC 0.979 against Alpaca prompts and 0.852 against XSTest's harmful-sounding safe prompts. Random head sets separate comparably, as W1 notes.

Summary for Reviewer ExAu

New in this discussion:

  • Zeroing the whole S³ selection moves refusal no more than random heads do (W2).

Changed in the revised PDF:

  • Appendix G: one sentence on zeroing the whole selection.

Not changed, and why:

  • "Detection" stays the paper's word for the harm-specific response S³ measures. The paper does not claim these heads are the model's only or necessary detector (W1).

We hope these answers address your concerns, and we would be grateful if you would reconsider your rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response: new evidence and changes in the revised PDF

We thank the three reviewers for close reviews that engage with the appendices as well as the main text. The table lists the concerns that drew new evidence or a change in the PDF. The per-reviewer replies answer the rest: the headline count and the selection rule (r3Rk), correlates and base decodability (ExAu), and novelty and rigor (r3Rk, hDV2).

Refusal mass is the probability the model assigns to refusal openings at its first output token.

Concern (raised by)What we didResultWhere
Is the null ablation evidence of separation? (hDV2, ExAu, r3Rk)In a new experiment, zeroed all S³-selected heads at once on Gemma-3-4B and Qwen3-8BRefusal mass moves no more than for random sets of the same size: 0.50 to 0.34 (69 of 200 random sets cut it at least as much); 0.40 to 0.38 (65 of 200)This discussion; one sentence in Appendix G
The nulls on Qwen3.5-4B and Gemma-4-31B (hDV2)Re-scored the sets scored at 200 prompts per dataset at the primary budget of 500Instruct above abliterated on all three: Qwen3.5-4B p = 7.0e-3, Qwen3.5-9B p = 1.3e-4, Gemma-4-31B p = 2.3e-16Appendix G
The erased set depends on the abliteration recipe (r3Rk)Scored Qwen3-8B under five recipes, two built with HereticInstruct above abliterated under all five (Holm p ≤ 0.033); on Gemma-3-4B, Heretic's p = 1.0 stands as printedThis discussion; Appendix G
Terms used before they are defined (r3Rk)Defined presence and register at first use; labeled the two base-probe ranges by contrastBenign Alpaca prompts or harm-matched XSTest promptsFigure 1 caption; introduction

What the paper claims

  • Safety fine-tuning raises the harm specificity of attention heads whose base outputs already separate harm. Abliteration lowers it on the same heads.
  • Zeroing those heads does not reproduce abliteration's refusal collapse. Paired with the S³ movement, this is the separation the paper reports.
  • The paper does not claim that these heads are the model's only or necessary detector, or that the probe is a guardrail. The claim rests on paired tests of the same named heads across checkpoints; the head counts describe the anatomy.

Changed in the revised PDF

  • Figure 1 caption: presence defined.
  • Introduction: register defined; the two base-probe ranges labeled by contrast.
  • Section 4.2: a pointer to Appendix G for the selection re-derived on the independent re-score.
  • Discussion: the prediction about defenses that suppress execution marked as untested.
  • Appendix G: one sentence on zeroing the whole selection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.