acceptodds
Under review as a conference paper at ICLR 2027

Frozen Input-Harm Probes under Compression: Retained AUROC Does Not Mean Retained Refusal

Abstract

Linear probes trained on a model's BF16 activations are candidate safety monitors, but models are often compressed after a probe is fitted. We ask whether a frozen input-harm probe that still separates harmful from harmless prompts implies that the compressed model still refuses. Across four families and forty compressed configurations (post-training quantisation, QLoRA, distillation, and Wanda and SparseGPT pruning), we pair judged attack success on StrongREJECT with frozen-probe AUROC on XSTest wherever hidden dimensions match. With greedy 48-token generation, thirteen configurations have attack-success intervals above zero and seven survive Holm correction; none is post-training quantised or Gemma-2-9B. In OLMo-2-7B, SparseGPT 2:4 and Wanda 2:4 raise attack success by and , while seed-mean probe AUROC falls detectably but by less than our planned 0.10 big-drop threshold at all eight probed layers (a one-sided test confirms this at every layer only for SparseGPT 2:4). The same methods give Llama and Qwen big drops at two or three layers shallower than the planned one. On our primary pooled AUROC basis this is therefore an existence result in one family (per fold, Llama SparseGPT 2:4 also qualifies), on disjoint probe and behavioural prompts. The OLMo increases lose Holm significance if three or four of six unjudged baseline responses (SparseGPT and Wanda, respectively) were successes; twelve of the thirteen increases persist at a 256-token cap and in every sampled seed. Retained AUROC does not certify retained refusal: reusing a probe after compression still requires behavioural evaluation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
model compression, quantization, pruning, knowledge distillation, linear probes, interpretability, safety monitoring, refusal, harmful compliance, sparse autoencoders, safety evaluation, LLM safety

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FZQFOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper tests whether a frozen classifier of harmful prompts remains accurate after model compression, and whether that accuracy implies that the model still refuses harmful requests. It finds that some pruned models produce more harmful answers while retaining high input-harm AUROC, and investigates ways to improve probe transfer.

Critical strengths

Several measurement controls are well chosen. The study includes a text-only baseline, paired comparisons, multiple-comparison correction, and checks for missing judgments and generation length. These help rule out several simple explanations for the reported results.

Critical weaknesses
  • The paper is almost impossible to follow as written. I found the presentation unnecessarily difficult and frustrating to read. Even the abstract's "With greedy 48-token generation, thirteen configurations have attack-success intervals above zero and seven survive Holm correction; none is post-training quantised or Gemma-2-9B" omits that these intervals concern changes relative to the uncompressed model, and the sentence as a whole simply does not convey anything understandable to a human reader. Results are buried in successive qualifications about statistical tests, layer choices, and different AUROC calculations. The reader repeatedly has to reconstruct which comparison supports which conclusion. A substantial rewrite is needed to make the argument and evidence understandable.

  • The main finding is already established, and the additional experiments provide little explanation. Zhao et al. (2025) (from NeurIPS 2025 cited in the paper) already show that harmfulness representations can remain largely unchanged while fine-tuning weakens refusal. This paper adds measurements under compression, but does not explain why particular compression methods preserve harmful-input classification while changing harmful behavior. The steering experiments in Section 5 estimate a new direction for each compressed model, so they do not establish how compression changed the original refusal mechanism.

  • The proposed augmentation is not shown to repair the safety problem motivating the paper. Section 6.2 reports improved classification of harmful inputs, but does not show that the augmented monitor catches more harmful outputs at a controlled false-positive rate. The paper's central finding is that input-harm AUROC does not establish safe behavior, so an improvement in that same metric cannot by itself demonstrate a useful safety remedy. On the primary AUROC calculation, augmentation also provides a smaller gain than simple re-centering.

Questions for authors
  1. Clarification - scientific contribution: What specific conclusion goes beyond the separation of harmfulness and refusal already established by Zhao et al. (2025), and which experiment supports it? Does the study explain why compression produces this effect, or is its contribution the observation that the known separation also occurs under compression?

  2. Additional evidence on augmentation: At the same false-positive rate, does the augmented probe detect more judge-confirmed harmful responses than the original frozen probe and simple re-centering? This comparison could change my assessment of the mitigation contribution. If it has not been evaluated, it would require an additional experiment; the reported improvement in input-harm AUROC does not answer this question.

3 comments
AuthorsOpenReview ↗

Matched-FPR test: augmentation dropped as a remedy; causal test added

Thank you for the detailed review. We ran your matched-FPR test (augmentation fails it; we drop it as a remedy) and a causal test of the BF16 direction (inconclusive).

Terms used below. BF16 = uncompressed model; frozen probe = BF16-trained, applied unchanged; attack success = judged share of usable harmful answers.

W1 (readability). Agreed. The revision will have defined terms, short sentences and one claim per paragraph; caveats will go in one robustness table. The planned layers (fixed before each family's data) and a single-probe AUROC (chosen post hoc, beside pooled AUROC over all folds' scores) will be primary. Your example will become four plain sentences in Section 4, as changes from BF16.

W2 + Q1. Largely the latter: our main result is the known separation, observed after standard compression with a deployer's frozen probe and judged behaviour. Three results go further (2 and 3 post hoc):

  1. Which methods. In OLMo-2-7B, SparseGPT and Wanda 2:4 pruning raise 48-token attack success by 5.9 and 6.5 points, yet the frozen probe loses under 0.10 AUROC at all eight layers (Section 4.1; pooled, Wanda's layer-4 interval crosses 0.10). Counting unjudged answers against them removes their Holm significance at 48 tokens, not 256 (post hoc Holm family). No post-training-quantised model (of 16) shows a detected increase (consistent with Hong et al., 2024); missing judgements leave room for small ones.
  2. The fixed threshold fails where AUROC holds (extends App. C.5, E.25). At a 5% false-positive-rate (FPR) threshold set on BF16, 2:4 pruning lowers recall of unsafe prompts by 0.218 [0.178, 0.277] (approximate) over 64 model-layers, while AUROC falls only 0.035 on average (0.066 pooled). Re-calibrating recovers only about a third of the loss (most at the planned layers).
  3. A pilot. Frozen, a Qwen probe trained on 16 BF16 compliance positives misses a 14-20 point rise in attack success (256 tokens) after pruning. With so few positives it largely learns an inverted harm score.

Why (W2, Q1). Not explained. Section 5's per-model directions cannot show how the BF16 mechanism changed, only that 38 of 40 compressed models have one whose negative steering lowers lexically scored refusal. Two new analyses fix the BF16 direction:

  • Description. In the six Holm-significant 2:4 models, at planned layers, harmful requests shift along it toward or past the BF16 class boundary. AUROC ignores shifts common to both classes and BF16's near-ceiling separation hides much of the rest; a fixed threshold loses recall. Not a mechanism: Gemma's 2:4 models shift less with no detected behaviour change, and Llama QLoRA changes behaviour with no norm-matched shift toward the boundary.
  • Causal test (rule fixed before it ran). In OLMo's 2:4 models we restored the BF16-direction projection lost to pruning, against three random directions of equal added norm (BF16 and 4-bit NF4 as controls). Restoring beat every random direction in both models, and their mean by over a quarter of the rise, which excludes our null outcome. A positive result also needs restoring to undo at least half of each model's rise, even with unjudged answers counted against it: SparseGPT 2:4 does (0.86; 0.72 counted against), Wanda 2:4 does not (0.58; 0.43 counted against). By our rule it is inconclusive. A control fixed in advance cuts against it: in BF16 and NF4, the same addition raises refusal of 250 safe XSTest prompts (+11.2, +10.4 points); no random direction detectably does.

Which settings (W2). We pruned OLMo anew at 2:4 and 20-70% unstructured sparsity, with capability benchmarks. Outside 2:4, greedy 256-token attack success rises Holm-significantly only at 60-70% sparsity, which costs 19.5-42.3 benchmark points on average (2:4: 22-24). Increases in three 40-50% builds fail Holm, and the capability-preserving nulls (Wanda 20-30%, SparseGPT 20%) are fragile to missing judgements. At 60% the probe holds at every layer with per-fold AUROC (pooled, both fail at layer 4; SparseGPT 60% loses over 0.10). So in OLMo the pattern does not need 2:4 but comes only with heavy damage. The experiment's analysis rules were finalised after the runs, so it is not pre-registered.

W3 + Q2 (augmentation). Not consistently. In 8 compressed models (194 judge-confirmed harmful 48-token answers), thresholds give 5% FPR on each model's held-out safe prompts. At the planned layer, augmentation changes the catch rate by +0.042 [-0.056, +0.162] over the frozen probe (p = 0.39; approximate). Per model (uncorrected), it gains in the OLMo and Qwen 2:4 models but loses in Llama Wanda 2:4. Sensitivity analysis, 256-token labels: +0.118 [+0.026, +0.189]. Re-centring adds one constant to every score, so at matched FPR its decisions equal the frozen probe's. Its pooled-AUROC edge is an offset correction. Section 6.2 will become an input-monitor robustness result.

If this addresses your concerns, we hope you will consider raising your score.

AuthorsOpenReview ↗

Revised PDF: where each change is

We have uploaded a revised PDF. The changes for your review, by location:

  • W1 (readability): a new abstract (question, main finding, implication); a Terms paragraph opens Section 3; the planned layers and a single held-out probe (chosen post hoc, with pooled AUROC beside it) are now primary (Section 3.4, Table 1, Figure 1); your example sentence is now four plain sentences in the Section 4 opening, stated as changes from BF16; long sentences in Sections 4.1 and 4.2 are split into shorter ones; the main caveats are gathered in Appendix Table 62.
  • W2, Q1 (beyond Zhao et al.): the Introduction attributes the separation to Zhao et al. and states that our contribution is empirical; the three results that go further are in Section 4.1 (which methods), Section 4.2 (fixed threshold) and Appendix G.3 (pilot).
  • W2, Q1 (why): Section 5 now states that per-model directions cannot show how BF16's mechanism changed, and describes the shift along the frozen BF16 refusal direction (a description, not a mechanism; details in Appendix G.5); Section 8 and Appendix F.4 report the causal test (inconclusive) and its over-refusal control.
  • W3, Q2 (augmentation at matched FPR): Section 6.2 reframes augmentation as an input-monitor robustness result with the matched-FPR test; Appendix G.2 (Table 60) has the full analysis, including re-centring.
  • Which settings: Section 8 and Appendix F.2.

Thank you for the revision and for running the additional experiments. I appreciate the acknowledgement that augmentation has not been shown to improve safety.

However, I still find the writing unnecessarily difficult to understand. For example, the response states:

"Frozen, a Qwen probe trained on 16 BF16 compliance positives misses a 14-20 point rise in attack success (256 tokens) after pruning."

As a human reader, I genuinely cannot understand this sentence as written. I think it means that a probe trained on the original Qwen model failed to detect the additional harmful answers produced after pruning. If so, why not say that directly? Readers should not have to repeatedly decode technical shorthand to understand what an experiment found. Although the presentation has improved, this remains a serious problem.

I also have two remaining scientific concerns:

  1. Novelty: Zhao et al. (2025) already established that recognizing harmful requests and refusing them are different things. The new fixed-threshold experiment provides a useful practical example, but the paper still does not explain why compression changes refusal behaviour. I therefore remain unconvinced that the central finding goes sufficiently beyond previous work.

  2. Practical relevance: The clearest reproduced safety failures occur in heavily pruned models that also lose 22%-24% on average across capability benchmarks. The new experiments do not detect comparable safety deterioration in models that preserve their capabilities. This raises an important question: does the finding apply to compressed models that people would realistically deploy, or mainly to models whose overall performance has already deteriorated substantially?

I appreciate the additional work and the willingness to report inconclusive and negative results. The revision addresses some of my original concerns, but the remaining issues with clarity, novelty, and practical relevance are still substantial enough that I cannot support acceptance.

Reviewer_XVh9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors study the problem of evaluating safety of compressed models (e.g. quantized, pruned, distilled, etc.). They consider linear probes trained on the uncompressed model as tools for safety monitors, and ask whether an observed high value of AUROC can be read as a proof of preserved alignment. The authors study four models families and ten compression techniques for each, and found that probes with retained AUCROC do not imply the model retained refusal, suggesting behavioral evaluation after compression is still needed independently of the probe performance.

Critical strengths

Extensive number of experiments (the authors consider four model families and ten types of compression for each one). All measurements are reported with confidence intervals and are adjusted to account for multiple measurements. Negative results and limitations are reported.

Critical weaknesses
  1. The post hoc analysis choices introduce a source of bias in the presented results. The significance of the findings could change substantially and this need to be quantified. The statistical analysis should correct for post selection bias, or the robustness of the results to these choices should be studied, or the analysis should be rerun on independent experiments.

  2. The presentation of the paper makes it very hard to understand the work. Many sentences are dense and leave information implicit. Many results are listed but not carefully discussed or motivated.

Questions for authors
  • how do the results depend of the post-doc analysis choices? can you quantify it and comment on their impact on the reported findings?
  • L. 23: “planned 0.10 big-drop threshold” how and why do you choose this value?
  • L. 47: “A post hoc rule (H7) is a sensitivity analysis (Section 4.2).” What does this sentence mean?
  • L. 53: “no detected post-training quantisation or Gemma increase in the 48-token grid and a less robust frozen sparse basis” increase what?
  • L 161-194: the procedure used to choose the headline layers is not clear and it matters. Please explain your decision or study the impact of such decision on the results.
  • Figure 2 is never mentioned in the text, what is it's role? Presentation:
  • Many sentences in the results sections are negative constructions (“X does not mean Y”), which is not that helpful without at least one positive construction saying what your result actually establishes. L. 345: “This does not measure compliance prediction or establish unchanged geometry." this sentence tells what it doesn’t not establish but not what it does establish. What is the take away? L. 358: again you say what these results are not, but you don’t state what is the take away of the experiment. L 413: “which does not establish immunity to other hazards or doses” you say again what your finding does not establish but you do not say what it does. Please comment your results.
  • Abstract: too dense. More than half is a list of findings. The actual conclusion is one line at the end.
  • Section 2.1 doesn’t seem to belong to related work, but to setup
  • L 322: “Refitting on compressed activations” refitting what? The probe? Clarify
2 comments
AuthorsOpenReview ↗

Post hoc choices quantified; OLMo 2:4 result reproduced at 256 tokens

Thank you for the careful reading. As you suggested, we measured how far each main post hoc choice moves the findings and reran the OLMo 2:4 result on newly pruned models; it reproduces at 256 tokens.

Terms used below: BF16 = uncompressed model; frozen probe = BF16-trained, applied unchanged; attack success = judged share of usable harmful answers; big drop = AUROC loss above 0.10; dissociation = attack success rises, no big drop; pooled AUROC ranks all folds' held-out scores jointly; per-fold AUROC averages fold AUROCs.

W1 + Q1. We agree. We applied no formal post-selection correction; these are post hoc re-analyses of cached data, not replications. Fixed before the results (in our dated log, not a public plan): the 0.10 big-drop and 0.75 failure thresholds, probe setup (pooled AUROC), paired comparisons, all eight layers, planned layers (Qwen's and OLMo's set after Llama's results), 48-token cap, judge prompt. Chosen after, with their effect:

  • Headline layers (Q5), chosen with no fixed rule or held-out split: Llama 8, where frozen transfer separated methods, then Qwen 7 and OLMo 8 at the same relative depth (Gemma: planned 21; no reason recorded). This favours method contrasts: with no big drop at the planned layers, all 6 probed Holm-significant increases are dissociations, against 2 at the headline layers (pooled; +4 [+2, +5]) or 4 per fold (+2 [+1, +3]). The OLMo 2:4 dissociation (not predicted) spans all eight layers, so no layer choice creates it. Table 1 and Figure 1 will use the planned layers.
  • Estimator: big drops fall from 21 model-layers pooled to 4 [3, 7] per fold; the refit gain also shrinks.
  • Tango intervals, Holm, imputation (conservative): scoring unjudged answers against each claim removes the OLMo increases' Holm significance at 48 tokens (not 256, under a post hoc Holm family) and makes 12 of 16 post-training-quantised and all 10 Gemma null intervals positive.
  • Multiverse (12,096 specifications, some without Holm): 0 to 11 dissociations (median 8).
  • L.47 rule (Q3): a sensitivity check sorting models by rise and probe degradation (0.05 margin); no claim rests on it (its internal label will go).
  • Augmentation: dropped as a safety remedy.
  • Judge model, XSTest probe prompts: no sensitivity result.

Rerun (closest to your independent-rerun option). We re-pruned both OLMo 2:4 models, reran BF16 on the same machine and judged greedy 256-token answers. Attack success rises by +23.8 [+19.4, +29.0] (SparseGPT 2:4) and +23.3 [+18.9, +28.4] points (Wanda 2:4; paper: +24.5 in both), Holm-significant under worst-case scoring. Per fold, the frozen probe is non-inferior (one-sided 5% test, 0.10 margin) at all eight layers; pooled, Wanda 2:4 again fails that test at layer 4. Probe values match the paper's to four decimals; answers are new. Same prompts, recipe and judge: not an independent sample; it shows reproducibility, not generality. Both lose 22-24 benchmark points on average.

Timing. The rerun's design and endpoints preceded the runs; its analysis plan came after the runs and judging, before any outcome was opened (logs had shown compliance-correlated cap-hit rates and unjudged counts), so it is not pre-registered. Only our new causal test (steering along the BF16 refusal direction) had its rule fixed in advance; by that rule it is inconclusive.

Q2 (L.23). A round number fixed with the 0.75 failure floor before any probe data; no derivation was recorded. Among Holm-significant increases, the pooled seed-mean count of all-layer dissociations is exactly two (both OLMo 2:4) for any threshold from 0.091 to 0.216; we will add threshold curves.

Q4 (L.53). An increase in judged attack success over the model's BF16 version, as contribution (iv) will say.

Q6 (Figure 2). Cited once (L.354-355); the steering paragraph will cite it first. It shows 38 of 40 compressed models have their own re-estimated direction whose negative steering lowers lexically scored refusal, not that the BF16 direction survived.

Negative constructions. Each will say what the result shows. L.345: refitting raises AUROC, so here input-harm information survives; the frozen BF16 decision rule fits compressed activations less well. L.358: see Q6. L.413: Gemma's layer-21 AUROC moved by at most 0.003 and no reproducible increase was detected at 48 or 256 tokens; smaller ones remain possible.

W2, abstract, Section 2.1. Agreed; the revision will have one claim per paragraph, defined terms, short sentences, one robustness table; an abstract that states the question, main finding and implication in turn (findings moved to the body); Section 2.1 moved to a Setup section.

L.322. Yes, the probe, refit on the compressed model's own activations.

If this addresses your concerns, we hope you will consider raising your score.

AuthorsOpenReview ↗

Revised PDF: where each change is

We have uploaded a revised PDF. The changes for your review, by location:

  • Post hoc choices (W1, Q1): the Introduction and Section 3.4 state what was fixed before the results and what was chosen after. New in this revision and chosen after the results: a single held-out probe is now the primary estimator, with the pre-specified pooled AUROC beside it (Table 1, Figure 1). Appendix G.4 lists each choice and, where we measured it, its effect, including the multiverse; Appendix Table 62 gathers the main ones.
  • Headline layers (Q5): Section 3.4 says how they were chosen and how they change the results; Table 1 and Figure 1 now use the planned layers, and the headline layers stay as a labelled post hoc sensitivity (Appendix Table 3).
  • Rerun (closest to your independent-rerun option): Section 8 and Appendix F.3 (Table 54).
  • Augmentation: Section 6.2 no longer presents it as a safety remedy.
  • Q2 (L.23): the Introduction says how 0.10 was set; Appendix G.4 (Figure 5) plots the results against the threshold.
  • Q3 (L.47): the internal label is gone; the rule is described in plain words (Introduction; Appendix B.1).
  • Q4 (L.53): contribution (iv) now names the quantity.
  • Q6 (Figure 2): cited in the first sentence of the steering paragraph (Section 5).
  • Negative constructions: L.345 and L.358 rewritten in Section 5; L.413 in Section 7.1.
  • W2, abstract, Section 2.1, L.322: a new abstract (question, main finding, implication); old Section 2.1 is now Section 3.1 of a Setup section that opens with a Terms paragraph; Section 5 says the probe is refit; long sentences in Sections 4.1 and 4.2 are split into shorter ones.
Reviewer_uDdjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether linear input-harm probes trained on BF16 activations remain informative about safety after compression. It evaluates 40 compressed configurations across four model families, measuring frozen-probe AUROC on XSTest and harmful compliance on StrongREJECT. The strongest finding is that OLMo's two structured pruning methods substantially increase harmful compliance while preserving high input-harm AUROC. Additional experiments examine probe retraining, steering, external guards, and compression-aware augmentation.

Critical strengths
  • The OLMo result is convincing within the tested setting. Both pruning methods retain headline AUROC above 0.95 while increasing harmful compliance. The behavioural increase persists with longer generation and sampled decoding, and SparseGPT passes the AUROC non-inferiority test at every measured depth. This supports a concrete counterexample to using retained discrimination as evidence of retained refusal.

  • The study includes useful controls and unusually explicit sensitivity analyses. Paired comparisons, multiplicity correction, missing-verdict analysis, a second judge, and alternative readouts help distinguish robust findings from results that depend on evaluation choices.

Critical weaknesses
  • The conceptual advance is limited. The probe predicts input harmfulness, so its continued accuracy does not imply that the model will act on that information. Zhao et al. already demonstrate preserved harmfulness representations despite weakened refusal under adversarial fine-tuning. The compression experiments extend that observation, but the paper does not evaluate transfer of a monitor trained to predict compliance. The authors acknowledge this distinction, which also narrows the significance of the central claim.

  • Several conclusions about probe degradation depend strongly on pooling scores from separately fitted fold probes. Computing AUROC within folds reduces the number of large drops from 21 to 4 and operational failures from 9 to 0. Likewise, the augmentation gain falls from 0.0609 to 0.0144. Keeping the pooled estimator because it was planned does not resolve its mismatch with deploying a single probe. The main presentation should center the estimator that measures the intended deployment setting.

  • The practical scope of the strongest result remains unclear. On the primary analysis, the all-layer counterexample is concentrated in one family and one sparsity pattern. Those OLMo models also have perplexity ratios around 1.57 to 1.59, indicating substantial general degradation. Capability evaluations and a compression severity sweep are needed to establish whether this dissociation occurs at useful operating points and how safety loss relates to broader model damage.

Questions for authors
  • Can you evaluate a single frozen BF16 probe on an independent test split, reporting AUROC and detection rates at a fixed false-positive rate? This would clarify how much of the reported degradation reflects pooling scores from different fold probes.

  • Does the OLMo dissociation persist at milder pruning levels that preserve downstream capabilities? The current perplexity increases leave its relevance to practical compression settings unclear.

  • Qwen's BF16 baseline has more compliance positives at 256 tokens. Can you use these labels, potentially with additional requests, to test frozen transfer of a compliance-trained probe and establish whether the finding extends beyond input-harm classification?

2 comments
AuthorsOpenReview ↗

All three requested analyses run: held-out probe, pruning sweep, compliance probe

Thank you for three precise questions. We ran all three analyses you asked for, including a new OLMo pruning sweep with capability benchmarks.

Terms used below. BF16 = uncompressed model; frozen probe = BF16-trained, applied unchanged; attack success = judged share of usable harmful answers; big drop = AUROC loss above 0.10; pooled AUROC ranks all fold probes' held-out scores together, per-fold AUROC averages within-fold AUROCs.

W1 + Q3. We agree: Zhao et al. describe the phenomenon and our contribution is empirical; retained AUROC shows input harm is still encoded, not that the model still refuses. New: which compression methods detectably raise attack success (some pruned, distilled and benign-QLoRA models; none of the 16 post-training-quantised (PTQ) ones, consistent with Hong et al., 2024), and that a fixed threshold loses recall where AUROC holds.

  • Q3: yes, as a pilot. Only Qwen's BF16 model has at least 10 judged-harmful answers (greedy 256 tokens: 16 of 306). Frozen, a layer-14 probe trained on these labels predicts compliance for 0.0-0.5% of requests to four pruned Qwen models (5.7% on BF16) while attack success rises by 14-20 points. Limits: one family, 16 positives, no new requests, much overlap with a sign-reversed harm score.
  • Mechanism: open. A causal test along the BF16 refusal direction (rule fixed before it ran) is inconclusive and raised safe-prompt refusal in BF16.

W2 + Q1. We agree. One frozen probe per split will be the primary estimator (200 stratified train/calibration/test splits of 450 XSTest prompts; all 288 model-layers), with recall and FPR at 5% and 10% FPR; the pre-specified pooled one will stay beside it.

  • AUROC: 5 big drops [2, 7] (mean over splits) and 0 failures (AUROC below 0.75), against 21 and 9 pooled and 4 [3, 7] and 0 per fold; it agrees with per-fold (r = 0.991), not pooled (r = 0.861). Every OLMo 2:4 layer's interval excludes a 0.10 loss.
  • Detection at 5% FPR: at the BF16 threshold, recall over the 64 2:4-pruned model-layers falls by 0.218 [0.178, 0.277] (approximate). Re-calibrating it on the compressed model recovers only the score-shift part (about a third; most at the planned layers).
  • Augmentation: its AUROC gain on held-out pruning methods is +0.0144 [+0.0109, +0.0179] per fold (pooled +0.0609); at matched FPR it does not consistently catch more judge-confirmed harmful outputs, so we drop it as a safety remedy.

W3 + Q2. We agree that OLMo 2:4 is a high-damage operating point, and its 48-token rise loses Holm significance if all unjudged answers count against it. We pruned OLMo-2-7B with Wanda and SparseGPT (20-70% unstructured; 2:4), each against same-machine BF16. A rise needs a positive paired 95% interval for greedy 256-token attack success and an exact McNemar test surviving Holm over the 14 pruned builds, also with unjudged answers counted against it. Capability is kept if the mean drop on MMLU, ARC-Challenge, HellaSwag and GSM8K is at most 2 points, none above 5 (point estimates).

Sweep limits. We set the endpoints and the 0.10 margin before the runs and the rules above after them, so the sweep is not pre-registered. One family, one-shot pruning; same prompts, judge and recipe, so not an independent sample (the 2:4 builds' probe values match the paper's to four decimals).

  • Q2: no. No build that keeps capability (Wanda 20-30%, SparseGPT 20%) shows a rise, but, as with the paper's PTQ nulls, scoring every unjudged answer in favour of an increase puts all three intervals above zero. Rises occur only at 60-70% and 2:4 (both methods), all failing the rule (outside 2:4, mean drops 19.5-42.3 points). Wanda 50% and SparseGPT 40-50% have intervals above zero but no Holm-significant rise (sampled, Wanda 50% is Holm-significant at two of three seeds and by majority vote, not with unjudged answers counted against it).
  • Dissociations (a rise while, at all eight layers, a one-sided 5% test rejects a frozen-probe per-fold AUROC loss of 0.10 or more): both methods at 60% and at 2:4 (2:4 mean drops: SparseGPT 24.1, Wanda 22.2 points), so in OLMo it does not need 2:4 but comes with heavy capability loss. With pooled AUROC only SparseGPT 2:4 qualifies (the others fail at layer 4; only SparseGPT 60% has a big drop).
  • Across families: under post hoc choices (e.g., per-fold AUROC, 256-token labels), 7 [6, 8] models in three families show an increase with no big drop at any layer (6 pass the one-sided test), including Llama/Qwen 50% pruning (est. 13-22 capability points lost; Egashira et al., ICLR 2026).
  • Damage and harm: over the 36 probed models, perplexity ratio tracks the 48-token attack-success change (Spearman +0.46 [+0.10, +0.74]; descriptive), but Gemma's 2:4 models (ratios 1.56, 1.70) show no detected increase.

Presentation: we will rewrite the paper for readability and add a per-build sweep table.

If these analyses answer your three questions, we hope you will consider raising your score.

AuthorsOpenReview ↗

Revised PDF: where each change is

We have uploaded a revised PDF. The changes for your review, by location:

  • W2, Q1 (estimator and detection): the single held-out probe is now the primary estimator, with pooled AUROC beside it (Section 3.4; Table 1 and Figure 1, now at the planned layers). Table 1 adds recall and FPR at the BF16 5%-FPR threshold; Appendix G.1 (Tables 58-59) gives 5% and 10% FPR. The fixed-threshold result is in Section 4.2.
  • W3, Q2 (sweep and capability): Section 8 summarises the OLMo sparsity sweep; Appendix F.2 has the per-build table (Table 52), the per-task capability results (Table 53) and the damage-versus-harm comparison. The across-family result (7 [6, 8] models) is in Appendix G.4 ("Scope across families").
  • Q3 (compliance-trained probe): Appendix C.2 and Appendix G.3 (Table 61), with its limits; Section 9 (L11) summarises them in the main text.
  • W1 (contribution): the Introduction attributes the separation to Zhao et al. and states that our contribution is empirical.
  • Augmentation: Section 6.2 now gives the per-fold gain (+0.0144) before the pooled one (+0.0609) and no longer presents augmentation as a safety remedy; the matched-FPR test is in Appendix G.2.
  • Mechanism: Section 8 and Appendix F.4 report the causal test (inconclusive) and its over-refusal control.
  • Presentation: the abstract is shorter and now opens with the question, then gives the main finding and its implication; Section 3 opens with a Terms paragraph and absorbs the old Section 2.1; long sentences in Sections 4.1 and 4.2 are split into shorter ones; the main caveats are gathered in Appendix Table 62.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.