- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper tests whether a frozen classifier of harmful prompts remains accurate after model compression, and whether that accuracy implies that the model still refuses harmful requests. It finds that some pruned models produce more harmful answers while retaining high input-harm AUROC, and investigates ways to improve probe transfer.
Critical strengths
Several measurement controls are well chosen. The study includes a text-only baseline, paired comparisons, multiple-comparison correction, and checks for missing judgments and generation length. These help rule out several simple explanations for the reported results.
Critical weaknesses
-
The paper is almost impossible to follow as written. I found the presentation unnecessarily difficult and frustrating to read. Even the abstract's "With greedy 48-token generation, thirteen configurations have attack-success intervals above zero and seven survive Holm correction; none is post-training quantised or Gemma-2-9B" omits that these intervals concern changes relative to the uncompressed model, and the sentence as a whole simply does not convey anything understandable to a human reader. Results are buried in successive qualifications about statistical tests, layer choices, and different AUROC calculations. The reader repeatedly has to reconstruct which comparison supports which conclusion. A substantial rewrite is needed to make the argument and evidence understandable.
-
The main finding is already established, and the additional experiments provide little explanation. Zhao et al. (2025) (from NeurIPS 2025 cited in the paper) already show that harmfulness representations can remain largely unchanged while fine-tuning weakens refusal. This paper adds measurements under compression, but does not explain why particular compression methods preserve harmful-input classification while changing harmful behavior. The steering experiments in Section 5 estimate a new direction for each compressed model, so they do not establish how compression changed the original refusal mechanism.
-
The proposed augmentation is not shown to repair the safety problem motivating the paper. Section 6.2 reports improved classification of harmful inputs, but does not show that the augmented monitor catches more harmful outputs at a controlled false-positive rate. The paper's central finding is that input-harm AUROC does not establish safe behavior, so an improvement in that same metric cannot by itself demonstrate a useful safety remedy. On the primary AUROC calculation, augmentation also provides a smaller gain than simple re-centering.
Questions for authors
-
Clarification - scientific contribution: What specific conclusion goes beyond the separation of harmfulness and refusal already established by Zhao et al. (2025), and which experiment supports it? Does the study explain why compression produces this effect, or is its contribution the observation that the known separation also occurs under compression?
-
Additional evidence on augmentation: At the same false-positive rate, does the augmented probe detect more judge-confirmed harmful responses than the original frozen probe and simple re-centering? This comparison could change my assessment of the mitigation contribution. If it has not been evaluated, it would require an additional experiment; the reported improvement in input-harm AUROC does not answer this question.
3 comments
Revised PDF: where each change is
We have uploaded a revised PDF. The changes for your review, by location:
- W1 (readability): a new abstract (question, main finding, implication); a Terms paragraph opens Section 3; the planned layers and a single held-out probe (chosen post hoc, with pooled AUROC beside it) are now primary (Section 3.4, Table 1, Figure 1); your example sentence is now four plain sentences in the Section 4 opening, stated as changes from BF16; long sentences in Sections 4.1 and 4.2 are split into shorter ones; the main caveats are gathered in Appendix Table 62.
- W2, Q1 (beyond Zhao et al.): the Introduction attributes the separation to Zhao et al. and states that our contribution is empirical; the three results that go further are in Section 4.1 (which methods), Section 4.2 (fixed threshold) and Appendix G.3 (pilot).
- W2, Q1 (why): Section 5 now states that per-model directions cannot show how BF16's mechanism changed, and describes the shift along the frozen BF16 refusal direction (a description, not a mechanism; details in Appendix G.5); Section 8 and Appendix F.4 report the causal test (inconclusive) and its over-refusal control.
- W3, Q2 (augmentation at matched FPR): Section 6.2 reframes augmentation as an input-monitor robustness result with the matched-FPR test; Appendix G.2 (Table 60) has the full analysis, including re-centring.
- Which settings: Section 8 and Appendix F.2.
Thank you for the revision and for running the additional experiments. I appreciate the acknowledgement that augmentation has not been shown to improve safety.
However, I still find the writing unnecessarily difficult to understand. For example, the response states:
"Frozen, a Qwen probe trained on 16 BF16 compliance positives misses a 14-20 point rise in attack success (256 tokens) after pruning."
As a human reader, I genuinely cannot understand this sentence as written. I think it means that a probe trained on the original Qwen model failed to detect the additional harmful answers produced after pruning. If so, why not say that directly? Readers should not have to repeatedly decode technical shorthand to understand what an experiment found. Although the presentation has improved, this remains a serious problem.
I also have two remaining scientific concerns:
-
Novelty: Zhao et al. (2025) already established that recognizing harmful requests and refusing them are different things. The new fixed-threshold experiment provides a useful practical example, but the paper still does not explain why compression changes refusal behaviour. I therefore remain unconvinced that the central finding goes sufficiently beyond previous work.
-
Practical relevance: The clearest reproduced safety failures occur in heavily pruned models that also lose 22%-24% on average across capability benchmarks. The new experiments do not detect comparable safety deterioration in models that preserve their capabilities. This raises an important question: does the finding apply to compressed models that people would realistically deploy, or mainly to models whose overall performance has already deteriorated substantially?
I appreciate the additional work and the willingness to report inconclusive and negative results. The revision addresses some of my original concerns, but the remaining issues with clarity, novelty, and practical relevance are still substantial enough that I cannot support acceptance.
Matched-FPR test: augmentation dropped as a remedy; causal test added
Thank you for the detailed review. We ran your matched-FPR test (augmentation fails it; we drop it as a remedy) and a causal test of the BF16 direction (inconclusive).
Terms used below. BF16 = uncompressed model; frozen probe = BF16-trained, applied unchanged; attack success = judged share of usable harmful answers.
W1 (readability). Agreed. The revision will have defined terms, short sentences and one claim per paragraph; caveats will go in one robustness table. The planned layers (fixed before each family's data) and a single-probe AUROC (chosen post hoc, beside pooled AUROC over all folds' scores) will be primary. Your example will become four plain sentences in Section 4, as changes from BF16.
W2 + Q1. Largely the latter: our main result is the known separation, observed after standard compression with a deployer's frozen probe and judged behaviour. Three results go further (2 and 3 post hoc):
Why (W2, Q1). Not explained. Section 5's per-model directions cannot show how the BF16 mechanism changed, only that 38 of 40 compressed models have one whose negative steering lowers lexically scored refusal. Two new analyses fix the BF16 direction:
Which settings (W2). We pruned OLMo anew at 2:4 and 20-70% unstructured sparsity, with capability benchmarks. Outside 2:4, greedy 256-token attack success rises Holm-significantly only at 60-70% sparsity, which costs 19.5-42.3 benchmark points on average (2:4: 22-24). Increases in three 40-50% builds fail Holm, and the capability-preserving nulls (Wanda 20-30%, SparseGPT 20%) are fragile to missing judgements. At 60% the probe holds at every layer with per-fold AUROC (pooled, both fail at layer 4; SparseGPT 60% loses over 0.10). So in OLMo the pattern does not need 2:4 but comes only with heavy damage. The experiment's analysis rules were finalised after the runs, so it is not pre-registered.
W3 + Q2 (augmentation). Not consistently. In 8 compressed models (194 judge-confirmed harmful 48-token answers), thresholds give 5% FPR on each model's held-out safe prompts. At the planned layer, augmentation changes the catch rate by +0.042 [-0.056, +0.162] over the frozen probe (p = 0.39; approximate). Per model (uncorrected), it gains in the OLMo and Qwen 2:4 models but loses in Llama Wanda 2:4. Sensitivity analysis, 256-token labels: +0.118 [+0.026, +0.189]. Re-centring adds one constant to every score, so at matched FPR its decisions equal the frozen probe's. Its pooled-AUROC edge is an offset correction. Section 6.2 will become an input-monitor robustness result.
If this addresses your concerns, we hope you will consider raising your score.