- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper explains the reason why post-hoc safety is fragile compared with the safety knowledge learned in the pretraining stage. Using a weight-space classification, the author found that most of the safety training in the post-training stage falls in the suppression region instead of the erasure region, explaining why post-hoc safety is so fragile. The author also proposes a kernel-immobility lemma to explain why post-hoc safety masks rather than removes the capability. In the end, the author proposes a mitigation strategy that uses contibuous safety co-training, supassing the baseline result.
Critical strengths
- The question of explaining the robustness of safety training in the pretraining stage and post-training is interesting and important.
- The experiments are rigorous and comprehensive.
Critical weaknesses
- The paper clarity is a very big issue. The writing is poor and hard to follow. Specifically, the paper uses a lot of compressed short paraphrases and headers (e.g., in line 195 , what's the meaning of " a diffuse one"??, in line 247, "100 benign fine-tuning examples cost Qwen 34.8 pp of AdvBench refusal (Fig. 1b, top row)", what's the meaning of cost xx refusal??? ), with a lot of terminology used without definition. For example, what exactly is the meaning of "safety install"? What's the meaning of "geometric instrument"? In line 237, what's the meaning of peaks? If we really want to find a peak value, why is the safety-install band an interval instead of a specific layer? What is "gate-attack" in figure1?
- The headline results is just an artifact of using regex safety checker. In figrue 1, the author claimed that compared with the basline model, whose refusal rate dropped by 35% and 38% for Qwen and Llama models, cotrained model has less than 6% drop of refusal rate. However, this results comes from the regex-based checker (see line 075). In the full results listed in table 25 and 26 (line 1637, 1638), if we use classifier based approach, the refusal rate drop for Qwen and Llama is merely 7.8% and 11.8, while for cotrained model, the refusal rate drop becomes 10.6% to 27.0%, even worse than the baseline.
Questions for authors
I have listed my questions in the weakness section