- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to effectively apply the training paradigm of on-policy self-distillation (OPSD) for safety alignment while preserving utility by reducing over-refusals. The authors claim the existing OPSD safety alignment methods (such as OPSA) condition a frozen teacher model on coarse, label-level privileged information (a hard refusal for harmful queries and a direct-answer instruction for benign ones), resulting in increased over-refusal rate (ORR) and degradation in safe completion rate (SCR) on borderline health requests. The paper proposes Query-Specific Safety Rubrics in OPSD (QSSR-OPSD) to address these failure modes, using query-specific safety rubrics built dynamically for each query from 100 RuleAdapter rules by GPT-5.4. The teacher then sees a meta instruction, a label-conditioned safety statement, and the rubrics, while the student sees only the query. The authors argue, via a KL-regularized penalty minimization framework, that operationalizing principles into rubrics shifts probability mass toward safe, useful responses primarily by suppressing unsafe completions. Across four models and various alignment methods, QSSR-OPSD achieves the highest composite safety score (CSS) and SCR while preserving general reasoning performance.
Critical strengths
-
The paper offers a comprehensive, strong evaluation and shows clear improvement on the important safety-alignment trade-off. The authors consider harmful compliance, over-refusal, and safe completion together rather than in isolation. The evaluation covers four models with different sizes and model families across diverse safety datasets, which makes the results more reliable.
-
The paper provides useful evidence on how privileged information (PI) design can affect OPSD's effect on safety alignment. The ablation of PI and the comparison of rubric-conditioned SFT and OPSD may give practical guidance to future works applying OPSD to safety alignment.
Critical weaknesses
-
The measurement of utility is limited. Helpfulness is assessed through a single health benchmark via SCR. On general benign queries, ORR records only whether the model refused, so a vague or incorrect answer counts as success. On harmful queries, the label policy asks for detailed safe alternatives, but the trained student is not evaluated on whether it provides them. Evaluating answer quality on general benign and harmful queries would directly test the paper’s main claim.
-
QSSR-OPSD receives supervision that other OPSD-based baselines do not. It adds per-query guidance generated by GPT-5.4 across the training set, so part of its advantages may come from distilling a stronger model’s judgment rather than from the method itself. The self-generated rubric ablation (Table 10) is encouraging but is reported only for Qwen3-8B, so it remains unclear how much the gains depend on the external model.
Questions for authors
-
Can you evaluate answer quality on general benign queries and on harmful queries outside health? This would directly test whether the student provides useful safe alternatives.
-
Can you report self-generated rubric results on the other backbones? How much does the performance depend on a strong external model like GPT-5.4?