- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies safety degradation on the condition that an aligned student acquires domain capabilities through on-policy distillation from a stronger but also more unsafe teacher. The authors propose SafeOPD, using a small harmful/benign calibration set to identify regions of the teacher’s hidden-state space associated with safety-relevant prefixes. At student-visited prefixes assigned to these regions, it modifies the teacher distribution through a KL-constrained projection toward the frozen initial student.
The paper shows some experiments about SafeOPD, covering mathematical and medical specialization of a Qwen3-4B student. Under the given settings SafeOPD reduces safety errors relative to vanilla on-policy distillation while maintaining the reported capability gains.
Critical strengths
-
Several experimental results show that SafeOPD is a useful distillation setting and demonstrates consistent mitigation across two domains. The distinction between teacher capability and teacher safety is practically consequential: a domain expert can provide valuable supervision while weakening an already aligned student. As reported, SafeOPD reduces average safety error from 22.58% to 13.71% in mathematics and from 16.35% to 11.76% in medicine. The corresponding reported capability averages increase from 45.12% to 46.00% and from 51.66% to 51.89%. The evaluation includes several safety benchmarks and a benign over-refusal control. These results support a useful improvement over vanilla OPD under the studied teacher–student asymmetry.
-
The intervention is interpretable and operationally economical. The method reuses the initial student as a safety reference and requires a small calibration set. Its constrained projection has a clear closed-form solution, allowing the strength of the initial-model anchor to vary with the teacher–reference disagreement. The frozen-rollout comparison also provides useful evidence that training on evolving student prefixes contributes to the observed degradation.
Critical weaknesses
-
Table 6 reports a safety error of 15.44% with random calibration, compared with 22.58% for vanilla OPD and 13.71% for full SafeOPD. Thus, random calibration retains approximately 80.5% of the full method’s mathematical-domain improvement over vanilla OPD. It seems that most of the improvement survives removal of the harmful/benign calibration signal.
More theoretically, at a fixed prefix, if we write the projected target as , where is the domain teacher and is the initial student, then , which suggests a global initial-model KL baseline and random routing matched for intervention frequency and strength. The existing random-calibration ablation does not provide those matched comparisons. Since the author claims that SafeOPD’s specific increment is learned prefix-state routing during specialization, combined with constrained target interpolation, this increment may need direct experimental isolation. -
Calculated from reported averages, SafeOPD removes 48.1% of the additional mathematical-domain error and 37.6% of the additional medical-domain error introduced by vanilla OPD. However, there are still 9.58 and 7.63 percentage points remaining. I think a safety/capability curve is important here, which can show or predict that how much specialization is attainable near the initial safety level.
Questions for authors
-
What safety–capability frontier results from varying the projection radius and routing threshold? In particular, can the method approach the initial 4.13% average safety error while retaining meaningful specialization gains?
-
Does calibrated state routing outperform a global initial-model KL anchor and frequency/strength-matched random routing at comparable capability?
-
What fraction of benign and harmful prefixes receives intervention, and how does intervention relate to subsequent harmful continuations? A report of achieved student–reference KL and routing coverage may give a good target-level construction to behavioral safety.
5 comments
Table 3: Per-benchmark results for mathematical distillation under different KL constraints (%).
(a) Capability
| Setting | OlympiadBench | SuperGPQA | AIME25 avg@32 | MMLU-Pro-1K | GPQA-Diamond | IFBench strict | IFEval strict | Cap. Avg ↑ |
|---|---|---|---|---|---|---|---|---|
| Hard Replacement (ε = 0) | 49.70 | 31.71 | 28.85 | 59.80 | 46.97 | 26.33 | 80.04 | 46.20 |
| ε = 0.002 (Ours) | 49.41 | 33.03 | 27.29 | 59.20 | 46.97 | 25.67 | 80.41 | 46.00 |
| ε = 0.01 | 50.74 | 32.51 | 27.29 | 61.20 | 46.97 | 24.67 | 80.78 | 46.31 |
| ε = 0.05 | 50.45 | 31.55 | 27.60 | 60.30 | 45.45 | 24.00 | 81.52 | 45.84 |
| ε = 0.1 | 51.19 | 31.02 | 27.60 | 61.70 | 44.95 | 24.00 | 81.52 | 46.00 |
(b) Safety.
| Setting | HarmBench ↓ | WJ-harmful ↓ | BeaverTails ↓ | HEx-Phi ↓ | Safety Avg ↓ | WJ-benign ↑ |
|---|---|---|---|---|---|---|
| Hard Replacement (ε = 0) | 11.56 | 33.55 | 3.14 | 18.48 | 16.69 | 100.00 |
| ε = 0.002 (Ours) | 9.38 | 29.25 | 2.29 | 13.94 | 13.71 | 99.60 |
| ε = 0.01 | 9.38 | 30.80 | 2.29 | 13.94 | 14.10 | 99.60 |
| ε = 0.05 | 8.75 | 32.35 | 2.43 | 14.85 | 14.59 | 100.00 |
| ε = 0.1 | 8.75 | 33.80 | 2.57 | 14.85 | 14.99 | 100.00 |
The projection-radius comparison addresses part of the requested trade-off. We do not claim full recovery of the initial safety level.
(Q3): Intervention coverage, student–reference divergence, and subsequent harmful continuations.
Reply: We report intervention coverage and trigger positions separately for clean and harmful training prompts in Table 4. Routing Rate is the fraction of all response positions selected by the router; Constraint Trigger Rate is the fraction of routed positions where KL exceeds . Harmful-prompt trajectories have higher routing and trigger rates, while most clean-prompt positions retain the original teacher supervision. Triggers occur throughout harmful responses and most frequently in the first quarter of clean responses.
Table 4: Analysis of constraint trigger rates and token position distributions during SafeOPD mathematical distillation (%). Trigger rates are measured separately on clean and harmful prompts. Token positions are normalized by response length and divided into four equal intervals.
| Prompt Type | Routing Rate (%) | Constraint Trigger Rate (%) | 0–25% | 25–50% | 50–75% | 75–100% |
|---|---|---|---|---|---|---|
| Clean | 0.0961% | 12.11% | 49.10 | 22.92 | 15.94 | 12.05 |
| Harmful | 5.51% | 75.59% | 28.56 | 20.24 | 22.69 | 28.52 |
We appreciate the reviewer’s thoughtful questions and helpful suggestions, which have helped us better explain the method and provide a more complete assessment of its benefits and limitations.
Dear Reviewer bRem,
We would like to thank you once again for your time and insightful feedback on our work.
We hope that our responses and additional experimental results have satisfactorily addressed your concerns. We would greatly appreciate your feedback on whether any issues remain unresolved.
Should you have any further questions or concerns, we would be more than happy to provide additional clarification. We look forward to continuing our discussion with you.
Thank you again for your consideration.
We thank Reviewer
bRemfor the detailed assessment and helpful comments. We address the concerns about state selection and the safety–capability trade-off below.Reply: The random-label control retains coherent benign text and the initial-student constraint, allowing it to provide a partial safety anchor. As shown in Table 2, unstructured text leads to worse capability and safety, whereas informative calibration improves both outcomes over random-label calibration. This supports the additional value of the calibration distribution and safety labels.
We also compare with a global KL constraint applied at every token position. As shown in Table 1, under our chosen ϵ and other hyperparameter settings, global constraints substantially limit capability gains in both domains. Together with Table 2, this supports the benefit of selective, informative calibration under the evaluated settings. The random-label control does not match intervention frequency and strength, so it does not fully isolate the effect of state selection.
Table 1: Comparison with global KL regularization across mathematical and medical distillation tasks (%). Higher capability and lower safety scores indicate better performance.
Table 2: Calibration ablations on mathematical distillation (%). Math Avg averages OlympiadBench, SuperGPQA, and AIME25; Safety Avg averages the four safety benchmarks.
Reply: The KL constraint threshold ϵ controls how far the supervision target can deviate from the frozen initial student at safety-relevant states. The corresponding interpolation weight αs is determined adaptively at each state. We examine sensitivity to ϵ in mathematical distillation.
(1) Sensitivity to the KL Constraint. Increasing ϵ generally increases safety error, while average capability remains similar.
(2) Comparison with Hard Replacement. We consider a baseline that switches the source of supervision at safety-relevant states. Let πsafe denote the frozen initial student, πT the frozen domain teacher, and πθ the trainable student. Let πHR denote the supervision target, defined at each student-generated prefix (x,y<t) as
\begin{cases} \pi_{\mathrm{safe}}(\cdot\mid x,y_{ < t}), & \text{if }(x,y_{ < t})\text{ is identified as safety-relevant},\\\\ \pi_T(\cdot\mid x,y_{ < t}), & \text{otherwise}. \end{cases}$$ The student is trained with the same reverse-KL distillation objective, using $\pi_{\mathrm{HR}}$ in place of $\pi_T$: $$\mathcal{J}_{\mathrm{HR}}(\theta) =\mathbb{E}_{x\sim\mathcal{D},\\;y\sim\pi_\theta(\cdot\mid x)} \left[ \sum_{t=1}^{T}D_{\mathrm{KL}}\\!\left( \pi_\theta(\cdot\mid x,y_{ < t}) \\,\Vert\\, \pi_{\mathrm{HR}}(\cdot\mid x,y_{ < t}) \right) \right].$$ Thus, the frozen initial student serves as the teacher at safety-relevant states, while the original domain teacher provides supervision elsewhere. This corresponds to $\epsilon=0$ at routed states. The comparison in Table 3 tests whether retaining domain-teacher information through a positive-radius KL projection offers a better safety–capability trade-off than this direct switch.