acceptodds
Under review as a conference paper at ICLR 2027

SafeOPD: Safety-Preserving On-Policy Distillation from Domain-Specialized Teachers

Abstract

On-policy distillation (OPD) has emerged as an effective paradigm for transferring advanced reasoning capabilities from stronger teachers to smaller students. We find, however, that even a small fraction of harmful training prompts can substantially weaken the safety alignment of an already aligned student, despite continued gains in mathematical and medical reasoning capabilities. We term this failure mode on-policy safety forgetting. To address this problem, we introduce SafeOPD, a general OPD framework for safety-preserving post-training of large language models (LLMs). SafeOPD incorporates two key designs. First, we identify safety-relevant states using a small calibration set, locating where teacher supervision may conflict with the student's existing safety behavior. Second, we selectively constrain teacher supervision at these states using the initial aligned student, while preserving the original distillation signal elsewhere. This local intervention preserves the student's safety behavior without broadly weakening capability transfer. Experiments across the domains of mathematical and medical reasoning show that SafeOPD preserves the initial student’s safety alignment while retaining most of the capability gains obtained from the corresponding specialized teacher. Overall, our results highlight the importance of incorporating safety preservation directly into the distillation process, rather than treating safety alignment as a separate post-training stage.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
on-policy distillation, knowledge distillation, safety alignment, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bRemOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies safety degradation on the condition that an aligned student acquires domain capabilities through on-policy distillation from a stronger but also more unsafe teacher. The authors propose SafeOPD, using a small harmful/benign calibration set to identify regions of the teacher’s hidden-state space associated with safety-relevant prefixes. At student-visited prefixes assigned to these regions, it modifies the teacher distribution through a KL-constrained projection toward the frozen initial student.

The paper shows some experiments about SafeOPD, covering mathematical and medical specialization of a Qwen3-4B student. Under the given settings SafeOPD reduces safety errors relative to vanilla on-policy distillation while maintaining the reported capability gains.

Critical strengths
  1. Several experimental results show that SafeOPD is a useful distillation setting and demonstrates consistent mitigation across two domains. The distinction between teacher capability and teacher safety is practically consequential: a domain expert can provide valuable supervision while weakening an already aligned student. As reported, SafeOPD reduces average safety error from 22.58% to 13.71% in mathematics and from 16.35% to 11.76% in medicine. The corresponding reported capability averages increase from 45.12% to 46.00% and from 51.66% to 51.89%. The evaluation includes several safety benchmarks and a benign over-refusal control. These results support a useful improvement over vanilla OPD under the studied teacher–student asymmetry.

  2. The intervention is interpretable and operationally economical. The method reuses the initial student as a safety reference and requires a small calibration set. Its constrained projection has a clear closed-form solution, allowing the strength of the initial-model anchor to vary with the teacher–reference disagreement. The frozen-rollout comparison also provides useful evidence that training on evolving student prefixes contributes to the observed degradation.

Critical weaknesses
  1. Table 6 reports a safety error of 15.44% with random calibration, compared with 22.58% for vanilla OPD and 13.71% for full SafeOPD. Thus, random calibration retains approximately 80.5% of the full method’s mathematical-domain improvement over vanilla OPD. It seems that most of the improvement survives removal of the harmful/benign calibration signal.
    More theoretically, at a fixed prefix, if we write the projected target as p~∝paq01−a\widetilde p \propto p^{a}q_0^{1-a}, where pp is the domain teacher and q0q_0 is the initial student, then DKL(q∥p~)=aDKL(q∥p)+(1−a)DKL(q∥q0)+log⁡ZD_{\mathrm{KL}}(q\|\widetilde p) = aD_{\mathrm{KL}}(q\|p) + (1-a)D_{\mathrm{KL}}(q\|q_0) + \log Z, which suggests a global initial-model KL baseline and random routing matched for intervention frequency and strength. The existing random-calibration ablation does not provide those matched comparisons. Since the author claims that SafeOPD’s specific increment is learned prefix-state routing during specialization, combined with constrained target interpolation, this increment may need direct experimental isolation.

  2. Calculated from reported averages, SafeOPD removes 48.1% of the additional mathematical-domain error and 37.6% of the additional medical-domain error introduced by vanilla OPD. However, there are still 9.58 and 7.63 percentage points remaining. I think a safety/capability curve is important here, which can show or predict that how much specialization is attainable near the initial safety level.

Questions for authors
  1. What safety–capability frontier results from varying the projection radius and routing threshold? In particular, can the method approach the initial 4.13% average safety error while retaining meaningful specialization gains?

  2. Does calibrated state routing outperform a global initial-model KL anchor and frequency/strength-matched random routing at comparable capability?

  3. What fraction of benign and harmful prefixes receives intervention, and how does intervention relate to subsequent harmful continuations? A report of achieved student–reference KL and routing coverage may give a good target-level construction to behavioral safety.

5 comments

We thank Reviewer bRem for the detailed assessment and helpful comments. We address the concerns about state selection and the safety–capability trade-off below.

(W1,Q2): Contribution of calibrated state routing beyond the initial-student anchor and random calibration.

Reply: The random-label control retains coherent benign text and the initial-student constraint, allowing it to provide a partial safety anchor. As shown in Table 2, unstructured text leads to worse capability and safety, whereas informative calibration improves both outcomes over random-label calibration. This supports the additional value of the calibration distribution and safety labels.

We also compare with a global KL constraint applied at every token position. As shown in Table 1, under our chosen ϵ\epsilon and other hyperparameter settings, global constraints substantially limit capability gains in both domains. Together with Table 2, this supports the benefit of selective, informative calibration under the evaluated settings. The random-label control does not match intervention frequency and strength, so it does not fully isolate the effect of state selection.

Table 1: Comparison with global KL regularization across mathematical and medical distillation tasks (%). Higher capability and lower safety scores indicate better performance.

MethodMath Cap. ↑Math Safety ↓Medical Cap. ↑Medical Safety ↓
Initial Student42.674.1347.444.13
Global KL Regularization42.963.8847.704.30
Vanilla OPD45.1222.5850.2116.35
SafeOPD (Ours)46.0013.7150.1311.76

Table 2: Calibration ablations on mathematical distillation (%). Math Avg averages OlympiadBench, SuperGPQA, and AIME25; Safety Avg averages the four safety benchmarks.

Calibration SettingMath Avg ↑Safety Avg ↓
Vanilla OPD (no calibration)35.3222.58
Random-label calibration35.8015.44
Unstructured text + random labels29.5024.41
SafeOPD36.5813.71

(W2,Q1): Safety–capability trade-off under different projection radii and routing thresholds.

Reply: The KL constraint threshold ϵ\epsilon controls how far the supervision target can deviate from the frozen initial student at safety-relevant states. The corresponding interpolation weight αs\alpha_s is determined adaptively at each state. We examine sensitivity to ϵ\epsilon in mathematical distillation.

(1) Sensitivity to the KL Constraint. Increasing ϵ\epsilon generally increases safety error, while average capability remains similar.

(2) Comparison with Hard Replacement. We consider a baseline that switches the source of supervision at safety-relevant states. Let πsafe\pi_{\mathrm{safe}} denote the frozen initial student, πT\pi_T the frozen domain teacher, and πθ\pi_\theta the trainable student. Let πHR\pi_{\mathrm{HR}} denote the supervision target, defined at each student-generated prefix (x,y<t)(x,y_{ < t}) as

\begin{cases} \pi_{\mathrm{safe}}(\cdot\mid x,y_{ < t}), & \text{if }(x,y_{ < t})\text{ is identified as safety-relevant},\\\\ \pi_T(\cdot\mid x,y_{ < t}), & \text{otherwise}. \end{cases}$$ The student is trained with the same reverse-KL distillation objective, using $\pi_{\mathrm{HR}}$ in place of $\pi_T$: $$\mathcal{J}_{\mathrm{HR}}(\theta) =\mathbb{E}_{x\sim\mathcal{D},\\;y\sim\pi_\theta(\cdot\mid x)} \left[ \sum_{t=1}^{T}D_{\mathrm{KL}}\\!\left( \pi_\theta(\cdot\mid x,y_{ < t}) \\,\Vert\\, \pi_{\mathrm{HR}}(\cdot\mid x,y_{ < t}) \right) \right].$$ Thus, the frozen initial student serves as the teacher at safety-relevant states, while the original domain teacher provides supervision elsewhere. This corresponds to $\epsilon=0$ at routed states. The comparison in Table 3 tests whether retaining domain-teacher information through a positive-radius KL projection offers a better safety–capability trade-off than this direct switch.

Table 3: Per-benchmark results for mathematical distillation under different KL constraints (%).

(a) Capability

SettingOlympiadBenchSuperGPQAAIME25 avg@32MMLU-Pro-1KGPQA-DiamondIFBench strictIFEval strictCap. Avg ↑
Hard Replacement (ε = 0)49.7031.7128.8559.8046.9726.3380.0446.20
ε = 0.002 (Ours)49.4133.0327.2959.2046.9725.6780.4146.00
ε = 0.0150.7432.5127.2961.2046.9724.6780.7846.31
ε = 0.0550.4531.5527.6060.3045.4524.0081.5245.84
ε = 0.151.1931.0227.6061.7044.9524.0081.5246.00

(b) Safety.

SettingHarmBench ↓WJ-harmful ↓BeaverTails ↓HEx-Phi ↓Safety Avg ↓WJ-benign ↑
Hard Replacement (ε = 0)11.5633.553.1418.4816.69100.00
ε = 0.002 (Ours)9.3829.252.2913.9413.7199.60
ε = 0.019.3830.802.2913.9414.1099.60
ε = 0.058.7532.352.4314.8514.59100.00
ε = 0.18.7533.802.5714.8514.99100.00

The projection-radius comparison addresses part of the requested trade-off. We do not claim full recovery of the initial safety level.

(Q3): Intervention coverage, student–reference divergence, and subsequent harmful continuations.

Reply: We report intervention coverage and trigger positions separately for clean and harmful training prompts in Table 4. Routing Rate is the fraction of all response positions selected by the router; Constraint Trigger Rate is the fraction of routed positions where KL exceeds ϵ\epsilon. Harmful-prompt trajectories have higher routing and trigger rates, while most clean-prompt positions retain the original teacher supervision. Triggers occur throughout harmful responses and most frequently in the first quarter of clean responses.

Table 4: Analysis of constraint trigger rates and token position distributions during SafeOPD mathematical distillation (%). Trigger rates are measured separately on clean and harmful prompts. Token positions are normalized by response length and divided into four equal intervals.

Prompt TypeRouting Rate (%)Constraint Trigger Rate (%)0–25%25–50%50–75%75–100%
Clean0.0961%12.11%49.1022.9215.9412.05
Harmful5.51%75.59%28.5620.2422.6928.52

We appreciate the reviewer’s thoughtful questions and helpful suggestions, which have helped us better explain the method and provide a more complete assessment of its benefits and limitations.

Dear Reviewer bRem,

We would like to thank you once again for your time and insightful feedback on our work.

We hope that our responses and additional experimental results have satisfactorily addressed your concerns. We would greatly appreciate your feedback on whether any issues remain unresolved.

Should you have any further questions or concerns, we would be more than happy to provide additional clarification. We look forward to continuing our discussion with you.

Thank you again for your consideration.

Reviewer_kyotOpenReview ↗
Rating
2
Confidence
4
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper investigates on-policy safety forgetting, where distillation from a less safety-aligned, domain-specialized teacher can degrade the safety alignment of an initially aligned student. To address this issue, the paper proposes SafeOPD, which identifies safety-relevant states through teacher hidden-state clustering using a small calibration set. At these states, teacher supervision is selectively constrained toward the frozen initial student, while standard OPD is applied elsewhere. Experiments on mathematical and medical reasoning tasks, with artificially contaminated training data, show that SafeOPD mitigates safety degradation compared to vanilla OPD while maintaining comparable capability performance.

Critical strengths
  • This paper addresses on-policy safety forgetting, an important problem where capability distillation can degrade the safety alignment of an already aligned student.

  • The proposed method selectively constrains teacher supervision using the initial student, without requiring an additional safety reward model or safety teacher, while retaining most of the capability gains.

Critical weaknesses
  • Why is safety preservation during OPD preferable to a simple two-stage pipeline consisting of vanilla OPD followed by safety re-alignment? Without comparing against this natural baseline under matched computational and data budgets, accounting for the additional overhead of the proposed method (Offline Safety-state Calibration), the necessity of the proposed framework remains unclear.

  • While artificially mixing harmful and benign data is a common evaluation protocol in the harmful fine-tuning literature, these studies typically motivate the setting through an explicit adversarial threat model (e.g., [1]). In contrast, the proposed setting assumes imperfect filtering of researcher-controlled domain-specific training data without sufficiently justifying why such contamination is unavoidable. Moreover, the paper evaluates only one contamination ratio per domain, rather than systematically varying contamination levels as in prior work. Given that simple data filtering achieves substantially better safety and comparable or better capability (Table 1,2,3,4), the necessity of the proposed framework remains insufficiently established.

[1] https://arxiv.org/abs/2310.03693

  • The calibration ablation provides limited evidence for the effectiveness of safety-relevant state identification. Random calibration already achieves approximately 80% of the safety improvement obtained by SafeOPD over vanilla OPD. This raises the possibility that much of the improvement comes from generic regularization toward the initial student, rather than accurate identification of safety-critical states.
Questions for authors
  1. Is there empirical evidence that high-KL prefixes are visited more frequently during on-policy training and directly contribute to safety forgetting?

  2. How can we verify that the identified teacher hidden-state clusters capture decision-critical prefixes leading to unsafe behavior, rather than merely reflecting harmful prompt semantics?

  3. Given the close relevance of existing localized OPD methods (e.g., [1]) , could the authors provide a direct comparison under the same experimental setting? Such a comparison would help better contextualize the effectiveness and practical advantages of SafeOPD.

[1] https://arxiv.org/abs/2606.02530

4 comments

We appreciate Reviewer kyot’s thoughtful evaluation and constructive suggestions. Below, we respond to the concerns and questions raised in the review.

(W1): Lack of Comparison with Two-Stage Safety Re-alignment

Reply: We added a comparison with Vanilla OPD + SafeSteer, giving both methods access to the same amount of safety data, to test whether a separate safety-alignment stage offers a simpler alternative. Table 1 reports the full computational cost, including SafeOPD’s offline calibration. SafeOPD provides better safety with comparable capability at a lower total cost, supporting the value of addressing safety during distillation.

Table 1: Comparison with two-stage safety re-alignment on mathematical distillation. Computational costs are measured in total GPU-hours, including offline calibration and all training stages. Higher capability and lower safety scores indicate better performance.

MethodCalibration (GPU-h)OPD (GPU-h)Re-alignment (GPU-h)Total (GPU-h) ↓Math Cap. ↑Math Safety ↓
Vanilla OPD + SafeSteer015.262.8018.0645.6920.22
SafeOPD (Ours)0.4015.60016.0046.0013.71

(W2): Insufficient Justification of the Contamination Setting and Advantages over Data Filtering

Reply: In practical post-training, some unsafe prompts may remain in domain-specific datasets even after safety filtering. Our experiments examine this residual contamination. We added experiments with contamination ratios of 1%, 2%, and 5%, keeping the total training data size and optimization settings unchanged. As shown in Table 2, increasing contamination worsens safety and modestly reduces capability for both methods, but SafeOPD consistently achieves better safety and capability, with a smaller decline in safety as contamination increases.

Table 2: Impact of harmful data contamination on mathematical distillation (%). Higher capability and lower safety scores indicate better performance.

Contamination RatioMethodMath Cap. ↑Safety ↓
1%Vanilla OPD45.1222.58
SafeOPD46.0013.71
2%Vanilla OPD44.5824.58
SafeOPD45.5213.98
5%Vanilla OPD44.1527.42
SafeOPD45.1215.43

(W3): Insufficient Evidence for Effective Safety-Relevant State Identification

Reply: Random-label calibration retains coherent benign text and the constraint toward the initial safe student, which can explain its partial safety benefit. Table 3 shows that replacing coherent text with unstructured text worsens both capability and safety, while informative calibration further improves both outcomes over random-label calibration. These results support the value of the calibration distribution and safety labels, although they do not fully isolate the effect of state selection.

Table 3: Calibration ablations on mathematical distillation (%). Math Avg averages OlympiadBench, SuperGPQA, and AIME25; Safety Avg averages the four safety benchmarks.

Calibration SettingMath Avg ↑Safety Avg ↓
Vanilla OPD (no calibration)35.3222.58
Random-label calibration35.8015.44
Unstructured text + random labels29.5024.41
SafeOPD36.5813.71

(Q1): Insufficient empirical evidence linking high-KL prefix visitation to on-policy safety forgetting.

Reply: We clarify that this concern is partly addressed by the controlled comparison in Figure 3 (Section 3.1). The on-policy student exhibits greater safety degradation than the frozen-rollout control, while the KL analysis shows that teacher–student disagreement is concentrated at a small fraction of prefixes. Appendix B provides examples of the resulting harmful compliance, and Appendix C describes the measurement protocol. Together, these findings support the role of evolving rollout distributions in safety forgetting and motivate selective constraints on teacher supervision.

(Q3): Lack of direct comparison with existing localized OPD methods under matched experimental settings.

Reply: We added a direct comparison with SafeSteer [2], applied after Vanilla OPD. Both methods have access to the same amount of safety data and are evaluated under the same protocol. As shown in Table 4, SafeOPD achieves better safety performance with comparable capability. Table 1 also accounts for the full computational cost, including offline calibration, and shows a lower total cost for SafeOPD.

The two methods localize supervision differently: SafeSteer applies reverse-KL supervision to selected safety-related vocabulary tokens from an activation-steered teacher, whereas SafeOPD selects prefix states and constrains the teacher target toward the initial safe student at those states. The comparison supports the practical value of this approach when transferring domain expertise from a less safety-aligned teacher.

Table 4: Comparison with the reproduced SafeSteer baseline on mathematical distillation (%). SafeSteer and SafeOPD use the same amount of safety data and the same evaluation protocol. Higher capability and lower safety scores indicate better performance.

MethodMath Cap. ↑Safety Avg ↓
Initial Student42.674.13
Vanilla OPD45.1222.58
Vanilla OPD + SafeSteer45.6920.22
SafeOPD (Ours)46.0013.71

[2] Li H, An J, Song Z, et al. SafeSteer: Localized On-Policy Distillation for Efficient Safety Alignment. arXiv preprint arXiv:2606.02530, 2026.

Ethics Concern: The manuscript includes examples of harmful instructions.

Reply: We will add a clear content warning before these examples, noting that they contain potentially harmful content and are included solely to analyze model safety behavior.

We are grateful to the reviewer for the careful evaluation and constructive comments, which have guided us in strengthening the experimental analysis and clarifying the scope of our conclusions.

Dear Reviewer kyot,

Thank you again for your valuable time and constructive feedback.

We hope our responses and additional experiments have addressed your concerns. Please let us know if there are any remaining concerns or questions that we could further clarify.

We appreciate your consideration and look forward to further discussion with you.

Thank you to the authors for the detailed rebuttal and additional experiments.

  • [W1] Two-Stage Safety re-alignment. Thank you for the additional comparison with SafeSteer under matched safety-data budgets. The results are convincing and substantially address my concern.

  • [W2] Contamination Setting. The additional results across contamination ratios are encouraging. However, I remain unconvinced of the practical motivation for deliberately contaminated training data, particularly in domain-specific datasets where filtering may be feasible. Naive data filtering still appears preferable, especially in the medical setting (Table 3 & 4) where it achieves both better safety and capability.

  • [W3] Safety-State Identification. The additional ablation is helpful but does not establish the necessity of accurate safety-state identification over simpler regularization. Moreover, the descriptions of random calibration in the original paper and rebuttal appear different despite identical reported results. Clarification of the exact calibration procedure would be helpful.

  • [Q1] High-KL Prefixes. The controlled comparison provides useful supporting evidence for on-policy safety forgetting. However, it does not directly establish that increased visitation of high-KL prefixes causally drives safety degradation.

  • [Q2] Safety-Critical States. This concern remains unresolved. It is still unclear whether the identified clusters capture decision-critical prefixes or merely harmful-prompt semantics.

  • [Q3] Comparison with SafeSteer. I appreciate the direct comparison under matched experimental settings. The results sufficiently address this concern.

Based on the rebuttal results, I'll raise both the Informativeness and Soundness scores from 1 to 2, while keeping my overall rating unchanged.

Reviewer_zbXuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors are dealing with one of the practical problems that arise when using the On-Policy Distillation (OPD). They wonder whether in case when the student model is safer than the teacher at the very beginning of OPD, could teacher's token-level supervision cause degradation of the safety level of the student model. The authors call this process "on-policy safety forgetting," and offer two supporting facts: firstly, Kullback-Leibler (KL) divergence between the teacher and student is highly concentrated on certain prefixes (first 10% of tokens contribute approximately 54% of the total KL); secondly, Attack Success Rate (ASR) grows with rollout depth, and the fastest increase is observed under on-policy sampling.

As the solution to the problem mentioned above, the authors offer a novel method called "SafeOPD." At first, 200 harmful and 200 benign prompts are processed by the teacher model, and then final-layer hidden states obtained as a result are separated into 16 clusters using K-means. The clusters that consist of at least 35% harmful prompts are considered to be "safety-relevant." Then, when training the student model, the authors calculate which cluster corresponds to the prefix: for standard clusters, teacher's distribution is used like in the normal case, while for safety-relevant clusters, teacher's distribution is modified to be closer to the frozen initial student model (by solving the optimization problem under KL constraint, thus getting the geometric interpolation). The experiment was conducted in two fields: mathematics (using DAPO-Math-17K and 181 harmful prompts) and medicine (using II-Medical-RL and 837 harmful prompts). Qwen3-4B-SafeRL model was used as the student model, while 8B-parameter models trained via GRPO were used as the teachers for each domain.

Critical strengths
  1. The problem of asymmetric configuration – where the teacher model is better in terms of capabilities but worse in terms of safety alignment when compared to the student – is an issue that necessarily emerges in practice with the use of OPD. It is shown in Table 1 where the teacher model demonstrates 50.05% ASR on the WJ-harmful benchmark while ASR of the student is 6.20%. Importantly, the authors are not stopping here and are able to distinguish the cause by introducing a control group with "frozen" rollout distribution. They show that while the state distribution is fixed (and thus policy drift becomes a major factor), ASR increases much slower. On the contrary, the on-policy group shows ASR growing rapidly (the increase is 6.1 times greater than initial student's ASR and 1.7 times greater than control group ASR at 256 tokens). The on-policy sampling, therefore, amplifies unsafe behavior. Appendix B mentions that the on-policy student may "refuse a request initially and comply with it further down the sequence" – an important insight for the practice of safety evaluation.

  2. Rather than doing the safety alignment separately after the training is completed, the paper introduces intervention directly during the distillation process – precisely in the moment when teacher's supervision is applied – which, conceptually, is the right way to do it. This method requires just 400 offline calibration prompts and eliminates the need for safety-specific preferences data, safety reward model and any extra "safety-aligned" teachers. Moreover, the cluster assignments remain frozen through training process and the calibration dataset is never replayed – such cost structure is very convenient from the engineering standpoint. As far as I know, the particular method – of prefixes clustering based on teacher's hidden states and corresponding weighting of supervision – is novel.

Critical weaknesses
  1. Although "KL Projection" may seem complicated, it in fact consists of a weighted combination of two terms involving KL-divergence, and the genuinely innovative part of the method—the choice of weights on a per-state basis—is never tested using an ablation study.

Set αs\alpha_s equal to 1/(1+λs)1/(1+\lambda_s). When this is substituted back into the single-state loss given by Equation 9, an exact identity results (a fact which I have confirmed numerically; the residual is of the order of machine precision).

DKL(qs∥p~s)=αs⋅DKL(qs∥ps)+(1−αs)⋅DKL(qs∥q0,s)+log⁡ZsD_{KL}(q_s \parallel \tilde{p}_s) = \alpha_s \cdot D_{KL}(q_s \parallel p_s) + (1-\alpha_s) \cdot D_{KL}(q_s \parallel q_{0,s}) + \log Z_s

In this case, log⁡Zs=(αs−1)⋅Dαs(ps∥q0,s)\log Z_s = (\alpha_s-1) \cdot D_{\alpha_s}(p_s \parallel q_{0,s})—that is, a Rényi divergence term—which, for a given prefix, has no dependence on θ\theta. As a result, the training signal can be viewed as comprising two parts: one that replicates the teacher and another that remains near the original 'safe' student, the weights being determined by the present state. The 'constrained projection' does not add any new supervisory signal apart from these two components.

Now, it is true that the paper does make a non-trivial incremental contribution by selecting αs\alpha_s on a per-state basis through clustering of prefixes. However, this particular contribution is never tested by removing it. A vital baseline that is missing is "global regularization"—that is, applying a fixed α\alpha to all states—in order to assess how much real benefit is obtained through per-state routing. Moreover, the value ϵ=0.002\epsilon=0.002 is very tight (this corresponds to αs≈10−2\alpha_s \approx 10^{-2}, which means the teacher is left with only a few percent of the weight). In practice, this causes the method to act almost like a hard switch between the teacher and the initial student; yet the only tunable parameter, ϵ\epsilon, is never subjected to a hyperparameter sweep.

  1. In the medical field, the approach involving simple data filtering performs better than the one proposed in terms of both safety (it provides better safety) and capability (it achieves higher performance)—a point which the paper conveniently omits.

If we look at Tables 3 and 4, the filtering approach yields a Safety Avg of 7.17% and a Medical Avg of 52.02%, while SafeOPD reaches 11.76% and 51.89% respectively. Section 4.2.2 compares only with V-OPD and says nothing regarding the 'Filtering' column. In the mathematics domain the results are also similar in that the safety level under 'Filtering' is almost completely kept at 4.40% compared to the initial 4.13%, with a loss in performance of just 1.36 points. The paper's main value proposition can be summed up as 'trading 9.31 points of safety for 1.36 points of capability'—yet the abstract states that it 'preserves' safety. Since the 'Safety Avg' has increased by a factor of 3.3, this is an instance of 'partial mitigation', not 'preservation'.

  1. The results are derived from single runs without any analysis of variance; the differences observed are probably merely due to noise.

There are no random seeds and no error bars; GPQA-Diamond comprises just 198 questions, which results in a standard error (SE) of about 3.5% per run, yet the three methods differ by 9 points in this column, a gap greater than any of the differences in 'Capability Avg' throughout the paper. Moreover, the training process did not finish a full epoch (it only reached about 44%), so of the 181 harmful prompts, each one would have been encountered only around 80 times; therefore, changing the random seed would very likely cause substantial fluctuations.

Questions for authors

Let αs=1/(1+λs)\alpha_s = 1/(1+\lambda_s). Can you confirm whether DKL(qs∥p~s)=αs⋅DKL(qs∥ps)+(1−αs)⋅DKL(qs∥q0,s)+log⁡ZsD_{KL}(q_s \parallel \tilde{p}_s) = \alpha_s \cdot D_{KL}(q_s \parallel p_s) + (1-\alpha_s) \cdot D_{KL}(q_s \parallel q_{0,s}) + \log Z_s holds? If it does, please include a comparison with a "globally fixed α\alpha regularization" baseline—quantify exactly how much is gained by selecting weights on a per-state basis. In the medical domain, "Filtering" outperforms SafeOPD in both safety and capability; in the mathematics domain, SafeOPD trades 9.31 points of safety for 1.36 points of capability. How should the practical value of SafeOPD be interpreted? Please perform a sweep of ϵ\epsilon (from 10−410^{-4} to 10−110^{-1}) and report the actual distribution of αs\alpha_s for safe states. At ϵ=0.002\epsilon = 0.002, what percentage of the original teacher weight remains, and how does this compare to a hard replacement with q0,sq_{0,s}? Please report the constraint trigger rate and the distribution of token positions where triggers occur, specifically for "clean" prompts. Additionally, please include cross-source calibration experiments (e.g., calibrating on BeaverTails and evaluating on HarmBench). Please run experiments with multiple seeds and report the mean and standard deviation. Are the differences in "Capability Avg" between the groups statistically significant?

6 comments

We sincerely thank Reviewer zbXu for the careful reading, positive assessment, and constructive feedback. We address each concern point-by-point below.

(W1,Q1): Algebraic equivalence of KL projection and its implications for novelty.

Reply: We agree that, at a fixed prefix, the projected-target loss admits the stated weighted-KL decomposition, with the normalization term independent of the trainable student parameters. Thus, the projection does not introduce an additional supervision source beyond the domain teacher and the frozen initial student. Our contribution lies in representation-based calibration to identify safety-relevant states and constraint-determined, state-dependent weighting of these two supervision sources. SafeOPD applies the KL-constrained projection only at the identified states and preserves the original teacher supervision elsewhere; at each selected state, the interpolation coefficient is determined adaptively by the constraint rather than specified as a fixed regularization weight.

(W1,Q2): Ablation of state-dependent regularization against a global initial-student KL anchor.

Reply: To assess the importance of selective constraints, we additionally evaluate a global KL-constrained baseline that applies the constraint at every token position. As shown in Table 1, under our chosen ϵ\epsilon and other hyperparameter settings, applying the KL constraint globally substantially limits capability gains in both domains.

Table 1: Comparison with global KL regularization across mathematical and medical distillation tasks (%). Higher capability and lower safety scores indicate better performance.

MethodMath Cap. ↑Math Safety ↓Medical Cap. ↑Medical Safety ↓
Initial Student42.674.1347.444.13
Global KL Regularization42.963.8847.704.30
Vanilla OPD45.1222.5850.2116.35
SafeOPD (Ours)46.0013.7150.1311.76

(W1,Q4): Sensitivity to the KL constraint threshold.

Reply: The KL constraint threshold ϵ\epsilon controls how far the supervision target can deviate from the frozen initial student at safety-relevant states. The corresponding interpolation weight αs\alpha_s is determined adaptively at each state. We examine sensitivity to ϵ\epsilon in mathematical distillation.

Sensitivity to the KL Constraint. Increasing ϵ\epsilon generally increases safety error, while average capability remains similar.

(W1,Q5): Continuous projection versus hard replacement.

Reply: We consider a baseline that switches the source of supervision at safety-relevant states. Let πsafe\pi_{\mathrm{safe}} denote the frozen initial student, πT\pi_T the frozen domain teacher, and πθ\pi_\theta the trainable student. Let πHR\pi_{\mathrm{HR}} denote the supervision target, defined at each student-generated prefix (x,y<t)(x,y_{ < t}) as

\begin{cases} \pi_{\mathrm{safe}}(\cdot\mid x,y_{ < t}), & \text{if }(x,y_{ < t})\text{ is identified as safety-relevant},\\\\ \pi_T(\cdot\mid x,y_{ < t}), & \text{otherwise}. \end{cases}$$ The student is trained with the same reverse-KL distillation objective, using $\pi_{\mathrm{HR}}$ in place of $\pi_T$: $$\mathcal{J}_{\mathrm{HR}}(\theta) =\mathbb{E}_{x\sim\mathcal{D},\\;y\sim\pi_\theta(\cdot\mid x)} \left[ \sum_{t=1}^{T}D_{\mathrm{KL}}\\!\left( \pi_\theta(\cdot\mid x,y_{ < t}) \\,\Vert\\, \pi_{\mathrm{HR}}(\cdot\mid x,y_{ < t}) \right) \right].$$ Thus, the frozen initial student serves as the teacher at safety-relevant states, while the original domain teacher provides supervision elsewhere. This corresponds to $\epsilon=0$ at routed states. As shown in Table 2, SafeOPD achieves better safety performance than Hard Replacement while maintaining comparable capability. **Table 2: Per-benchmark results for mathematical distillation under different KL constraints (%).** **(a) Capability** | Setting | OlympiadBench | SuperGPQA | AIME25 avg@32 | MMLU-Pro-1K | GPQA-Diamond | IFBench strict | IFEval strict | Cap. Avg ↑ | | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | Hard Replacement (ε = 0) | 49.70 | 31.71 | **28.85** | 59.80 | **46.97** | **26.33** | 80.04 | 46.20 | | **ε = 0.002 (Ours)** | 49.41 | **33.03** | 27.29 | 59.20 | **46.97** | 25.67 | 80.41 | 46.00 | | ε = 0.01 | 50.74 | 32.51 | 27.29 | 61.20| **46.97** | 24.67 | 80.78 | **46.31** | | ε = 0.05 | 50.45 | 31.55 | 27.60 | 60.30 | 45.45 | 24.00 | **81.52**| 45.84 | | ε = 0.1 | **51.19** | 31.02 | 27.60 | **61.70** | 44.95 | 24.00 | **81.52** | 46.00 | **(b) Safety.** | Setting | HarmBench ↓ | WJ-harmful ↓ | BeaverTails ↓ | HEx-Phi ↓ | Safety Avg ↓ | WJ-benign ↑ | | --- | ---: | ---: | ---: | ---: | ---: | ---: | | Hard Replacement (ε = 0) | 11.56 | 33.55 | 3.14 | 18.48 | 16.69 | **100.00** | | **ε = 0.002 (Ours)** | 9.38 | **29.25** | **2.29** | **13.94** | **13.71** | 99.60 | | ε = 0.01 | 9.38 | 30.80 | **2.29** | **13.94** | 14.10 | 99.60 | | ε = 0.05 | **8.75** | 32.35 | 2.43 | 14.85 | 14.59 | **100.00** | | ε = 0.1 | **8.75** | 33.80 | 2.57 | 14.85 | 14.99 | **100.00** |

(W1,Q6): Constraint trigger rates and token positions, particularly on clean prompts.

Reply: We analyze intervention coverage and token positions during mathematical distillation, separately for clean and harmful training prompts. Routing Rate measures the proportion of all response-token positions selected by the safety-state router; Constraint Trigger Rate measures the proportion of these selected positions at which KL exceeds ε and projection is required.

As shown in Table 3, harmful prompts have higher routing coverage and a larger proportion of routed positions that trigger the constraint than clean prompts. Intervention is therefore more frequent on harmful-prompt trajectories, while the overwhelming majority of positions on clean-prompt trajectories retain the original teacher supervision.

We also examine where the constraint is triggered within a response. For clean prompts, triggers occur most frequently in the first quarter, while harmful-prompt trajectories show triggers throughout the response. This analysis describes where the constraint is activated and complements the paper’s early-token analysis of the magnitude of teacher–student disagreement.

Table 3: Analysis of constraint trigger rates and token position distributions during SafeOPD mathematical distillation (%). Trigger rates are measured separately on clean and harmful prompts. Token positions are normalized by response length and divided into four equal intervals.

Prompt TypeRouting Rate (%)Constraint Trigger Rate (%)0–25%25–50%50–75%75–100%
Clean0.0961%12.11%49.1022.9215.9412.05
Harmful5.51%75.59%28.5620.2422.6928.52

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.