acceptodds
Under review as a conference paper at ICLR 2027

Recovering General Capabilities via Uncertainty-Calibrated Multi-Teacher On-Policy Distillation

Abstract

Specializing large language models to vertical domains improves domain-specific behavior but often degrades general capabilities. We study this trade-off in Multi-Teacher On-Policy Distillation (MOPD), where a specialized model learns from domain and general teachers on its own sampled trajectories. Standard MOPD faces two limitations: ordinary on-policy sampling rarely exposes tokens with large positive teacher–student advantages, and advantage sign alone does not establish whether the proposed update direction is reliable. We propose Uncertainty-Calibrated MOPD (UCMOPD), which addresses these limitations through two complementary mechanisms. Golden-Gain Enhancement combines higher-temperature exploration with a standard-temperature anchor and retains trajectories whose positive learning signal matches or exceeds the prompt-specific anchor. Teacher-Endorsement Filtering then uses centered log-likelihood (CLL) to estimate each retained token's plausibility relative to the teacher's uncertainty and probabilistically preserves updates whose directions are supported by that endorsement. Across role-playing and medical-domain specialization, UCMOPD improves the general-capability average over standard MOPD by and , respectively, while maintaining vertical-domain performance. Component ablations and diagnostic analyses support the intended roles of the two mechanisms: exposing and selecting stronger positive signals at the trajectory level and validating update directions through teacher endorsement at the token level.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, multi-teacher distillation, domain specialization, general capability recovery, uncertainty calibration, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kYJXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper improves Multi-Teacher On-Policy Distillation (MOPD) using an uncertainty quantification method. It points out that MOPD does not sufficiently explore tokens with high positive advantages and that it is difficult to determine the appropriate learning direction based only on the sign of the teacher–student log-likelihood ratio. The proposed Uncertainty-Calibrated MOPD (UCMOPD) has two components. First, Golden-Gain Enhancement generates diverse trajectories through high-temperature sampling and selects useful trajectories based on positive advantages. Second, Teacher-Endorsement Filtering evaluates the reliability of token-level updates using Centered Log-Likelihood (CLL), which combines teacher entropy and log-likelihood, and probabilistically filters the learning signals. The method is compared with MOPD-based baselines in role-playing and medical-domain specialization experiments.

Critical strengths
  1. I think the motivation is very clear, specially Figure 1 concretely illustrates the problems that can arise when determining the appropriateness of token-level updates based only on the sign of the advantage in existing MOPD.

  2. Methodology looks simple and intuiative. Sections 3.2 and 3.3 effectively utilize learning signals by combining trajectory selection through high-temperature exploration with teacher entropy-based token filtering.

  3. The initial experimental results looks promising. Table 2 shows that the proposed method achieves higher general capability recovery than existing MOPD while maintaining or improving domain-specific performance in both role-playing and medical specialization.

  4. Ablation studies Tables 3–4 and Appendix Table 7 looks proper to validate the effectiveness of each component and show that the performance improvements are maintained even under the same rollout budget.

Critical weaknesses
  1. The proposed method seems to be limited to heuristic weight adjustments. For example, see Eq. 6 of DistLLM (ICML 2024) and Section 3.2 of CSD (ICLR 2026). Their gradients also differ from those of existing losses (e.g., KL), but they do not directly modify the gradients. Instead, they propose different loss functions that converge to the target distribution and analyze their gradients. In contrast, simply adjusting gradient weights, as in the proposed method, does not guarantee matching to the original target distribution. In other words, what is the optimal solution of the student model when Algorithm 1 is optimized to convergence?

  2. The proposed method is developed based on the LMOPDL_{MOPD} loss, which is reverse-KL-based, as mentioned in line 201. Could it also be applied to other divergences proposed in DistiLLM or CSD?

  3. In Table 2, the proposed method shows large improvements on some benchmarks but performs worse than existing methods on others. For example, in the medical setting, IF-Eval improves significantly, but IFBench shows no improvement. In the role-playing setting, LiveCodeBench v5 improves, while AIME25 performs worse than the original MOPD. Further analysis and discussion are needed to explain which benchmark characteristics cause these differences and which types of capability recovery the proposed method is particularly effective for.

Questions for authors

Please see weakness section.

6 comments
AuthorsOpenReview ↗

Response to Reviewer kYJX (1)

Thank you for recognizing the motivation, the simplicity of the method, and the component and rollout-budget comparisons. We appreciate your questions about the optimization target, compatibility with other distillation losses, and differences across benchmarks. We address them separately below.

W1. What is the optimal student distribution when Algorithm 1 converges?

We agree that changing gradient weights requires analyzing the resulting update, rather than assuming that the original objective is preserved. To answer your question precisely: in a realizable setting with independent logits at each prefix, the routed teacher is the unique interior stationary distribution of Algorithm 1's expected distillation update, under the conditions below.

First, consider the complete algorithm at a fixed prefix, with a uniquely defined, full-support routed teacher distribution pp and student distribution q=softmax⁡(z)q=\operatorname{softmax}(z). Assume independent conditional logits, positive loss averaging, and no auxiliary training objective. Write the expected descent direction as

b_i=\kappa_i(q)\log\frac{p_i}{q_i},$$ where $\kappa_i\geq0$ incorporates token exposure, trajectory selection, CLL retention, and positive averaging factors. Importantly, $\kappa_i>0$ whenever $q_i<p_i$: Algorithm 1 always retains a temperature-1, top-p-1 anchor, and its positive-advantage CLL keep probability $s_i=\min(p_i e^{H[p]},1)$ is strictly positive. If $q\neq p$, normalization implies that some token is underestimated and another is overestimated. Consequently, some $b_i>0$, while some $b_j\leq0$. But stationarity $u=0$ requires $b_i/q_i=\sum_j b_j$ for every token, which is impossible with these different signs. At $q=p$, all advantages vanish. Thus, **$q^*=p$ is the only interior stationary distribution in this idealized setting**, including when some negative-advantage updates are always filtered. The result applies at prefixes with positive anchor visitation; it does not constrain unseen prompts. Second, an explicit objective can be given for the isolated CLL component. At a fixed prefix, with fresh samples $i\sim q$, no trajectory selection, and normalization independent of the Bernoulli masks, define $$a_i(q_i)= \begin{cases} s_i,&q_i\leq p_i,\\ 1-s_i,&q_i>p_i, \end{cases} \qquad F(q;p)=\sum_i a_i(q_i) \left[q_i\log\frac{q_i}{p_i}-q_i+p_i\right].$$ Its gradient equals the expected CLL-masked surrogate gradient: $$\nabla_\theta F =-\mathbb E_{i\sim q}\left[ a_i(q_i)\log\frac{p_i}{q_i}\nabla_\theta\log q_i \right].$$ The branch boundary is differentiable because both the bracketed term and its derivative vanish at $q_i=p_i$. Each summand is nonnegative. If $q\neq p$, at least one underestimated token has $a_i=s_i>0$ and contributes strictly positively. Therefore, $F$ has the unique global minimizer $q=p$ on the probability simplex. **Therefore, under the idealized conditions above, if Algorithm 1's expected update converges to an interior stationary point, the student distribution must match the corresponding teacher distribution. For the isolated CLL component, we additionally provide an explicit objective with the same unique optimal distribution.** We will add these derivations and their conditions to the appendix.
AuthorsOpenReview ↗

Response to Reviewer kYJX (2)

W2. Can the method extend to the losses in DistiLLM or CSD?

The candidate-generation framework and teacher-only endorsement score are transferable, whereas trajectory scoring and direction-dependent filtering require analysis under the new loss and adaptation where necessary. Specifically, high-temperature exploration and a standard-temperature anchor can generate candidates independently of the distillation loss. CLL's endorsement sis_i also depends only on the teacher. In contrast, Golden-Gain currently scores trajectories using positive reverse-KL log-probability gaps, and CLL's signed keep rule uses their proposed update directions. These two parts require loss-specific analysis and adaptation where necessary.

For DistiLLM, the losses are skew KL and skew reverse KL:

\qquad D_{\rm SRKL}=D_{\rm KL}(q\|n),\quad n=(1-\alpha)p+\alpha q.$$ At a fixed prefix, their gradients admit the score-function representation $\nabla_\theta D=-\mathbb E_{i\sim q}[A_i\nabla_\theta\log q_i]$, with coefficients $$A_i^{\rm SKL}=(1-\alpha)\left(\frac{p_i}{m_i}-1\right), \qquad A_i^{\rm SRKL}=\log\frac{n_i}{q_i} +\alpha\left(\frac{q_i}{n_i}-1\right).$$ Here, constant baselines are chosen so that the coefficients vanish at $q=p$. For $0\leq\alpha<1$, both coefficients have the same sign as $p_i-q_i$, although their magnitudes differ from $\log(p_i/q_i)$. Thus, CLL's teacher-only endorsement score and signed keep rule can be applied to these derived coefficients. To make the filtered optimization objectives explicit, define the following per-token scalar terms, with $p_i>0$ and $q_i>0$: $$\begin{aligned} \phi_i^{\rm SKL}(q_i;p_i) &=p_i\log\frac{p_i}{m_i}+(1-\alpha)(q_i-p_i),\\ \phi_i^{\rm SRKL}(q_i;p_i) &=q_i\log\frac{q_i}{n_i}-(1-\alpha)(q_i-p_i). \end{aligned}$$ The added linear terms sum to zero on the probability simplex, so without filtering, $\sum_i\phi_i^\ell=D_\ell$, where $\ell\in\{\mathrm{SKL},\mathrm{SRKL}\}$. Differentiating with respect to each $q_i$ gives $$\begin{aligned} \frac{\partial\phi_i^{\rm SKL}}{\partial q_i} &=(1-\alpha)\left(1-\frac{p_i}{m_i}\right)=-A_i^{\rm SKL}, &\quad\frac{\partial^2\phi_i^{\rm SKL}}{\partial q_i^2} &=\frac{(1-\alpha)^2p_i}{m_i^2}>0,\\ \frac{\partial\phi_i^{\rm SRKL}}{\partial q_i} &=\log\frac{q_i}{n_i}+\alpha\left(1-\frac{q_i}{n_i}\right)=-A_i^{\rm SRKL}, &\quad\frac{\partial^2\phi_i^{\rm SRKL}}{\partial q_i^2} &=\frac{(1-\alpha)^2p_i^2}{q_i n_i^2}>0. \end{aligned}$$ Both scalar terms and their first derivatives vanish at $q_i=p_i$. Strict convexity therefore implies that each term is nonnegative and is zero only at $q_i=p_i$. Using the piecewise retention probability $a_i(q_i)$ from W1, define $$F_\ell(q;p)=\sum_i a_i(q_i)\phi_i^\ell(q_i;p_i), \qquad \ell\in\{\mathrm{SKL},\mathrm{SRKL}\}.$$ Under the fixed-prefix, standard on-policy sampling, and mask-independent normalization conditions of the isolated CLL analysis in W1, the gradients of these objectives exactly equal the expected surrogate gradients after the corresponding Bernoulli filtering: $$\nabla_\theta F_\ell =\sum_i a_i(q_i)\frac{\partial\phi_i^\ell}{\partial q_i}\nabla_\theta q_i =-\mathbb E_{i\sim q} \left[a_i(q_i)A_i^\ell\nabla_\theta\log q_i\right].$$ Although $a_i$ may jump at the branch boundary, both $\phi_i^\ell$ and its first derivative vanish there, so $F_\ell$ remains differentiable. At $q=p$, $F_\ell=0$. If $q\neq p$, normalization guarantees at least one $q_i<p_i$, whose retention probability is $a_i=s_i>0$ and whose scalar term is strictly positive. Thus, both filtered objectives have the unique global minimizing distribution $q=p$, even if some negative-update retention probabilities are zero. We will add these derivations and their conditions to the appendix. Golden-Gain's candidate-generation and anchor-comparison framework can also be reused; one candidate trajectory score corresponding to the new update coefficients is their mean over positive values. CSD instead matches pairwise logit differences. Its native gradient makes the required adaptation explicit. Let $\delta_i=f_S(i)-f_T(i)$, and let its factorized, normalized weights be $w_{ij}=\omega_{1i}\omega_{2j}$. Holding these weights fixed during differentiation, Section 3.2 gives $$g_i=\frac{\partial L_{\rm CSD}}{\partial f_S(i)} =\omega_{1i}(\delta_i-\bar\delta_{\omega_2}) +\omega_{2i}(\delta_i-\bar\delta_{\omega_1}), \qquad \bar\delta_\omega=\sum_j\omega_j\delta_j.$$ Since $\sum_i g_i=0$, its gradient has the exact score-function representation $$\nabla_\theta L_{\rm CSD} =-\mathbb E_{i\sim q}[A_i^{\rm CSD}\nabla_\theta\log q_i], \qquad A_i^{\rm CSD}=-g_i/q_i.$$
AuthorsOpenReview ↗

Response to Reviewer kYJX (3)

The representation uses zero baseline. This coefficient, rather than log⁡(pi/qi)\log(p_i/q_i), is a candidate signal for direction-dependent filtering: retain its positive branch with probability sis_i and its negative branch with probability 1−si1-s_i. Its sign can differ from the reverse-KL advantage because CSD compares centered logit residuals across the vocabulary. A corresponding candidate trajectory score is the mean positive AytCSDA_{y_t}^{\rm CSD}, using the same anchor comparison as Golden-Gain.

In independent-logit space, under the same fixed-prefix, standard on-policy sampling, and mask-independent normalization conditions as the isolated CLL analysis in W1, strictly positive CSD pairwise weights preserve the teacher as the unique interior stationary distribution of the signed-filtered expected update above. However, the filtered direction need not decrease the original CSD loss at every step, and the factor 1/qi1/q_i can introduce high sampling variance.

W3. Why do improvements differ across benchmarks, and which capabilities benefit?

We agree that the averages alone do not capture the trade-offs. The differences you identify between medical IF-Eval/IFBench and role-playing LiveCodeBench v5/AIME25 are present. Comparing capability types reveals more specific patterns. All changes below are relative to vanilla MOPD:

Capability / benchmarkRole-playing settingMedical setting
Coding: LiveCodeBench v5+4.66+3.59
Coding: LiveCodeBench v60.00+3.43
Scientific problem solving: GPQA-Diamond+4.04+1.64
Structured logical reasoning: ZebraLogic+5.10+1.10
Instruction following: IF-Eval+2.96+23.47
Generalization to new constraints: IFBench+1.02-1.02
Competition mathematics: HMMT25+1.04+1.66
Competition mathematics: AIME25-3.75+1.88

Coding, scientific problem solving, and structured logical reasoning show relatively consistent gains. LiveCodeBench v5, GPQA, and ZebraLogic improve in both settings, while LiveCodeBench v6 is unchanged in role-playing and improves in the medical setting. These are therefore capability areas that benefit relatively consistently in our experiments. LiveCodeBench evaluates code correctness through execution tests, GPQA tests challenging scientific question answering, and ZebraLogic tests constraint-based logical reasoning. These results also show that the benefits extend beyond a single instruction-following benchmark.

Recovery on standard verifiable instruction following is substantial, while generalization to new constraints is weaker. IFEval evaluates 25 types of verifiable instructions, whereas IFBench specifically tests generalization to 58 new, diverse, and challenging out-of-domain constraints. IFEval improves in both settings, especially by 23.47 points in the medical setting; IFBench instead decreases by 1.02 points in that setting. Thus, the evidence more directly supports recovery of the instruction-following abilities covered by IFEval than comparable improvement in generalization to new constraints. The benchmark-design difference offers a plausible interpretation, although establishing whether it causes the gap in our model requires constraint-level analysis.

Gains in competition mathematics depend more on the setting. AIME25 tests competition-level mathematical problem solving: its score falls from 47.29 to 43.54 in role-playing but rises from 54.37 to 56.25 in the medical setting; HMMT25 improves in both. This differs from the patterns on coding, GPQA, and ZebraLogic. Coding improvements therefore do not imply simultaneous gains in competition mathematics, nor does the role-playing AIME regression imply a general deterioration in mathematical reasoning. MMLU-Redux and MMLU-Pro also lack consistent improvements across settings, so we will not characterize the results as uniform gains in general knowledge.

One interpretation consistent with our design is that UCMOPD improves the student's utilization of existing teacher feedback, without directly optimizing generalization to out-of-domain constraints or final mathematical-answer correctness. This may help explain why some capabilities recover while others do not improve simultaneously. We will distinguish this explanatory hypothesis from the patterns directly observed in the table.

Our contribution is to identify and improve learning-signal utilization in vanilla MOPD for capability recovery. Vanilla MOPD is therefore the central control for evaluating the proposed changes, and the capability patterns above refer to that comparison. The general averages increase from 52.09 to 54.43 and from 51.81 to 55.88 in the two settings, while domain averages are maintained or improved. We will expand the discussion to highlight the gains in coding, scientific problem solving, structured logical reasoning, and instruction-following recovery on IFEval, while reporting the trade-offs on benchmarks such as IFBench and AIME.

Reviewer_Tww2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to recover the general capabilities of domain-specialized language models through multi-teacher on-policy distillation. It proposes UCMOPD, which combines Golden-Gain Enhancement for trajectory-level selection and Teacher-Endorsement Filtering based on centered log-likelihood (CLL) for token-level filtering, aiming to improve general capabilities while preserving domain-specific performance.

Critical strengths

The paper addresses an important and practical problem of recovering general capabilities while preserving domain-specific performance. The proposed method is well-designed, combining trajectory-level and token-level selection to improve the utilization of distillation signals. In particular, the prompt-specific anchor provides a reasonable criterion for selecting informative high-temperature trajectories, while the entropy-centered CLL offers an adaptive way to estimate teacher endorsement. The experiments are comprehensive, covering both role-playing and medical domains, with evaluations on 13 general-capability benchmarks and comparisons against several competitive baselines. The results demonstrate improvements in aggregate general-capability performance over standard MOPD while maintaining domain-specific performance. Moreover, the ablation studies and the rollout-budget-controlled comparison in the role-playing setting provide additional evidence for the effectiveness of the proposed method.

Critical weaknesses
  1. The paper refers to negative-advantage updates on high-confidence teacher tokens as “mis-suppression” and positive-advantage updates on low-confidence tokens as “mis-reinforcement.” However, such updates can be valid corrections under the reverse-KL objective, since their directions depend on the relative probabilities assigned by the teacher and student. Although the paper acknowledges that these conflicts are not ground-truth error labels, it does not sufficiently explain why such updates should be considered unreliable learning signals. Improvements from filtering them may instead reflect a useful bias introduced by the proposed method.

  2. In the role-playing setting, Table 10 and Figure 5 do not rule out a simpler explanation for the reported gains: the asymmetry in CLL's retention rates (81.88% positive vs. 9.88% negative) may itself improve performance by substantially reducing the contribution of negative-advantage updates and rebalancing the positive-to-negative update ratio. As reported, the contribution of uncertainty-aware, per-token selection is therefore not isolated from the effect of asymmetric retention rates. Although Tables 3 and 4 provide component ablations and comparisons with alternative endorsement criteria, neither includes a sign-only, rate-matched control, so this alternative explanation remains open.

Questions for authors
  1. Could the authors clarify why these direction–endorsement conflicts should be regarded as unreliable learning signals, despite being consistent with the reverse-KL objective? Is there empirical evidence that filtering them addresses harmful updates rather than simply introducing a beneficial bias?

  2. Could the authors add a sign-only random filtering baseline that ignores teacher confidence and CLL scores but retains positive- and negative-advantage updates at rates matched to CLL's? Matching the per-step retention schedule (Figure 5) and reporting retained-token counts and update magnitude would make the comparison more informative. This would help determine whether CLL's token-level selection matters beyond its aggregate retention rates.

  3. The paper reports performance results without standard deviations or confidence intervals across multiple random seeds. Could the authors provide multi-seed results to assess the stability of the proposed method, particularly given the relatively small improvement over CaMOPD in the role-playing setting?

  4. In the medical setting (Table 2), the improvements of UCMOPD appear to be concentrated in a few benchmarks, particularly IF-Eval and LiveBench, while the gains on other benchmarks are relatively limited or even negative. Could the authors provide further analysis to explain why the improvements vary so substantially across different benchmarks?

Reviewer_u6sVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a variant of multi-teacher on-policy distillation (MOPD), UCMOPD, motivated by two concerns: ordinary on-policy sampling provides limited exposure to tokens with large positive teacher–student log-probability gaps, and the sign of this gap alone may not establish the reliability of the proposed update. The authors augment vanilla MOPD with higher-temperature exploration rollouts that are kept only when their mean positive advantage matches a standard-temperature rollout of the same prompt, and with a per-token Bernoulli mask whose keep probability is the teacher’s entropy-centered log-likelihood of the sampled token, flipped by the advantage sign. Experiments cover role-playing and medical specialization on thirteen general benchmarks plus domain metrics, against SFT, vanilla MOPD, SelecTKD, ReOPOLD and CaMOPD. Compared with vanilla MOPD, the reported advantages are most pronounced on the general-capability average in the medical setting, where they are concentrated in IF-Eval and LiveBench; in the role-playing setting the method is level with CaMOPD on both averages.

Critical strengths

S1. Appropriate baselines and controls. The paper evaluates two specialization settings across 13 general benchmarks, includes the closest prior method (CaMOPD) alongside SelecTKD and ReOPOLD, provides component ablations, and reports an eight-rollout budget-matched MOPD comparison for role-playing.

S2. An important problem with a clear method. Recovering general capabilities using a frozen general-purpose teacher and a small set of public prompts is practically important. The two components are clearly described and straightforward to integrate into a standard MOPD loop.

S3. Transparent diagnostics. Expected vs. realized keep rates (Table 10, Fig. 5), kept-vs-dropped trajectory statistics (Table 8), and the B.4 statistics on negative-advantage tokens make clear what the filters actually do during training.

Critical weaknesses

W1. The medical setting is misreported. Appendix A.1 (lines 765–766) and Table 5 state that the student and both teachers share the same Qwen3 architecture and 8B parameter count. However, the cited II-Medical-7B-Preview checkpoint is, per its model card, fine-tuned from Qwen2.5-7B-Instruct and further trained with DAPO. Consequently (i) the medical experiment is cross-family distillation from a Qwen3-8B teacher into a Qwen2.5-7B student, not the "homogeneous, same-scale" setup claimed; (ii) the "Base" column (Qwen3-8B) is not the model the student was specialized from, so the Base→SFT drop cannot be read as specialization-induced forgetting, nor the gain as recovery toward the student’s own base.

W2. Aggregate results in Table 2 are inconsistent. For example, the mean of the four role-playing metrics for MOPD is 42.86, not the reported 41.22: (43.57 + 34.91 + 41.45 + 51.50) / 4 = 42.8575. The value 41.22 is also reused in Table 3, Table 9, and the main text. If the per-metric scores are correct, UCMOPD’s gain over MOPD on this vertical is 2.14 points, not 3.78, while the gain attributed to Teacher-Endorsement Filtering alone falls from 2.14 to 0.50 points. The authors should clarify which values are incorrect, recalculate the affected gains, and revisit the related claims.

W3. Gains are small and not attributed to the proposed mechanisms. UCMOPD exceeds CaMOPD by only 0.16 points on the role-playing general average; in the medical setting, the margin over CaMOPD comes almost entirely from IF-Eval and LiveBench. The eight-rollout MOPD control (53.31 vs. 52.09 for vanilla MOPD) suggests that roughly half of the gain over vanilla MOPD is attributable to the larger rollout budget, and no equivalent control is reported for the medical setting. Teacher-Endorsement Filtering alone improves the general average by just 0.08 points (Tables 3 and 9), while the hard top-k variants perform worse than vanilla MOPD (Table 4). The diagnostics show that the selection rules behave as designed, but do not establish that the removed updates were harmful. Teacher endorsement alone does not show that suppressing a teacher-endorsed token is harmful when the student already overestimates its probability. A positive-only OPD control and a Golden-Gain-only row are needed before the contribution of the token-level mechanism can be assessed.

Questions for authors

Please see the Critical Weaknesses section.

Although the paper is fluently written, the method is simple and clearly designed, and the experiments are reasonably thorough, the weakness above lower my confidence in the overall reliability of the reported results.

4 comments
AuthorsOpenReview ↗

Response to Reviewer u6sV (1)

Thank you for recognizing the practical importance of capability recovery, the clarity of our method, and the baseline comparisons and diagnostic analyses. Your comments identify reporting inconsistencies and important questions about the source of the improvements. We address these separately below.

W1. Medical checkpoint provenance and interpretation

You are correct that our description of the medical setting as homogeneous, same-scale Qwen3 distillation is inaccurate. We used the public II-Medical-7B-Preview, which is based on Qwen2.5-7B-Instruct and further trained with DAPO, for student initialization and the frozen medical teacher. The general teacher is Qwen3-8B. We will correct Appendix A.1 and Table 5 to describe this cross-family setting and the corresponding model sizes.

We also agree that Qwen3-8B is a general-teacher reference, rather than the medical student's pre-specialization checkpoint. Its performance difference from the medical initialization therefore cannot be attributed directly to specialization-induced forgetting. We will revise the “Base” description and the related interpretation in Section 4.2. The medical experiment evaluates improvement in the specialized student's general capabilities while preserving its medical performance, rather than restoration toward its own pre-specialization checkpoint.

W2. Inconsistent role-playing entries in Table 2

Thank you for identifying this inconsistency. For vanilla MOPD, the error is in the four per-metric entries printed in Table 2. The correct scores are 42.74, 32.74, 38.29, and 51.13, respectively. Their mean is 41.225 at the displayed precision. Consequently, the role-playing improvements are approximately 3.78 points for UCMOPD and 2.14 points for CLL-only, rather than the smaller improvements obtained by averaging the incorrectly printed entries.

We will correct these entries and reconcile rounding across the tables and text. Our audit also identified a SelecTKD aggregate that should be 41.79 based on its listed scores.

W3. Magnitude and distribution of the improvements

We agree that the role-playing general-average difference from CaMOPD is small: 54.43 versus 54.27. We will describe this as comparable performance and avoid implying a substantial advantage over CaMOPD in this setting.

In the medical setting, UCMOPD improves the general average over CaMOPD from 54.26 to 55.88, while obtaining a medical average of 60.95 versus 59.40, close to the specialized initialization's 60.96. We agree that the general-average advantage over CaMOPD is concentrated in IF-Eval and LiveBench. We will make this concentration explicit rather than suggest uniform improvements across benchmarks.

The comparison with vanilla MOPD is broader: excluding IF-Eval and LiveBench, UCMOPD still improves the average of the remaining 11 general benchmarks by approximately 1.27 points. Thus, the coverage of the gains differs depending on the baseline, and we will distinguish these comparisons more clearly.

Examining capability types reveals relatively consistent gains over vanilla MOPD in coding, scientific problem solving, and structured logical reasoning: LiveCodeBench v5, GPQA, and ZebraLogic improve by 4.66/3.59, 4.04/1.64, and 5.10/1.10 points, respectively, in the role-playing/medical settings. IF-Eval also improves in both settings, especially by 23.47 points in the medical setting. However, medical IFBench decreases by 1.02 points and role-playing AIME25 decreases by 3.75 points, indicating that gains in generalization to new constraints and competition mathematics depend more on the setting. These patterns show that the gains over vanilla MOPD extend beyond IF-Eval and LiveBench.

Our main contribution is to identify and improve learning-signal utilization in vanilla MOPD for capability recovery: standard sampling provides limited exposure to strong positive learning opportunities, and the teacher–student relative-probability correction indicated by the advantage needs to be distinguished from the teacher's own endorsement of a token. Therefore, vanilla MOPD is the central control for evaluating whether these changes are effective, while comparisons with CaMOPD and other baselines assess the resulting domain–general performance trade-off. Relative to vanilla MOPD, UCMOPD improves the general/domain averages by approximately 2.33/3.78 points in role-playing and the general average by approximately 4.07 points in the medical setting while maintaining medical performance.

AuthorsOpenReview ↗

Response to Reviewer u6sV (2)

W3. Contribution of rollout budget

We agree that additional sampling contributes to the improvement over the original MOPD configuration. Importantly, the budget-matched comparison in Appendix Table 7 shows a remaining advantage at eight responses per prompt: UCMOPD achieves 54.43/45.00 in general/role-playing averages, compared with 53.31/43.38 for vanilla MOPD. These correspond to improvements of 1.12 and 1.62 points under the same response-count budget.

The medical setting uses four responses per prompt, rather than eight. The results therefore demonstrate improvements under both four-response and eight-response configurations. However, we agree that this cross-setting observation does not replace a medical budget-matched control. We will restrict the budget-controlled conclusion to the role-playing experiment and avoid attributing the full medical improvement solely to the proposed selection mechanisms.

W3. CLL versus hard top-k filtering and positive-only updates

The hard top-k variants in Table 4 are design controls for a simple implementation of teacher endorsement. They test whether membership in a fixed teacher top-k set is sufficient for selecting updates. CLL instead uses a continuous score whose reference probability adapts to the teacher distribution's entropy.

In the tested configurations, CLL obtains higher general and role-playing averages than both hard top-k variants. We interpret this as evidence favoring the continuous CLL criterion over these fixed-rank alternatives. We do not claim that teacher-endorsement filtering is beneficial regardless of how endorsement is defined.

Following your suggestion, we added a positive-only OPD baseline, which starts from vanilla MOPD and discards all negative-advantage updates without introducing Golden-Gain Enhancement.

MethodGeneral averageRole-playing average
Vanilla MOPD52.0941.22
Positive-only OPD51.7441.13
CLL-only52.1743.36

Discarding all negative updates does not reproduce the CLL-only results. CLL-only improves over positive-only OPD by approximately 0.43 points in the general average and 2.2 points in the role-playing average. This comparison supports selective filtering over blanket removal of negative updates. We acknowledge that it does not, by itself, isolate token-specific CLL selection from the effect of changing the positive-to-negative update balance.

W3. Component attribution and the interpretation of teacher endorsement

Golden-Gain Enhancement and CLL operate at different levels: the former explores and selects trajectories, while the latter selects token updates within retained trajectories. The cumulative ablation in Table 3 evaluates the addition of Golden-Gain Enhancement to a CLL-enabled configuration: the general/role-playing averages increase from 52.17/43.36 to 54.43/45.00. The comparison between vanilla MOPD and CLL-only separately evaluates CLL without Golden-Gain Enhancement.

These comparisons support the usefulness of the two components in the evaluated configurations. However, they do not quantify the incremental effect of CLL when full Golden-Gain Enhancement is already enabled. We have not completed the additional Golden-Gain-only run within the rebuttal computational budget. We will make this limitation explicit and avoid claiming that the components are independent or that their interaction has been fully isolated.

Finally, we agree with your distinction between teacher endorsement and update correctness. Suppressing a teacher-endorsed token can be a valid reverse-KL correction when the student overestimates its probability. Our endorsement statistics are descriptive measures, not ground-truth labels of harmful updates. We will revise the “mis-suppression/mis-reinforcement” terminology and distinguish the empirical benefits of the filtering rule from a claim that every removed update is harmful.

Thank you for the response, which has addressed some of my concerns. However, the errors in Table 2 reduce my confidence. Given these issues and the lack of code to support reproducibility, I decide to to maintain my original score.

Thank you for considering our response. We understand that the inconsistencies in Table 2 undermine confidence in the experimental reporting, and we apologize for these errors.

We have corrected the four role-playing scores for vanilla MOPD to 42.74, 32.74, 38.29, and 51.13, and the SelecTKD role-playing average to 41.79, including its corresponding point in the trade-off figure. These corrections do not change the reported UCMOPD scores or the general-capability results. The revised manuscript also clarifies the provenance of the medical checkpoint and the cross-family experimental setting.

We would also like to clarify that code release is subject to institutional restrictions, which prevent us from making it publicly available. To support reproducibility, the revised appendix documents the model checkpoints and their roles, data construction, teacher routing, and training configurations, including the corrected medical setup. We recognize that these details do not fully substitute for access to the implementation and understand your concern.

We appreciate your careful scrutiny and respect your decision. We hope that the corrections and clarifications help restore confidence in our experimental reporting, and would be grateful if you could reconsider your overall assessment in light of the corrected results and the paper’s contributions.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to Reviewer Tww2 (1)

Thank you for recognizing the practical importance of the problem, the design of the two selection mechanisms, and the experimental coverage. We appreciate your distinction between teacher endorsement and reverse-KL correctness, and your suggestion for a retention-matched control. We respond to each weakness and question below.

Critical Weakness 1: Interpreting direction–endorsement conflicts

We agree that these conflicts do not establish that an update is incorrect under reverse KL. When the student assigns a higher probability than the teacher to an otherwise plausible token, reducing that probability can be a valid distribution-matching correction. We will revise the “mis-suppression/mis-reinforcement” terminology and avoid presenting teacher endorsement as a ground-truth label of update correctness.

Our motivation is to introduce a teacher-dependent preference over which distribution-matching updates receive training effort during capability recovery. A useful analogy is teacher-generated SFT: conditional on a teacher-sampled token, cross-entropy training reinforces that token even if its current student probability exceeds its teacher probability. This illustrates how a teacher-supported token can receive reinforcement in a sampled-target objective. It does not invalidate reverse KL: averaged over teacher samples, the forward-KL/SFT gradient also corrects student overestimation.

CLL uses teacher-relative plausibility to favor reinforcement of endorsed tokens and suppression of weakly endorsed tokens. We therefore agree that its empirical benefit may be understood as a useful inductive bias. CLL is not equivalent to teacher-generated SFT, and the present evidence does not establish that every filtered update is harmful. We will distinguish this motivation from claims about update correctness throughout the paper.

Critical Weakness 2: Asymmetric retention versus token-specific selection

We agree that the asymmetric retention rates provide a plausible alternative explanation. Table 10 and Figure 5 characterize CLL's candidate counts and retention dynamics, but do not isolate token-specific selection from the positive-to-negative update balance. The component ablations and alternative endorsement criteria do not replace a sign-only random control matched to the per-step retention rates.

We added the positive-only baseline described under Question 2 below. It tests whether removing all negative updates can reproduce CLL's results, but does not fully resolve the retention-matching question. We will state this distinction explicitly and narrow our component-attribution claims accordingly.

AuthorsOpenReview ↗

Response to Reviewer Tww2 (2)

Question 1: Why regard these signals as unreliable, and is there evidence that they are harmful?

We will no longer use “unreliable” to imply that these updates are invalid reverse-KL corrections. Direction–endorsement conflict measures disagreement between a relative teacher–student correction and a teacher-internal plausibility criterion. It does not provide an external correctness judgment.

A concrete comparison with teacher-generated SFT helps explain both directional cases. Consider a fixed prefix and a three-token example with teacher probabilities (0.80, 0.15, 0.05) and student probabilities (0.82, 0.15, 0.03). Denote the first and third tokens by x and z, respectively. Suppose the teacher samples x as the SFT target. The following comparisons concern hypothetical student-sampled updates at this same prefix.

Negative advantage on a strongly endorsed token. If the student samples x, its advantage log(0.80/0.82) is negative, proposing suppression. In contrast, the SFT target x gives a logit gradient p_S(x)-1=-0.18, so gradient descent reinforces x. CLL assigns maximal endorsement to x in this example and filters its negative-advantage update. It filters the proposed suppression, rather than reinforcing x directly.

Positive advantage on a weakly endorsed token. If the student instead samples z, its advantage log(0.05/0.03) is positive, proposing reinforcement despite its low teacher probability. For the same teacher-generated SFT target x, the gradient with respect to z's logit is p_S(z)=0.03, so gradient descent suppresses z. CLL assigns z a low endorsement score and makes its positive-advantage update unlikely to be retained. Again, filtering this reinforcement is not a direct negative update on z.

Our concern is therefore that advantage sign alone describes relative distribution mismatch without separately representing teacher endorsement. The comparison between vanilla MOPD and CLL-only provides empirical support for introducing a teacher-endorsement-based selection preference: in the evaluated setting, filtering the selected updates improves downstream performance. In particular, CLL-only improves the role-playing average from 41.22 to 43.36 while maintaining a comparable general average (52.09 versus 52.17). These results support the practical benefit of withholding the selected updates during training.

We distinguish this aggregate evidence from a claim that every discarded update is individually harmful. We will revise the terminology while retaining the experimentally supported claim that CLL filtering improves the observed capability trade-off.

Question 2: Sign-only random filtering with matched retention rates

We agree that the requested sign-only random baseline, matched to the per-step positive and negative retention rates, is the appropriate control for separating token-specific selection from asymmetric retention. We also understand that retained-token counts and update magnitudes are needed to assess the comparability of the two procedures. The CLL statistics already reported in Table 10 and Figure 5 do not substitute for this comparison.

As an additional control, we evaluated positive-only OPD, which starts from vanilla MOPD and discards all negative-advantage updates without Golden-Gain Enhancement:

MethodGeneral averageRole-playing average
Vanilla MOPD52.0941.22
Positive-only OPD51.7441.13
CLL-only52.1743.36

CLL-only exceeds positive-only OPD by approximately 0.43 points in the general average and 2.2 points in the role-playing average. Thus, blanket removal of negative updates does not reproduce the CLL-only results. However, this comparison does not rule out the possibility that a less extreme, rate-matched random filter could achieve similar gains.

We have not completed the requested rate-matched training run within the rebuttal computational budget. We plan to include it in a follow-up evaluation. In the present revision, we will explicitly acknowledge that the contribution of token-specific CLL selection beyond asymmetric retention remains unresolved.

AuthorsOpenReview ↗

Response to Reviewer Tww2 (3)

Question 3: Multi-seed stability and the small difference from CaMOPD

We agree that the 0.16-point role-playing general-average difference from CaMOPD is insufficient to establish reliable superiority without uncertainty estimates across repeated training runs. We will describe this result as comparable performance and avoid claims of statistical significance or stable superiority.

Our primary goal is to analyze and improve learning-signal utilization in vanilla MOPD. A central contribution is distinguishing the relative teacher–student correction indicated by advantage from teacher-internal token endorsement, and investigating whether endorsement-guided selection improves capability recovery. Accordingly, vanilla MOPD is the primary reference for evaluating the proposed changes, while the additional baselines contextualize the resulting performance trade-offs.

We have not completed additional independent training runs with different seeds within the rebuttal computational budget and plan to include them in a follow-up evaluation. We will explicitly acknowledge that the stability of the reported gains, including those over vanilla MOPD, remains to be assessed across training seeds.

Question 4: Uneven improvements across medical benchmarks

We agree that the gains are uneven. In particular, the general-average improvement over CaMOPD is concentrated almost entirely in IF-Eval and LiveBench: excluding these two benchmarks leaves an average difference of approximately 0.014 points across the remaining 11 benchmarks. We will report this concentration explicitly.

For the IF-Eval/IFBench difference, the benchmark designs help distinguish the capabilities being recovered: IFEval evaluates 25 types of verifiable instructions, whereas IFBench specifically tests generalization to 58 new, diverse, and challenging out-of-domain constraints. Relative to vanilla MOPD, medical IF-Eval improves by 23.47 points, while IFBench decreases by 1.02 points. Thus, the evidence supports substantial recovery of the instruction-following abilities covered by IFEval, without simultaneous improvement in generalization to new constraints.

Relative to vanilla MOPD, gains also extend beyond instruction following: medical LiveCodeBench v5/v6, GPQA, and ZebraLogic improve by 3.59/3.43, 1.64, and 1.10 points, respectively. Together with the improvements on LiveCodeBench v5, GPQA, and ZebraLogic in role-playing, the results identify coding, scientific problem solving, and structured logical reasoning as areas with relatively consistent gains across settings; gains in generalization to new constraints and competition mathematics depend more on the setting.

Our objective is to improve the aggregate capability-recovery trade-off of vanilla MOPD. In the medical setting, the general average increases from 51.81 to 55.88, while the medical average remains close to the specialized initialization (60.95 versus 60.96). Excluding IF-Eval and LiveBench, the remaining 11 general benchmarks still improve over vanilla MOPD by approximately 1.27 points on average. These results support an improvement in the aggregate trade-off in the reported setting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.