- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper improves Multi-Teacher On-Policy Distillation (MOPD) using an uncertainty quantification method. It points out that MOPD does not sufficiently explore tokens with high positive advantages and that it is difficult to determine the appropriate learning direction based only on the sign of the teacher–student log-likelihood ratio. The proposed Uncertainty-Calibrated MOPD (UCMOPD) has two components. First, Golden-Gain Enhancement generates diverse trajectories through high-temperature sampling and selects useful trajectories based on positive advantages. Second, Teacher-Endorsement Filtering evaluates the reliability of token-level updates using Centered Log-Likelihood (CLL), which combines teacher entropy and log-likelihood, and probabilistically filters the learning signals. The method is compared with MOPD-based baselines in role-playing and medical-domain specialization experiments.
Critical strengths
-
I think the motivation is very clear, specially Figure 1 concretely illustrates the problems that can arise when determining the appropriateness of token-level updates based only on the sign of the advantage in existing MOPD.
-
Methodology looks simple and intuiative. Sections 3.2 and 3.3 effectively utilize learning signals by combining trajectory selection through high-temperature exploration with teacher entropy-based token filtering.
-
The initial experimental results looks promising. Table 2 shows that the proposed method achieves higher general capability recovery than existing MOPD while maintaining or improving domain-specific performance in both role-playing and medical specialization.
-
Ablation studies Tables 3–4 and Appendix Table 7 looks proper to validate the effectiveness of each component and show that the performance improvements are maintained even under the same rollout budget.
Critical weaknesses
-
The proposed method seems to be limited to heuristic weight adjustments. For example, see Eq. 6 of DistLLM (ICML 2024) and Section 3.2 of CSD (ICLR 2026). Their gradients also differ from those of existing losses (e.g., KL), but they do not directly modify the gradients. Instead, they propose different loss functions that converge to the target distribution and analyze their gradients. In contrast, simply adjusting gradient weights, as in the proposed method, does not guarantee matching to the original target distribution. In other words, what is the optimal solution of the student model when Algorithm 1 is optimized to convergence?
-
The proposed method is developed based on the loss, which is reverse-KL-based, as mentioned in line 201. Could it also be applied to other divergences proposed in DistiLLM or CSD?
-
In Table 2, the proposed method shows large improvements on some benchmarks but performs worse than existing methods on others. For example, in the medical setting, IF-Eval improves significantly, but IFBench shows no improvement. In the role-playing setting, LiveCodeBench v5 improves, while AIME25 performs worse than the original MOPD. Further analysis and discussion are needed to explain which benchmark characteristics cause these differences and which types of capability recovery the proposed method is particularly effective for.
Questions for authors
Please see weakness section.
6 comments
Response to Reviewer kYJX (2)
W2. Can the method extend to the losses in DistiLLM or CSD?
The candidate-generation framework and teacher-only endorsement score are transferable, whereas trajectory scoring and direction-dependent filtering require analysis under the new loss and adaptation where necessary. Specifically, high-temperature exploration and a standard-temperature anchor can generate candidates independently of the distillation loss. CLL's endorsement also depends only on the teacher. In contrast, Golden-Gain currently scores trajectories using positive reverse-KL log-probability gaps, and CLL's signed keep rule uses their proposed update directions. These two parts require loss-specific analysis and adaptation where necessary.
For DistiLLM, the losses are skew KL and skew reverse KL:
\qquad D_{\rm SRKL}=D_{\rm KL}(q\|n),\quad n=(1-\alpha)p+\alpha q.$$ At a fixed prefix, their gradients admit the score-function representation $\nabla_\theta D=-\mathbb E_{i\sim q}[A_i\nabla_\theta\log q_i]$, with coefficients $$A_i^{\rm SKL}=(1-\alpha)\left(\frac{p_i}{m_i}-1\right), \qquad A_i^{\rm SRKL}=\log\frac{n_i}{q_i} +\alpha\left(\frac{q_i}{n_i}-1\right).$$ Here, constant baselines are chosen so that the coefficients vanish at $q=p$. For $0\leq\alpha<1$, both coefficients have the same sign as $p_i-q_i$, although their magnitudes differ from $\log(p_i/q_i)$. Thus, CLL's teacher-only endorsement score and signed keep rule can be applied to these derived coefficients. To make the filtered optimization objectives explicit, define the following per-token scalar terms, with $p_i>0$ and $q_i>0$: $$\begin{aligned} \phi_i^{\rm SKL}(q_i;p_i) &=p_i\log\frac{p_i}{m_i}+(1-\alpha)(q_i-p_i),\\ \phi_i^{\rm SRKL}(q_i;p_i) &=q_i\log\frac{q_i}{n_i}-(1-\alpha)(q_i-p_i). \end{aligned}$$ The added linear terms sum to zero on the probability simplex, so without filtering, $\sum_i\phi_i^\ell=D_\ell$, where $\ell\in\{\mathrm{SKL},\mathrm{SRKL}\}$. Differentiating with respect to each $q_i$ gives $$\begin{aligned} \frac{\partial\phi_i^{\rm SKL}}{\partial q_i} &=(1-\alpha)\left(1-\frac{p_i}{m_i}\right)=-A_i^{\rm SKL}, &\quad\frac{\partial^2\phi_i^{\rm SKL}}{\partial q_i^2} &=\frac{(1-\alpha)^2p_i}{m_i^2}>0,\\ \frac{\partial\phi_i^{\rm SRKL}}{\partial q_i} &=\log\frac{q_i}{n_i}+\alpha\left(1-\frac{q_i}{n_i}\right)=-A_i^{\rm SRKL}, &\quad\frac{\partial^2\phi_i^{\rm SRKL}}{\partial q_i^2} &=\frac{(1-\alpha)^2p_i^2}{q_i n_i^2}>0. \end{aligned}$$ Both scalar terms and their first derivatives vanish at $q_i=p_i$. Strict convexity therefore implies that each term is nonnegative and is zero only at $q_i=p_i$. Using the piecewise retention probability $a_i(q_i)$ from W1, define $$F_\ell(q;p)=\sum_i a_i(q_i)\phi_i^\ell(q_i;p_i), \qquad \ell\in\{\mathrm{SKL},\mathrm{SRKL}\}.$$ Under the fixed-prefix, standard on-policy sampling, and mask-independent normalization conditions of the isolated CLL analysis in W1, the gradients of these objectives exactly equal the expected surrogate gradients after the corresponding Bernoulli filtering: $$\nabla_\theta F_\ell =\sum_i a_i(q_i)\frac{\partial\phi_i^\ell}{\partial q_i}\nabla_\theta q_i =-\mathbb E_{i\sim q} \left[a_i(q_i)A_i^\ell\nabla_\theta\log q_i\right].$$ Although $a_i$ may jump at the branch boundary, both $\phi_i^\ell$ and its first derivative vanish there, so $F_\ell$ remains differentiable. At $q=p$, $F_\ell=0$. If $q\neq p$, normalization guarantees at least one $q_i<p_i$, whose retention probability is $a_i=s_i>0$ and whose scalar term is strictly positive. Thus, both filtered objectives have the unique global minimizing distribution $q=p$, even if some negative-update retention probabilities are zero. We will add these derivations and their conditions to the appendix. Golden-Gain's candidate-generation and anchor-comparison framework can also be reused; one candidate trajectory score corresponding to the new update coefficients is their mean over positive values. CSD instead matches pairwise logit differences. Its native gradient makes the required adaptation explicit. Let $\delta_i=f_S(i)-f_T(i)$, and let its factorized, normalized weights be $w_{ij}=\omega_{1i}\omega_{2j}$. Holding these weights fixed during differentiation, Section 3.2 gives $$g_i=\frac{\partial L_{\rm CSD}}{\partial f_S(i)} =\omega_{1i}(\delta_i-\bar\delta_{\omega_2}) +\omega_{2i}(\delta_i-\bar\delta_{\omega_1}), \qquad \bar\delta_\omega=\sum_j\omega_j\delta_j.$$ Since $\sum_i g_i=0$, its gradient has the exact score-function representation $$\nabla_\theta L_{\rm CSD} =-\mathbb E_{i\sim q}[A_i^{\rm CSD}\nabla_\theta\log q_i], \qquad A_i^{\rm CSD}=-g_i/q_i.$$Response to Reviewer kYJX (3)
The representation uses zero baseline. This coefficient, rather than , is a candidate signal for direction-dependent filtering: retain its positive branch with probability and its negative branch with probability . Its sign can differ from the reverse-KL advantage because CSD compares centered logit residuals across the vocabulary. A corresponding candidate trajectory score is the mean positive , using the same anchor comparison as Golden-Gain.
In independent-logit space, under the same fixed-prefix, standard on-policy sampling, and mask-independent normalization conditions as the isolated CLL analysis in W1, strictly positive CSD pairwise weights preserve the teacher as the unique interior stationary distribution of the signed-filtered expected update above. However, the filtered direction need not decrease the original CSD loss at every step, and the factor can introduce high sampling variance.
W3. Why do improvements differ across benchmarks, and which capabilities benefit?
We agree that the averages alone do not capture the trade-offs. The differences you identify between medical IF-Eval/IFBench and role-playing LiveCodeBench v5/AIME25 are present. Comparing capability types reveals more specific patterns. All changes below are relative to vanilla MOPD:
| Capability / benchmark | Role-playing setting | Medical setting |
|---|---|---|
| Coding: LiveCodeBench v5 | +4.66 | +3.59 |
| Coding: LiveCodeBench v6 | 0.00 | +3.43 |
| Scientific problem solving: GPQA-Diamond | +4.04 | +1.64 |
| Structured logical reasoning: ZebraLogic | +5.10 | +1.10 |
| Instruction following: IF-Eval | +2.96 | +23.47 |
| Generalization to new constraints: IFBench | +1.02 | -1.02 |
| Competition mathematics: HMMT25 | +1.04 | +1.66 |
| Competition mathematics: AIME25 | -3.75 | +1.88 |
Coding, scientific problem solving, and structured logical reasoning show relatively consistent gains. LiveCodeBench v5, GPQA, and ZebraLogic improve in both settings, while LiveCodeBench v6 is unchanged in role-playing and improves in the medical setting. These are therefore capability areas that benefit relatively consistently in our experiments. LiveCodeBench evaluates code correctness through execution tests, GPQA tests challenging scientific question answering, and ZebraLogic tests constraint-based logical reasoning. These results also show that the benefits extend beyond a single instruction-following benchmark.
Recovery on standard verifiable instruction following is substantial, while generalization to new constraints is weaker. IFEval evaluates 25 types of verifiable instructions, whereas IFBench specifically tests generalization to 58 new, diverse, and challenging out-of-domain constraints. IFEval improves in both settings, especially by 23.47 points in the medical setting; IFBench instead decreases by 1.02 points in that setting. Thus, the evidence more directly supports recovery of the instruction-following abilities covered by IFEval than comparable improvement in generalization to new constraints. The benchmark-design difference offers a plausible interpretation, although establishing whether it causes the gap in our model requires constraint-level analysis.
Gains in competition mathematics depend more on the setting. AIME25 tests competition-level mathematical problem solving: its score falls from 47.29 to 43.54 in role-playing but rises from 54.37 to 56.25 in the medical setting; HMMT25 improves in both. This differs from the patterns on coding, GPQA, and ZebraLogic. Coding improvements therefore do not imply simultaneous gains in competition mathematics, nor does the role-playing AIME regression imply a general deterioration in mathematical reasoning. MMLU-Redux and MMLU-Pro also lack consistent improvements across settings, so we will not characterize the results as uniform gains in general knowledge.
One interpretation consistent with our design is that UCMOPD improves the student's utilization of existing teacher feedback, without directly optimizing generalization to out-of-domain constraints or final mathematical-answer correctness. This may help explain why some capabilities recover while others do not improve simultaneously. We will distinguish this explanatory hypothesis from the patterns directly observed in the table.
Our contribution is to identify and improve learning-signal utilization in vanilla MOPD for capability recovery. Vanilla MOPD is therefore the central control for evaluating the proposed changes, and the capability patterns above refer to that comparison. The general averages increase from 52.09 to 54.43 and from 51.81 to 55.88 in the two settings, while domain averages are maintained or improved. We will expand the discussion to highlight the gains in coding, scientific problem solving, structured logical reasoning, and instruction-following recovery on IFEval, while reporting the trade-offs on benchmarks such as IFBench and AIME.
Response to Reviewer kYJX (1)
Thank you for recognizing the motivation, the simplicity of the method, and the component and rollout-budget comparisons. We appreciate your questions about the optimization target, compatibility with other distillation losses, and differences across benchmarks. We address them separately below.
W1. What is the optimal student distribution when Algorithm 1 converges?
We agree that changing gradient weights requires analyzing the resulting update, rather than assuming that the original objective is preserved. To answer your question precisely: in a realizable setting with independent logits at each prefix, the routed teacher is the unique interior stationary distribution of Algorithm 1's expected distillation update, under the conditions below.
First, consider the complete algorithm at a fixed prefix, with a uniquely defined, full-support routed teacher distribution p and student distribution q=softmax(z). Assume independent conditional logits, positive loss averaging, and no auxiliary training objective. Write the expected descent direction as
b_i=\kappa_i(q)\log\frac{p_i}{q_i},$$ where $\kappa_i\geq0$ incorporates token exposure, trajectory selection, CLL retention, and positive averaging factors. Importantly, $\kappa_i>0$ whenever $q_i<p_i$: Algorithm 1 always retains a temperature-1, top-p-1 anchor, and its positive-advantage CLL keep probability $s_i=\min(p_i e^{H[p]},1)$ is strictly positive. If $q\neq p$, normalization implies that some token is underestimated and another is overestimated. Consequently, some $b_i>0$, while some $b_j\leq0$. But stationarity $u=0$ requires $b_i/q_i=\sum_j b_j$ for every token, which is impossible with these different signs. At $q=p$, all advantages vanish. Thus, **$q^*=p$ is the only interior stationary distribution in this idealized setting**, including when some negative-advantage updates are always filtered. The result applies at prefixes with positive anchor visitation; it does not constrain unseen prompts. Second, an explicit objective can be given for the isolated CLL component. At a fixed prefix, with fresh samples $i\sim q$, no trajectory selection, and normalization independent of the Bernoulli masks, define $$a_i(q_i)= \begin{cases} s_i,&q_i\leq p_i,\\ 1-s_i,&q_i>p_i, \end{cases} \qquad F(q;p)=\sum_i a_i(q_i) \left[q_i\log\frac{q_i}{p_i}-q_i+p_i\right].$$ Its gradient equals the expected CLL-masked surrogate gradient: $$\nabla_\theta F =-\mathbb E_{i\sim q}\left[ a_i(q_i)\log\frac{p_i}{q_i}\nabla_\theta\log q_i \right].$$ The branch boundary is differentiable because both the bracketed term and its derivative vanish at $q_i=p_i$. Each summand is nonnegative. If $q\neq p$, at least one underestimated token has $a_i=s_i>0$ and contributes strictly positively. Therefore, $F$ has the unique global minimizer $q=p$ on the probability simplex. **Therefore, under the idealized conditions above, if Algorithm 1's expected update converges to an interior stationary point, the student distribution must match the corresponding teacher distribution. For the isolated CLL component, we additionally provide an explicit objective with the same unique optimal distribution.** We will add these derivations and their conditions to the appendix.