- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper makes analyses on on-policy distillation to answer several questions such as stronger reference/model scale impacts. Based on unified any-to-any formulation to cover several OPD variants, various empirical analyses are provided. For example, it was demonstrated that model scale alone as a reference model does not always help, whereas task-specific models are consistently beneficial.
Critical strengths
This paper first formulate several OPD methods and analyzes any-to-any OPD with different teacher/student models. Several empirical findings include: model scale as a reference model does not always help; task-specific models can consistently benefit distillation; the transfer role of expert and filter role of reference; matching reference benefits. Relatively rich benchmarks are considered across math and coding tasks.
Critical weaknesses
Fig. 2 overlap ratio shows comparison of task-specific or agnostic models, whereas the scale-only impact should compare the case with smaller target model like 1.7B-RL as in Fig. 1(a). All experiments consider alpha=beta and one learning rate, fixed step. However, different proxy pairs have significantly different log-ratio probability. So, we should sweep those hyperparameters to have a sufficiently converged performance to compare. Experiments use a limited token coverage within top 16 tokens. The paper does not provide sensitivity experiments for the token cutoff, the fraction of retained tokens for each source–target pair, or the resulting approximation errors in policy ratios and KL diagnostics. The authors associate successful transfer with increasing expert overlap and generalization to validation examples. However, both quantities are observed during or after target optimization. The paper does not provide a measurable criterion available before training that predicts whether a particular proxy pair will transfer to a given target. The target sweep covers Qwen3 and Qwen2.5 variants, without testing independently developed model families. Most experiments focus on mathematical reasoning; the code results cover a small selection of representative configurations rather than the complete matrix. No training-seed variability, confidence intervals, or significance tests are reported for the transfer comparisons. There are also numerical inconsistencies: the Qwen3-14B/Qwen3-4B source gain is reported as +1.11 in Table 2 but +1.72 in Table 3.
Questions for authors
The state distribution depends on the target policy, but Eq. (8) ignores its gradient. Under what assumptions does this surrogate optimization approximate the original objective, and how might this approximation affect transferability? The authors claim that matching the reference model's generation improves transfer. However, changing the reference also changes the transferred policy difference . How can the effects of generation compatibility and changes in the transferred learning signal be disentangled? Task-trained proxy pairs generally have larger task-performance gains than scale-only pairs. Could the observed transfer differences primarily reflect the proxy performance gap , rather than the source of the policy difference? Can the authors provide comparisons controlling for ? All experiments use a fixed transfer strength . Could be evalluated by tuning this ratio or normalizing the transferred log-probability differences? The initial proxy–target distance is measured using trajectories generated by the optimized target, making it dependent on the training outcome. Would the conclusions regarding the relationship between policy distance and transferability remain unchanged if distances were evaluated on a shared, fixed set of states before training?
11 comments
Response to Reviewer Nge2 [2/3]
Response to Weakness 5
We agree that both families tested are Qwen. During the rebuttal period our compute was prioritised for the sensitivity analyses the reviewers requested, so an additional independently developed family is not included here; we will add one in the revision.
Response to Weakness 6
We agree that the code results are a targeted check rather than a full matrix. Their purpose is to test whether the findings obtained on math survive a change of domain, not to re-run the grid, and the configurations we did run reproduce the ordering observed on math. Extending the code matrix is a scope we acknowledge as future work.
Response to Weakness 7
We now report bootstrap 95% confidence intervals (CI), computed by resampling each benchmark's evaluation set with replacement 1000 times over per-sample Mean@16, so that the reported comparisons carry an explicit measure of evaluation uncertainty.
| Pair | Mean@16 | 95% CI | Outcome |
|---|---|---|---|
| Qwen3-4B-RL / Qwen3-4B → Qwen3-8B | 45.27 | [40.06, 50.04] | success |
| Qwen3-4B-RL / Qwen2.5-3B → Qwen3-8B | 47.52 | [42.34, 52.11] | success after reference swap |
| Qwen3-4B-RL / Qwen3-4B → Qwen2.5-7B | 0.05 | [0.00, 0.13] | failure |
| Qwen3-4B-RL / Qwen2.5-3B → Qwen2.5-7B | 8.53 | [5.96, 11.15] | recovered by reference swap |
| Qwen3-1.7B-RL / Qwen3-1.7B → Qwen3-8B | 28.21 | [23.69, 32.60] | success |
| Qwen3-1.7B-RL / Qwen3-1.7B → Qwen2.5-7B | 0.15 | [0.05, 0.27] | failure |
The intervals for successful and failing configurations are disjoint by a wide margin.
Response to Weakness 8
The reviewer is right that the two tables disagree. The correct value is +1.72: Table 3, row Qwen3-14B / Qwen3-4B-Base, reports 18.43 → 19.54 (+1.72), and Table 6 gives the same two scores. Table 2 mistakenly used the Qwen3-8B value 18.43 as the Qwen3-4B baseline in that row. Table 3 is correct, and we will fix Table 2.
Response to Question 1
Eq. (8) differentiates the state-wise objective: for a state drawn from , the loss is the per-state divergence at and the gradient is taken with held fixed. The term does not appear because is not part of that objective — the surrogate is exact for it, not an approximation of it.
Removing the term does not move the optimum. The state-wise objective is zero at every state the target visits exactly when matches the teacher's distribution on those states, and that same condition is the global minimum of the trajectory-level objective, where the omitted term vanishes. The term therefore affects the update direction away from the optimum, i.e. the optimization path, but not the optimum. The state distribution is updated implicitly rather than through the gradient: states are re-sampled from the current target at every step, so each iteration re-solves the per-state problem on the new state distribution — the same treatment used in PPO and GRPO. Appendix B.3 derives both gradients explicitly.
Response to Reviewer Nge2 [3/3]
Response to Question 2
We agree, and the sentence in §4.4 is incorrect as written. Since , replacing the proxy reference changes the transferred log-ratio by construction, so generation compatibility cannot be separated from the learning signal. Fixed in this comparison: the proxy expert (the same Qwen3-4B-RL checkpoint), the target and its initialization, the data, the steps, and . Not fixed: the reference, and with it both the magnitude and the direction of .
We will replace the sentence "To test whether the family split is causal, we hold the difference fixed and change only the reference's family" with:
"We fix the proxy expert and vary only the proxy reference. Because the reference enters , this changes the transferred log-ratio as well as the reference's model generation; the comparison thus reflects their combined effect on transfer."
Response to Question 3
This is controlled in Table 3. In the block labelled "Two RL increments with similar proxy gains", is +4.80 and +5.39 while is +9.62 and +25.03: proxy gains are nearly identical and target gains differ by a factor of 2.6. We agree that is strongly associated with across Table 2 (Pearson r = 0.96, Fig. 1b) and we state this in §4.1; the controlled comparison shows it is not sufficient.
Response to Question 5
On Question 5: is measured on target-visited states because distillation is on-policy — is evaluated only on states the target itself generates during training. A distance measured on a fixed shared state set would score each pair on states the training signal never reaches, so it could rank pairs differently from the states where transfer actually acts. is used in the paper as a diagnostic of what happened, not as a pre-training predictor, and the claim it supports is negative: even measured on exactly those states, distance does not separate successful from failing pairs.
We will add these analyses to the revision.
Response to Reviewer Nge2 [1/3]
We thank the reviewer for the constructive assessment. The requested sensitivity analyses are below and will be added to the revision.
Response to Weakness 1
Fig. 2 compares two pairs at the same expert scale, chosen because their induced updates are of comparable magnitude: the task-trained pair Qwen3-4B-RL / Qwen3-1.7B and the scale-only pair Qwen3-4B / Qwen3-1.7B (KInduced 0.3531 vs. 0.4181, Table 2). The target's overlap with the task-trained expert rises from 0.75 to about 0.81 and overtakes the scale-only curve within 20 steps, while the scale-only overlap drifts down to 0.78.
The 1.7B-scale case has no non-trivial scale-only counterpart, because Qwen3-1.7B against itself is a zero difference. What that scale does provide is the task-trained pair itself, Qwen3-1.7B-RL / Qwen3-1.7B → Qwen3-8B, already reported in Table 2 (first row): 18.43 → 28.21 (+9.78), KInduced=0.0773. Task-trained differences therefore transfer at the smaller scale as well, and the scale-only control has to come from a larger scale, which is why Fig. 2 uses the 4B/1.7B pair.
Response to Weakness 2 & Question 4
We swept λ ∈ {0.5, 1, 1.5, 2} on two successful and two failing configurations, with additional λ ∈ {0.75, 0.95, 1.1} for the first pair.
The trend is consistent with prior work[1][2]: λ materially changes the final score, and its best value depends on the proxy pair — 1.0 for Qwen3-4B-RL / Qwen3-4B and 1.5 for Qwen3-1.7B / Qwen3-1.7B, the latter collapsing to 0.00 at λ = 0.5. The success/failure split, however, is unchanged across the sweep: the failing pairs stay at or below 0.39 (pre-distillation 1.79) and 3.29 (pre-distillation 0.77) at every λ tested.
[1] Fu D, Wu R, Yang Y, et al. Proxy OPD: On-Policy Distillation with Transferable Relative Proxy Update[J]. arXiv preprint arXiv:2607.11505, 2026.
[2] Feng S, Gao H, Chi H, et al. Weak-to-Strong Generalization via Direct On-Policy Distillation[J]. arXiv preprint arXiv:2607.05394, 2026.
Response to Weakness 3
We swept the top-k cutoff over k ∈ {0, 4, 16, 32, 64} on Qwen3-4B-RL / Qwen3-4B → Qwen3-8B.
The score spans 1.08 over the whole range, and the curve is flat up to k = 16: k = 16 costs 40.0 min/step and scores 0.83 below k = 64, which costs 46.4 min/step. Beyond k = 16 the remaining gain is therefore bought at a rising price, as OPD preparation time grows from 9.9 to 15.0 min/step.
We also measured how much probability mass the student's top-16 set retains, on 500 randomly drawn training-set samples, for four representative pairs.
Across these four pairs, the student's top-16 set already retains at least 95.7% of each frozen model's probability mass, so top-16 is sufficient for computing the reward.
Response to Weakness 4
We tested a broad set of candidate a-priori criteria and none of them separates successful from failing pairs: KBase, KInduced, KTarget, the pre-training expert–target overlap, and combinations of these. All three distances are computed for every configuration and reported in Tables 7–10, with the evaluation protocol in Appendix C.
Two failures of these criteria illustrate this. The failing cross-generation pair Qwen3-4B-RL / Qwen3-1.7B → Qwen2.5-7B (Table 10, row 22) carries the largest induced distance in our grid (KInduced=0.6416) together with a small target-side distance (KTarget=0.0397), so it is extreme in one coordinate and unremarkable in the other. Conversely, the two pairs with similar KInduced (0.3531 vs. 0.4181, Table 2) differ by more than 22 points in target gain (+25.85 vs. +3.42).