acceptodds
Under review as a conference paper at ICLR 2027

Scaling Properties of Capability Transfer in Same-Family On-Policy Distillation

Abstract

*Reinforcement learning (RL)* can induce substantial reasoning capabilities in large language models (LLMs), but how much of this capability transfers across model scales, and how quickly, remains unclear. We study the scaling properties of *on-policy distillation (OPD)* across *weak-to-strong*, *same-base*, and *strong-to-weak* teacher–student setups. We find that early OPD training dynamics uniformly exhibit a regular *useful-transfer* regime, in which held-out accuracy (the *gold score*, ) rises approximately linearly in , the square root of token-level reverse KL divergence from the student initialization. In every observed weak-to-strong pair, the student's peak gold score exceeds its teacher's own, so a compact RL expert can transfer capability to a much larger student via OPD. To estimate OPD outcomes, we fit *power laws* for how and the slope of the useful-transfer regime scale with student and teacher parameter counts and with teacher gold score. These laws show that peak gold score improves with teacher scale only up to roughly the student's scale, and that at a matched gold score smaller teachers transfer better, so a teacher's score alone does not define its supervision value. We also study the scaling effects of two OPD variants, bootstrapping weak-to-strong OPD, and the degree of on-policy supervision.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, scaling law, weak-to-strong generalization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9MYtOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the scaling properties of OPD across 25 teacher–student combinations within the Qwen2.5 family (0.5B–14B). It identifies a regular "useful-transfer regime" where gold score increases approximately linearly in \sqrt{token-mean KL}, followed by a noisy phase. Joint power laws in student scale, teacher scale, and teacher score predict peak gold score and useful-transfer rate. Key findings include: (1) in every weak-to-strong pair the student surpasses the teacher; (2) at matched score, smaller teachers transfer better; (3) off-policy cold start harms weak-to-strong OPD; (4) bootstrapping chains do not improve over direct transfer from the smallest expert. The study also compares Vanilla-OPD and Delta-OPD objectives.

Critical strengths

The 5×5 full grid plus two OPD variants, on-policy degree comparison, and bootstrapping chains constitute a thorough empirical investigation.

Critical weaknesses

All experiments use Qwen2.5 (0.5B–14B) on a single math reasoning mixture (GSM8K + MATH). Whether the fitted power-law coefficients transfer to other model families (e.g., Llama), other tasks (e.g., code generation, general instruction following), or different post-training recipes (e.g., different RL algorithms, different SFT data) is entirely untested.

Questions for authors

Clarification:

  1. For the intermediate-teacher validation, you compare the 3B checkpoint at step 58 with the 1.5B RL endpoint. Was step 58 the earliest available checkpoint, or was it selected because it approximately matched the 1.5B endpoint's score? If the former, how sensitive is the joint law's prediction to the exact score of this checkpoint?

Future Work:

  1. You note that OPD with full-vocabulary loss accelerates weak-to-strong regression relative to sampled-token loss. Could this observation connect to the implicit-reward overoptimization mechanism you identify, does the full-vocabulary loss provide a more exploitable proxy?
3 comments

We thank the reviewer for the close reading and the two precise questions. Revision letters refer to the global response; revised text is marked in green in the updated PDF.

Q1a. Checkpoint selection — step 58 is the earliest saved checkpoint, no checkpoint search took place, and the cell was excluded from every fit.

It was the earliest available checkpoint, with one nuance we now state explicitly. Our RL runs save one checkpoint per epoch (58 updates), so step 58 is the first checkpoint that exists, and no other intermediate checkpoint was evaluated. That its score of 66.0 approximately matches the 1.5B endpoint's 63.6 is coincidental — though a score-matched teacher is indeed what the comparison requires: had the score not matched, we would have retrained the 3B teacher to produce a score-matched intermediate checkpoint, rather than searched among later, higher-scoring saves. So the comparison is conditioned on score matching by design, but involved no selection among candidate checkpoints. The cell "is held out from every fit" (§H, Table 11), so it influenced no coefficient. The revision states this chronology in §H.

Q1b. Score sensitivity — ≈0.72 points of predicted peak per point of teacher score, and the predicted ordering is stable (Revision C).

The joint peak law gives this in closed form. With 1−Gpeak=A N~S−α(N~Teff)−β(1−GTeff)ζ1-G_{\text{peak}} = A\,\tilde N_S^{-\alpha}(\tilde N_T^{\text{eff}})^{-\beta}(1-G_T^{\text{eff}})^{\zeta} and the fitted Vanilla-OPD ζ=0.95\zeta = 0.95,

∂Gpeak∂GT=ζ 1−Gpeak1−GT≈0.95×0.2590.340≈0.72\frac{\partial G_{\text{peak}}}{\partial G_T} = \zeta\,\frac{1-G_{\text{peak}}}{1-G_T} \approx 0.95 \times \frac{0.259}{0.340} \approx 0.72

at the validation point (GT=66.0G_T = 66.0, predicted Gpeak=74.1G_{\text{peak}} = 74.1). A one-point error in the intermediate teacher's measured score therefore shifts the predicted peak by ≈0.72 points, while the observed gap between the two students is 3.7 points — so the qualitative conclusion (the joint law orders the 3B-intermediate student below the 1.5B-endpoint student; the scale-only and score-only laws order it above) would reverse only if the measured teacher score were off by several points. We emphasize that this is the fitted formula's local sensitivity, not a new training experiment; it now appears in §H (Revision C).

Coefficient transfer — the limitation moves into the introduction (Revision A).

We agree. The revision also makes a distinction explicit: what we hypothesize may recur across settings is the functional form (multiplicative power law in student scale, capped teacher scale, and teacher remaining error, with the linear-in-KL\sqrt{\mathrm{KL}} useful-transfer regime) and the fitting protocol (including breaking the scale–score collinearity with bootstrapped-chain teachers); the coefficients are specific to Qwen2.5, our math mixture, and our recipe, as is standard for scaling-law constants. The within-family evidence cannot establish that hypothesis, so the revision states it as a hypothesis, with the three held-out validations (global response, point 2) as in-scope support.

Full-vocabulary loss — plausibly a denser, more exploitable proxy; stated as a hypothesis linked to Figure 9 (Revision D).

We find this connection plausible and thank the reviewer for articulating it. The full-vocabulary reverse KL supplies gradient signal on every token of the vocabulary at every student-visited prefix, including tokens where a weak teacher's conditional distribution is itself miscalibrated; the sampled-token estimator only propagates signal through tokens the student actually samples. The full-vocabulary objective is therefore a denser — and plausibly more exploitable — proxy for the teacher-induced implicit reward. This is consistent with the signature in Figure 9, where the proxy keeps improving while gold score regresses, and with our observation (§K) that full-vocabulary loss accelerates and intensifies post-peak weak-to-strong regression. Our logs do not contain the per-token diagnostics needed to test the mechanism (e.g., whether regression concentrates on tokens with large teacher–student support mismatch), so the revision states it as an explicit hypothesis connected to Figure 9, and we agree it is a promising direction for future work.

AuthorsOpenReview ↗

Follow-up on Reviewer Concerns

Dear Reviewer,

Thank you for your time and participation in the review phase.

Since the discussion is coming to an end in a day, we would like to kindly ask if you consider your concerns sufficiently addressed and would like to update your assessment of our work. We completely understand your busy schedules, and we would be sincerely grateful if you could take some time and leave a reply.

Thank you again for your time and contributions.

Sincerely, Authors

I thank the authors for the clarifications. Q1a and Q1b are adequately addressed. I also appreciate the explicit scoping of coefficient transfer and the honest reframing of the full-vocabulary-loss connection as a hypothesis.

However, my main concern remains: the scaling claims are supported only within Qwen2.5 and one math mixture, and the response does not provide cross-family, cross-task, or cross-recipe evidence. Moving the limitation to the introduction improves clarity but does not resolve the external-validity concern. I therefore maintain my rating.

Reviewer_82kXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the scaling properties of capability transfer through on-policy distillation (OPD), considering weak-to-strong, same-base, and strong-to-weak teacher–student settings. The authors characterize early-stage transfer dynamics using a square-root KL coordinate and propose empirical power laws to predict peak student performance and initial transfer rates from teacher and student model sizes and teacher accuracy.

Experiments are conducted on Qwen2.5 models ranging from 0.5B to 14B parameters, using mathematics reasoning tasks. The paper also compares Vanilla-OPD with Delta-OPD and investigates the effects of off-policy cold starts and multi-stage distillation.

Critical strengths
  1. Systematic empirical characterization. The paper studies OPD across multiple teacher–student configurations and provides a relatively comprehensive analysis of transfer dynamics, including comparisons between Vanilla-OPD and Delta-OPD.

  2. Potentially useful predictive framework. Modeling the relationship between teacher/student scale and achievable student performance is an interesting direction. If the proposed relationships generalize, they could help guide teacher selection and post-training decisions.

Critical weaknesses
  1. Limited model scale and outdated experimental setting. All experiments are conducted on Qwen2.5 Base models, with the largest model being only 14B parameters. Qwen2.5 was released in 2024, and these base checkpoints do not represent modern, reasoning-specialized models. Although the authors apply RL to obtain reasoning experts, it remains unclear whether the observed transfer dynamics hold for more recent reasoning models, especially at substantially larger scales. This limitation is particularly concerning given the paper's focus on scaling properties.

  2. Insufficient evidence for the generalizability of the proposed scaling laws. Although the authors evaluate held-out model sizes within Qwen2.5, the predictive formulas are not validated on independent model families, different task domains, or substantially larger models. All main experiments are restricted to a single model family and a mathematics reasoning setting. Consequently, it is unclear whether the fitted relationships reflect general scaling principles of OPD or merely empirical regularities specific to the chosen experimental setup. Stronger out-of-distribution and large-scale validation is needed to substantiate the central claims.

Questions for authors
  1. Do the proposed scaling laws generalize to more recent reasoning-specialized model families, such as Qwen3 Thinking models, without refitting the coefficients?

  2. How well do the fitted formulas extrapolate beyond 14B parameters, for example to 32B or 70B-scale students? Does the same functional form remain predictive?

  3. Do these scaling relationships hold beyond mathematics reasoning, such as in coding or other reasoning-intensive tasks? Evidence of cross-domain transfer would substantially strengthen the paper's central claims.

2 comments

We thank the reviewer for the careful summary and for acknowledging the "systematic empirical characterization" and the "potentially useful predictive framework" for teacher selection. The three questions mark the boundary of what our evidence supports; we answer each directly below. Revision letters refer to the global response; revised text is marked in green in the updated PDF.

Q1. Qwen3 Thinking without refitting is out of scope; what we hypothesize transfers is the functional form and protocol, with constants refit per setting (Revision A).

Our intended claim is narrower, and the revised introduction states it explicitly: we hypothesize that the functional form (a multiplicative power law in student scale, capped teacher scale, and teacher remaining error, together with the linear-in-KL\sqrt{\mathrm{KL}} useful-transfer regime) and the fitting protocol (including the use of bootstrapped-chain teachers to break the scale–score collinearity) may carry over to other settings, while the fitted constants are specific to Qwen2.5 Base, our math mixture, and our recipe and would need refitting per setting. This aligns with how constants behave elsewhere in the scaling-law literature (e.g., pretraining-law constants differ between Kaplan et al. and Hoffmann et al. with data and tokenization choices, and the overoptimization laws of Gao et al. are refit per reward model). A further caveat specific to the reviewer's example is now in the limitations: our students are base models whose reasoning capability is largely induced by the transfer itself, whereas a reasoning-specialized initialization such as Qwen3 Thinking starts with substantial reasoning behavior, so even the qualitative dynamics (e.g., the cold-start effect) may differ there.

Q2. Extrapolation beyond 14B; the revision publishes the law's Qwen2.5-32B/72B predictions as labeled, falsifiable extrapolations (Revision B).

Within the family, extrapolation is tested over one octave (fits on 0.5B–7B predict the held-out 14B student within 0.7 accuracy points for Vanilla-OPD and 0.2 for Delta-OPD; Tables 2 and 12). Rather than leave the question open, the revised §H adds a table (Table 13) of the joint law's predicted peaks for the larger released Qwen2.5 Base models, clearly labeled as extrapolation beyond the fitted range: distilling from our 1.5B RL endpoint (gold score 63.6) under the canonical recipe, the Vanilla-OPD joint law using full-precision coefficients predicts peak gold scores of 85.5 for a Qwen2.5-32B student and 88.7 for Qwen2.5-72B, with the fitted 14B cell (predicted 81.4, observed 82.1) as an in-range anchor. Running these cells is beyond our resources in this cycle — our single completed 14B-student/14B-teacher cell required ≈1,050 GPU-hours (131 hours on 8 GPUs), and the cost grows with student scale — so we offer the predictions as concrete, falsifiable claims for future validation.

Q3. Beyond mathematics reasoning; the limitation moves into the introduction and abstract (Revision A).

We agree that cross-domain evidence would substantially strengthen the central claims, and we consider it the most important follow-up experiment.

Qwen2.5's age and the 14B ceiling — chosen so a full 5×5 grid on a common recipe is feasible; the revision states this scope plainly.

The study requires five scales sharing a common pretraining recipe with per-scale compute-optimal tuning, all small enough that the full 5×5 OPD grid plus RL expert training is feasible — the grid as a whole, not any single run, is the binding cost. We agree that this choice does not establish relevance to frontier reasoning models, and the revision will not imply otherwise.

We hope Revisions A and B, together with the paper's three forms of held-out validation (leave-one-scale-out, largest-scale extrapolation, and the out-of-sample intermediate-teacher comparison; global response, point 2), clarify what the paper does and does not establish. We would be glad to run additional analyses on our existing logs during the discussion period if that would be helpful.

AuthorsOpenReview ↗

Follow-up on Reviewer Concerns

Dear Reviewer,

Thank you for your time and participation in the review phase.

Since the discussion is coming to an end in a day, we would like to kindly ask if you consider your concerns sufficiently addressed and would like to update your assessment of our work. We completely understand your busy schedules, and we would be sincerely grateful if you could take some time and leave a reply.

Thank you again for your time and contributions.

Sincerely, Authors

Reviewer_E3QAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies scaling properties of on-policy distillation (OPD) across five Qwen2.5 model scales. It characterizes early accuracy growth in a square-root-KL coordinate and fits scaling laws for peak accuracy and transfer rate. It also examines alternative distillation objectives, off-policy cold starts, and bootstrapped transfer. This paper concludes that OPD outcomes can be estimated from student scale, teacher scale, and teacher score before training, so RL expertise can be trained once at small scale and transferred predictably across a model family.

Critical strengths
  • The experiments systematically evaluate 25 teacher–student size combinations, covering weak-to-strong, same-base, and strong-to-weak distillation.
  • The cold-start and bootstrapping results provide useful observations for designing distillation pipelines.
  • Although this is mainly an empirical paper, the paper offers solid findings and practical guidance that are valuable to the community amid the current research frenzy.
Critical weaknesses
  • Evaluation is limited to GSM8K and MATH. Given the current state of progress in the community on informal math, such experiments are insufficient, especially for papers that are driven by experimental results. I suggest the authors perform more experiments on more challenging datasets.
  • Evaluation is limited to the Qwen2.5 series; I think this is also insufficient for an empirical paper.
  • The experiments cover only informal math tasks. It is unclear whether the findings hold for other tasks, such as coding or scientific reasoning.
Questions for authors

None

3 comments

We thank the reviewer for recognizing the systematic coverage of the 25 teacher–student combinations and, especially, for highlighting the cold-start and bootstrapping results as "useful observations for designing distillation pipelines" — the findings we most hope practitioners will use (an off-policy SFT phase harms weak-to-strong OPD increasingly with the capability gap; bootstrapped chains never beat direct transfer from the smallest expert, despite higher-scoring intermediate teachers). Revision letters refer to the global response; revised text is marked in green in the updated PDF.

The three stated weaknesses all ask for extensions of the evidence base — harder math, a second model family, non-math domains — so we answer them together: the extensions are valuable, we cannot run them in this cycle, and we believe the controlled single-family study is a complete contribution on its own terms.

One family, one recipe — a deliberate control, not an oversight; the scope statement moves into the introduction (Revision A).

Fitting scaling laws across teacher–student pairs requires that the cells differ only in scale. Mixing families would confound parameter count with tokenizer, data mixture, and architecture choices, making the central objects of the paper — the joint power laws and the matched-score teacher comparison — unidentifiable with any feasible number of runs. The same control sets the cost of any extension: our single completed 14B×14B cell took ≈1,050 GPU-hours, and a second family requires retraining the RL experts plus a comparable grid.

Within its scope, the study is predictive, not merely descriptive — three validations on data the fits never saw.

Leave-one-scale-out RMSE is halved against the scale-only baseline (Table 2); extrapolation predicts the held-out largest student and teacher scales within 0.7/0.4 accuracy points; and an out-of-sample intermediate-teacher run shows a higher-scoring 3B teacher producing a worse 7B student than the 1.5B endpoint — a reversal that only the joint law predicts and that teacher score alone would get wrong. We believe an empirical paper earns its conclusions through falsifiable predictions within its stated scope, rather than by breadth with weaker controls.

The requested extensions — acknowledged, and made actionable for the community (Revisions A and B).

The revised PDF names cross-family, cross-domain, and harder-benchmark validation as the directly actionable next experiments, states that the functional forms and the fitting protocol (including the collinearity-breaking bootstrapped chains) are the components we expect to recur, and adds the joint law's extrapolated predictions for 32B and 72B Qwen2.5 students (85.5 and 88.7 from the 1.5B expert) as explicit falsifiable claims (Revision B), so the laws can be tested by groups with the requisite compute.

Presentation — §5 is restructured, and we would welcome pointers for more (Revision E).

Independent of specific pointers, the revised §5 now states the two functional families back-to-back and moves the exponent-identification discussion after both, which we suspect was the densest part of the original text. If other particular sections or passages drove the presentation score, we would be grateful to know which, and will revise them as well.

AuthorsOpenReview ↗

Follow-up on Reviewer Concerns

Dear Reviewer,

Thank you for your time and participation in the review phase.

Since the discussion is coming to an end in a day, we would like to kindly ask if you consider your concerns sufficiently addressed and would like to update your assessment of our work. We completely understand your busy schedules, and we would be sincerely grateful if you could take some time and leave a reply.

Thank you again for your time and contributions.

Sincerely, Authors

Thanks for the authors' rebuttal. Since the experiments I requested were not conducted, I will maintain my rating of 2.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global response

We thank the Area Chair and all three reviewers for their contributions and careful reading. We are encouraged that they find "a thorough empirical investigation" (9MYt), "solid findings and practical guidance that are valuable to the community" (E3QA), and a "systematic empirical characterization" with a "potentially useful predictive framework" (82kX) — and that all three assess soundness as having only minor or fixable concerns, with no result, measurement, or fitting procedure disputed.

The shared concern is scope — whether the fitted relationships extend to other model families, other tasks, and larger or more recent models. Meanwhile, no reviewer question the value of our established contribution. We state our position once here, and each individual reply references the lettered revisions below.

1. The claims were scoped deliberately. The title specifies same-family on-policy distillation, and §B states: "Whether the fitted coefficients transfer to other families, tasks, or post-training recipes is untested." We chose a single pretrained family and a common recipe so that the 25 teacher–student cells differ only in scale — this controlled design is what makes the grid interpretable and the laws fittable at all. We do not claim that our fitted coefficients transfer across families and domains; rather, we advocate for transferability of the functional form of the power law. On other model families and domains, the power law coefficients are expected to be fitted from scratch under the same functional form.

2. Within scope, the laws are validated in three ways on held-out data.

  • Leave-one-scale-out: conditioning on teacher score halves the scale-only baseline's RMSE (3.43 → 1.66 Vanilla-OPD; 2.47 → 0.82 Delta-OPD; Table 2).
  • Largest-scale extrapolation: fits without the largest student or teacher scale predict the held-out 14B cells within 0.7 (Vanilla) and 0.4 (Delta) accuracy points.
  • Out-of-sample intermediate-teacher validation (excluded from every fit): a 3B checkpoint scoring above the 1.5B endpoint (66.0 vs 63.6) yields a worse 7B student (73.8 vs 77.5; joint-law prediction: 74.1), a reversal that only the joint law orders correctly. This matched-score reversal is also a practical teacher-selection result that teacher score alone cannot provide.

3. Why no second family or domain in this cycle. A single completed 14B-student/14B-teacher cell required ≈1,050 GPU-hours (131 hours on 8 GPUs); a second family amounts to a total cost of 15,800 H200 GPU hours, which we cannot afford. This explains the absence of those experiments; the revisions below answer the underlying concern by cautiously clarifying the claims and making the scaling laws falsifiable.

Revisions in the updated PDF (cited as A–E in our individual replies):

  • A. Scope statement moved into the introduction and abstract — all fitted coefficients are specific to Qwen2.5 + math reasoning + our recipe; what we hypothesize transfers is the functional form and the fitting protocol, with constants refit per setting.
  • B. Falsifiable extrapolated predictions — a new table (§H, Table 13) of the joint law's predicted peaks for Qwen2.5-32B and 72B students (85.5 and 88.7 from the 1.5B expert, using full-precision coefficients), explicitly labeled as extrapolation beyond the fitted range.
  • C. Sensitivity analysis — the joint law's local sensitivity to teacher score around the intermediate-teacher validation point, added to §H.
  • D. Full-vocabulary hypothesis — the §K observation restated as an explicit hypothesis linked to the implicit-reward-overoptimization evidence of Figure 9.
  • E. Presentation revisions — §5 now states the two functional families back-to-back, with the identification discussion following both.

We respond to each reviewer individually below; the revised PDF, with every new or reworded passage set in green for easy inspection, accompanies this response.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.