- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the scaling properties of OPD across 25 teacher–student combinations within the Qwen2.5 family (0.5B–14B). It identifies a regular "useful-transfer regime" where gold score increases approximately linearly in \sqrt{token-mean KL}, followed by a noisy phase. Joint power laws in student scale, teacher scale, and teacher score predict peak gold score and useful-transfer rate. Key findings include: (1) in every weak-to-strong pair the student surpasses the teacher; (2) at matched score, smaller teachers transfer better; (3) off-policy cold start harms weak-to-strong OPD; (4) bootstrapping chains do not improve over direct transfer from the smallest expert. The study also compares Vanilla-OPD and Delta-OPD objectives.
Critical strengths
The 5×5 full grid plus two OPD variants, on-policy degree comparison, and bootstrapping chains constitute a thorough empirical investigation.
Critical weaknesses
All experiments use Qwen2.5 (0.5B–14B) on a single math reasoning mixture (GSM8K + MATH). Whether the fitted power-law coefficients transfer to other model families (e.g., Llama), other tasks (e.g., code generation, general instruction following), or different post-training recipes (e.g., different RL algorithms, different SFT data) is entirely untested.
Questions for authors
Clarification:
- For the intermediate-teacher validation, you compare the 3B checkpoint at step 58 with the 1.5B RL endpoint. Was step 58 the earliest available checkpoint, or was it selected because it approximately matched the 1.5B endpoint's score? If the former, how sensitive is the joint law's prediction to the exact score of this checkpoint?
Future Work:
- You note that OPD with full-vocabulary loss accelerates weak-to-strong regression relative to sampled-token loss. Could this observation connect to the implicit-reward overoptimization mechanism you identify, does the full-vocabulary loss provide a more exploitable proxy?
3 comments
Follow-up on Reviewer Concerns
Dear Reviewer,
Thank you for your time and participation in the review phase.
Since the discussion is coming to an end in a day, we would like to kindly ask if you consider your concerns sufficiently addressed and would like to update your assessment of our work. We completely understand your busy schedules, and we would be sincerely grateful if you could take some time and leave a reply.
Thank you again for your time and contributions.
Sincerely, Authors
I thank the authors for the clarifications. Q1a and Q1b are adequately addressed. I also appreciate the explicit scoping of coefficient transfer and the honest reframing of the full-vocabulary-loss connection as a hypothesis.
However, my main concern remains: the scaling claims are supported only within Qwen2.5 and one math mixture, and the response does not provide cross-family, cross-task, or cross-recipe evidence. Moving the limitation to the introduction improves clarity but does not resolve the external-validity concern. I therefore maintain my rating.
We thank the reviewer for the close reading and the two precise questions. Revision letters refer to the global response; revised text is marked in green in the updated PDF.
Q1a. Checkpoint selection — step 58 is the earliest saved checkpoint, no checkpoint search took place, and the cell was excluded from every fit.
It was the earliest available checkpoint, with one nuance we now state explicitly. Our RL runs save one checkpoint per epoch (58 updates), so step 58 is the first checkpoint that exists, and no other intermediate checkpoint was evaluated. That its score of 66.0 approximately matches the 1.5B endpoint's 63.6 is coincidental — though a score-matched teacher is indeed what the comparison requires: had the score not matched, we would have retrained the 3B teacher to produce a score-matched intermediate checkpoint, rather than searched among later, higher-scoring saves. So the comparison is conditioned on score matching by design, but involved no selection among candidate checkpoints. The cell "is held out from every fit" (§H, Table 11), so it influenced no coefficient. The revision states this chronology in §H.
Q1b. Score sensitivity — ≈0.72 points of predicted peak per point of teacher score, and the predicted ordering is stable (Revision C).
The joint peak law gives this in closed form. With 1−Gpeak=AN~S−α(N~Teff)−β(1−GTeff)ζ and the fitted Vanilla-OPD ζ=0.95,
∂GT∂Gpeak=ζ1−GT1−Gpeak≈0.95×0.3400.259≈0.72
at the validation point (GT=66.0, predicted Gpeak=74.1). A one-point error in the intermediate teacher's measured score therefore shifts the predicted peak by ≈0.72 points, while the observed gap between the two students is 3.7 points — so the qualitative conclusion (the joint law orders the 3B-intermediate student below the 1.5B-endpoint student; the scale-only and score-only laws order it above) would reverse only if the measured teacher score were off by several points. We emphasize that this is the fitted formula's local sensitivity, not a new training experiment; it now appears in §H (Revision C).
Coefficient transfer — the limitation moves into the introduction (Revision A).
We agree. The revision also makes a distinction explicit: what we hypothesize may recur across settings is the functional form (multiplicative power law in student scale, capped teacher scale, and teacher remaining error, with the linear-in-KL useful-transfer regime) and the fitting protocol (including breaking the scale–score collinearity with bootstrapped-chain teachers); the coefficients are specific to Qwen2.5, our math mixture, and our recipe, as is standard for scaling-law constants. The within-family evidence cannot establish that hypothesis, so the revision states it as a hypothesis, with the three held-out validations (global response, point 2) as in-scope support.
Full-vocabulary loss — plausibly a denser, more exploitable proxy; stated as a hypothesis linked to Figure 9 (Revision D).
We find this connection plausible and thank the reviewer for articulating it. The full-vocabulary reverse KL supplies gradient signal on every token of the vocabulary at every student-visited prefix, including tokens where a weak teacher's conditional distribution is itself miscalibrated; the sampled-token estimator only propagates signal through tokens the student actually samples. The full-vocabulary objective is therefore a denser — and plausibly more exploitable — proxy for the teacher-induced implicit reward. This is consistent with the signature in Figure 9, where the proxy keeps improving while gold score regresses, and with our observation (§K) that full-vocabulary loss accelerates and intensifies post-peak weak-to-strong regression. Our logs do not contain the per-token diagnostics needed to test the mechanism (e.g., whether regression concentrates on tokens with large teacher–student support mismatch), so the revision states it as an explicit hypothesis connected to Figure 9, and we agree it is a promising direction for future work.