- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies that in strong-to-weak on-policy distillation (OPD), the teacher's contrast among the student's likely next tokens decays along a rollout, even though the mean advantage stays non-zero. This is an interesting observation. The proposed LDA-OPD measures the teacher's top-1/2 margin M_t within the student's top-K tokens and averages it per sentence. A profiled RSS-BIC test finds one downward change point. The loss after it is zeroed, and the prefix is rescaled to keep the loss mass. With a Qwen3-30B-A3B teacher, Qwen3-1.7B rises from 36.8% (OPD) to 40.1% math AVG, against 39.4% for the best fixed prefix. Results for 4B/8B students, Gemma3, code, GPQA and IFEval show similar trends.
Critical strengths
- The proposed approach is simple, practical yet effective. A drop-in loss mask that removes the need for a cutoff sweep at no extra cost.
- The proposed method has consistent empirical gains, Improving over full OPD and fixed-prefix baselines across three scales, two model families and four task types.
- The paper is easy to follow. The motivation and method are clearly presented. It's appreciated that the code is released.
Critical weaknesses
-
The empirical evidence seems not to fully support the theory. App. A.2 links useful updates to the teacher–student contrast, but M_t depends only on the teacher. Var_{C_t}(A_t) is never tested, and the KL signal closest to the theory does worse (37.8%, Table 3). Fig. 1a (late mean advantage ≈ −0.3) and Case 2 (A.4) suggest the released suffix still carries corrective signal.
-
Gains sit within hindsight choices and noise. The lead is 0.7 pts over the best fixed prefix (1.7B) and 0.5 over FastOPD at 8B (Table 4). K=4 is the test-set "peak", the same hindsight choice the paper faults in fixed-prefix oracles. Seed runs compare only with FastOPD-2048. MATH500 uses 1 sample per problem, and the GPQA gain (42.4 vs 41.2) is called "significant" without a rigorous statistical test.
-
The ablations seem confounded. Rescaling alone adds 0.8 pts, more than the 0.6 lead over FastOPD without it, and it is unclear whether the baselines rescale. All Table 3 alternatives (single runs) fall below plain fixed prefixes (38.6–39.4%). Entropy rises as contrast falls, so a downward-change detector may misplace its release.
Questions for authors
-
Can the authors compare Var_{C_t}(A_t) and student-weighted reverse KL as release signals
-
Can the authors measure gradient SNR on released vs retained tokens to test A.2 empirically
-
Can the authors apply the same rescaling to the fixed-prefix baselines
4 comments
Q2
Thank you for this insightful suggestion. In our exploratory measurements of directional gradient SNR, we found that the reference direction was sensitive to the sampled trajectories and their token counts. We also lacked an independently justified basis for selecting a direction that represents useful optimization progress. We therefore evaluated conditional gradient SNR using the action-centered gradient component defined in Appendix A.2. On the initial Qwen3-1.7B checkpoint, we sampled 1,024 states from each region across 256 student-generated responses and enumerated the student’s top-4 candidate actions at each state. At each state, we centered the candidate-action advantages under the locally renormalized student distribution. We defined the signal as the squared norm of the expected centered gradient and the noise as the expected squared deviation of the advantage-weighted action gradients from that mean. The pooled conditional SNR is the sum of these signals divided by the sum of the corresponding noise terms. This full-vector measurement requires no reference projection direction and uses equal numbers of sampled states from both regions.
The mean teacher margin was 12.295 for the retained prefix and 8.072 for the released suffix, while the mean action-wise advantage variance was 1.432 and 0.865, respectively. The released suffix also had a 79.1% lower mean squared norm of the centered expected gradient than the retained prefix. The pooled conditional gradient SNR point estimate was also higher for retained states: 0.548 versus 0.283. These measurements provide empirical evidence of a weaker average local action-dependent signal in the released region, consistent with the motivation in Appendix A.2.
As an additional experiment, we applied direct supervision only to valid response tokens after the cutoff. This suffix-only variant achieves an in-domain Math-AVG of 34.7%, compared with 24.9% for the base student, 36.8% for full OPD, and 40.1% for LDA-OPD. Thus, suffix supervision retains learning value, although relying exclusively on it yields smaller gains in this setting. We will refine our claims to avoid broadly characterizing the released suffix as uninformative or harmful. Concurrent work has also reported related observations (e.g., arXiv:2605.27028).
Q3
Thank you for the suggestion. Please see our response to the third weakness.
Dear Reviewer hSy8,
Since the discussion phase is ending soon, we would like to kindly ask you to take a look at our rebuttal. We have carefully addressed all the concerns raised in your review and provided point-by-point responses in the rebuttal. If our response has addressed your concerns, we would be grateful if you could consider updating your assessment during the discussion phase.
Thank you very much for your time and thoughtful feedback. We would be happy to provide any further clarifications.
Best regards, The Authors
Dear Reviewer,
Thank you so much for your thoughtful initial reviews!
With only two days left before the discussion window closes, I wanted to send a friendly reminder to check out the authors' rebuttal. They have put a lot of effort into answering the specific questions you raised.
Whenever you have a spare moment today, please drop a brief reply in the thread to let the authors know if their responses have addressed your concerns, or if any questions remain. Your timely feedback makes a huge difference in ensuring a fair and constructive review process!
Warm regards
W1
Regarding "Var_{C_t}(A_t)" and "student-weighted reverse KL": We added experiments on the 1.7B student using Var_{C_t}(A_t) and student-weighted reverse KL as release signals, obtaining Math-AVG scores of 39.1% and 38.3%, below the margin signal's 40.1%.
While the advantage variance theoretically provides an upper bound on the magnitude of the centered update, achieving this bound is not guaranteed in practice due to factors such as training stability and the size of the candidate set. Furthermore, we acknowledge that, given the student's candidate set, the margin relies solely on teacher probabilities and is not strictly equivalent to the teacher–student advantage variance. Instead, we treat it as an empirically effective proxy.
We will clarify this distinction in the revised manuscript (And we will incorporate this and all other updates into the revised PDF once all reviewer feedback has been collected). Thank you very much for this insightful suggestion. Regarding "released suffix still carries corrective signal": The additional suffix-only result is 34.7% (please see our response to Reviewer AwDc's first weakness), above the base student's 24.9%. This also indicates that suffix supervision can retain learning value, although training with suffix-only supervision is less effective.
W2
We added three Math-AVG experiments of FastOPD-4096 on the 1.7B student, obtaining 39.4%, 38.6%, and 38.3%, below the 40.07% reported for LDA-OPD in Table 8. On the OOD benchmark GPQA, the three FastOPD-2048 experiments yield 41.2%, 41.4%, and 40.4%, while LDA-OPD yields 42.4%, 41.9%, and 42.9%, with means of 41.0% and 42.4%, respectively. We will replace “significant” with “a higher observed mean accuracy.” Thank you also for the feedback on our “sweep-free” claim. We will qualify the corresponding wording.
W3
Regarding "ablations seem confounded": We additionally evaluated FastOPD-2048 with reweighting on the 4B student, using the same per-sample reweighting formula and a loss denominator based on all valid response tokens, as in LDA-OPD. It achieves 52.7%, while LDA-OPD achieves 53.9%, a gain of 1.2 percentage points. Without reweighting, FastOPD-2048 and LDA-OPD achieve 52.5% and 53.1%, respectively. Thus, LDA-OPD consistently retains its performance advantage under both settings.
Regarding "below plain fixed prefixes": We posit that an adaptive selection mechanism does not inherently guarantee superiority over fixed prefixes; rather, its effectiveness depends critically on the choice of the underlying guiding signal.
Regarding "a downward-change detector": Our implementation already accounts for the direction of the entropy signal. We first transform the sentence-level teacher entropy within the candidate set as Hi′=maxjHj−Hi, then apply log1p and detect a downward change. An increase in the original entropy thus corresponds to a decrease in the transformed signal.
Q1
Thank you for the suggestion. Please see our response to the first weakness.