- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Prospective Supervision Weighting (PROW) for token-level weighting in on-policy distillation. Using the same student rollout, PROW combines each token’s immediate learning opportunity with a prospective term that estimates how the teacher-directed correction shifts later learning opportunities, then reallocates a fixed per-token optimization budget. Controlled interventions show that states with similar immediate gains can have very different downstream effects. PROW yields modest but fairly consistent gains over OPD, TA-OPD, and IW-OPD across Qwen3 teacher-student pairs and reasoning benchmarks.
Critical strengths
- The problem framing is clear and supported. Because the student writes its own future prefixes, a local update changes the supervision encountered later.
- The method is practical. PROW reuses standard OPD rollouts and adds no extra sampling, reward models, or critics. The single-rollout estimate of the prospective term tracks the 32-rollout reference well, which supports its use in standard pipelines.
Critical weaknesses
- The statistical reliability of the main results is unclear. Most gaps in Table 1 are under one point, and AIME uses only 30 problems. The paper reports a single seed and no variance or significance tests for Avg@8, so the win counts 23 of 30 cannot be separated from noise.
- The key ablations are missing. There is no random-weight control that keeps the same clipping and KL budget, and no direct comparison with existing lookahead methods, even though they are the closest related work. Since G-only and PROW differ by small margins in Figure 4, and PROW adds several empirical steps, it is hard to tell whether gains come from the prospective signal itself or from these design choices or from reweighting variance in general.
Questions for authors
- Could the author report means and standard deviations over at least three seeds, with confidence intervals or paired tests for AIME and AMC?
- Could you add a random-weight control under the same clipping, KL budget, and normalization, and a direct comparison with lookahead-based methods?
- What fraction of tokens have a zero prospective score since it is set to zero when y_t is outside B_t, and how much do PROW and G-only weights differ on those tokens?
- Were the baselines hyperparameters inherited from their original papers, or tuned under the same budget as PROW?
- Table 1 reports 33.00 Avg@8 for the Qwen3-4B-Base student init on Competition-MATH, while Table 2 reports 5.00 for the same student on GPQA-Diamond. Are these evaluated under identical settings?
3 comments
Dear Reviewer,
Thank you for the time and effort you’ve dedicated to reviewing our work. We have carefully addressed your concerns during the discussion phase. As the discussion period is nearing its close, we would greatly appreciate it if you could take a moment to review our responses and confirm whether they satisfactorily resolve your questions. Thank you once again for your thoughtful feedback and support.
Warm regards,
Paper's Authors
Dear Reviewer,
We would like to kindly follow up once more, as the discussion period is about to close. We would be very grateful if you could take a moment to revisit our responses and share any remaining concerns or questions.
We sincerely appreciate your time and consideration, especially during this busy period.
Best regards, The Authors
We thank the reviewer for recognizing the clear problem framing and the practicality of PROW. We address each concern below.
W1/Q1: Seeds and significance. We have updated the paper to report all main configurations over three training seeds (mean ± std), with Welch's tests and Holm–Bonferroni correction, in Appendix F.2 (Tables 6–8).
W2/Q2: Random-weight control. Regarding the reviewer's concern, we added a random-weight control whose weights are a random permutation of PROW's weights (Qwen3-4B → Qwen3-1.7B, Competition-MATH, three epochs, three seeds). It uses the same clipping and KL budget as PROW. The Avg@8 results are reported below:
Random reweighting falls below uniform OPD on every benchmark. PROW's gains therefore come from where the weight is placed, not from reweighting variance. The controlled intervention in Section 5.3 isolates the signal further: shuffling Dt within matched-Gt strata reduces the selected-state gain from 8.7 to 6.6, close to G-only (6.4) (Fig. 3b). The signal is therefore informative because it is state-specific. Additionally, Figure 4 shows that the combined score is highest or tied-highest in all 24 cases (20 strict) versus G-only and prospective-only, highlighting the gain from prospective information.
W2/Q2: Lookahead methods. We respectfully clarify that the lookahead methods discussed in Appendix A address a different problem. They change the objective or the supervised tokens and pay for future information with extra computation:
PROW, by contrast, uses the prospective signal in a lightweight way. It is computed in closed form from the existing rollout and logits, with no extra rollouts, forward passes or external validation, and only reweights the unchanged OPD loss. The appropriate baselines are therefore token-reweighting methods such as IW-OPD and TA-OPD, which we already compare against. We have revised Related Work and added Table 3, which makes explicit that the problems differ and that PROW obtains the prospective signal in a lightweight way.
At the reviewer's request, we nevertheless compare with the closest lookahead method, TOPD. We note that this comparison is not meant to position TOPD as a competing baseline, but to quantify the cost of obtaining future information. During training, TOPD's trajectory-aware update takes ≈90% of its step time (vs. ≈24% for PROW), making it roughly 3× more expensive per training token. We therefore compare both at the training time of PROW's full run, where PROW outperforms TOPD on all 10 metrics (results in our general comment).
Q3: Zero prospective scores. We have included this in Appendix F.7 (Table 11). Only 0.140% of sampled tokens fall outside B_t and have their prospective score set to zero. On these tokens, PROW and G-only weights differ by 0.102 on average (median 0.046). These tokens carry only 0.169% of the total PROW weight mass, so zeroing has a negligible effect on optimization.
Q4: Baseline hyperparameters. We clarify that all methods share the same training setting (Table 4). For IW-OPD and TA-OPD, we inherited the already-tuned hyperparameters from their original papers to reproduce their results. Similarly, PROW's hyperparameters are fixed across all teacher–student configurations and corpora, with no per-setting tuning.
Q5: Base-model evaluation. Thank you for the question. We inspected the responses of the Qwen3-4B-Base checkpoint and found that, for most GPQA-Diamond prompts, they are repetitive or meaningless, with no extractable answer, so they receive a low score. After post-training with an on-policy distillation framework such as PROW, the model recovers its accuracy. On Competition-MATH, under exactly the same evaluation configuration, the Base model’s responses contain an extractable answer much more often than on GPQA-Diamond, which yields the score of 33.0. The two scores are therefore not inconsistent with each other.