acceptodds
Under review as a conference paper at ICLR 2027

Beyond Pointwise Signals: Prospective Supervision Weighting in On-Policy Distillation

Abstract

On-policy distillation (OPD) for large language models (LLMs) provides dense teacher supervision along student-generated trajectories, yet existing token-weighting methods judge each position by signals local to the current state. This overlooks a defining property of OPD: because the student generates its own trajectories, learning at one state changes which states, and hence which supervision, it encounters later. Through controlled single-state interventions, we show that updates with nearly identical immediate learning gains can have opposite downstream effects. We therefore introduce Prospective Supervision Weighting (PROW), which scores each token by its immediate learning opportunity together with a prospective term estimating how the teacher-directed correction at that token shifts the student toward continuations offering more learning opportunity. PROW computes both terms from standard OPD rollouts and only reallocates a fixed per-token optimization budget, requiring no additional rollouts, reward models, or learned critics. Across reasoning benchmarks and model scales, PROW improves standard OPD and existing token-weighting baselines, while controlled interventions confirm that its prospective signal captures effects beyond local learning value.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Knowledge Distillation, Large Language Models, Supervision Weighting, Trajectory-Aware Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yHpzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Prospective Supervision Weighting (PROW) for token-level weighting in on-policy distillation. Using the same student rollout, PROW combines each token’s immediate learning opportunity with a prospective term that estimates how the teacher-directed correction shifts later learning opportunities, then reallocates a fixed per-token optimization budget. Controlled interventions show that states with similar immediate gains can have very different downstream effects. PROW yields modest but fairly consistent gains over OPD, TA-OPD, and IW-OPD across Qwen3 teacher-student pairs and reasoning benchmarks.

Critical strengths
  • The problem framing is clear and supported. Because the student writes its own future prefixes, a local update changes the supervision encountered later.
  • The method is practical. PROW reuses standard OPD rollouts and adds no extra sampling, reward models, or critics. The single-rollout estimate of the prospective term tracks the 32-rollout reference well, which supports its use in standard pipelines.
Critical weaknesses
  • The statistical reliability of the main results is unclear. Most gaps in Table 1 are under one point, and AIME uses only 30 problems. The paper reports a single seed and no variance or significance tests for Avg@8, so the win counts 23 of 30 cannot be separated from noise.
  • The key ablations are missing. There is no random-weight control that keeps the same clipping and KL budget, and no direct comparison with existing lookahead methods, even though they are the closest related work. Since G-only and PROW differ by small margins in Figure 4, and PROW adds several empirical steps, it is hard to tell whether gains come from the prospective signal itself or from these design choices or from reweighting variance in general.
Questions for authors
  • Could the author report means and standard deviations over at least three seeds, with confidence intervals or paired tests for AIME and AMC?
  • Could you add a random-weight control under the same clipping, KL budget, and normalization, and a direct comparison with lookahead-based methods?
  • What fraction of tokens have a zero prospective score since it is set to zero when y_t is outside B_t, and how much do PROW and G-only weights differ on those tokens?
  • Were the baselines hyperparameters inherited from their original papers, or tuned under the same budget as PROW?
  • Table 1 reports 33.00 Avg@8 for the Qwen3-4B-Base student init on Competition-MATH, while Table 2 reports 5.00 for the same student on GPQA-Diamond. Are these evaluated under identical settings?
3 comments

We thank the reviewer for recognizing the clear problem framing and the practicality of PROW. We address each concern below.

W1/Q1: Seeds and significance. We have updated the paper to report all main configurations over three training seeds (mean ± std), with Welch's tests and Holm–Bonferroni correction, in Appendix F.2 (Tables 6–8).

  • Overall: PROW has the highest or tied-highest three-seed mean Avg@8 in 29/30 setting–benchmark cells (28 strict).
  • AIME (paired test across settings): over all 12 AIME24/25 cells, PROW exceeds OPD in 12/12, by +4.37 on average (95% CI [2.80, 5.93], paired t p < 0.001). It exceeds the strongest baseline in 11/12, by +1.61 on average (95% CI [0.69, 2.53], p = 0.003).
  • AMC23: PROW exceeds OPD in 5/6 settings, by +3.54 on average (95% CI [0.44, 6.64], p = 0.032).

W2/Q2: Random-weight control. Regarding the reviewer's concern, we added a random-weight control whose weights are a random permutation of PROW's weights (Qwen3-4B → Qwen3-1.7B, Competition-MATH, three epochs, three seeds). It uses the same clipping and KL budget as PROW. The Avg@8 results are reported below:

BenchmarkOPDRandomPROW
Compe-MATH59.14±0.3253.87±0.3162.97±0.21
MATH-50060.25±0.5456.47±0.4262.76±0.60
AIME2416.53±1.2012.22±1.9220.42±1.82
AIME2518.33±1.2514.17±1.5020.14±1.88
AMC2345.00±2.6740.94±0.8352.81±1.74

Random reweighting falls below uniform OPD on every benchmark. PROW's gains therefore come from where the weight is placed, not from reweighting variance. The controlled intervention in Section 5.3 isolates the signal further: shuffling D^t\widehat{D}_t within matched-GtG_t strata reduces the selected-state gain from 8.7 to 6.6, close to G-only (6.4) (Fig. 3b). The signal is therefore informative because it is state-specific. Additionally, Figure 4 shows that the combined score is highest or tied-highest in all 24 cases (20 strict) versus G-only and prospective-only, highlighting the gain from prospective information.

W2/Q2: Lookahead methods. We respectfully clarify that the lookahead methods discussed in Appendix A address a different problem. They change the objective or the supervised tokens and pay for future information with extra computation:

  • SFD runs additional candidate forward passes.
  • FutureBridge-OPD samples additional student continuations.
  • TOPD adds a trajectory-aware update.

PROW, by contrast, uses the prospective signal in a lightweight way. It is computed in closed form from the existing rollout and logits, with no extra rollouts, forward passes or external validation, and only reweights the unchanged OPD loss. The appropriate baselines are therefore token-reweighting methods such as IW-OPD and TA-OPD, which we already compare against. We have revised Related Work and added Table 3, which makes explicit that the problems differ and that PROW obtains the prospective signal in a lightweight way.

At the reviewer's request, we nevertheless compare with the closest lookahead method, TOPD. We note that this comparison is not meant to position TOPD as a competing baseline, but to quantify the cost of obtaining future information. During training, TOPD's trajectory-aware update takes ≈90% of its step time (vs. ≈24% for PROW), making it roughly 3× more expensive per training token. We therefore compare both at the training time of PROW's full run, where PROW outperforms TOPD on all 10 metrics (results in our general comment).

Q3: Zero prospective scores. We have included this in Appendix F.7 (Table 11). Only 0.140% of sampled tokens fall outside B_t and have their prospective score set to zero. On these tokens, PROW and G-only weights differ by 0.102 on average (median 0.046). These tokens carry only 0.169% of the total PROW weight mass, so zeroing has a negligible effect on optimization.

Q4: Baseline hyperparameters. We clarify that all methods share the same training setting (Table 4). For IW-OPD and TA-OPD, we inherited the already-tuned hyperparameters from their original papers to reproduce their results. Similarly, PROW's hyperparameters are fixed across all teacher–student configurations and corpora, with no per-setting tuning.

Q5: Base-model evaluation. Thank you for the question. We inspected the responses of the Qwen3-4B-Base checkpoint and found that, for most GPQA-Diamond prompts, they are repetitive or meaningless, with no extractable answer, so they receive a low score. After post-training with an on-policy distillation framework such as PROW, the model recovers its accuracy. On Competition-MATH, under exactly the same evaluation configuration, the Base model’s responses contain an extractable answer much more often than on GPQA-Diamond, which yields the score of 33.0. The two scores are therefore not inconsistent with each other.

Dear Reviewer,

Thank you for the time and effort you’ve dedicated to reviewing our work. We have carefully addressed your concerns during the discussion phase. As the discussion period is nearing its close, we would greatly appreciate it if you could take a moment to review our responses and confirm whether they satisfactorily resolve your questions. Thank you once again for your thoughtful feedback and support.

Warm regards,

Paper's Authors

Dear Reviewer,

We would like to kindly follow up once more, as the discussion period is about to close. We would be very grateful if you could take a moment to revisit our responses and share any remaining concerns or questions.

We sincerely appreciate your time and consideration, especially during this busy period.

Best regards, The Authors

Reviewer_bjkFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PROW assigns different training weights to tokens in on-policy distillation (OPD), considering both how much can be learned at the current step and how learning there changes subsequent learning opportunities.

Critical strengths

1.Both immediate learning opportunity and prospective learning effect are derived from a small teacher-directed perturbation of the student distribution, giving the two scores a common mathematical basis. 2.The prospective score reuses existing student-generated reasoning without additional rollouts or a learned critic, allowing integration into the original OPD training workflow.

Critical weaknesses

1.The main experiments do not compare against methods that also use future information, such as the near-future guidance approach discussed in Appendix A, leaving the new scoring method's advantage over these approaches unclear. 2.Training uses only random seed 42, and the 14B-teacher-to-4B-student configuration exceeds the strongest baseline on MMLU-Pro by just 0.3 percentage points, with no repeated training runs to establish whether this small gain is stable. 3.Although no additional rollouts are needed, the paper does not report the extra training time or GPU memory required for candidate-token scoring and weight computation, making the actual training cost difficult to assess.

Questions for authors

1.If distillation loss is always evaluated using the frozen pre-update model, can the prospective score still predict changes in the teacher-student gap on content generated after the update? 2.Do the accuracy differences relative to the baselines on AIME and MMLU-Pro persist across different training random seeds? 3.Could the authors report the total probability mass retained by the Top-K candidates and the fraction of prospective scores set to zero because the sampled token falls outside the candidate set?

3 comments

We thank the reviewer for recognizing PROW's unified derivation and its integration into OPD without extra rollouts or critics.

W1: Lookahead methods. We respectfully clarify that lookahead methods address a different problem (Appendix A, Table 3). They change the objective or the supervised tokens, and pay for future information with extra computation:

  • SFD runs additional candidate forward passes.
  • FutureBridge-OPD samples additional student continuations.
  • TOPD adds a trajectory-aware update.

PROW, by contrast, uses the prospective signal in a lightweight proxy, and its reliability is shown by the similar results from 1-rollout and 32-rollout estimates (Appendix F.4). It is computed in closed form from the existing rollout and logits, with no extra rollouts, forward passes or external validation, and only reweights the unchanged OPD loss. The appropriate baselines are therefore token-reweighting methods such as IW-OPD and TA-OPD. To our knowledge, PROW is the first method to bring prospective information into OPD as a lightweight token reweighting. We have revised Related Work accordingly.

At the reviewer's request, we nevertheless compare with the closest lookahead method, TOPD. Because the two methods solve different problems, this comparison is meant to quantify the cost of obtaining future information, not to position TOPD as a competing baseline. During training, TOPD's trajectory-aware update takes ≈90% of its step time (vs. ≈24% for PROW), making it roughly 3× costlier per training token. We therefore compare both at the training time of PROW's full run (Qwen3-4B → Qwen3-1.7B, Competition-MATH, three seeds; Avg@8 / Pass@8):

BenchmarkTOPDPROW
Compe-MATH59.07±0.95 / 72.67±0.6462.97±0.21 / 74.20±0.53
MATH-50060.87±0.61 / 72.67±1.0362.76±0.60 / 74.07±0.31
AIME2415.56±1.92 / 32.22±9.6220.42±1.82 / 43.33±3.33
AIME2514.44±1.92 / 32.22±6.9420.14±1.88 / 48.89±3.85
AMC2342.71±1.72 / 80.00±4.3352.81±1.74 / 85.83±1.44

Under the same training time, PROW outperforms TOPD on all 10 metrics, with gains of up to +10.10 Avg@8 (AMC23).

W2/Q2: Seeds. We repeated all main configurations with three training seeds and report mean ± std in Appendix F.2 (Tables 6–8).

  • MMLU-Pro (14B→4B): the gain is stable. PROW reaches 62.12±0.07 Avg@8 versus 61.72±0.05 for the strongest baseline. PROW also achieves the best MMLU-Pro Avg@8 in all three configurations.
  • AIME: across all 12 AIME24/25 Avg@8 cells in Table 6, PROW exceeds OPD in 12/12 (mean +4.37) and achieves the highest or tied-highest mean in 12/12.
  • Overall: across all 30 setting–benchmark cells, PROW achieves the highest or tied-highest three-seed mean Avg@8 in 29/30 (28 strict) and exceeds OPD in 29/30 with an average gain of +3.47 points.

W3: Cost. We already include this in Appendix F.6 (Table 10). Under identical hardware and batch settings, mean time per optimizer step and peak memory are:

s/stepPeak mem (GiB)
OPD5.9359.92
IW-OPD6.22 (+4.9%)59.79
TA-OPD5.53 (−6.9%)33.35
PROW6.26 (+5.6%)60.43

PROW requires no additional rollouts or teacher forward passes, so teacher scoring time is identical to OPD (0.90 s/step). The extra 0.33 s/step comes only from the closed-form score and the allocation, about 12 minutes over a full training run. Peak memory rises by only 0.51 GiB (+0.9%).

Q1: Post-update content. We clarify that Section 5.3 of the original submission already tests exactly this. The two intervention studies use different evaluators by design:

  • Section 2 (Fig. 2) uses a frozen pre-update evaluator to isolate which states are visited.
  • Section 5.3 (Eq. 14) measures D_KL(p_θ₁‖q) with the post-update student on continuations sampled after the update.

Under this measure, among high-G_t states the downstream gain is −3.6, 3.2 and 10.6 for D^t\widehat{D}_t < 0, = 0 and > 0 (Fig. 3a). Selecting states with PROW raises the realized gain from 6.4 to 8.7 (95% CI of the difference: [0.8, 3.8]). We will make this distinction explicit in the writing.

Q3: Top-K coverage. We have included these statistics in Appendix F.7 (Table 11), with K = 16:

  • Mass retained: A_t retains 99.83% of the student mass, and B_t = TopK₁₆(p_t) ∪ TopK₁₆(q_t) retains 99.85% of both the student and the teacher mass. Coverage is stable throughout training (minimum at any step: 99.73%).
  • Zeroed scores: only 0.140% of sampled tokens fall outside B_t and have their prospective score set to zero. This fraction stays low throughout training (at most 0.251% at any step).

Thus, the Top-K union captures nearly all probability mass of both distributions, and zeroing affects a negligible fraction of tokens.

Dear Reviewer,

Thank you for the time and effort you’ve dedicated to reviewing our work. We have carefully addressed your concerns during the discussion phase. As the discussion period is nearing its close, we would greatly appreciate it if you could take a moment to review our responses and confirm whether they satisfactorily resolve your questions. Thank you once again for your thoughtful feedback and support.

Warm regards,

Paper's Authors

Dear Reviewer,

We would like to kindly follow up once more, as the discussion period is about to close. We would be very grateful if you could take a moment to revisit our responses and share any remaining concerns or questions.

We sincerely appreciate your time and consideration, especially during this busy period.

Best regards, The Authors

Reviewer_1Xd6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose Prospective Supervision Weighting (PROW), a token-level weighting method for OPD. Rather than evaluating tokens strictly by immediate pointwise signals, PROW combines an immediate learning opportunity metric with a prospective term that estimates how a local correction shifts the model's probability toward branches with better downstream learning opportunities. The method allocates a fixed token-level optimization budget using a KL-constrained Gibbs weighting and is evaluated on various mathematical and reasoning benchmarks.

Critical strengths

Conceptual Novelty: The theoretical motivation to explicitly model how a local, teacher-directed correction alters access to future learning opportunities in OPD is reasonable and addresses the limitation of current pointwise approaches.

Computational Efficiency: Utilizing a single-rollout estimator with restricted Top-K union supports allows the method to function without requiring extra rollouts, reward models, or learned critics.

Critical weaknesses

Limited Model Diversity: The primary empirical evaluation is restricted almost entirely to the Qwen model family (e.g., Qwen3-14B, Qwen3-4B, Qwen3-1.7B). This narrow scope makes it difficult to assess how well the prospective weighting mechanism generalizes to other architectural families.

Narrow Task Scope: The experimental validation focuses strictly on single-turn mathematical and reasoning benchmarks. The evaluation completely omits multi-turn agentic tasks, where downstream trajectory effects and prospective supervision would theoretically be the most impactful.

Marginal Empirical Gains: The performance improvements on key datasets like Competition-MATH are not highly significant compared to standard OPD and baselines like IW-OPD or TA-OPD. The modest gains raise concerns about whether the added algorithmic complexity translates to meaningful practical benefits.

Questions for authors
  1. Can you provide empirical results on multi-turn agentic tasks to better demonstrate the utility of prospective weighting over longer, interactive trajectories?

  2. How does PROW perform on entirely different model families outside of the Qwen architecture evaluated in the main text?

  3. Given the modest improvements on Competition-MATH, can you provide statistical significance testing to confirm these gains fall outside of standard run-to-run variance?

3 comments

We thank the reviewer for recognizing PROW's conceptual novelty and computational efficiency. We take the concerns about evaluation breadth, statistical reliability, and practical benefit seriously, and have expanded the revision accordingly.

Q1/W2. We agree that interactive, multi-turn tasks are a particularly relevant test of prospective supervision. We therefore added ALFWorld, distilling Qwen3-4B into Qwen3-1.7B for one epoch with identical training settings across methods. We evaluate 140 In Domain and 134 Out Of Domain tasks, with eight episodes per task, reporting Avg@8 success rate (%) over three training seeds:

MethodID SROOD SR
OPD5.18 ± 0.457.71 ± 0.44
IW-OPD8.60 ± 0.446.62 ± 0.19
TA-OPD9.08 ± 0.496.84 ± 0.24
PROW11.43 ± 0.548.49 ± 0.19

PROW leads on both splits: +6.25 points over OPD and +2.35 over the strongest baseline on ID tasks. On OOD tasks, IW-OPD and TA-OPD both underperform standard OPD, whereas PROW improves on it by +0.78 points. This is consistent with our motivation: reweighting by immediate signals alone can help on seen tasks yet hurt transfer, while accounting for downstream learning opportunities helps in both cases. We will extend this evaluation to additional teacher–student pairs in the final version.

Q2/W1. We would like to clarify that the original submission already included Llama-3.1-8B-Instruct → Llama-3.2-3B-Instruct results in Appendix F.1 (Table 5). To strengthen the cross-family evidence, we additionally evaluated Gemma-2-9B → Gemma-2-2B under the same Competition-MATH training protocol and six-benchmark evaluation (Table 5):

  • Gemma-2: PROW achieves the best Avg@8 on 5/6 benchmarks and the best Pass@8 on 6/6.
  • Llama-3: PROW is best or tied-best on 4/6 benchmarks for both Avg@8 and Pass@8.
  • Qwen3.5-9B → Qwen3.5-4B: PROW is best or tied-best in Avg@8 on all six benchmarks.

Q3/W3. We reran all methods with three independent training seeds and report mean ± sample standard deviation in Appendix F.2 (Tables 6-8), including Welch's two-sided tests with Holm–Bonferroni correction on Competition-MATH.

The gains lie well outside run-to-run variance. On Competition-MATH Avg@8, PROW improves over the strongest baseline in all six settings (two-sided sign test p = 0.031). In the 14B→4B and 4B→1.7B configurations, the four gains are +2.42 to +3.42 points, with Welch t-statistics of 7.6–13.5. Three remain significant after Holm–Bonferroni correction (p_Holm ≤ 0.004), and the fourth reaches p_Holm = 0.050. Since seed standard deviations are 0.2–0.8 points, these gains are several times the noise. In these settings PROW also exceeds standard OPD, by +2.71 to +4.63 points.

These improvements are not limited to Competition-MATH. Across all 30 benchmark–setting comparisons, PROW achieves the highest or tied-highest three-seed mean Avg@8 in 29/30 cases and outperforms standard OPD in 29/30, with an average gain of +3.47 points. This is comparable to gains reported by recent OPD methods such as EOPD (Jin et al., ICML 2026), which reports average improvements of +0.99 to +1.80 points over OPD.

The intervention experiments further support the role of prospective weighting. Figure 2 shows that states with similar immediate learning gains can have opposite downstream effects. When controlling for immediate learning opportunity, adding the prospective signal increases the downstream gain of selected states from 6.4 to 8.7 (Fig. 3), showing that it captures useful information beyond the immediate signal.

These benefits come at a small computational cost. PROW leaves the OPD loss, student rollouts, and teacher scoring unchanged, requiring no additional forward passes. Under matched conditions, PROW takes 6.26 s/step, compared with 5.93 s/step for OPD and 6.22 s/step for IW-OPD. This amounts to only 5.6% additional training time, while peak GPU memory increases by just 0.51 GiB (+0.9%).

Thus, the gains are not only reproducible across seeds but also consistent across benchmarks, supported by the intervention results, and achieved with limited additional training cost.

Dear Reviewer,

Thank you for the time and effort you’ve dedicated to reviewing our work. We have carefully addressed your concerns during the discussion phase. As the discussion period is nearing its close, we would greatly appreciate it if you could take a moment to review our responses and confirm whether they satisfactorily resolve your questions. Thank you once again for your thoughtful feedback and support.

Warm regards,

Paper's Authors

Dear Reviewer,

We would like to kindly follow up once more, as the discussion period is about to close. We would be very grateful if you could take a moment to revisit our responses and share any remaining concerns or questions.

We sincerely appreciate your time and consideration, especially during this busy period.

Best regards, The Authors

Comments on the paper

2 comments

We thank all reviewers for their constructive feedback and are encouraged that they found the problem framing clear and well supported (yHpz), the motivation conceptually novel (1Xd6), the two scores grounded in a common mathematical basis (bjkF), and PROW practical, with no extra rollouts, reward models or critics (1Xd6, bjkF, yHpz). Below we summarize our main clarifications and the new supporting results.

1. PROW addresses a different problem from lookahead methods (bjkF, yHpz)

Lookahead methods use future information to change what is trained: the objective, the training data, or the set of supervised tokens. They pay for this information with additional computation:

  • SFD adds a lookahead term to the objective and runs extra candidate forward passes.
  • FutureBridge-OPD samples additional student continuations to validate teacher interventions.
  • TOPD adds a trajectory-aware update that distributes guidance over future segments.
  • Prune-OPD removes supervision after detected trajectory drift.

PROW asks a different question: under the unchanged OPD loss, how should a fixed per-token budget be allocated? It uses prospective information in a lightweight way, and its reliability has been verified: a single rollout closely matches a 32-rollout reference (Appendix F.4). It is computed in closed form from the existing rollout and logits, with no extra rollouts, forward passes or external validation, and only reweights the OPD loss. The appropriate baselines are therefore token-reweighting methods (IW-OPD, TA-OPD). To our knowledge, PROW is the first method to bring prospective information into OPD as a lightweight token reweighting. We revised Related Work (Appendix A) and added Table 3 to highlight this.

TOPD results. At the reviewers' request, we made our best effort to compare with TOPD, the lookahead method closest to PROW. Like PROW, it keeps the student's own rollouts and uses future context to decide how each token is supervised. The purpose is to quantify the cost of future information, not to treat TOPD as a baseline. During training, TOPD's trajectory-aware update takes ≈90% of its step time (vs. ≈24% for PROW), making it roughly 3× costlier per training token. For a fair comparison, we therefore compare both at the training time of PROW's full run (Qwen3-4B → Qwen3-1.7B, Competition-MATH, three seeds; Avg@8 / Pass@8):

BenchmarkTOPDPROW
Compe-MATH59.07±0.95 / 72.67±0.6462.97±0.21 / 74.20±0.53
MATH-50060.87±0.61 / 72.67±1.0362.76±0.60 / 74.07±0.31
AIME2415.56±1.92 / 32.22±9.6220.42±1.82 / 43.33±3.33
AIME2514.44±1.92 / 32.22±6.9420.14±1.88 / 48.89±3.85
AMC2342.71±1.72 / 80.00±4.3352.81±1.74 / 85.83±1.44

At the same training time, PROW beats TOPD on all 10 metrics, by up to +10.10 Avg@8 (AMC23).

2. Statistical reliability (1Xd6, bjkF, yHpz)

We repeated all main configurations with three training seeds (Appendix F.2, Tables 6–8). The improvements are stable: PROW remains best in nearly all setting–benchmark cells, with seed variation well below its gains. Welch's tests with Holm–Bonferroni correction confirm them.

3. Gains come from the prospective signal (yHpz)

To test whether the gains come from the prospective signal rather than reweighting variance, we ran a random-weight control. It uses PROW's own weights, randomly permuted, with the same clipping, KL budget and normalization. The control falls below uniform OPD on every benchmark (e.g., 53.87 vs. 59.14 on Competition-MATH; PROW: 62.97). The gains therefore come from where the prospective signal places the weight, not from reweighting variance.

4. Broader evaluation (1Xd6)

PROW generalizes beyond math reasoning and Qwen models:

  • Agentic, multi-turn tasks (ALFWorld, Qwen3-4B → Qwen3-1.7B): PROW achieves the best success rate on both ID and OOD splits, improving over TA-OPD / IW-OPD by +2.35 / +2.83 (ID) and +1.65 / +1.87 (OOD). On OOD, it is the only weighting method that improves over OPD: immediate-signal weighting helps on seen tasks but hurts transfer, while accounting for downstream learning opportunities helps in both. We will extend this evaluation to additional teacher-student pairs in the final version.
  • Other model families: Beyond the Llama and Qwen3.5 results already in Table 5 of the original submission, we added Gemma-2-9B → 2B. PROW again achieves the best Avg@8 on 5/6 and the best Pass@8 on 6/6 benchmarks.

5. Cost and coverage (bjkF, yHpz)

  • Training cost (Appendix F.6): PROW adds only +5.6% time per step and +0.51 GiB peak memory over OPD, with identical teacher scoring time.
  • Coverage (Appendix F.7): the Top-16 union retains 99.85% of both the student and teacher mass, and only 0.140% of sampled tokens fall outside it.

We believe these results address the main concerns: PROW is a lightweight, principled and effective way to bring prospective information into OPD.

Dear Area Chair and Reviewers,

We sincerely thank you for your time and effort in reviewing our paper. We would like to respectfully follow up on our rebuttal. We have carefully addressed the reviewers' concerns and provided additional clarifications and supporting evidence to resolve the issues raised in the initial reviews.

We kindly encourage the reviewers to revisit our rebuttal and reconsider their initial assessments in light of our detailed responses. We sincerely hope that the clarifications and evidence provided can help resolve the original concerns and be reflected in updated scores where appropriate.

We would also greatly appreciate it if the Area Chair could carefully consider our rebuttal and the additional evidence when making the final assessment.

Thank you once again for your valuable time, feedback, and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.