acceptodds
Withdrawn from ICLR 2027

Beyond the Local View: Predicting Correction Gains for Sparse On-Policy Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

On-policy distillation guides a student on its own prefixes, but local uncertainty and teacher–student disagreement do not directly measure the benefit of a correction. We define correction gain as the change in eventual success probability when a teacher candidate replaces the student’s sampled token at a fixed prefix, with the current student continuing both branches. Correction-Gain Selection (CGS) learns candidate values from offline paired interventions and uses their predicted difference to select 20% of response positions for distillation. The predictor remains frozen while its inputs follow the evolving student, and training retains the original trajectories. Across five mathematical benchmarks, CGS improves macro-average accuracy over the strongest matched-budget sparse baseline by 1.73 and 1.69 percentage points for Qwen3-1.7B and DeepSeek-R1-Distill-Qwen-1.5B, and by 1.81 points for Qwen3-8B to Qwen3-4B-Base distillation. Paired diagnostics in the two smaller-student settings connect this allocation to correction gains and examine predictor reuse as the student changes. The approach turns correction experience into reusable guidance about where teacher supervision helps the current student.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, knowledge distillation, large language models, mathematical reasoning, token selection, supervision allocation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_cHJ6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Correction-Gain Selection (CGS), a sparse on-policy distillation method that estimates the incremental benefit of teacher corrections. The approach trains a candidate-value predictor using offline paired interventions and selects 20% of response positions based on the predicted difference between teacher- and student-candidate success probabilities. Experiments across five mathematical reasoning benchmarks and three teacher–student configurations demonstrate improvements over conventional sparse selection baselines, accompanied by analyses of correction gains, predictor calibration, and supervision allocation.

Critical strengths

S1. The paper identifies a limitation of existing sparse OPD methods, where local uncertainty or teacher–student disagreement may not reflect the actual benefit of supervision. Estimating the incremental correction gain provides an intuitive alternative for identifying potentially useful supervision positions.

S2. CGS outperforms the evaluated sparse selection baselines across three teacher–student configurations and five mathematical reasoning benchmarks. The results suggest that outcome-informed supervision selection can complement conventional local uncertainty and divergence measures.

Critical weaknesses

W1. Based on my inspection, the provided source code does not offer a reproducible end-to-end workflow, and its correspondence to several key methodological components described in the paper is difficult to establish. This raises substantial concerns about implementation transparency and the ability to independently verify the reported experimental results.

W2. The presentation of this paper need improvement. The paper frequently uses overly compressed explanations and introduces several technical quantities without sufficiently developing their relationships, making the methodological narrative difficult to follow. In particular, the extensive diagnostic analyses and experimental protocols are fragmented across the main text and appendices, while several figures contain dense information that is not adequately interpreted.

W3. Lack of justification for heuristic design choices. The proposed method relies on several seemingly arbitrary heuristic choices, such as selecting 0.2L scores and retaining the top 16 candidates, without sufficient theoretical or empirical justification. It remains unclear why these specific values are preferred over other alternatives. Moreover, the absence of systematic hyperparameter sensitivity analyses (e.g., varying the selection ratio or top-k) raises concerns about the robustness and generalizability of the method. These limitations make it difficult to determine whether the reported improvements stem from the proposed methodology itself or from carefully tuned heuristic choices.

W4. CGS requires Full OPD checkpoints, substantial offline intervention labeling, and additional predictor training before the final sparse distillation stage. The reported comparison of candidate-branch counts does not account for the complete computational pipeline, making it difficult to assess whether the proposed method offers meaningful end-to-end efficiency advantages.

Questions for authors
  1. Why is the supervision budget fixed at 20% of response positions? Could the authors provide sensitivity analyses across different selection ratios, such as 5%, 10%, 30%, and 50%, and explain how zero-gain positions should be handled?

  2. What motivates the choice of top-16 tokens in Equation (3)? Could the authors compare different truncation values to examine whether the reported improvements are robust to the approximation of the KL objective?

  3. Could the authors provide additional evidence connecting predicted correction gain with the actual improvement resulting from reverse-KL updates, rather than only the outcomes of forced-token interventions?

Reviewer_yyFDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes CGS, which selects important positions for OPD based on the predicted improvement in final answer correctness. CGS trains a value predictor using offline paired interventions from full OPD checkpoints and uses the difference between teacher and student candidate values to select 20% of positions for distillation. The authors conducted experiments on three teacher-student configurations across five math benchmarks and show that CGS improves average accuracy over the baselines.

Critical strengths
  • The motivation is convincing. Selecting supervision positions based on whether teacher guidance really helps the student reach the correct answer makes sense.
  • On average, CGS improves performance over the compared sparse OPD baselines.
  • The authors provide additional analyses with paired interventions to understand how the selected positions affect the final outcome.
Critical weaknesses

I think the most important part of the paper is whether the proposed correction gain is a reliable signal for selecting the token positions that need teacher supervision. However, there are several concerns about the method itself and also how the authors justify that it works.

  • First of all, the method requires the full training of OPD first and then trains a value predictor based on its checkpoints. This adds a huge compute cost since there are essentially two training phases. This added complexity is of course a weakness, especially when the actual improvement is relatively small. The paper also lacks a proper analysis of the computational cost (Appendix G shows some analysis but it does not measure the overall training cost including the full OPD training required to obtain the checkpoints).
  • Also, the most concerning point is how imperfect the value predictor is. The whole method relies on the predicted correction gain, but it is not well justified how reliable this prediction could be.
    • First of all, the value predictor is trained using Full OPD student checkpoints at rounds 0, 112, and 225. But how can it be expected to predict the values of prefixes generated from a student that is newly updated with sparse OPD? The training trajectories and the actual CGS trajectories are different, and it is not clear how well the predictor generalizes to this distribution shift. Although the authors provide some checkpoint analyses, these do not directly measure prediction accuracy on the evolving CGS student during the sparse OPD run.
    • Also, in general, predicting whether a partial reasoning trajectory will eventually succeed is itself a very difficult problem. It requires estimating the outcome of the remaining reasoning process, which is closely related to solving or verifying the problem itself. Therefore, it is unclear how accurately such values can be predicted, especially with only one binary outcome per candidate during training.
    • The authors do provide some predictor analyses, but the pairwise winner accuracy is only around 60-61%, and the direction AUC is around 0.63-0.64. These results do not seem particularly strong, especially when the whole method depends on comparing the two candidate values. More importantly, there is a lack of direct analysis on how accurately the predictor ranks the actual correction gains on newly generated CGS trajectories.
  • Another important concern is the connection between the proposed correction gain and the actual OPD training objective. Correction gain measures the change in final success probability when the student token is replaced by the teacher's argmax token. However, the actual training does not replace the token and instead applies reverse KL to the teacher distribution on the original trajectory. These are two very different things, and it is not clear why a token replacement that improves the final outcome necessarily indicates that applying reverse KL at that position will improve training. There is almost no theoretical or empirical justification for this connection.
  • Another concerning point is that a large portion of the selected positions actually have zero predicted correction gain. According to Table A5 in Appendix, around 34% and 24% of the positions selected by CGS have zero predicted gain for Qwen and DeepSeek, respectively. This means that these positions are selected using Local-KL as the tie-breaking criterion, so the fallback option. This means that a substantial portion of the supervision is still selected based on Local-KL rather than the proposed correction gain. It is unclear how much of the improvement actually comes from correction-gain selection and how much comes from this Local-KL fallback.
  • The main result tables do not provide error bars or standard deviations. For math benchmarks like AIME and AMC, there are only 30-40 problems per benchmark, and the results can be quite noisy. Also, the evaluation is focused entirely on math reasoning.
  • The baselines are also relatively weak. The authors mainly compare against Random, Entropy, and Local-KL for sparse supervision selection. However, there are more recent methods for token selection and on-policy distillation, such as TIP and TA-OPD, which are discussed in the related work but not included in the experiments.
  • The writing is also very poor in some places, making the paper hard to follow. For example, in the first paragraph of the introduction, the authors move from OPD distribution shift to reasoning trajectories and alternative solution paths, and then to teacher-student disagreement, without clearly connecting these ideas. The connection between each sentence is weak, and it is hard to understand the main motivation from this paragraph. Please improve the flow of the paper in general.
Questions for authors

See the weaknesses above for the details.

  • How much additional compute does the full pipeline require?
  • How accurate is the predictor on evolving CGS trajectories?
  • Why should this token replacement gain predict reverse-KL training benefits? Isn't it different?
  • Does correction gain outperform using teacher value alone?
  • How much improvement comes from Local-KL fallback positions?
Reviewer_sN3xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Correction-Gain Selection (CGS), which estimates how much a teacher candidate token improves the continuing student's probability of answering correctly relative to its original token at the same prefix. A predictor trained on offline paired interventions remains frozen during training and selects the highest-scoring positions on student trajectories. Across several teacher–student configurations and mathematical benchmarks, CGS improves accuracy over the baseline.

Critical strengths
  • The student relative criterion is well motivated an distinguishes incremental teacher help from absolute candidate success.

  • The reported improvements are consistent across multiple model configurations, with a common distillation objective across selection methods.

Critical weaknesses
  • The robustness of the accuracy gains is unclear because of independent training runs. The confidence intervals for performance differences are not reported.

  • Stronger ablations are absent to isolate the benefit of the value difference, e.g., against teacher-candidate value alone. The connection between single-token intervention gains and distribution-level distillation also remains insufficiently established.

Questions for authors

See the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.