- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes Correction-Gain Selection (CGS), a sparse on-policy distillation method that estimates the incremental benefit of teacher corrections. The approach trains a candidate-value predictor using offline paired interventions and selects 20% of response positions based on the predicted difference between teacher- and student-candidate success probabilities. Experiments across five mathematical reasoning benchmarks and three teacher–student configurations demonstrate improvements over conventional sparse selection baselines, accompanied by analyses of correction gains, predictor calibration, and supervision allocation.
Critical strengths
S1. The paper identifies a limitation of existing sparse OPD methods, where local uncertainty or teacher–student disagreement may not reflect the actual benefit of supervision. Estimating the incremental correction gain provides an intuitive alternative for identifying potentially useful supervision positions.
S2. CGS outperforms the evaluated sparse selection baselines across three teacher–student configurations and five mathematical reasoning benchmarks. The results suggest that outcome-informed supervision selection can complement conventional local uncertainty and divergence measures.
Critical weaknesses
W1. Based on my inspection, the provided source code does not offer a reproducible end-to-end workflow, and its correspondence to several key methodological components described in the paper is difficult to establish. This raises substantial concerns about implementation transparency and the ability to independently verify the reported experimental results.
W2. The presentation of this paper need improvement. The paper frequently uses overly compressed explanations and introduces several technical quantities without sufficiently developing their relationships, making the methodological narrative difficult to follow. In particular, the extensive diagnostic analyses and experimental protocols are fragmented across the main text and appendices, while several figures contain dense information that is not adequately interpreted.
W3. Lack of justification for heuristic design choices. The proposed method relies on several seemingly arbitrary heuristic choices, such as selecting 0.2L scores and retaining the top 16 candidates, without sufficient theoretical or empirical justification. It remains unclear why these specific values are preferred over other alternatives. Moreover, the absence of systematic hyperparameter sensitivity analyses (e.g., varying the selection ratio or top-k) raises concerns about the robustness and generalizability of the method. These limitations make it difficult to determine whether the reported improvements stem from the proposed methodology itself or from carefully tuned heuristic choices.
W4. CGS requires Full OPD checkpoints, substantial offline intervention labeling, and additional predictor training before the final sparse distillation stage. The reported comparison of candidate-branch counts does not account for the complete computational pipeline, making it difficult to assess whether the proposed method offers meaningful end-to-end efficiency advantages.
Questions for authors
-
Why is the supervision budget fixed at 20% of response positions? Could the authors provide sensitivity analyses across different selection ratios, such as 5%, 10%, 30%, and 50%, and explain how zero-gain positions should be handled?
-
What motivates the choice of top-16 tokens in Equation (3)? Could the authors compare different truncation values to examine whether the reported improvements are robust to the approximation of the KL objective?
-
Could the authors provide additional evidence connecting predicted correction gain with the actual improvement resulting from reverse-KL updates, rather than only the outcomes of forced-token interventions?