- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper points out that incorrect student-generated trajectories can provide more useful training signals than correct ones in OPD and OPSD. Motivated by this observation, the authors propose ReNIO, a trajectory reweighting method that aims to emphasize likely incorrect trajectories without requiring final-answer correctness labels. ReNIO uses the student-to-teacher log probability ratio to identify potentially pivotal tokens and aggregates their scores into trajectory-level weights for the distillation objective. Since these weights can be computed from partial reasoning trajectories, the method preserves the efficiency advantage of prefix-based distillation.
Critical strengths
-
The paper provides a clear empirical motivation for the proposed method. Through exploratory experiments, the authors demonstrate that training exclusively on incorrect student-generated trajectories consistently outperforms training solely on correct ones under both OPD and OPSD settings.
-
Beyond reporting performance differences, the authors further analyze the resulting reasoning behaviors, observing that correct-only training tends to yield shorter reasoning chains and fewer reflection-style markers, whereas incorrect-only training better preserves exploratory and self-corrective behaviors. These findings highlight the distinct roles that correct and incorrect trajectories may play during distillation, providing a valuable starting point for investigating how training weights should be distributed across student-generated outputs.
-
The paper presents a relatively comprehensive experimental evaluation of the proposed method. The authors evaluate ReNIO across multiple model scales and two model families under both OPD and OPSD settings, covering both mathematical reasoning and code generation tasks. The results demonstrate consistent improvements over the corresponding distillation baselines in terms of average performance.
-
The paper investigates the effectiveness of short-prefix training, analyzes how the proposed weighting mechanism distributes importance between correct and incorrect trajectories, and conducts ablation studies on its key components. These experiments provide solid empirical support for the effectiveness and applicability of the proposed approach.
Critical weaknesses
-
The connection between the motivating observation and the proposed optimization mechanism is not sufficiently justified. The paper motivates ReNIO by showing that incorrect student-generated trajectories can be more beneficial for distillation than correct ones. However, the proposed method relies on student–teacher log probability ratios to estimate trajectory importance. To justify this connection, the authors conduct a small-scale toy experiment on 200 GSM8K samples, showing that incorrect trajectories and first-error tokens exhibit higher average log ratios. While this provides preliminary evidence of a correlation, it is not sufficient to establish that the proposed signal can reliably identify informative negative trajectories. In particular, higher student–teacher disagreement does not necessarily imply greater learning value. Therefore, the empirical justification for using log ratios as a proxy for negative trajectory importance remains limited.
-
The experimental comparisons are insufficient to establish the advantages of the proposed weighting mechanism. Although the paper reports improvements over standard OPD and OPSD, these comparisons mainly demonstrate the effectiveness of introducing trajectory reweighting rather than the necessity of ReNIO's specific design. More comparisons with existing reweighting methods would help clarify whether the proposed pivotal-token selection and aggregation mechanism provides additional benefits.
Questions for authors
-
In Equation (6), the trajectory weight is computed by averaging the clipped log ratios over the selected pivotal tokens. Consequently, a trajectory containing a single high-ratio token can receive the same weight as one containing many such tokens, provided their average log ratios are equal. Given the paper's motivation that incorrect trajectories contain informative deviations from the teacher, why should the number of identified pivotal tokens have no direct influence on the trajectory weight? Is this normalization intended to prevent trajectories with more disagreement tokens from being overemphasized, or does the method assume that the average disagreement strength is more informative than its cumulative magnitude?
-
Appendix D.2 reports that trajectory-level weighting outperforms a token-level weighting variant using the same student–teacher ratio signal. Could the authors further clarify the underlying reason for this difference? Since ReNIO identifies a small subset of pivotal tokens as the primary source of informative disagreement, why is it more beneficial to increase the distillation weight of the entire trajectory rather than concentrate the additional supervision on these positions?
3 comments
rebbuttal response
I thank the authors for their detailed response and additional experiments. I appreciate the authors' efforts to address my concerns. Overall, I will maintain my rating.
We sincerely thank the reviewer for the follow-up and for engaging with our additional experiments.
We thank the reviewer for the thoughtful feedback and respond to each point below.
W1: Can the proposed signal reliably identify informative negative trajectories?
We agree that correlation with incorrectness alone does not establish learning value. We therefore additionally compare uniform and ReNIO weighting within incorrect-only trajectories on Qwen3-1.7B.
W2: More experiments to show ReNIO's specific design(pivotal-token selection, aggregation mechanism)
We already compare ReNIO with SCOPE [1] in Appendix D.3: ReNIO achieves higher average accuracy in all four evaluated configurations; for Qwen3-1.7B, the averages are 40.56 vs. 37.41 under OPSD. SCOPE first determines the correctness of each trajectory and then weights correct and incorrect trajectories using student and teacher perplexity, respectively. In contrast, ReNIO jointly incorporates teacher and student information into its weights, providing a more comprehensive signal. It also implicitly emphasizes incorrect trajectories, whose importance is demonstrated by our motivating experiments, while preserving the efficiency advantage of prefix-based training. More specifically, we provide the following explanations and experiments to justify pivotal-token selection and aggregation mechanism.
Q1: information about the count of the pivotal tokens "why should the number of pivotal tokens have no direct influence on the trajectory weight? Is this normalization intended to prevent trajectories with more disagreement tokens from being overemphasized, or does the method assume that the average disagreement strength is more informative than its cumulative magnitude?"
The reason is the former. Using cumulative log ratio, after exp operation, trajectory with more pivotal tokens will easily reach a very large magnitude. In our observation, this will make weight distribution collapse(most weights in batch become 0).
We also try only using the count of pivotal tokens to get trajectory weight in Qwen3 1.7B:
Q2: Why token weighting worse than trajectory weighting Token-level weighting changes the relative emphasis across positions within a trajectory, whereas trajectory-level weighting preserves that relative weighting and changes the trajectory’s overall contribution. One possible interpretation is that prioritizing more informative trajectories matters more than directly concentrating supervision on individual high-ratio tokens but ignoring trajectory importance. Yet we dont have a definitive explanation for the performance difference.
[1] Zheng et al(2026). SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting. ArXiv, abs/2604.10688.