acceptodds
Under review as a conference paper at ICLR 2027

ReNIO: Reweighting Negative Trajectory Importance for LLM On-Policy Distillation

Abstract

On-policy distillation (OPD) improves LLM reasoning by training a student model on its own generated outputs, but standard OPD treats all student-generated outputs (SGOs) equally regardless of their informativeness. We observe a consistent asymmetry in controlled filtering experiments: in both OPD and on-policy self distillation (OPSD), training only on incorrect SGOs outperforms training only on correct ones. Our further analysis suggests that models trained on correct-only SGOs tend to generate shorter reasoning traces and show weaker reflection behavior, while incorrect SGOs better preserve exploratory reasoning near the model's capability boundary. To exploit this signal without requiring full answer-containing rollouts, we introduce ReNIO, which Reweights Negative trajectory Importance for LLM On-policy distillation. By using the student-to-teacher probability ratio, ReNIO identifies pivotal tokens leading to wrong reasoning traces and aggregates their information into a normalized sample weight, inherently assigning larger weights to likely negative trajectories without observing the correctness of final-answer. Since Re-NIO only uses prefix-conditioned token probabilities, it preserves OPD's prefix training advantage over full-rollout reinforcement learning. Across both mathematical reasoning and code generation tasks, ReNIO improves both OPD and OPSD, with representative relative gains of up to 8.90% for Qwen3-1.7B and 10.00% for R1-Distill-Qwen-7B on mathematical reasoning benchmarks

Primary area:
generative models
Keywords:
LLM, Distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_66ihOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper points out that incorrect student-generated trajectories can provide more useful training signals than correct ones in OPD and OPSD. Motivated by this observation, the authors propose ReNIO, a trajectory reweighting method that aims to emphasize likely incorrect trajectories without requiring final-answer correctness labels. ReNIO uses the student-to-teacher log probability ratio to identify potentially pivotal tokens and aggregates their scores into trajectory-level weights for the distillation objective. Since these weights can be computed from partial reasoning trajectories, the method preserves the efficiency advantage of prefix-based distillation.

Critical strengths
  1. The paper provides a clear empirical motivation for the proposed method. Through exploratory experiments, the authors demonstrate that training exclusively on incorrect student-generated trajectories consistently outperforms training solely on correct ones under both OPD and OPSD settings.

  2. Beyond reporting performance differences, the authors further analyze the resulting reasoning behaviors, observing that correct-only training tends to yield shorter reasoning chains and fewer reflection-style markers, whereas incorrect-only training better preserves exploratory and self-corrective behaviors. These findings highlight the distinct roles that correct and incorrect trajectories may play during distillation, providing a valuable starting point for investigating how training weights should be distributed across student-generated outputs.

  3. The paper presents a relatively comprehensive experimental evaluation of the proposed method. The authors evaluate ReNIO across multiple model scales and two model families under both OPD and OPSD settings, covering both mathematical reasoning and code generation tasks. The results demonstrate consistent improvements over the corresponding distillation baselines in terms of average performance.

  4. The paper investigates the effectiveness of short-prefix training, analyzes how the proposed weighting mechanism distributes importance between correct and incorrect trajectories, and conducts ablation studies on its key components. These experiments provide solid empirical support for the effectiveness and applicability of the proposed approach.

Critical weaknesses
  1. The connection between the motivating observation and the proposed optimization mechanism is not sufficiently justified. The paper motivates ReNIO by showing that incorrect student-generated trajectories can be more beneficial for distillation than correct ones. However, the proposed method relies on student–teacher log probability ratios to estimate trajectory importance. To justify this connection, the authors conduct a small-scale toy experiment on 200 GSM8K samples, showing that incorrect trajectories and first-error tokens exhibit higher average log ratios. While this provides preliminary evidence of a correlation, it is not sufficient to establish that the proposed signal can reliably identify informative negative trajectories. In particular, higher student–teacher disagreement does not necessarily imply greater learning value. Therefore, the empirical justification for using log ratios as a proxy for negative trajectory importance remains limited.

  2. The experimental comparisons are insufficient to establish the advantages of the proposed weighting mechanism. Although the paper reports improvements over standard OPD and OPSD, these comparisons mainly demonstrate the effectiveness of introducing trajectory reweighting rather than the necessity of ReNIO's specific design. More comparisons with existing reweighting methods would help clarify whether the proposed pivotal-token selection and aggregation mechanism provides additional benefits.

Questions for authors
  1. In Equation (6), the trajectory weight is computed by averaging the clipped log ratios over the selected pivotal tokens. Consequently, a trajectory containing a single high-ratio token can receive the same weight as one containing many such tokens, provided their average log ratios are equal. Given the paper's motivation that incorrect trajectories contain informative deviations from the teacher, why should the number of identified pivotal tokens have no direct influence on the trajectory weight? Is this normalization intended to prevent trajectories with more disagreement tokens from being overemphasized, or does the method assume that the average disagreement strength is more informative than its cumulative magnitude?

  2. Appendix D.2 reports that trajectory-level weighting outperforms a token-level weighting variant using the same student–teacher ratio signal. Could the authors further clarify the underlying reason for this difference? Since ReNIO identifies a small subset of pivotal tokens as the primary source of informative disagreement, why is it more beneficial to increase the distillation weight of the entire trajectory rather than concentrate the additional supervision on these positions?

3 comments

We thank the reviewer for the thoughtful feedback and respond to each point below.

W1: Can the proposed signal reliably identify informative negative trajectories?

We agree that correlation with incorrectness alone does not establish learning value. We therefore additionally compare uniform and ReNIO weighting within incorrect-only trajectories on Qwen3-1.7B.

MethodAIME24AIME25HMMT25Avg
OPSD, incorrect-only51.6741.9423.6139.07
OPSD, incorrect-only + ReNIO55.0040.5626.6740.74
OPD, incorrect-only57.5041.1128.6142.41
OPD, incorrect-only + ReNIO57.2243.3329.1743.24
These results provide additional evidence that ReNIO's weighting signal can be useful even after correctness-based selection.

W2: More experiments to show ReNIO's specific design(pivotal-token selection, aggregation mechanism)

We already compare ReNIO with SCOPE [1] in Appendix D.3: ReNIO achieves higher average accuracy in all four evaluated configurations; for Qwen3-1.7B, the averages are 40.56 vs. 37.41 under OPSD. SCOPE first determines the correctness of each trajectory and then weights correct and incorrect trajectories using student and teacher perplexity, respectively. In contrast, ReNIO jointly incorporates teacher and student information into its weights, providing a more comprehensive signal. It also implicitly emphasizes incorrect trajectories, whose importance is demonstrated by our motivating experiments, while preserving the efficiency advantage of prefix-based training. More specifically, we provide the following explanations and experiments to justify pivotal-token selection and aggregation mechanism.

  • pivotal-token selection Removing selection reduces Qwen3-1.7B OPD accuracy from 42.04 to 40.83 (§4.5); Appendix D.4 provides threshold sensitivity results.
  • aggregation mechanism:ReNIO aggregates PSPT\frac{P_{S}}{P_{T}} ratios using their geometric mean, an alternative is to use the arithmetic mean. Geometric averaging reduces the relative influence of individual large ratios compared with arithmetic averaging. Our ablation further shows that geometric aggregation achieves higher accuracy under both OPD and OPSD:
MethodAIME24AIME25HMMT25Avg
OPSD+arithmetic mean54.1738.8923.8938.98
OPSD+geometric mean57.7842.7827.7842.78
OPD+arithmetic mean54.1740.2826.1140.19
OPD+geometric mean54.7243.0628.3342.04

Q1: information about the count of the pivotal tokens "why should the number of pivotal tokens have no direct influence on the trajectory weight? Is this normalization intended to prevent trajectories with more disagreement tokens from being overemphasized, or does the method assume that the average disagreement strength is more informative than its cumulative magnitude?"

The reason is the former. Using cumulative log ratio, after exp operation, trajectory with more pivotal tokens will easily reach a very large magnitude. In our observation, this will make weight distribution collapse(most weights in batch become 0).

We also try only using the count of pivotal tokens to get trajectory weight in Qwen3 1.7B:

MethodAIME24AIME25HMMT25Avg
OPSD+ReNIO57.7842.7827.7842.78
OPSD+count weight52.2240.8325.8339.63
OPD+ReNIO54.7243.0628.3342.04
OPD+count weight52.7841.9425.0039.91
Count-based weighting also performs substantially worse than ReNIO. The average score of OPD+count weighting (39.91) is even lower than that of unweighted OPD (40.37). This suggests that pivotal-token count alone may not be an effective weighting signal in this setting.

Q2: Why token weighting worse than trajectory weighting Token-level weighting changes the relative emphasis across positions within a trajectory, whereas trajectory-level weighting preserves that relative weighting and changes the trajectory’s overall contribution. One possible interpretation is that prioritizing more informative trajectories matters more than directly concentrating supervision on individual high-ratio tokens but ignoring trajectory importance. Yet we dont have a definitive explanation for the performance difference.

[1] Zheng et al(2026). SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting. ArXiv, abs/2604.10688.

Reviewer_66ihOpenReview ↗

rebbuttal response

I thank the authors for their detailed response and additional experiments. I appreciate the authors' efforts to address my concerns. Overall, I will maintain my rating.

We sincerely thank the reviewer for the follow-up and for engaging with our additional experiments.

Reviewer_6cTXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ReNIO, a trajectory reweighting method for on-policy distillation and on-policy self-distillation. Motivated by experiments in which training on incorrect student-generated trajectories outperforms training on correct ones, ReNIO selects tokens with large student-to-teacher log-probability ratios, aggregates their clipped values into trajectory weights, and normalizes these weights within each batch. The weights require neither completed answers nor correctness labels, preserving training with truncated prefixes. Experiments cover mathematical reasoning and code generation with Qwen3 and R1-Distill-Qwen models. The paper reports improvements over corresponding unweighted baselines, supported by component ablations, alternative weighting comparisons, and a comparison with SCOPE.

Critical strengths
  1. ReNIO uses probabilities already available during distillation and does not require final-answer correctness. This makes it directly applicable to truncated student trajectories. Table 4 and Figure 4(a) support this practical benefit: for Qwen3-1.7B OPD with 1024-token prefixes, the mathematical average improves from 40.37 to 42.04 while the measured time remains approximately 5 seconds per step. Improving existing distillation with negligible measured overhead is a consequential contribution.

  2. Across the seven model/distillation configurations in Tables 2–3, both mathematical and coding averages improve over their corresponding baselines. The evaluation also includes component ablations, token-versus-trajectory weighting, the reversed probability ratio, and SCOPE under a matched generation-length limit. These comparisons provide substantially better support than an isolated improvement on one model and help establish the usefulness of the particular weighting design.

Critical weaknesses
  1. Mathematical-average improvements are approximately 1.30–1.95 percentage points, while coding-average improvements range from 0.41 to 1.89 points, but the paper does not report training-run variability or uncertainty for the paired comparisons. There are also unexplained differences between apparently comparable baselines: the OPD “Both” condition in Figure 1 reports approximately 29.2 on HMMT25, whereas OPD(4096) in Table 4 reports 25.83, although Appendix B describes a 4096-token limit for the motivating experiment. These may be different runs or settings, but the difference is material relative to the reported gains. Together with the unclear definition of Avg@12/Avg@4, this limits confidence in the magnitude and reproducibility of the improvements.

  2. Appendix B describes filtering on-policy samples by final-answer correctness, but does not specify whether batches are replenished after filtering, how many trajectories and supervised tokens are retained, or how the loss is normalized. Filtering can change both the amount of supervision and the prompt distribution, so keeping the nominal optimization settings unchanged does not establish that correctness is the relevant explanatory factor. The weighting ablations support the proposed recipe, but the reflection-marker and teacher-entropy analyses remain indirect evidence for the stronger mechanism claims. This limits the explanatory contribution concerning why negative trajectories are especially useful, even if the practical gains hold.

Questions for authors
  1. How exactly are Avg@12 and Avg@4 computed? Please reconcile the unfiltered “Both” results in Figure 1 with the 4096-token baselines in Table 4, identifying any differences in seeds, checkpoints, or training settings. How many independent training runs underlie the main results, and can you provide uncertainty estimates for representative paired improvements? Training-run variability and evaluation-sampling uncertainty should be distinguished.

  2. After correctness filtering, are batches replenished, and what are the retained trajectory and supervised-token counts in each condition? How is the loss reduced across tokens and trajectories? If the supervision budgets differ, does the incorrect-only advantage persist when they are matched? This would directly affect my assessment of the paper’s central motivating observation.

  3. Which divergence and token/sequence reduction are used in the reported experiments? For R1 self-distillation, what teacher/student conditioning difference produces the learning signal when the teacher is fixed to the student’s initialization? Appendix B specifies the thinking-mode difference for Qwen3, but the corresponding R1 setup is unclear.

3 comments

We thank the reviewer for the careful reading. We address the questions in order.

Q1 / W1: Metric definition, Figure 1 vs. Table 4, and multi-seed results

(a) Definition of Avg@12 / Avg@4. Avg@k is the mean accuracy over k independently sampled responses per problem, averaged over problems; it is not pass@k (best-of-k). We have clarified this definition in Section 4.1 of the revised PDF.

(b) Figure 1 “Both” vs. Table 4 OPD(4096). Thank you for identifying this discrepancy. We checked the underlying experiment records and confirmed that the HMMT25 score for the “Both” condition was incorrectly transcribed in the original Figure 1. The correct value is 25.83, as reported in Table 4. Yet this correction does not change our main finding that incorrect-only training outperforms correct-only training.

(c) Multi-seed results. The paper reported a single seed (42). We repeated the runs with seeds 123 and 2026. The table below reports the three-benchmark average as mean ± sample standard deviation; paired gains compare ReNIO and the baseline using the same seed.For evaluation, averaging 12 responses per problem reduces evaluation-sampling noise.

ModelMethodAIME24AIME25HMMT25Avg
Qwen3-1.7BOPD53.61 ± 0.7341.57 ± 2.3725.93 ± 2.4940.37 ± 1.11
OPD + ReNIO55.46 ± 2.0542.04 ± 0.9027.04 ± 1.4041.51 ± 1.07
Δ per seed+0.27 / +0.83 / +4.45 (+1.85)+3.91 / −0.28 / −2.22 (+0.47)+0.83 / +2.50 / +0.00 (+1.11)+1.66 / +1.02 / +0.74 (+1.14 ± 0.47)
Qwen3-1.7BOPSD52.69 ± 0.8940.18 ± 2.3625.46 ± 1.8539.44 ± 1.21
OPSD + ReNIO57.13 ± 1.3740.46 ± 2.0126.57 ± 1.6341.39 ± 1.49
Δ per seed+4.70 / +6.39 / +2.23 (+4.44)+0.84 / −1.67 / +1.67 (+0.28)+0.28 / +3.33 / −0.28 (+1.11)+1.97 / +2.68 / +1.20 (+1.95 ± 0.74)
Qwen3-4BOPSD75.00 ± 0.5967.32 ± 1.6743.05 ± 0.7461.79 ± 0.96
OPSD + ReNIO75.68 ± 0.3168.34 ± 0.7445.07 ± 1.3963.03 ± 0.76
Δ per seed+0.27 / +0.94 / +0.83 (+0.68)+0.28 / +2.22 / +0.56 (+1.02)+3.34 / +1.95 / +0.76 (+2.02)+1.30 / +1.70 / +0.72 (+1.24 ± 0.50)

The mean gains are relatively consistent with the paper. ReNIO also improves the three-benchmark's average in every seed of every setting, which supports both the effectiveness and the stability of the gain.

Q2 / W2: Supervision budget in the correctness-filtering experiment

We acknowledge that The original experiment filtered trajectories without replenishing the batch, resulting in unequal supervision budgets. The revised protocol retains 32 trajectories per update, one per prompt, by repeatedly sample responses for each prompt to obtain one with the target correctness label.(Appendix B)

We also compare performance against cumulative supervised tokens. Incorrect-only training still outperforms correct-only training, with the advantage generally persisting at comparable token budgets (Figures 1a–b).

Q3: Distillation divergence, loss reduction, and teacher–student differences in OPSD

(a) Divergence. Following [1], all main experiments use forward KL (FKL). Prior work notes that the relative effectiveness of FKL and reverse KL (RKL) is task-dependent [2]. We therefore additionally compare them under OPD with Qwen3-1.7B:

DivergenceAIME24AIME25HMMT25Avg
FKL54.4439.1627.5040.37
RKL47.2235.8324.1735.74

FKL outperforms RKL by 4.63 percentage points on the three-benchmark average. The magnitude of this gap is comparable to that reported in Table 3 of [1], providing additional empirical support for our choice of FKL in this setting.

(b) Reduction. The divergence is computed at every token position, averaged over the tokens within each trajectory, and then averaged over the trajectories in the batch. The ReNIO weight multiplies the per-trajectory mean.

(c) Teacher–student difference in OPSD. Following [1], the frozen self-teacher additionally receives a reference solution to the training problem and supervises the student’s reasoning conditioned on it; the student sees only the problem. The extra teacher-side prompt is:

Here is a reference solution to this problem:
=== Reference Solution Begin ===
{solution}
=== Reference Solution End ===

This privileged information is the source of the learning signal in all OPSD experiments, for both Qwen3 and R1-Distill-Qwen. The thinking-mode difference is a separate configuration choice that applies only to Qwen3 models; R1-Distill-Qwen has no such switch and relies on the privileged reference solution alone.

[1] Zhao et al. (2026). Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. [2] Agarwal, R., Vieillard, N., Zhou, Y., Stańczyk, P., Ramos, S., Geist, M., & Bachem, O. (2023). On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. International Conference on Learning Representations.

Thank you for the detailed response and additional experiments. The multi-seed results and revised filtering experiment largely address my concerns about the reliability of the gains and unequal supervision budgets. Overall, I will maintain my rating of 3.

We sincerely thank the reviewer for the positive assessment and for confirming it after reading our response.

Reviewer_rZatOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The authors propose ReNIO, a method for OPSD/OPD that reweights trajectories as a function of how token distributions between the student and teacher differ. More specifically, ReNIO measures the ratio between the student’s and teacher’s probability of each token in a student-generated trajectory. Tokens where this log-ratio exceeds a threshold (i.e. where the student is confident but the teacher is not) are treated as "pivotal" tokens likely to steer the trajectory toward an incorrect answer. The clipped log-ratios of these tokens are combined (averaged and exponentiated) to give a trajectory-level weight and then used to scale the trajectory’s distillation loss. The aim is to emphasize likely-incorrect trajectories, which the authors find to be more useful than correct ones for OPD/OPSD, without rolling out to the final answer, which preserves the ability to train on short truncated prefixes (framed as a cost saving measure). Experiments on Qwen3 (1.7B, 8B) and DeepSeek-R1-Distill-Qwen (1.5B, 7B) models, evaluated on AIME24/25, HMMT25, HumanEval+ and MBPP+, show improvements over unweighted OPD/OPSD of typically 1-2 points on average, along with comparisons to GRPO.

The key contribution is thus a method that upweights negative rollouts in OPD/OPSD (which the authors argue is a good thing) while maintaining learning from partial prefixes that may not possess a final answer.

Critical strengths
  • Interesting problem setup. One of OPSD's benefits/assumptions is the ability to learn from partial prefixes (unlike RL, which requires complete trajectories due to its dependence on rewards), which reduces the cost of training. ReNIO allows us to reap the benefits of upweighting negative rollouts (which the authors show is beneficial) while also maintaining dependence only on partial prefixes.
Critical weaknesses
  • Motivation behind the method. The main benefit of using partial prefixes is a cost one, as framed in this paper. Indeed, it appears that ReNIO is no better than simple negative rollout filtering (Fig. 1: 42.9 OPD / 41.6 OPSD vs. Table 4: 42.3 / 40.6). However, this cost vs. accuracy tradeoff is not quantified. How does rollout cost trade off against potential loss of performance when we use ReNIO + partial prefixes? Without this comparison it is hard to properly assess the value of the method.
  • The main Qwen experiments are done with thinking enabled for the teacher and disabled for the student, following the OPSD setup. Why do we want the student with thinking mode disabled to learn from the distribution of the teacher with thinking mode enabled, when the sorts of test time behaviours we want from these models are very different? And why do we then evaluate the student with thinking mode enabled? Would it not be better for both student and teacher to use thinking mode since the idea is to train models to reason better on "challenging" math and code reasoning tasks? Furthermore, what behaviour changes (desirable or not) might we see as a result of this sort of training? I realize this is a challenge to the OPSD setup as well as the ReNIO one, but since they are the same I feel it is an appropriate one to make.
  • Results: gains appear to be small vs. the unweighted baselines (usually in the 1-2% range). Without error bars it is hard to know whether this is just noise.
  • The GRPO baseline appears to be very weak, and regresses from the base model in certain experiments (e.g. Qwen3-4B and R1-Distill-Qwen-7B for AIME2, Table 2). This points to potential issues with the GRPO baseline, either with the data or with the training process. Furthermore, GRPO details are not provided, so it is hard to know whether this baseline was conducted properly.
Questions for authors
  • Pivotal tokens: this is an interesting idea, and the finding that incorrect rollouts have much bigger mean log ratios is an interesting one. What do the pivotal tokens look like? If there are simple lexical "reflection-style markers" (e.g. "hmm", "wait") then a reasonable baseline would be to simply upweight trajectories that contain a larger proportion of such tokens. Furthermore, length also often correlates with correctness (usually inversely), so length weighting could also serve as a reasonable baseline.
  • What is the privileged information used for OPSD?

To raise my score, I would need to see the following:

  • Justification for the OPSD and OPD setups. More fundamentally, why distillation of thinking enabled teacher behaviour into a non-thinking student is a good idea, and what this conceptually helps to improve. The arguments/evidence given in the OPSD paper should at least be discussed explicitly since it is key to the setup, although even then I believe the justification there to be somewhat weak. I would prefer seeing experiments where these methods are applied to models with thinking mode enabled, as this best reflects how test time compute is scaled on reasoning tasks within the community today.
  • Error bars on results, or more convincing results.
  • Justification for the weak GRPO baselines, potentially with reruns if there are data/setup problems, although I realize this is not central to the paper (it is reasonable for ReNIO to "lose" to GRPO, so long as there are other clear benefits e.g. cost or ease of training, quantified).
  • Accuracy vs. cost tradeoff analysis, so that we can better understand when training on partial prefixes with ReNIO is a desirable thing to do.
5 comments

(Part 1/3)

We thank Reviewer rZat for the detailed and constructive review. We ran additional experiments and clarify both the evidence and its limitations below.

W1/Q6: Is ReNIO better than simple negative-rollout filtering? What is the accuracy–cost trade-off?

(a) ReNIO vs. incorrect-only. We additionally compare the two methods on Qwen3-4B and R1-Distill-Qwen-1.5B with a 4096-token train budget. In all new runs, the incorrect-only baseline replenishes its batches after filtering, so both methods train on the same number of trajectories per step.

ModelMethodAIME24AIME25HMMT25Avg
Qwen3-4BOPSD + Incorrect-only74.4470.2845.2863.33
OPSD + ReNIO75.5670.0045.8363.80
R1-Distill-Qwen-1.5BOPSD + Incorrect-only27.5023.8913.8921.76
OPSD + ReNIO30.0024.1715.0023.06
OPD + Incorrect-only28.6123.6114.4422.22
OPD + ReNIO31.3925.2813.8923.52

On Qwen3-4B and R1-Distill-Qwen-1.5B, ReNIO matches or exceeds it under both OPD and OPSD, at the same training budget. However, incorrect-only filtering must still generate every rollout to completion, it does not share ReNIO’s rollout-cost advantage.

(b) Accuracy vs. rollout cost. On Qwen3-1.7B, we vary the rollout token budget from 512 to 4096 tokens and report mean AIME24/25 Avg@12.

Token budgetOPDOPD + ReNIOOPSDOPSD + ReNIO
51245.1448.0645.6946.53
102446.8048.8947.5050.28
204848.3348.4745.1446.67
409649.1649.1745.2848.20
These results suggest that ReNIO improves the accuracy–token-budget frontier, ReNIO is therefore most useful when the rollout budget is tight.

W2/Q3: Why a non-thinking student with a thinking teacher?

We follow the setup of [1], which also uses a non-thinking student and a thinking teacher while training. [2] reports that when both think, self-distillation can end up below the base model, and our results confirm this: (For evaluating, following [1], we enable model thinking.)

MethodStudent modeAIME24AIME25HMMT25Avg
Qwen3 1.7B–51.3336.0023.3336.89
OPSDthinking50.0035.8320.8335.56
OPSD + ReNIOthinking51.6134.8922.8336.44 (+0.88)
OPSDnon-thinking52.6940.1825.4639.44
OPSD + ReNIOnon-thinking57.1340.4626.5741.39 (+1.95)
OPDthinking49.1735.2825.2836.57
OPD + ReNIOthinking50.5636.6725.6137.61 (+1.04)
OPDnon-thinking53.6141.5725.9340.37
OPD + ReNIOnon-thinking55.4642.0427.0441.51 (+1.14)

With a thinking student, OPD and OPSD fall below the base model and trail the non-thinking-student runs by 3.8–3.9 points. We therefore train with a non-thinking student. One plausible factor is that Qwen3’s two modes share all weights, so corrections learned on shorter non-thinking rollouts, which a fixed prefix budget covers more fully, transfer to thinking-mode inference. Importantly, ReNIO and OPD/OPSD baselines share this setting, so the comparison is fair. And ReNIO still improves over its baseline when the student thinks (+0.88 / +1.04), so our core conclusion does not depend on this choice.

In OPSD, the teacher is the frozen student given the reference answer (privileged information), and it guides the student's reasoning based on that answer. Hence privileged reference solutions provide supervision distinct from the teacher–student thinking-mode difference. 

[1] Zhao et al. (2026). Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models.

[2] Kaur et al. (2026). Rethinking On-Policy Self-Distillation for Thinking Models.

(Part 2/3)

W3/Q4: Error bars

We repeated the main runs with seeds 123 and 2026, and combined them with the seed-42 results reported in the paper, both with 1024 training tokens. Entries are mean ± std over the three seeds. The Δ rows give the paired per-seed improvement (ReNIO − baseline, same seed) in absolute points, listed as seed 42 / 123 / 2026.

ModelMethodAIME24AIME25HMMT25Avg
Qwen3-1.7BOPD53.61 ± 0.7341.57 ± 2.3725.93 ± 2.4940.37 ± 1.11
OPD + ReNIO55.46 ± 2.0542.04 ± 0.9027.04 ± 1.4041.51 ± 1.07
Δ per seed+0.27 / +0.83 / +4.45 (+1.85)+3.91 / −0.28 / −2.22 (+0.47)+0.83 / +2.50 / +0.00 (+1.11)+1.66 / +1.02 / +0.74 (+1.14 ± 0.47)
Qwen3-1.7BOPSD52.69 ± 0.8940.18 ± 2.3625.46 ± 1.8539.44 ± 1.21
OPSD + ReNIO57.13 ± 1.3740.46 ± 2.0126.57 ± 1.6341.39 ± 1.49
Δ per seed+4.70 / +6.39 / +2.23 (+4.44)+0.84 / −1.67 / +1.67 (+0.28)+0.28 / +3.33 / −0.28 (+1.11)+1.97 / +2.68 / +1.20 (+1.95 ± 0.74)
Qwen3-4BOPSD75.00 ± 0.5967.32 ± 1.6743.05 ± 0.7461.79 ± 0.96
OPSD + ReNIO75.68 ± 0.3168.34 ± 0.7445.07 ± 1.3963.03 ± 0.76
Δ per seed+0.27 / +0.94 / +0.83 (+0.68)+0.28 / +2.22 / +0.56 (+1.02)+3.34 / +1.95 / +0.76 (+2.02)+1.30 / +1.70 / +0.72 (+1.24 ± 0.50)

The mean gains are consistent with the paper. ReNIO also improves the three-benchmark average in every seed of every setting, which supports both the effectiveness and the stability of the gain.

W4/Q5: Weak GRPO baseline "GRPO baseline regresses from the base model in certain experiments (e.g. Qwen3-4B and R1-Distill-Qwen-7B for AIME2".

That may because non enough training steps for GRPO. We reran GRPO on Qwen3-4B and R1-Distill-Qwen-7B (seed 42), extending training from 300 to 500 steps, which yields larger gains. We compare it with OPSD+ReNIO from the paper (also seed 42), and will rerun GRPO in the remaining settings as well. The GRPO hyperparameters are listed in Appendix B of the paper, which is following[1].

ModelMethodAIME24AIME25
Qwen3-4BBase74.4465.28
GRPO75.5667.78
OPSD + ReNIO75.8369.17
R1-Distill-Qwen-7BBase56.1140.83
GRPO57.7841.11
OPSD + ReNIO55.8342.78

On Qwen3-4B, GRPO remains below OPSD+ReNIO on both AIME24 and AIME25. On R1-Distill-Qwen-7B, GRPO is slightly better on average (49.44 vs. 49.31). We consider this small gap acceptable given ReNIO's substantially lower training cost(4.59s for ReNIO, 29.9s for GRPO, per training step, described in Figure 4(a)).

[1] Zhao et al. (2026). Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models.

(Part 3/3)

Q1: What do pivotal tokens look like? Marker-based and length-based baselines

We categorized the pivotal tokens selected during training on Qwen3-1.7B, and report their proportion.

CategoryExamplesOPDOPSD
Math+, 3, \pi29.34%29.56%
Style (incl. reflection)Therefore, and, but, hmm5.88%6.97%
Format**, ###, \n6.83%8.55%
Othersthe, we, compute57.94%54.91%

Most of them are Math and Others tokens. Style tokens (including reflection markers) make up only 6–7% of pivotal tokens, hence ReNIO’s weights are not mainly driven by reflection markers.

We also add the two suggested baselines: weighting trajectory with reflection-style markers's proportion or trajectory length. Trajectory length information requires a full-length rollout, so the length-weighting runs use a 4096-token budget(enough for full length). This also gives up the prefix-training advantage that ReNIO preserves. The marker-weighting runs use a 1024-token budget.

Qwen3-1.7B, 4096 tokensAIME24AIME25HMMT25Avg
OPD + length weighting51.9440.0025.0038.98
OPD + ReNIO56.3941.9428.6142.31
OPSD + length weighting53.5638.7825.0639.13
OPSD + ReNIO53.3343.0625.2840.56
Qwen3-1.7B, 1024 tokensAIME24AIME25HMMT25Avg
OPD + marker weighting50.2840.5625.8338.89
OPD + ReNIO54.7243.0628.3342.04
OPSD + marker weighting53.8938.3324.9839.07
OPSD + ReNIO57.7842.7827.7842.78

Both baselines are worse than ReNIO. ReNIO assigns weights at a finer grain: it uses token-level information from both the teacher and the student, rather than only the trajectory length or the number of reflection markers.

Q2: What is the privileged information in OPSD?

The privileged information is the reference solution to the training problem, and the teacher then guides the student's reasoning according to the reference solution. For math tasks, the reference solution is the worked reference solution from the dataset; for coding tasks, it is a reference code implementation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.