- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how on-policy self-distillation affects internal exploration during reasoning post-training. Using intermediate-layer predictive distributions, the authors observe declining semantic exploration under OPSD and introduce a retained-exploration statistic combining semantic diversity with cross-layer retention. RESD uses this teacher-side statistic to control the direction and magnitude of token-level self-distillation corrections within a GRPO objective. The training recipe additionally assigns nonzero advantages to all-correct/all-wrong groups and applies the modifications during the first 60 steps before reverting to standard GRPO. Experiments on three models and five mathematical reasoning benchmarks report substantial improvements, supported by component ablations, continuation analyses, and training-token-budget comparisons.
Critical strengths
-
Substantial and reasonably consistent empirical gains. Table 2 reports the best Avg@12 results on 14 of 15 model–benchmark combinations, with macro-average improvements of 8.72, 5.98, and 10.56 percentage points over the strongest reported baseline for the three models. Table 7 also shows improvements in average Pass@12. The inclusion of both Qwen and SmolLM models makes these results more persuasive than evidence from a single backbone. The magnitude and consistency of the reported gains make the complete training recipe worth investigating.
-
An interesting internal-state signal with several forms of supporting evidence. The paper goes beyond presenting a new weighting heuristic: it examines layer-wise training dynamics, semantic token categories, downstream continuation diversity, and alternative routing statistics. Tables 3 and 6 provide useful evidence that both the signal construction and correction direction matter within the proposed recipe. Appendix E also includes a meaningful comparison against cumulative training-generated tokens, partially addressing the concern that longer training trajectories alone explain the gains.
Critical weaknesses
-
The contribution relative to the closest existing directional self-distillation method is insufficiently established. The introduction and Section 5 clearly identify the conceptual distinction between output-level signals and internal retained exploration. However, the underlying combination of verifier-based advantages and dense teacher–student corrections is already established, and DASD already uses signed token-level supervision to attract or repel the student relative to the privileged teacher. The main potential methodological contribution is therefore the internal routing statistic and its justification, rather than the combination of RLVR and self-distillation or directional teacher guidance itself.
This makes the missing DASD comparison particularly consequential. Appendix B.2 transparently explains that its reproduction was unstable, but this does not establish the advantage of the proposed internal signal over the closest output-level alternative. Moreover, Tables 3 and 6 do not clearly specify the entropy source, layer, or routing direction. In the DASD formulation, Section 4, high student output entropy induces repulsion and low entropy induces attraction. Simply replacing Q with H in Algorithm 1 while retaining its routing sign would implement the opposite entropy rule. A faithful DASD comparison, or a controlled comparison with an appropriately directed output-entropy router, is needed to establish the incremental contribution.
-
The experimental controls do not yet isolate the source of the improvement, and an important configuration discrepancy needs clarification. RESD changes several aspects of training simultaneously: the internal router, nonzero advantages for homogeneous reward groups, and the 60-step intervention schedule. Algorithm 1 disables both the teacher correction and the modified base advantage after step 60. Consequently, comparison with baselines trained under their usual objectives for all 300 steps does not separate the value of the internal signal from the value of this early intervention. The existing ablations are informative, but a GRPO-plus-base-advantage control and baselines using the same distillation-to-GRPO schedule are missing. Importantly, the version without the base advantage still exceeds GRPO and RLSD on the three ablation benchmarks, so the results should not be dismissed as entirely explained by that modification.
There is also an apparent inconsistency between the shared rollout configuration in Section 4.1/Appendix B.2 and Table 8. For Qwen3-1.7B, GRPO and OPSD have similar response lengths, 2,372 and 2,337 tokens, but report 634K and 78K student tokens per step. These figures are approximately consistent with 256 versus 32 responses per step; the Qwen3-4B figures show a similar pattern. This may reflect different batch definitions or a reporting issue, but it must be reconciled before interpreting baseline fairness and efficiency.
Finally, the manuscript does not report independent training-run counts or uncertainty estimates, and checkpoint selection is unclear. These omissions particularly weaken conclusions based on small ablation differences and claims of training stability.
Additional Limitations and Interpretation
The evidence supports the effectiveness of the complete recipe more strongly than its proposed causal explanation. In Section 3.1.2 and Appendix A.2, entire suffixes are resampled from positions selected by different metrics without matching their relative positions, remaining horizons, or reasoning stages. Higher continuation diversity therefore does not isolate the causal importance of the selected token, and distinct-answer diversity includes incorrect answers. Similarly, higher Pass@k establishes improved sampling success but does not by itself establish more diverse successful reasoning strategies.
The common decoding settings in Table 5 and training-token comparison in Figure 14 are appropriate controls. However, they do not establish equal inference-token costs. Table 8 reports substantially longer training responses for RESD, while evaluation response lengths and performance under matched inference-token budgets remain unclear. Its average timing measurements also include many steps after the additional correction has been disabled.
The restriction to small models, LoRA, and mathematical tasks limits generalization, but I do not regard these choices as independent grounds for rejection. Likewise, the present evidence does not establish an implementation error or invalidate the reported performance gains.
Questions for authors
-
Closest baseline and novelty: What model distribution, layer, and routing direction define the entropy baseline in Tables 3 and 6? Can you provide a faithful DASD comparison, or a correctly directed student-output-entropy router under otherwise matched settings?
-
Configuration clarification: How many prompts and responses per prompt are processed in each training step for every method? How do these counts reconcile the shared configuration with the approximately eightfold GRPO–OPSD token-volume difference in Table 8?
-
Attribution: How much of the improvement remains when the base advantage and intervention schedule are matched? In particular, how do GRPO with the same homogeneous-group advantages, and OPSD/RLSD switched to GRPO after 60 steps, perform?
-
Statistical and selection protocol: How many independent training runs were performed? How were checkpoints, the 60-step horizon, and routing hyperparameters selected? Were independent validation data used, and can uncertainty estimates be provided for the principal comparisons?
-
Mechanism: Does Q retain its continuation-diversity advantage after matching cut positions, remaining horizons, and local uncertainty? Does the advantage also hold for valid or correct reasoning branches, rather than only for lexical diversity and distinct final answers?
-
Cost interpretation: What are the actual evaluation response lengths and inference-token costs? Can the active correction-stage overhead be reported separately from the subsequent GRPO stage, and can performance be compared under matched inference-token budgets?
5 comments
Response to Reviewer hiBs (Part 2)
Additional clarification and ablations on the correction horizon and base advantage. (Weakness 2 & Question 3)
The reviewer raises concerns about two design choices in RESD, the correction horizon and the base advantage.
First, we clarify that the correction horizon is a method-specific design choice in post-training. We apply the correction only during early training to encourage the student's exploration, since continuously driving exploration upward could instead increase uncertainty and destabilize training. This design is not unique to RESD: the RLSD baseline also adopts a 60-step early-correction schedule, consistent with both its official implementation and our RESD setting, whereas the other baselines do not use this design. Therefore, we do not consider it necessary or fair to impose an additional correction schedule on every baseline. Nevertheless, to address the reviewer's request for a distillation-to-GRPO comparison, we train OPSD for 60 steps and then switch to standard GRPO, with the results reported below.
For the base advantage, we also consider it a design choice, as explained in Section 3.2. Specifically, since RESD still applies token-level teacher corrections to all-correct and all-wrong groups, unlike standard GRPO where these groups receive zero advantages, we add a small signed base advantage to preserve the rollout-level correctness signal and stabilize their updates. Although we consider this additional control unnecessary for establishing our contribution, we include an ablation of GRPO with the same base advantage to further address the reviewer's concern.
| Method (Ablation settings) | AIME24 | AIME25 | HMMT25 | Avg. |
|---|---|---|---|---|
| Standard GRPO | 33.06 | 26.39 | 15.83 | 25.09 |
| Standard OPSD | 18.33 | 16.94 | 8.89 | 14.72 |
| GRPO + Base Advantage | 28.06 | 26.94 | 12.78 | 22.59 |
| OPSD (60 steps) + GRPO | 32.78 | 25.28 | 16.94 | 25.00 |
| RESD | 44.17 | 32.22 | 22.50 | 32.96 |
These results show that neither adding the base advantage to GRPO nor applying a 60-step OPSD-to-GRPO schedule reproduces RESD's gains, while RESD substantially outperforms both controls.
Furthermore, regarding the reviewer's observation that "the version without the base advantage still exceeds GRPO and RLSD on the three ablation benchmarks, so the results should not be dismissed as entirely explained by that modification," we emphasize that we have never attributed RESD's performance gains to the base-advantage modification. The base advantage is merely a supplementary design choice, whereas the core component of RESD is its internal-exploration-guided token-level teacher supervision. Thus, the fact that RESD without the base advantage still outperforms GRPO and RLSD on all three benchmarks is positive evidence that the self-distillation correction alone is already effective enough to surpass these baselines. Separately, Table 6 demonstrates the benefit of the base advantage itself, that the full RESD configuration outperforms the variant without it on all three benchmarks.
Rollout configuration of the baselines. (Weakness 2 & Question 2)
The difference between GRPO and OPSD in average response length and tokens generated per step is not an inconsistency, but results from different rollout settings. GRPO and the other baselines use a rollout , whereas for standard OPSD, we follow its original paper and official implementation and set , since pure KL-based self-distillation does not require a rollout group to compute a rollout-level advantage.
Response to Reviewer hiBs (Part 3)
Multi-run results and parameter-selection protocol. (Weakness 2 & Question 4)
We did not report statistical significance or multi-seed results because the performance improvements of our method reported in Section 4 are too substantial to be plausibly attributed to random seed selection or statistical noise. Nevertheless, to further address your concerns about statistical results, we conducted three independent runs (seeds 42, 43, and 44) for GRPO, OPSD, and RESD on Qwen3-1.7B. The results are reported as mean standard deviation below.
| Method | AIME24 | AIME25 | HMMT25 | Avg. |
|---|---|---|---|---|
| GRPO | ||||
| OPSD | ||||
| RESD (Ours) |
These multi-seed comparisons further substantiate the effectiveness of our method, with RESD consistently outperforming both GRPO and OPSD across all three benchmarks.
Regarding the selection protocol, we select the checkpoint with the highest Avg@12 on AIME24 for each model and training method, and use it for evaluation on the remaining four benchmarks. The routing hyperparameters and the 60-step correction horizon are determined based on preliminary experiments and observed training dynamics, and fixed throughout the main experiments. The choice of the key design parameters is further examined through ablation and sensitivity analyses in Tables 3 and 6, which demonstrate the effectiveness of our default configuration and the robustness of the proposed routing strategy.
Detailed continuation-diversity analysis of . (Question 5)
The reviewer asks whether retains its continuation-diversity advantage within comparable position ranges and among final-correct continuations. We conduct two additional analyses using normalized edit distance defined in Section 3.1.
(i) Relative-position analysis. We divide cut positions, normalized by the original rollout length, into three equal-width intervals and calculate continuation diversity within each interval.
| Relative cut position | Random | ||||
|---|---|---|---|---|---|
| First third | 0.7874 | 0.7770 | 0.7837 | 0.7886 | 0.7842 |
| Middle third | 0.7023 | 0.6968 | 0.7039 | 0.7416 | 0.7355 |
| Final third | 0.5540 | 0.5494 | 0.5125 | 0.6115 | 0.4776 |
(ii) Final-correct analysis. We calculate pairwise edit distance only among continuations with verified correct final answers.
| Metric | Random | ||||
|---|---|---|---|---|---|
| Normalized edit distance over final-correct continuations | 0.6423 | 0.6435 | 0.6427 | 0.6961 | 0.6438 |
achieves the highest edit distance (i) in all three position intervals and (ii) in the final-correct subset. These results show that the diversity advantage of persists across rollout stages and extends to successful continuations.
We provide these analyses only to address the reviewer's request, but do not consider them necessary components of the main paper. Important reasoning forks can naturally occur early where decisions shape the subsequent reasoning, so a position preference is expected. Greater diversity among final-correct continuations is also not a requirement for an important fork, since its importance lies in how alternative continuations lead to different outcomes, including both correct and incorrect answers. Our original analysis therefore measures diversity across all continuations and final answers at the selected positions.
Training efficiency and evaluation budgets. (Question 6)
Our existing experiments already show that RESD learns more effectively under matched training budgets and achieves higher evaluation accuracy under the same generation allowance, while forcing equal evaluation response lengths would unfairly penalize useful reasoning without clarifying either conclusion.
Appendix E compares performance under matched training-token budgets, and our additional analysis compares matched training-time budgets (added in Appendix E of current revised submission). Both show that RESD's gains persist after controlling for training compute.
For evaluation, the reported results already use the same 16K maximum response length across methods, the advantage of RESD therefore holds under the same test-time generation allowance.
A shared generation limit we set (aligning with commonly applied standard) gives every method the same opportunity to reason. How effectively a model uses that allowance is part of its reasoning ability. Equalizing actual token use would instead treat longer reasoning as a cost that must be removed, even when it contributes to a correct solution. We do not consider this an appropriate requirement for evaluating reasoning improvements.
Response to Reviewer hiBs (Part 1)
We appreciate the reviewer's feedback. Below, we address each of the main concerns raised in the "Critical Weaknesses" and "Questions For Authors" sections.
Main contribution of our work compared to related self-distillation methods. (Weakness 1)
First, we consider it unfair to characterize the contribution of our work as "insufficiently established."
In the Introduction (~line 50), we explicitly review and summarize how existing self-distillation methods augment RLVR objectives with dense token-level self-distillation signals. We also thoroughly explain why we leverage internal reasoning states, rather than final-prediction-based signals (e.g., entropy), to guide self-teacher supervision. In Section 5, we further discuss the motivation and key differences between our method and existing work, including DASD: "These methods mainly rely on rollout outcomes or output-level signals, whereas RESD regulates self-distillation based on whether teacher predictions internally retain rich reasoning alternatives."
Therefore, we believe that our contribution relative to closely related work has already been sufficiently established. Our contribution is clearly distinguished from "the combination of RLVR and self-distillation or directional teacher guidance itself" as stated in the review.
DASD comparison and entropy-routing ablation. (Weakness 1 & Question 1)
We have clarified in Appendix B.2 that DASD was not included as a comparable baseline because no official implementation was publicly available, and our own paper-based implementation exhibited unstable training. Since the reviewer requests a direct comparison with DASD, we provide its results along with detailed experimental settings.
The detailed experimental settings for DASD are listed below, with each setting explicitly identified as either following the original DASD paper or adopted from our own experimental setup when it is not specified in the paper or requires adaptation for a fair comparison.
All customized settings above are consistent with those used for other baselines (GRPO, RLSD, etc.), without deliberate modifications. The main token-level advantage computation follows the description in the original DASD paper. Specifically, DASD computes entropy from the student's final-layer output distribution and adopts the opposite routing direction to RESD: high-entropy positions are routed away from the teacher, while low-entropy positions are routed toward the teacher. Therefore, we consider this implementation a fair comparison with RESD, as the remaining experimental settings are matched, and the main difference lies in the token-level advantage computation and its corresponding routing signal. We report the DASD results below (Avg@12, %).
As mentioned in Appendix B.2, DASD exhibits highly unstable training dynamics in our implementation. On Qwen3-1.7B, within approximately 40 training steps, entropy drops from 0.256 to 0.060, while the average response length increases from approximately 1,085 to 9,390 tokens. Meanwhile, the fraction of responses reaching the 16K-token generation limit rises from 0% to 55.5%, and continues to increase to over 90% by approximately step 80. These concurrent changes indicate a degeneration toward low-entropy, increasingly long, and unsuccessful outputs. We additionally disable the DASD correction after the first 60 steps to match the correction horizon used by RESD. However, this variant also fails to achieve non-zero accuracy on any of the three test benchmarks.
In conclusion, under our controlled experimental settings, the DASD degradation likely stems from the instability of its routing metric or routing-direction design, given that all other settings are consistent with our RESD method.
Regarding the entropy ablations in Tables 3 and 6, we believe the experimental configurations are already sufficiently clear. These are controlled ablation studies in which each variant modifies only the specified component of standard RESD, while keeping all other settings unchanged. Therefore, the entropy ablation naturally retains the same model distribution, layer, and routing direction as standard RESD.