acceptodds
Under review as a conference paper at ICLR 2027

Beyond Output-Level Guidance: Leveraging Internal Exploration for LLM Self-Distillation

Abstract

On-policy self-distillation (OPSD) provides dense token-level supervision for language model reasoning, but conditioning the self-teacher on privileged information may suppress the exploration needed by the student at inference time. By tracing middle-layer distributions throughout post-training from a interpretability perspective, we characterize how different training paradigms reshape the model's internal exploration–exploitation dynamics. In contrast to GRPO, OPSD progressively contracts latent exploration, and this contraction coincides with late-stage reasoning degradation. Motivated by this observation, we introduce retained exploration, a token-level signal that captures whether semantically diverse exploration persists across layers and identifies structurally important positions that shape subsequent reasoning trajectories. Based on this signal, we propose Retained-Exploration-Guided Self-Distillation (RESD), which preserves exploration-supported teacher guidance while counteracting potentially over-concentrated preferences at low-retained-exploration positions. Across three models and five mathematical reasoning benchmarks, RESD consistently outperforms strong RLVR and self-distillation baselines.

Primary area:
reinforcement learning
Keywords:
On-Policy Self-Distillation, Interpretability of Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_hiBsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how on-policy self-distillation affects internal exploration during reasoning post-training. Using intermediate-layer predictive distributions, the authors observe declining semantic exploration under OPSD and introduce a retained-exploration statistic combining semantic diversity with cross-layer retention. RESD uses this teacher-side statistic to control the direction and magnitude of token-level self-distillation corrections within a GRPO objective. The training recipe additionally assigns nonzero advantages to all-correct/all-wrong groups and applies the modifications during the first 60 steps before reverting to standard GRPO. Experiments on three models and five mathematical reasoning benchmarks report substantial improvements, supported by component ablations, continuation analyses, and training-token-budget comparisons.

Critical strengths
  1. Substantial and reasonably consistent empirical gains. Table 2 reports the best Avg@12 results on 14 of 15 model–benchmark combinations, with macro-average improvements of 8.72, 5.98, and 10.56 percentage points over the strongest reported baseline for the three models. Table 7 also shows improvements in average Pass@12. The inclusion of both Qwen and SmolLM models makes these results more persuasive than evidence from a single backbone. The magnitude and consistency of the reported gains make the complete training recipe worth investigating.

  2. An interesting internal-state signal with several forms of supporting evidence. The paper goes beyond presenting a new weighting heuristic: it examines layer-wise training dynamics, semantic token categories, downstream continuation diversity, and alternative routing statistics. Tables 3 and 6 provide useful evidence that both the signal construction and correction direction matter within the proposed recipe. Appendix E also includes a meaningful comparison against cumulative training-generated tokens, partially addressing the concern that longer training trajectories alone explain the gains.

Critical weaknesses
  1. The contribution relative to the closest existing directional self-distillation method is insufficiently established. The introduction and Section 5 clearly identify the conceptual distinction between output-level signals and internal retained exploration. However, the underlying combination of verifier-based advantages and dense teacher–student corrections is already established, and DASD already uses signed token-level supervision to attract or repel the student relative to the privileged teacher. The main potential methodological contribution is therefore the internal routing statistic and its justification, rather than the combination of RLVR and self-distillation or directional teacher guidance itself.

    This makes the missing DASD comparison particularly consequential. Appendix B.2 transparently explains that its reproduction was unstable, but this does not establish the advantage of the proposed internal signal over the closest output-level alternative. Moreover, Tables 3 and 6 do not clearly specify the entropy source, layer, or routing direction. In the DASD formulation, Section 4, high student output entropy induces repulsion and low entropy induces attraction. Simply replacing Q with H in Algorithm 1 while retaining its routing sign would implement the opposite entropy rule. A faithful DASD comparison, or a controlled comparison with an appropriately directed output-entropy router, is needed to establish the incremental contribution.

  2. The experimental controls do not yet isolate the source of the improvement, and an important configuration discrepancy needs clarification. RESD changes several aspects of training simultaneously: the internal router, nonzero advantages for homogeneous reward groups, and the 60-step intervention schedule. Algorithm 1 disables both the teacher correction and the modified base advantage after step 60. Consequently, comparison with baselines trained under their usual objectives for all 300 steps does not separate the value of the internal signal from the value of this early intervention. The existing ablations are informative, but a GRPO-plus-base-advantage control and baselines using the same distillation-to-GRPO schedule are missing. Importantly, the version without the base advantage still exceeds GRPO and RLSD on the three ablation benchmarks, so the results should not be dismissed as entirely explained by that modification.

    There is also an apparent inconsistency between the shared rollout configuration in Section 4.1/Appendix B.2 and Table 8. For Qwen3-1.7B, GRPO and OPSD have similar response lengths, 2,372 and 2,337 tokens, but report 634K and 78K student tokens per step. These figures are approximately consistent with 256 versus 32 responses per step; the Qwen3-4B figures show a similar pattern. This may reflect different batch definitions or a reporting issue, but it must be reconciled before interpreting baseline fairness and efficiency.

    Finally, the manuscript does not report independent training-run counts or uncertainty estimates, and checkpoint selection is unclear. These omissions particularly weaken conclusions based on small ablation differences and claims of training stability.

Additional Limitations and Interpretation

The evidence supports the effectiveness of the complete recipe more strongly than its proposed causal explanation. In Section 3.1.2 and Appendix A.2, entire suffixes are resampled from positions selected by different metrics without matching their relative positions, remaining horizons, or reasoning stages. Higher continuation diversity therefore does not isolate the causal importance of the selected token, and distinct-answer diversity includes incorrect answers. Similarly, higher Pass@k establishes improved sampling success but does not by itself establish more diverse successful reasoning strategies.

The common decoding settings in Table 5 and training-token comparison in Figure 14 are appropriate controls. However, they do not establish equal inference-token costs. Table 8 reports substantially longer training responses for RESD, while evaluation response lengths and performance under matched inference-token budgets remain unclear. Its average timing measurements also include many steps after the additional correction has been disabled.

The restriction to small models, LoRA, and mathematical tasks limits generalization, but I do not regard these choices as independent grounds for rejection. Likewise, the present evidence does not establish an implementation error or invalidate the reported performance gains.

Questions for authors
  1. Closest baseline and novelty: What model distribution, layer, and routing direction define the entropy baseline in Tables 3 and 6? Can you provide a faithful DASD comparison, or a correctly directed student-output-entropy router under otherwise matched settings?

  2. Configuration clarification: How many prompts and responses per prompt are processed in each training step for every method? How do these counts reconcile the shared configuration with the approximately eightfold GRPO–OPSD token-volume difference in Table 8?

  3. Attribution: How much of the improvement remains when the base advantage and intervention schedule are matched? In particular, how do GRPO with the same homogeneous-group advantages, and OPSD/RLSD switched to GRPO after 60 steps, perform?

  4. Statistical and selection protocol: How many independent training runs were performed? How were checkpoints, the 60-step horizon, and routing hyperparameters selected? Were independent validation data used, and can uncertainty estimates be provided for the principal comparisons?

  5. Mechanism: Does Q retain its continuation-diversity advantage after matching cut positions, remaining horizons, and local uncertainty? Does the advantage also hold for valid or correct reasoning branches, rather than only for lexical diversity and distinct final answers?

  6. Cost interpretation: What are the actual evaluation response lengths and inference-token costs? Can the active correction-stage overhead be reported separately from the subsequent GRPO stage, and can performance be compared under matched inference-token budgets?

5 comments
AuthorsOpenReview ↗

Response to Reviewer hiBs (Part 1)

We appreciate the reviewer's feedback. Below, we address each of the main concerns raised in the "Critical Weaknesses" and "Questions For Authors" sections.


Main contribution of our work compared to related self-distillation methods. (Weakness 1)

First, we consider it unfair to characterize the contribution of our work as "insufficiently established."

In the Introduction (~line 50), we explicitly review and summarize how existing self-distillation methods augment RLVR objectives with dense token-level self-distillation signals. We also thoroughly explain why we leverage internal reasoning states, rather than final-prediction-based signals (e.g., entropy), to guide self-teacher supervision. In Section 5, we further discuss the motivation and key differences between our method and existing work, including DASD: "These methods mainly rely on rollout outcomes or output-level signals, whereas RESD regulates self-distillation based on whether teacher predictions internally retain rich reasoning alternatives."

Therefore, we believe that our contribution relative to closely related work has already been sufficiently established. Our contribution is clearly distinguished from "the combination of RLVR and self-distillation or directional teacher guidance itself" as stated in the review.


DASD comparison and entropy-routing ablation. (Weakness 1 & Question 1)

We have clarified in Appendix B.2 that DASD was not included as a comparable baseline because no official implementation was publicly available, and our own paper-based implementation exhibited unstable training. Since the reviewer requests a direct comparison with DASD, we provide its results along with detailed experimental settings.

The detailed experimental settings for DASD are listed below, with each setting explicitly identified as either following the original DASD paper or adopted from our own experimental setup when it is not specified in the paper or requires adaptation for a fair comparison.

Parameter / SettingValue / DescriptionSource
Routing signal, computation details, and method-specific hyperparametersFollowing the original DASD paperDASD paper
Training datasetOpenThoughts MathOur official setting
Token-level correction weight β\beta0.30Our official setting
Batch size, learning rate, and other training settingsSame as Section 4.1Our official settings
Generation settings for training and evaluationSame as Appendix B.2Our official settings

All customized settings above are consistent with those used for other baselines (GRPO, RLSD, etc.), without deliberate modifications. The main token-level advantage computation follows the description in the original DASD paper. Specifically, DASD computes entropy from the student's final-layer output distribution and adopts the opposite routing direction to RESD: high-entropy positions are routed away from the teacher, while low-entropy positions are routed toward the teacher. Therefore, we consider this implementation a fair comparison with RESD, as the remaining experimental settings are matched, and the main difference lies in the token-level advantage computation and its corresponding routing signal. We report the DASD results below (Avg@12, %).

Training StepAIME24AIME25HMMT25
600.000.000.00
1000.000.000.00
2000.000.000.00
3000.000.000.00
300 (DASD correction disabled after step 60)0.000.000.00

As mentioned in Appendix B.2, DASD exhibits highly unstable training dynamics in our implementation. On Qwen3-1.7B, within approximately 40 training steps, entropy drops from 0.256 to 0.060, while the average response length increases from approximately 1,085 to 9,390 tokens. Meanwhile, the fraction of responses reaching the 16K-token generation limit rises from 0% to 55.5%, and continues to increase to over 90% by approximately step 80. These concurrent changes indicate a degeneration toward low-entropy, increasingly long, and unsuccessful outputs. We additionally disable the DASD correction after the first 60 steps to match the correction horizon used by RESD. However, this variant also fails to achieve non-zero accuracy on any of the three test benchmarks.

In conclusion, under our controlled experimental settings, the DASD degradation likely stems from the instability of its routing metric or routing-direction design, given that all other settings are consistent with our RESD method.

Regarding the entropy ablations in Tables 3 and 6, we believe the experimental configurations are already sufficiently clear. These are controlled ablation studies in which each variant modifies only the specified component of standard RESD, while keeping all other settings unchanged. Therefore, the entropy ablation naturally retains the same model distribution, layer, and routing direction as standard RESD.


AuthorsOpenReview ↗

Response to Reviewer hiBs (Part 2)

Additional clarification and ablations on the correction horizon and base advantage. (Weakness 2 & Question 3)

The reviewer raises concerns about two design choices in RESD, the correction horizon and the base advantage.

First, we clarify that the correction horizon is a method-specific design choice in post-training. We apply the correction only during early training to encourage the student's exploration, since continuously driving exploration upward could instead increase uncertainty and destabilize training. This design is not unique to RESD: the RLSD baseline also adopts a 60-step early-correction schedule, consistent with both its official implementation and our RESD setting, whereas the other baselines do not use this design. Therefore, we do not consider it necessary or fair to impose an additional correction schedule on every baseline. Nevertheless, to address the reviewer's request for a distillation-to-GRPO comparison, we train OPSD for 60 steps and then switch to standard GRPO, with the results reported below.

For the base advantage, we also consider it a design choice, as explained in Section 3.2. Specifically, since RESD still applies token-level teacher corrections to all-correct and all-wrong groups, unlike standard GRPO where these groups receive zero advantages, we add a small signed base advantage to preserve the rollout-level correctness signal and stabilize their updates. Although we consider this additional control unnecessary for establishing our contribution, we include an ablation of GRPO with the same base advantage to further address the reviewer's concern.

Method (Ablation settings)AIME24AIME25HMMT25Avg.
Standard GRPO33.0626.3915.8325.09
Standard OPSD18.3316.948.8914.72
GRPO + Base Advantage28.0626.9412.7822.59
OPSD (60 steps) + GRPO32.7825.2816.9425.00
RESD44.1732.2222.5032.96

These results show that neither adding the base advantage to GRPO nor applying a 60-step OPSD-to-GRPO schedule reproduces RESD's gains, while RESD substantially outperforms both controls.

Furthermore, regarding the reviewer's observation that "the version without the base advantage still exceeds GRPO and RLSD on the three ablation benchmarks, so the results should not be dismissed as entirely explained by that modification," we emphasize that we have never attributed RESD's performance gains to the base-advantage modification. The base advantage is merely a supplementary design choice, whereas the core component of RESD is its internal-exploration-guided token-level teacher supervision. Thus, the fact that RESD without the base advantage still outperforms GRPO and RLSD on all three benchmarks is positive evidence that the self-distillation correction alone is already effective enough to surpass these baselines. Separately, Table 6 demonstrates the benefit of the base advantage itself, that the full RESD configuration outperforms the variant without it on all three benchmarks.


Rollout configuration of the baselines. (Weakness 2 & Question 2)

The difference between GRPO and OPSD in average response length and tokens generated per step is not an inconsistency, but results from different rollout settings. GRPO and the other baselines use a rollout n=8n=8, whereas for standard OPSD, we follow its original paper and official implementation and set n=1n=1, since pure KL-based self-distillation does not require a rollout group to compute a rollout-level advantage.


AuthorsOpenReview ↗

Response to Reviewer hiBs (Part 3)

Multi-run results and parameter-selection protocol. (Weakness 2 & Question 4)

We did not report statistical significance or multi-seed results because the performance improvements of our method reported in Section 4 are too substantial to be plausibly attributed to random seed selection or statistical noise. Nevertheless, to further address your concerns about statistical results, we conducted three independent runs (seeds 42, 43, and 44) for GRPO, OPSD, and RESD on Qwen3-1.7B. The results are reported as mean ±\pm standard deviation below.

MethodAIME24AIME25HMMT25Avg.
GRPO30.19±2.5230.19 \pm 2.5224.17±2.2224.17 \pm 2.2213.24±2.2413.24 \pm 2.2422.53±2.2322.53 \pm 2.23
OPSD17.78±0.5517.78 \pm 0.5514.91±1.7814.91 \pm 1.788.06±1.008.06 \pm 1.0013.58±1.0113.58 \pm 1.01
RESD (Ours)45.46±1.1245.46 \pm 1.1233.89±1.6733.89 \pm 1.6722.31±0.8522.31 \pm 0.8533.89±0.8033.89 \pm 0.80

These multi-seed comparisons further substantiate the effectiveness of our method, with RESD consistently outperforming both GRPO and OPSD across all three benchmarks.

Regarding the selection protocol, we select the checkpoint with the highest Avg@12 on AIME24 for each model and training method, and use it for evaluation on the remaining four benchmarks. The routing hyperparameters and the 60-step correction horizon are determined based on preliminary experiments and observed training dynamics, and fixed throughout the main experiments. The choice of the key design parameters is further examined through ablation and sensitivity analyses in Tables 3 and 6, which demonstrate the effectiveness of our default configuration and the robustness of the proposed routing strategy.


Detailed continuation-diversity analysis of QQ. (Question 5)

The reviewer asks whether QQ retains its continuation-diversity advantage within comparable position ranges and among final-correct continuations. We conduct two additional analyses using normalized edit distance defined in Section 3.1.

(i) Relative-position analysis. We divide cut positions, normalized by the original rollout length, into three equal-width intervals and calculate continuation diversity within each interval.

Relative cut positionHHSESEQ′Q'QQRandom
First third0.78740.77700.78370.78860.7842
Middle third0.70230.69680.70390.74160.7355
Final third0.55400.54940.51250.61150.4776

(ii) Final-correct analysis. We calculate pairwise edit distance only among continuations with verified correct final answers.

MetricHHSESEQ′Q'QQRandom
Normalized edit distance over final-correct continuations0.64230.64350.64270.69610.6438

QQ achieves the highest edit distance (i) in all three position intervals and (ii) in the final-correct subset. These results show that the diversity advantage of QQ persists across rollout stages and extends to successful continuations.

We provide these analyses only to address the reviewer's request, but do not consider them necessary components of the main paper. Important reasoning forks can naturally occur early where decisions shape the subsequent reasoning, so a position preference is expected. Greater diversity among final-correct continuations is also not a requirement for an important fork, since its importance lies in how alternative continuations lead to different outcomes, including both correct and incorrect answers. Our original analysis therefore measures diversity across all continuations and final answers at the selected positions.


Training efficiency and evaluation budgets. (Question 6)

Our existing experiments already show that RESD learns more effectively under matched training budgets and achieves higher evaluation accuracy under the same generation allowance, while forcing equal evaluation response lengths would unfairly penalize useful reasoning without clarifying either conclusion.

Appendix E compares performance under matched training-token budgets, and our additional analysis compares matched training-time budgets (added in Appendix E of current revised submission). Both show that RESD's gains persist after controlling for training compute.

For evaluation, the reported results already use the same 16K maximum response length across methods, the advantage of RESD therefore holds under the same test-time generation allowance.

A shared generation limit we set (aligning with commonly applied standard) gives every method the same opportunity to reason. How effectively a model uses that allowance is part of its reasoning ability. Equalizing actual token use would instead treat longer reasoning as a cost that must be removed, even when it contributes to a correct solution. We do not consider this an appropriate requirement for evaluating reasoning improvements.

Reviewer_r1wHOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how on-policy self-distillation affects internal exploration in reasoning models. It proposes retained exploration, a middle-layer signal measuring whether semantically diverse alternatives persist across layers, and uses it to route self-teacher guidance.

Critical strengths
  1. The paper offers a novel internal-state perspective on self-distillation, beyond conventional output-level uncertainty or teacher–student disagreement.

  2. RESD achieves strong, consistent improvements across three models and five math benchmarks, outperforming the strongest baseline in 14 of 15 settings.

Critical weaknesses
  1. Middle-layer distributions are obtained using the final unembedding matrix, and cosine similarity between unembedding vectors is treated as semantic similarity. It remains unclear whether this reliably represents meaningful reasoning alternatives and the observed association between exploration contraction and performance degradation also does not establish causality.

  2. Evaluation is limited to relatively small models and mathematical reasoning, and RESD generates substantially longer responses and costs about 1.6× more time per training step than GRPO. A efficiency comparison of complete compute cost is needed.

Questions for authors
  1. How much of RESD’s improvement comes from retained-exploration routing versus its longer generated responses and additional training compute?

  2. Does the identified exploration layer and routing configuration transfer to larger models or non-mathematical tasks without model-specific tuning?

2 comments
AuthorsOpenReview ↗

Response to Reviewer r1wH

We appreciate the reviewer's thoughtful feedback. Below, we address each of the main concerns raised in the "Critical Weaknesses" and "Questions For Authors" sections.


Whether the retained exploration metric QQ reliably captures meaningful reasoning alternatives. (Weakness 1)

The construction of QQ follows established practices in LLM interpretability. Specifically, we use the unembedding matrix to obtain middle-layer distributions and cosine similarity between unembedding vectors to measure semantic diversity. Both choices are grounded in prior work cited in Section 2.1.

More importantly, we devote the entire Section 3.1 to directly validating whether high-QQ positions capture meaningful reasoning alternatives from two complementary perspectives. Section 3.1.1 shows that high-QQ positions are enriched for semantically important reasoning-control operations, while Section 3.1.2 demonstrates that resampling from these positions yields more diverse reasoning continuations and final answers.

Together, these results provide direct empirical evidence that QQ reliably identifies meaningful reasoning forks, rather than merely reflecting internal uncertainty or superficial token-level differences. We believe the analyses in Section 3.1 sufficiently establish the validity of QQ for the token-level supervision proposed in our method.


The association between internal exploration and the model performance. (Weakness 1)

We clarify that our work neither claims nor relies on a causal relationship between exploration contraction and performance degradation.

Concretely, as discussed in Section 2.2, this association between latent exploration and performance motivates us to regulate token-level teacher guidance according to internal exploration richness. Guided by this motivation, RESD improves model performance (Table 2) by encouraging the model's internal semantic exploration (Figure 7). Therefore, we see no need for a separate causal analysis, as the effectiveness of RESD itself provides strong empirical support for the association between exploration preservation and performance improvement.


Performance under matched training-token and training-time budgets. (Weakness 2 & Question 1)

The reviewer asks whether RESD's performance gains stem from longer generated responses or additional training compute. The original submission already includes a training-token-budget analysis (Figure 14 in Appendix E) to examine whether the gains can be explained by greater token consumption. To further address the concern about RESD's higher training time per step, we have added a training-time-budget analysis to the revised submission (Figure 15).

Both comparisons consistently show that RESD substantially outperforms GRPO under matched budgets. For example, under a 70M training-token budget, RESD achieves 40.79% vs. 27.38% on Qwen3-1.7B and 55.76% vs. 43.03% on Qwen3-4B. Under a 400-minute training-time budget, the corresponding results are 40.26% vs. 27.38% and 55.76% vs. 36.49%, respectively. These results demonstrate that RESD's performance gains do not come from simply generating more tokens or consuming more training time, but from its retained-exploration-guided routing design.


Generalization of RESD. (Weakness 2 & Question 2)

We further extend our method to the logical reasoning task to evaluate the generalization of RESD. Specifically, we conduct experiments on the Knights & Knaves dataset released by RLCSD. The training set contains 5,000 puzzles involving 4-8 characters. For evaluation, we use 500 held-out puzzles with 4-8 characters for in-domain testing, and 100 puzzles each with 9, 10, and 11 characters for out-of-domain testing.

We use Qwen3-1.7B and directly transfer RESD to this task, retaining the same training configurations, generation parameters, exploration layer, and routing configuration as in our mathematical reasoning experiments, without any task-specific modifications. We also include GRPO and OPSD as baselines. We evaluate predictions by verifying the complete knight/knave assignment and report Pass@1 for each test group and their unweighted average (directly applied from RLCSD). The results are summarized below.

Method4-8 Roles9 Roles10 Roles11 RolesAvg.
Base29.6012.0010.007.0014.65
GRPO47.4029.0017.0017.0027.60
OPSD26.4012.006.003.0011.85
RESD67.6056.0057.0031.0052.90

RESD consistently outperforms all baselines across both in-domain and out-of-domain test groups, achieving an average Pass@1 of 52.90%, substantially higher than GRPO (27.60%). These results demonstrate that RESD generalizes beyond mathematical reasoning, with the internal-exploration-based correction transferring directly to a different reasoning task without task-specific tuning.


Thank you for your response. I would like to update my score from 2 to 3. As this topic is somewhat outside my primary area of expertise, I believe the other reviewers' comments should also be given due consideration.

Reviewer_SpMoOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a potential weakness of on-policy self-distillation: a teacher given a reference solution may provide overly concentrated guidance, reducing the student’s exploration. To remedy this, this paper introduces retained exploration, a token-level score combining intermediate-layer uncertainty, semantic diversity, and persistence across subsequent layers. The proposed RESD method uses this score to steer the student toward the teacher at high-score positions and away at low-score positions, achieving consistent improvements over RLVR and self-distillation baselines.

Critical strengths
  • The paper is well written and well-motivated.
  • The proposed method achieved consistent improvements over the baselines across two model families.
  • Ablation studies support the design of the routing signal and correction direction.
Critical weaknesses
  • Statistical significance of the results is not reported.
  • The proposed method is highly related to DASD. The entropy-routing ablation is helpful, but does not replace a comparison against the complete DASD method.
Questions for authors

Please refer to my comments in the weakness part.

1 comment
AuthorsOpenReview ↗

Response to Reviewer SpMo

We appreciate the reviewer's thoughtful feedback. Below, we address each of the main concerns raised in the "Critical Weaknesses".


Statistical significance of the main experimental results. (Weakness 1)

We did not report statistical significance or multi-seed results because the performance improvements of our method reported in Table 2 are too substantial to be plausibly attributed to random seed selection or statistical noise. Nevertheless, to further address your concern about statistical results, we conducted three independent runs (seeds 42, 43, and 44) for GRPO, OPSD and RESD on Qwen3-1.7B. The results are reported as mean ±\pm standard deviation below.

MethodAIME24AIME25HMMT25Avg.
GRPO30.19±2.5230.19 \pm 2.5224.17±2.2224.17 \pm 2.2213.24±2.2413.24 \pm 2.2422.53±2.2322.53 \pm 2.23
OPSD17.78±0.5517.78 \pm 0.5514.91±1.7814.91 \pm 1.788.06±1.008.06 \pm 1.0013.58±1.0113.58 \pm 1.01
RESD (Ours)45.46±1.1245.46 \pm 1.1233.89±1.6733.89 \pm 1.6722.31±0.8522.31 \pm 0.8533.89±0.8033.89 \pm 0.80

These multi-seed comparisons further substantiate the effectiveness of our method, with RESD consistently outperforming both GRPO and OPSD across three benchmarks.


Comparison against the DASD baseline. (Weakness 2)

As clarified in Appendix B.2, DASD was not included as a comparable baseline because no official implementation was publicly available, and our own paper-based implementation exhibited unstable training. Since the reviewer still requests a direct comparison with DASD, we provide its results along with the detailed experimental settings.

The detailed experimental settings for DASD are listed below, with each setting explicitly identified as either following the original DASD paper or adopted from our own experimental setup when it is not specified in the paper or requires adaptation for a fair comparison.

Parameter / SettingValue / DescriptionSource
Routing signals, computation details, and hyperparameter settingsFollowing the original DASD paperDASD paper
Training datasetOpenThoughts MathOur official setting
Token-level correction weight β\beta0.30Our official setting
Batch size, learning rate, and other training settingsSame as Section 4.1Our official settings
Generation settings for training and evaluationSame as Appendix B.2Our official settings

All customized settings above are consistent with those used for other baselines (GRPO, RLSD, etc.), without deliberate modifications. The main token-level advantage computation strictly follows the description in the original DASD paper. The DASD performance are reported below.

Training StepAIME24AIME25HMMT25
600.000.000.00
1000.000.000.00
2000.000.000.00
3000.000.000.00

DASD exhibits highly unstable training dynamics in our implementation. On Qwen3-1.7B, within approximately 40 training steps, the entropy drops from 0.256 to 0.060, while the average response length increases from approximately 1,085 to 9,390 tokens. Meanwhile, the fraction of responses reaching the 16K-token generation limit rises from 0% to 55.5%, and continues to increase to over 90% by approximately step 80. These concurrent changes indicate a degeneration toward low-entropy, increasingly long, and unsuccessful outputs.

Moreover, DASD has already collapsed before step 60 when our correction phase ends. Under our experimental setting, this degradation likely stems from the instability of its routing metric or routing-direction design, given that all other settings are consistent with ours.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.