acceptodds
Under review as a conference paper at ICLR 2027

When to Use the Solution: Preserving Exploration in On-Policy Self-Distillation for Reasoning Models

Abstract

Does knowing the solution always lead to better supervision for reasoning models? On-policy self-distillation (OPSD) uses a solution-conditioned copy of the model to provide dense token-level supervision on student-generated trajectories. However, preferences formed with solution access may not be appropriate at every reasoning state, particularly when the model is still exploring competing continuations. Our experiments show that uniformly applying such supervision can degrade performance in thinking mode. Our token-level analysis further reveals that solution conditioning frequently changes the unprivileged self-teacher's leading candidate at high-entropy , whereas the leading candidate is usually preserved at low-entropy lock \~ states. These observations motivate adapting the supervision source to the uncertainty of the current reasoning state. We therefore propose Fork-Sensitive On-Policy Self-Distillation (FS-OPSD), which separates the decision of from . Specifically, the method consists of two components: (1) is proposed to mitigate exploration inhibition by selectively applying unprivileged self-teacher at forks and privileged self-teacher at locks. (2) is proposed to balance the weights of the selected source, ensuring supervision targets are appropriately distributed. Our evaluation across five thinking models and three mathematical reasoning benchmarks shows gains in averaged avg@16 up to 8.26% over vanilla OPSD and up to 2.36% over corresponding base models. Code is available at https://anonymous.4open.science/r/FS-OPSD-14A3/README.md.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on policy distillation, reasoning model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YVZoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper shows that vanilla OPSD often degrades thinking models, since solution-conditioned supervision applied at every token suppresses the student’s exploration. The authors use two frozen self-teachers: one conditioned on the reference solution (as in OPSD), and one without it. They classify each token position by the top-3 entropy of the solution-free teacher: at high-entropy (fork) states, the target is the solution-free teacher with temperature softening; at low-entropy (lock) states, the target is the solution-conditioned teacher; intermediate (neutral) states use the solution-free teacher without softening. On five thinking models and three math benchmarks, the method improves avg@16 by up to 8.26 points over OPSD and up to 2.36 points over the base model.

Critical strengths
  1. The motivation is clear and data-driven: Fig. 1 shows that solution conditioning overturns the self-teacher’s top token mainly at high-entropy forks and rarely at locks, directly justifying when to apply privileged supervision.

  2. The method is simple, adding only one extra teacher pass and an entropy gate with interpretable thresholds.

Critical weaknesses
  1. No comparison with recent OPSD variants. Purified OPSD[1] and AD-OPSD address the same thinking-collapse problem and are discussed in related work, but only vanilla OPSD is used as a baseline,
  2. Limited experimental scope. Experiments cover only math reasoning with a 4,096-token completion limit during training, so it is unclear whether the findings transfer to other domains or to longer reasoning traces where exploration matters most.
  3. Modest gains over Base and limited evaluation scale. Improvements over the base model are small for several models and the small-scale results in Appendix E are mixed, while each benchmark contains only 30 problems and no confidence intervals or multiple seeds are reported, so it is hard to assess whether the gains over Base are significant.
  4. Generality of the token-level analysis. The key observation motivating the method (Fig. 1) is based on 60 Qwen3-4B rollouts only; it is unclear whether the same fork/lock pattern holds for the other models, which weakens the justification for using a fixed τ_lock across all models.
  5. No computational cost analysis. The method requires two teacher forward passes instead of one, but no wall-clock or memory comparison with vanilla OPSD is reported.

[1] Shen, Zhanming, et al. "Purified OPSD: On-Policy Self-Distillation Without Losing How to Think." arXiv preprint arXiv:2607.02234 (2026).

[2] Peng, Keqin, et al. "Diagnosing and Mitigating Thinking Collapse in On-Policy Self-Distillation." arXiv preprint arXiv:2607.10805 (2026).

Questions for authors

Does FS-OPSD improve reasoning, or only preserve it?

The method’s supervision at every position is either the solution-free self-teacher (which is the student’s own initial distribution, softened) or the solution-conditioned self-teacher at locks. Neither source seems to provide a signal that could strengthen the student’s reasoning beyond its initial capability: at forks the student is anchored to its own prior, and at locks the solution only reweights candidates the student already prefers. This raises the possibility that the gains come mainly from the student learning to produce more concise or more decisive trajectories, rather than from improved reasoning.

Could the authors clarify where new capability is expected to come from under this design? Relatedly, does FS-OPSD change the average response length compared to Base and vanilla OPSD? If responses become shorter, this would support the interpretation above; if not, some analysis of what actually changes in the student’s trajectories would help substantiate the claim that exploration is preserved.

5 comments

Thank you for the review! We address each weakness and question below.

W1: No comparison with recent OPSD variants. Purified OPSD[1] and AD-OPSD address the same thinking-collapse problem and are discussed in related work, but only vanilla OPSD is used as a baseline,

Since the source code of Purified OPSD [1] and AD-OPSD [2] was not open-sourced, we did not previously compare our work against these two papers. Following the reviewer's suggestion, we have now implemented them based on the descriptions provided in the papers. And the results are shown below.

Methodaime24aime25hmmt25avgTraining time
Qwen3-4B72.0864.3841.0459.170
Vanilla OPSD67.9255.4236.0453.133h49min
Purified OPSD73.9667.2942.561.254h32min
AD-OPSD69.5861.0438.1356.254h3min
FS-OPSD74.5866.4643.1361.394h9min

As shown in the table, our method achieves comparable performance to Purified OPSD while requiring 8.5% less wall-clock training time (FS-OPSD: 4h9min and Purified OPSD: 4h32min). Compared with AD-OPSD, our method improves average accuracy by 5.14%, with consistent gains across three benchmarks. These additional comparisons demonstrate that FS-OPSD remains competitive with recent OPSD variants beyond the vanilla OPSD baseline.

[1] Shen, Zhanming, et al. "Purified OPSD: On-Policy Self-Distillation Without Losing How to Think." arXiv preprint arXiv:2607.02234 (2026).

[2] Peng, Keqin, et al. "Diagnosing and Mitigating Thinking Collapse in On-Policy Self-Distillation." arXiv preprint arXiv:2607.10805 (2026).


W2: Limited experimental scope. Experiments cover only math reasoning with a 4,096-token completion limit during training, so it is unclear whether the findings transfer to other domains or to longer reasoning traces where exploration matters most.

First we would like to provide further experiment results on math reasoning with a 8192-token completion limit during training. The results are shown below.

Methodaime24aime25hmmt25avg
Qwen3-4B72.0864.3841.0459.17
Vanilla OPSD67.0857.7134.3853.06
FS-OPSD72.5062.0842.0858.89

In this table we can see that the thinking collapse issue of vanilla OPSD happens the same in longer reasoning traces and our method significantly alleviates this problem. So the findings can transfer to longer reasoning traces.

Second we would like to provide further experiment results conducted on other science domains (physics, chemistry, biology). We seleted 10000 problems from MegaScience dataset, including 4000 physics, 2500 math, 2500 chemistry and 1000 biology, as our training set. Then we evaluate trained model on GPQA-Diamond, MMLU-Pro (categories: math, physics, chemistry and biology) and UGPhysics. We report avg@16 for GPQA-Diamond, avg@1 for MMLU-Pro and UGPhysics. The results are shown below.

MethodGPQA-DiamondMMLU-Pro-mathMMLU-Pro-physics-chemistry-biologyUGPhysicsavg
Qwen3-4B54.3288.8279.7324.0061.72
Vanilla OPSD53.8886.6871.7012.6856.24
FS-OPSD55.1889.9379.9222.6861.93

In this table we can see that the thinking collapse issue of vanilla OPSD happens in other science domains. Our method achieves higher performance than base model. So the findings can transfer to other domains.

AuthorsOpenReview ↗

Response (Part 2)

W3: Modest gains over Base and limited evaluation scale. Improvements over the base model are small for several models and the small-scale results in Appendix E are mixed, while each benchmark contains only 30 problems and no confidence intervals or multiple seeds are reported, so it is hard to assess whether the gains over Base are significant.

Here we would like to provide 95% paired-bootstrap confidence intervals for our main results (Table 1). The differences below are measured in percentage points of avg@16, averaged across AIME 2024, AIME 2025, and HMMT 2025.

ModelFS-OPSD − BaseFS-OPSD − vanilla OPSD
Qwen3-4B+2.22 [+0.42, +3.96]+8.26 [+5.56, +11.04]
Qwen3-8B+0.41 [−1.81, +2.71] †+7.08 [+4.44, +9.93]
R1-Distill-Qwen-7B+2.36 [+0.28, +4.51]+3.82 [+1.60, +6.04]
R1-Distill-Llama-8B+2.15 [+0.14, +4.31]+2.84 [+0.56, +5.28]
GLM-Z1-9B-0414+0.62 [−1.81, +3.13] †+4.51 [+1.53, +7.64]

FS-OPSD yields positive point estimates over Base across all five main models. The 95% paired-bootstrap confidence intervals support statistically significant gains for Qwen3-4B (+2.22 percentage points), R1-Distill-Qwen-7B (+2.36), and R1-Distill-Llama-8B (+2.15), providing evidence of improvements beyond the initial model’s performance. As a complementary comparison, FS-OPSD improves over vanilla OPSD by 2.84–8.26 percentage points across all five models, with every 95% confidence interval excluding zero.

These smaller gains should be viewed in the context of the problem studied: vanilla OPSD reduces average performance below Base on all five models. FS-OPSD consistently improves on this training approach and achieves statistically supported gains beyond Base in three models, demonstrating the practical value of adapting privileged supervision to the reasoning state.


W4: Generality of the token-level analysis. The key observation motivating the method (Fig. 1) is based on 60 Qwen3-4B rollouts only; it is unclear whether the same fork/lock pattern holds for the other models, which weakens the justification for using a fixed τ_lock across all models.

Here we would like to provide two more fork/lock pattern analysis for R1-Distill-Qwen-7B and R1-Distill-Llama-8B.

ModelSwitch:ForkNeutralLockDrop:ForkNeutralLock
Qwen3-4B43.5219.520.8629.401.350.066
R1-Distill-Qwen-7B37.2911.990.2205.850.2780.0057
R1-Distill-Llama-8B36.5610.970.2485.840.3260.0093

On these two models, the switch and drop share the same pattern as Qwen3-4B, indicating the same fork/lock pattern holds for different models.

As for τlock\tau_{lock}, the value is first derived mathematically (Appendix B). Then we combine concentration change point for calibration and choose τlock=0.22\tau_{lock}=0.22. The value is model-independent. We evaluated candidate thresholds from 0.19 to 0.26 in increments of 0.01. As shown below, all three evaluated models attain their lowest observed residual S(τ)S(\tau) at τ=0.22\tau=0.22, supporting our choice of a shared threshold across these models.

τ\tauQwen3-4BR1-Distill-Qwen-7BR1-Distill-Llama-8B
0.190.530.500.55
0.200.420.410.46
0.210.360.370.42
0.220.310.320.37
0.230.380.340.39
0.240.380.340.38
0.250.400.400.45
0.260.420.430.49
AuthorsOpenReview ↗

Response (Part 3)

W5: No computational cost analysis. The method requires two teacher forward passes instead of one, but no wall-clock or memory comparison with vanilla OPSD is reported.

We would like to provide computational cost comparison with vanilla OPSD. During training, we use vLLM as the inference engine, which makes the memory usage almost the same for both vanilla OPSD and FS-OPSD methods. Below we report training time required for different models.

Training:

Modelvanilla OPSDFS-OPSD
Qwen3-4B3h49min4h9min
Qwen3-8B4h59min5h32min
R1-Distill-Qwen-7B4h3min4h16min
R1-Distill-Llama-8B4h32min4h54min
Qwen3-4B5h7min5h34min

As can be seen from these two tables, our method requires only a small amount of additional training time compared to vanilla OPSD (less than 10%). But we have a relatively much better result than vanilla OPSD.

Reviewer_ceZ7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies issues with OPSD (a training routine wherein privileged information is shown to a teacher that then supervises a student's on policy rollouts). Specifically, the paper finds that this can suppress forking behavior that is important for modern thinking models, and the paper then proposes an updated objective to address this problem.

Critical strengths
  • The paper is well motivated, and principled in identifying weaknesses with some forms of OPSD setups.
  • The paper proposes an alternative distillation objective that on face improves performance noticeably.
  • The exposition of the paper is good, and it is a very timely/relevant topic.
Critical weaknesses
  • The OPSD numbers that the authors compare to seem different (and much worse) than the OPSD numbers quoted in the OPSD paper.
  • The forking/locking diagnosis isn't very novel as Kaur et. al and other works seem to identify the same issue.
  • The OPSD repo/updated paper seems to add some new loss term to deal with these forking/deliberation positions. Maybe this solves the issue entirely? I won't hold this as a critical weakness of the paper as this seems to be an update to the OPSD work.
  • Confidence intervals aren't provided in many of the measurements, which is important when working with such small benchmarks and margins.
  • In several settings the proposed method seems to present marginal gains over base model performance.
Questions for authors
  • How were your OPSD numbers calculated?
  • What exactly are you claiming as a novel contribution compared to Kaur et al., and the updated version of OPSD?
2 comments

Thank you for the review! We address each weakness and question below.

W1: The OPSD numbers that the authors compare to seem different (and much worse) than the OPSD numbers quoted in the OPSD paper.

Q1: How were your OPSD numbers calculated?

We would like to clarify that the OPSD numbers reported in our paper were calculated under our experimental protocol, with thinking mode enabled for both student and teacher during training, as shown in the table below. In the main experiments of Zhao et al. [1], thinking mode is disabled for student and enabled for teacher during training, while student evaluation uses thinking mode. Thus, their OPSD results and our OPSD baseline correspond to different training configurations.

MethodTeacher thinking (training)Student thinking (training)Student thinking (evaluation)
OPSD in [1]EnabledDisabledEnabled
OPSD baseline in our studyEnabledEnabledEnabled
FS-OPSD in our studyEnabledEnabledEnabled

[1] Siyan Zhao, Zhihui Xie, et al. "Self-distilled reasoner: On-policy self-distillation for large language models." arXiv preprint arXiv:2601.18734 (2026).


W2: The forking/locking diagnosis isn't very novel as Kaur et. al and other works seem to identify the same issue.

Q2: What exactly are you claiming as a novel contribution compared to Kaur et al., and the updated version of OPSD?

We build on Kaur et al.’s diagnosis of fork suppression and contribute an explicit state-dependent rule for selecting the supervision source during training. FS-OPSD (our method) uses solution-free self-teacher at fork and neutral states, admits the solution-conditioned teacher only at lock states, and softens fork targets while preserving their candidate ranking. This turns the fork–lock distinction into a concrete intervention on the distillation targets. We further provide theoretical derivation(Appendix B), empirical calibration (Fig. 3a), and sensitivity analysis (Table 4) for the thresholds used by this rule.


W3: The OPSD repo/updated paper seems to add some new loss term to deal with these forking/deliberation positions. Maybe this solves the issue entirely? I won't hold this as a critical weakness of the paper as this seems to be an update to the OPSD work.

Q2: What exactly are you claiming as a novel contribution compared to Kaur et al., and the updated version of OPSD?

We reproduced the updated OPSD under the setting that thinking mode is enabled for both student and teacher during the training phase. 1k and 4k represent the completion limit during training. We report avg@16 on AIME24, AIME25 and HMMT25. And below is the results.

Methodaime24aime25hmmt25avg
Qwen3-4B72.0864.3841.0459.17
Updated OPSD (1k)68.7558.9638.1355.28
FS-OPSD (1k)74.5865.0040.8360.14
Updated OPSD (4k)61.0457.2933.7550.69
FS-OPSD (4k)74.5866.4643.1361.39

In the results we can see that updated OPSD did not solve the thinking collapse issue. In our understanding, the pointwise divergence clipping method adopted by updated OPSD could be a useful mechanism for stability, but they still use teacher signal as the supervision target in all positions, which will lead to exploration suppression. Our method selects supervision source based on solution free uncertainty and reshape the supervision target. Compared to updated OPSD, our novel contribution focus on the selecion and construction of supervision signal.

AuthorsOpenReview ↗

Response (Part 2)

W4: Confidence intervals aren't provided in many of the measurements, which is important when working with such small benchmarks and margins.

W5: In several settings the proposed method seems to present marginal gains over base model performance.

Here we would like to provide 95% paired-bootstrap confidence intervals for our main results (Table 1). The differences below are measured in percentage points of avg@16, averaged across AIME 2024, AIME 2025, and HMMT 2025.

ModelFS-OPSD − BaseFS-OPSD − vanilla OPSD
Qwen3-4B+2.22 [+0.42, +3.96]+8.26 [+5.56, +11.04]
Qwen3-8B+0.41 [−1.81, +2.71] †+7.08 [+4.44, +9.93]
R1-Distill-Qwen-7B+2.36 [+0.28, +4.51]+3.82 [+1.60, +6.04]
R1-Distill-Llama-8B+2.15 [+0.14, +4.31]+2.84 [+0.56, +5.28]
GLM-Z1-9B-0414+0.62 [−1.81, +3.13] †+4.51 [+1.53, +7.64]

FS-OPSD yields positive point estimates over Base across all five main models. The 95% paired-bootstrap confidence intervals support statistically significant gains for Qwen3-4B (+2.22 percentage points), R1-Distill-Qwen-7B (+2.36), and R1-Distill-Llama-8B (+2.15), providing evidence of improvements beyond the initial model’s performance. As a complementary comparison, FS-OPSD improves over vanilla OPSD across all five models, with every 95% paired-bootstrap CI excluding zero.

These smaller gains should be viewed under the problem we study: vanilla OPSD reduces average performance below Base on all five models. FS-OPSD consistently improves on this training approach and achieves statistically supported gains beyond Base in three models, demonstrating the practical value of adapting privileged supervision to the reasoning state.

Reviewer_q2rCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the exploration–exploitation trade-off in diffusion RL fine-tuning. The authors propose E2Ada, which adds a dense exploration reward based on 1−SSIM between adjacent clean-image predictions. A noise-level predictor controls its weight, and adaptive termination skips late RL updates while keeping the remaining denoising steps. The paper presents theoretical arguments about optimal policies and entropy, then evaluates reward learning, diversity, image quality and training efficiency across diffusion and flow-matching models.

Critical strengths
  1. This evaluation extends beyond the mere maximization of training rewards. Table 1 presents twelve unoptimized metrics—including FID, LPIPS, and preference scores—for cases where the Average Aesthetic Score (AES) is roughly comparable; Figure 4 further illustrates how these metrics evolve as reward levels vary. For instance, E2Ada achieves an FID of 50.19 and an LPIPS of 0.651, whereas the corresponding figures for DDPO are 95.15 and 0.601, respectively. This provides compelling empirical evidence that, among the methods evaluated, E2Ada achieves a superior trade-off between quality and diversity.
  2. The experiments examine integration with various optimizers, as well as with diffusion and flow-matching models, covering aspects such as fixed/scheduled exploration, reward design, and an ablation study without ATS. Providing the standard deviation across three random seeds is highly useful. Table 4 distinguishes between the per-sample computational cost and the time required to reach the target: the full E2Ada approach achieves a PS of approximately 22.2 in 5.44 hours, whereas the variant without ATS requires 7.17 hours, and the baseline method takes 10.41 hours.
Critical weaknesses
  1. My main concern is that the theory does not yet justify the method. In Theorem 1, the authors combine results optimized for two different rewards, but this does not generally solve the problem of optimizing both rewards together. The difference can change which action the model prefers. Theorem 2 also does not show that the proposed weighting rule can produce the claimed shift between exploration and exploitation. The paper still needs to connect what the proofs assume with what the actual image-change reward does. These are central gaps in the claimed theoretical contribution.
  2. It is hard to tell how much E2Ada adds over the closest prior methods. AdaEnt already adjusts exploration and denoising steps, while AdaScope chooses which parts of denoising receive RL updates. Both are cited, but neither appears as a named comparison baseline. The simpler exploration and scheduling controls do not settle this comparison. The paper also needs clearer instructions for training the predictor and deciding when to stop updates. The empirical gains are useful, but the theoretical gaps and missing close-method comparison keep me at weak reject. A corrected derivation and clearer evidence of what E2Ada adds would make me more confident.
Questions for authors
  1. Can you correct the key steps in Theorems 1–2 and show that their conclusions apply to the exploration reward used in E2Ada?
  2. Under matched training settings, how much does E2Ada improve over AdaEnt or AdaScope, especially in image diversity and training time?
  3. The control condition in Figure 17 is not entirely clear. Compared to standard training that utilizes a final-image reward, what exactly is omitted in the "w/o backfilling" (without backfilling) setting?
  4. Can you specify the practical ATS stopping rule, including the predictor threshold and which RL updates are skipped once it is reached?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.