acceptodds
Under review as a conference paper at ICLR 2027

Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement

12 views

Abstract

On-policy distillation (OPD) offers dense token-level supervision as an alternative to the sparse outcome-level advantages of reinforcement learning with verifiable rewards (RLVR). However, the teacher scores student-generated trajectories that are inherently off-policy for it, so the reliability of its supervision, and hence the source of the student's improvement, remains unclear. We quantitatively analyze teacher supervision during OPD training and find substantial noise whose prevalence increases with teacher scale. Surprisingly, the student policy is insensitive to such noise, converging to comparable performance regardless of whether noisy supervision is retained or removed. Does OPD distill at all? By analyzing what drives its gains, we find that learning concentrates on low log-probability tokens, and using a single fixed negative advantage matches the performance of teacher-provided ones. This suggests that OPD works largely by suppressing low log-probability tokens, which requires no teacher. These findings motivate On-Policy Self-Adaptation (OPSA), a supervision-free method using entropy-adaptive negative advantages. It assigns stronger learning signals to high-entropy positions, suppressing tail tokens, and evenly redistributing probability mass among head tokens. Compared with the base Qwen3-1.7B, OPSA improves Avg@32 by 35.41 points on AIME24, corresponding to a 263% relative gain, and more than doubles Pass@32 across all three benchmarks. It also outperforms OPD by 16.77 points in Avg@32 on AIME24. Extensive experiments and analyses across model families and tasks further demonstrate its effectiveness and generalizability.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Training, Reinforcement Learning, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_8d3wOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates whether OPD improvements depend on teacher-specific supervision and proposes On-Policy Self-Adaptation (OPSA). OPSA applies entropy-adaptive negative advantages to the lowest log-p sampled tokens, without teacher feedback or verifiable rewards during parameter updates. The reported results show substantial improvements across mathematical reasoning, code generation, and multiple model families.

Critical strengths

The idea of self-adaptive is interesting. The teacher-free training recipe is simple and practically attractive, and the reported gains are substantial. Cross-model and cross-domain experiments strengthen the empirical contribution.

Critical weaknesses

The definition of noisy supervision is questionable. A negative advantage on a correct answer means that the teacher assigns it lower probability than the student, not necessarily that the guidance is wrong. Disagreement between advantage signs and answer correctness therefore does not directly establish unreliable supervision.

Matching OPD’s performance without a teacher does not show that OPD itself improves through the same mechanism. Moreover, filtering trajectories using final answer tokens still retains teacher guidance throughout the reasoning trace. These experiments do not sufficiently isolate the contribution of teacher information.

The contribution of entropy weighting needs better controls for advantage magnitude and loss normalization. The mathematical validation split used for checkpoint selection should also be identified, and the main comparisons should include results across independent training seeds.

Questions for authors

Could the observed gains partly reflect the tendency of reverse KL to concentrate probability on particular teacher modes, especially when teacher and student already share dominant modes? This does not exclude knowledge transfer, since the teacher may still determine which modes are reinforced or suppressed. Could the authors compare against a sharpened student snapshot with matched update magnitudes, and examine whether the conclusions also hold under forward KL?

Does entropy weighting still outperform a fixed negative advantage when token selection, loss normalization, and update magnitudes are matched?

8 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/3)

Thank you for your great efforts on the review and constructive comments.


W1: The definition of noisy supervision is questionable. A negative advantage on a correct answer means that the teacher assigns it lower probability than the student, not necessarily that the guidance is wrong. Disagreement between advantage signs and answer correctness therefore does not directly establish unreliable supervision.


Under our definition of noisy supervision, OPD will reinforce incorrect answer tokens or suppress correct ones. For example, even when the student assigns 99% probability to a correct answer token, a teacher probability of 98% still yields a negative advantage, discouraging the student from generating that correct token. We therefore consider such supervision undesirable from an optimization perspective. Moreover, in practice, the training of OPD only replaces the advantage computation in GRPO, where advantage signs are aligned with answer correctness. This motivates our definition of noisy supervision, which is also consistent with recent studies, including OPDVR [1] and RA-OPD [2], that emphasize aligning advantage directions with answer correctness.


W2: Matching OPD’s performance without a teacher does not show that OPD itself improves through the same mechanism. Moreover, filtering trajectories using final answer tokens still retains teacher guidance throughout the reasoning trace. These experiments do not sufficiently isolate the contribution of teacher information.


  1. We don't make such a strong claim in our paper. We acknowledge that we cannot conclusively establish whether OPD's improvements arise from exactly the same mechanism as OPSA. Therefore, in Lines 245–249, we deliberately use “may” to present this as a possible explanation rather than a definitive conclusion. Moreover, the comparable performance achieved without teacher supervision, together with the noisy-only experiments, provides supporting evidence for our hypothesis that OPD's gains may not primarily stem from teacher knowledge transfer.
  2. We conduct additional experiments to further isolate the contribution of teacher information. Inspired by GRPO, we introduce a new noisy-only setting, where OPD is trained only on tokens whose advantage signs oppose the update direction implied by response correctness. Specifically, for incorrect responses with positive advantages on final answer tokens, we retain only positive-advantage tokens; for correct responses with negative advantages on final answer tokens, we retain only negative-advantage tokens. OPD still achieves comparable performance under this setting, providing further evidence that its gains may not primarily depend on correctness-aligned teacher supervision. All experiments use Qwen3-1.7B as the student and Qwen3-4B-Instruct as the teacher.
    AIME24 Avg@420406080100
    Standard OPD25.0034.1732.5034.1731.67
    Standard OPD (forward kl)21.6728.3334.1730.8330.00
    Clean-only response, trained on whole response21.6725.0029.1730.0028.33
    Clean-only response, trained on whole response (forward kl)16.6724.1725.8334.1730.83
    Noisy-only response, trained on whole response22.5028.3333.3333.3335.00
    Noisy-only response, trained on whole response (forward kl)13.3325.8336.6728.3332.50
    Noisy-only response, trained on noisy tokens11.6721.6735.0031.6730.83
    Noisy-only response, trained on noisy tokens (forward kl)15.0031.6734.1729.1732.50
  3. Our analyses in Figures 4 and 5 are conducted entirely without teacher supervision, demonstrating that a fixed negative advantage can reproduce OPD's performance gains. These experiments therefore isolate the contribution of teacher information by showing that comparable improvements can be achieved without it.

References

[1] Lin et al. (2026). On-policy Distillation with Verifiable Reward. arXiv:2608.24696.

[2] Gan et al. (2026). When Teacher Guidance Misleads: Reward-Aligned On-Policy Distillation. arXiv:2608.27960.

AuthorsOpenReview ↗

Official Comment by Authors (2/3)

W3: The contribution of entropy weighting needs better controls for advantage magnitude and loss normalization. & Q2: Does entropy weighting still outperform a fixed negative advantage when token selection, loss normalization, and update magnitudes are matched?


  1. OPSA's improvements stem from entropy-adaptive advantages rather than advantage magnitude alone. To isolate the effect of entropy adaptation, we introduce a mean-matched fixed-negative baseline, which replaces OPSA's token-level advantages with their batch mean at each training step. We also evaluate fixed advantages of −0.5 and −1.0 to further ensure comparable update magnitudes across settings. All settings use identical lowest-20%-logp token selection and loss normalization. As shown in the table below, OPSA consistently outperforms these baselines, suggesting that its gains arise from the entropy-dependent assignment of advantages rather than their magnitude alone.

    MethodsAIME24AIME24AIME25AIME25HMMT25HMMT25
    avg@32pass@32avg@32pass@32avg@32pass@32
    Qwen3-1.7B13.4440.009.6930.005.7323.33
    +A=−0.5A=-0.531.3570.0020.7353.3313.6540.00
    +A=−1.0A=-1.033.4470.0021.7756.6714.6940.00
    +mean-matched fixed-negative-advantage28.8573.3324.5850.0018.2343.33
    +OPSA48.8580.0035.3166.6723.3350.00
  2. OPSA does not introduce any special loss normalization. Since OPSA computes the loss only on the 20% of tokens with the lowest student log-probabilities, we naturally normalize the loss by the number of selected tokens rather than the total response length. This ensures that the loss reflects the average contribution of the tokens actually used for optimization.

  3. The fixed-negative-advantage baseline uses exactly the same token selection strategy as OPSA. As explicitly stated in Line 238 of our paper, the fixed-negative-advantage experiments also compute the loss only on the 20% of tokens with the lowest student log-probabilities. Therefore, both settings use identical token selection strategies, ensuring that their performance differences are not caused by differences in token selection.


W3: The mathematical validation split used for checkpoint selection should also be identified, and the main comparisons should include results across independent training seeds.


  1. We follow the standard checkpoint selection strategy in RL post-training and apply it consistently across all methods. Specifically, we select the best-performing checkpoint on AIME24, using the metric of avg@4, following prior OPD and RL studies. All baselines and our method use the same selection strategy to ensure fair comparisons.

  2. We further report results using an entirely independent dataset, AMC23, for checkpoint selection. As shown in the table below, selecting checkpoints based on either AIME24 or AMC23 yields comparable results across different methods. This further demonstrates that our conclusions are robust to the choice of checkpoint selection benchmark. We note that the identical results under different checkpoint selection strategies for OPD and OPSA arise because both strategies select the same checkpoint for each method.

    MethodsAIME24AIME24AIME25AIME25HMMT25HMMT25
    avg@32pass@32avg@32pass@32avg@32pass@32
    Qwen3-1.7B13.4440.009.6930.005.7323.33
    +OPD$_{aime24}$32.0873.3320.5250.0013.8540.00
    +OPD$_{amc23}$32.0873.3320.5250.0013.8540.00
    +GRPO$_{aime24}$33.9670.0025.3150.0015.1043.33
    +GRPO$_{amc23}$31.5670.0026.3556.6714.6943.33
    +OPSA$_{aime24}$48.8580.0035.3166.6723.3350.00
    +OPSA$_{amc23}$48.8580.0035.3166.6723.3350.00
  3. Comparison of different training seeds. Following prior RL and OPD studies, we use a fixed random seed of 42 for training and rollout generation across all methods to ensure fair comparisons. To assess robustness despite the computational cost, we additionally evaluate OPD and OPSA with three seeds (41, 42, and 43). As shown in the table below, despite some variation across seeds, OPSA consistently and substantially outperforms OPD, demonstrating the robustness of our method.

    MethodsAIME24AIME24AIME25AIME25HMMT25HMMT25
    avg@32pass@32avg@32pass@32avg@32pass@32
    Qwen3-1.7B13.4440.009.6930.005.7323.33
    +OPD$_{41}$29.9070.0016.3543.3311.7740.00
    +OPD$_{42}$32.0873.3320.5250.0013.8540.00
    +OPD$_{43}$32.7170.0019.6946.6713.6540.00
    +OPSA$_{41}$44.9076.6727.8160.0021.1550.00
    +OPSA$_{42}$48.8580.0035.3166.6723.3350.00
    +OPSA$_{43}$49.4880.0035.9463.3322.0853.33
AuthorsOpenReview ↗

Official Comment by Authors (3/3)

Q1: Could the observed gains partly reflect the tendency of reverse KL to concentrate probability on particular teacher modes, especially when teacher and student already share dominant modes? This does not exclude knowledge transfer, since the teacher may still determine which modes are reinforced or suppressed. Could the authors compare against a sharpened student snapshot with matched update magnitudes, and examine whether the conclusions also hold under forward KL?


Our paper does not analyze OPD from the perspective of why reverse KL is effective. We are unsure of the intended question, but speculate that the reviewer is suggesting using a sharpened student snapshot as the teacher to examine whether OPD's gains persist when the student and teacher already share dominant modes. In our paper, reverse KL is used only in Section 2.3 to investigate the student's insensitivity to teacher supervision. Nevertheless, we provide additional forward-KL experiments for completeness.

  1. A sharpened student snapshot does not improve performance. We construct the teacher by applying temperature scaling (t=0.5) to a student snapshot's logits. Under this setting, approximately 95% of tokens receive non-negative advantages, including 60% with zero advantages. Despite sharing the same dominant modes as the student, this sharpened teacher fails to improve performance, suggesting that shared dominant modes alone are insufficient to explain OPD's gains. This is consistent with our finding that suppressing low-probability tokens through negative advantages is important for OPD's improvements.

    AIME24 Avg@420406080100
    Standard OPD25.0034.1732.5034.1731.67
    Standard OPD (forward kl)21.6728.3334.1730.8330.00
    Sharpened student as teacher13.3311.677.509.176.67
  2. Our findings in Section 2.3 remain valid under forward KL. We additionally conduct OPD experiments using forward KL and observe conclusions consistent with those under reverse KL. The corresponding results are reported in the experimental table in our response to W2 (in Official Comment by Authors (1/3)).

Reviewer_i9wnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates the role of teacher signals in sampled-log-ratio OPD. It examines disagreement between final-answer correctness and advantage signs, then develops OPSA through token-selection and advantage-replacement experiments. OPSA updates the lowest-log-probability 20% of sampled tokens within each response using entropy-dependent negative advantages, requiring neither teacher scores nor answer labels. Experiments cover mathematical reasoning, code generation, and cross-domain transfer across Qwen and Olmo models.

Critical strengths
  1. OPSA provides a simple training rule with substantial reported benefits. On Qwen3-1.7B, AIME24 Avg@32 rises from 13.44 to 48.85, versus 32.08 for OPD and 33.96 for GRPO. Gains also occur for Qwen3.5 and both Olmo variants, while direct code training improves LiveCodeBench and HumanEval+. This breadth gives the method value beyond the initial OPD diagnostic.
  2. The subsequent interventions provide useful evidence for training design: fixed negative advantages on low-probability sampled tokens achieve gains comparable to OPD, while ablations of high-probability-token updates, entropy-dependent weights, and selection fractions examine which positions and signals matter. The NSR comparison and inference-time sampling/length controls also test plausible alternatives. These results support selective negative updates independently of the initial noise diagnostic.
Critical weaknesses
  1. Final-answer “noise” is an insufficient proxy for supervision quality across a trajectory. A teacher conditioned on student reasoning may reward an incorrect final answer while penalizing an earlier erroneous step; negative advantages on correct answers can also reflect relative confidence. Such local disagreement is foreseeable. Its frequency and variation across teachers are descriptive findings, but do not establish the teacher's overall contribution. Section 2.3 selects trajectories by answer signals while retaining teacher advantages at other positions, so noisy-only training does not exclude useful intermediate supervision. Fixed-negative-advantage replacement more directly supports an effective teacher-free recipe, but still does not identify the dominant mechanism of standard OPD. The introduction and Section 2 should narrow their mechanism claims accordingly.
  2. The method's advantage over existing label-free training remains insufficiently established. Intuitor [1] is discussed but not evaluated, and EM-FT/EM-RL [2] already exploit intrinsic uncertainty without labels. OPSA's selection and weighting differ, but comparisons against OPD, GRPO, TTRL, and NSR do not isolate their added value over these alternatives. Also, B.2 specifies best-checkpoint selection without identifying the mathematical validation split or fully detailing baseline budgets. These gaps limit the interpretation of the reported margins.

[1] Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. Learning to Reason without External Rewards. In International Conference on Learning Representations (ICLR), 2026.

[2] Shivam Agarwal, Zimin Zhang, Lifan Yuan, Jiawei Han, and Hao Peng. The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning. In Advances in Neural Information Processing Systems, volume 38 (NeurIPS), 2025.

Questions for authors
  1. Could the authors clarify how the answer-level noise diagnostic accounts for useful teacher guidance at earlier reasoning positions, distinguish what the noisy-only and fixed-negative-advantage interventions establish, and scope the mechanism claims accordingly?
  2. Could the authors provide a matched comparison with EM-FT/EM-RL [2] or Intuitor [1], keeping prompts, checkpoint selection, and computational budgets comparable, to assess OPSA's particular token-selection and weighting rule?
  3. Could the authors identify the mathematical validation questions, their separation from final tests, and the baseline selection budgets? Could the authors also reconcile NSR Pass@32 in Table 10 versus Figure 11(b), where AIME24 is 73.33 versus 66.7 and HMMT25 is 43.33 versus 40.0?

[1] Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. Learning to Reason without External Rewards. In International Conference on Learning Representations (ICLR), 2026.

[2] Shivam Agarwal, Zimin Zhang, Lifan Yuan, Jiawei Han, and Hao Peng. The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning. In Advances in Neural Information Processing Systems, volume 38 (NeurIPS), 2025.

4 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We sincerely thank the reviewer for the thoughtful and thorough comments on our paper.


Weakness 1: Final-answer “noise” is an insufficient proxy for supervision quality across a trajectory. A teacher conditioned on student reasoning may reward an incorrect final answer while penalizing an earlier erroneous step; negative advantages on correct answers can also reflect relative confidence. Such local disagreement is foreseeable. Its frequency and variation across teachers are descriptive findings, but do not establish the teacher's overall contribution. Section 2.3 selects trajectories by answer signals while retaining teacher advantages at other positions, so noisy-only training does not exclude useful intermediate supervision. Fixed-negative-advantage replacement more directly supports an effective teacher-free recipe, but still does not identify the dominant mechanism of standard OPD. The introduction and Section 2 should narrow their mechanism claims accordingly.


  1. Our definition of noisy supervision is based on the optimization direction, rather than the teacher's underlying knowledge. A negative advantage on a correct answer token reflects lower teacher confidence, but it suppresses the probability of that correct token during optimization. We consider this update direction undesirable from the perspective of outcome-based RL. Therefore, we define noisy supervision based on whether answer-token advantage signs align with final answer correctness. Recent studies, including OPDVR [1] and RA-OPD [2], similarly highlight the importance of aligning teacher-induced updates with verifiable rewards.

  2. We conduct additional experiments to isolate correctness-misaligned teacher supervision. Our original noisy-only setting selects responses based on final-answer advantage signs but retains advantages at all positions. To address this limitation, we introduce a new noisy-only setting that further restricts training to tokens whose advantage signs oppose the update direction implied by response correctness. We compare four settings: (1) Standard OPD, using all tokens; (2) Clean-only, using all tokens from responses with correctness-aligned answer advantages; (3) Noisy-only, trained on whole response, using all tokens from responses with correctness-misaligned answer advantages; and (4) Noisy-only, trained on noisy tokens, using only correctness-misaligned advantage tokens within noisy responses. All experiments use Qwen3-1.7B as the student and Qwen3-4B-Instruct as the teacher. As shown below, OPD still achieves comparable improvements under sequence noisy-only training, suggesting that correctness-aligned teacher guidance is not necessary for these gains.

    AIME24 Avg@420406080100
    Standard OPD25.0034.1732.5034.1731.67
    Clean-only response, trained on whole response21.6725.0029.1730.0028.33
    Noisy-only response, trained on whole response22.5028.3333.3333.3335.00
    Noisy-only response, trained on noisy tokens11.6721.6735.0031.6730.83

    As shown in the table, both noisy-only settings achieve performance comparable to standard OPD. Notably, even when restricting training to tokens whose advantage directions are inconsistent with trajectory-level correctness, OPD still achieves substantial performance improvements and reaches similar accuracy. This finding suggests that student model may insensitive to teacher supervision in OPD training, the improvement may not excactly from aligning it behavior closer to the teacher.

  3. Our claims in the paper are already appropriately qualified rather than definitive. Our teacher-free fixed-negative experiments reproduce comparable improvements without teacher information, while the noisy-only experiments provide further supporting evidence. Based on these findings, we hypothesize that OPD's gains may not primarily stem from teacher knowledge transfer. As explicitly stated in Lines 245–249, we use “may” to present this as a possible explanation, rather than claiming to have conclusively identified the dominant mechanism of standard OPD.


References

[1] Lin et al. (2026). On-policy Distillation with Verifiable Reward. arXiv:2608.24696.

[2] Gan et al. (2026). When Teacher Guidance Misleads: Reward-Aligned On-Policy Distillation. arXiv:2608.27960.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

Weakness 2 & Question 2.


  1. We follow standard RL checkpoint selection practices and apply the same strategy to all methods. Following prior RL and OPD studies, we select the best checkpoint on AIME24 and evaluate it on AIME24, AIME25, and HMMT25. Applying identical selection criteria across all methods ensures consistent comparisons.

  2. We additionally evaluate checkpoint selection on an independent dataset, AMC23. To address potential selection bias, we report results using both AIME24 (our original protocol) and AMC23, which is independent of all three evaluation benchmarks. Both strategies are applied consistently across methods, with the subscript “AMC23” indicating checkpoints selected on AMC23.

  3. OPSA substantially outperforms other label-free methods under identical training settings. We additionally compare OPSA with Intuitor and EM-RL, which use sequence-level entropy-based advantages. In contrast, OPSA applies entropy-adaptive negative advantages to the lowest-20%-logp tokens, enabling fine-grained token-level optimization. Results are reported in the table below.

    MethodsAIME24AIME24AIME25AIME25HMMT25HMMT25
    avg@32pass@32avg@32pass@32avg@32pass@32
    Qwen3-1.7B13.4440.009.6930.005.7323.33
    +Intuitor13.8546.678.9630.005.4226.67
    +Intuitor$_{amc23}$12.4040.008.6530.005.2123.33
    +EM-RL13.3350.007.9240.007.2926.67
    +EM-RL$_{amc23}$13.3350.007.9240.007.2926.67
    +OPSA48.8580.0035.3166.6723.3350.00
    +OPSA$_{amc23}$48.8580.0035.3166.6723.3350.00

    Similar to TTRL, sequence-level label-free methods yield limited gains or even degrade performance, whereas OPSA achieves substantial improvements. Identical results under different checkpoint selection strategies for EM-RL and OPSA arise because both select the same checkpoints. We attribute OPSA's gains to fine-grained token-level suppression, which improves accuracy at low-entropy positions while preserving exploration among head tokens at high-entropy reasoning forks (Figure 6).


Question 1.


  1. Noise-related experiments suggest that the student is relatively insensitive to teacher supervision during OPD training. Both answer noisy-only and sequence noisy-only settings achieve similar performance to standard OPD. Moreover, using teachers of different sizes also leads to similar converged accuracy, suggesting that the specific supervision provided by the teacher may not be essential to OPD's improvements.

  2. Fixed-negative-advantage experiments demonstrate that uniform token-level suppression can reproduce OPD's improvements. We find that positive advantages lead to training collapse, while a fixed negative advantage achieves accuracy comparable to OPD with teacher-provided advantages. This suggests that OPD's performance gains may primarily arise from the suppression effect, since the fixed negative advantage reproduces OPD's improvement.


Question 3.


  1. We follow the standard checkpoint selection strategy in RL post-training and apply it consistently across all methods. Specifically, we select the best-performing checkpoint on AIME24, using the metric of avg@4, following prior OPD and RL studies. All baselines and our method use the same selection strategy to ensure fair comparisons.

  2. We further report results using an entirely independent dataset, AMC23, for checkpoint selection. As shown in the table below, selecting checkpoints based on either AIME24 or AMC23 yields comparable results across different methods. This further demonstrates that our conclusions are robust to the choice of checkpoint selection benchmark. We note that the identical results under different checkpoint selection strategies for OPD and OPSA arise because both strategies select the same checkpoint for each method.

    MethodsAIME24AIME24AIME25AIME25HMMT25HMMT25
    avg@32pass@32avg@32pass@32avg@32pass@32
    Qwen3-1.7B13.4440.009.6930.005.7323.33
    +OPD$_{aime24}$32.0873.3320.5250.0013.8540.00
    +OPD$_{amc23}$32.0873.3320.5250.0013.8540.00
    +GRPO$_{aime24}$33.9670.0025.3150.0015.1043.33
    +GRPO$_{amc23}$31.5670.0026.3556.6714.6943.33
    +OPSA$_{aime24}$48.8580.0035.3166.6723.3350.00
    +OPSA$_{amc23}$48.8580.0035.3166.6723.3350.00
  3. Thank you for pointing out the inconsistency between Table 10 and Figure 11(b). This discrepancy was caused by incorrect values in the plotting code. We have corrected the error and updated the figure in the revised manuscript.

Thank you for the detailed response and the additional experiments. The comparisons with Intuitor and EM-RL address my request for relevant baselines, and OPSA’s gains persist when checkpoints are selected independently on AMC23. I also checked the revised Figure 11(b) and consider the NSR discrepancy resolved. The control that matches the mean advantage and the results across three training seeds further strengthen the empirical case for OPSA.

My remaining concern is how the paper interprets the gains from standard OPD. The new experiment that updates only “noisy” tokens is informative, but the final answer’s correctness does not tell us whether the retained teacher signal is useful at an intermediate reasoning step. The fixed negative advantage experiment shows that a teacher-free update can achieve comparable gains in the tested setting. What drives the gains in standard OPD remains open. I appreciate the authors’ clarification that suppression is a hypothesis and encourage them to frame it consistently as such in the abstract and main text.

The additional evidence strengthens my support for acceptance. I am keeping my rating at 3 (Weak acceptance).

We thank the reviewer for their thoughtful follow-up and continued support. While our original noisy-only experiment could not rule out potentially useful teacher signals at intermediate reasoning steps, our new experiment addresses this limitation from the perspective of outcome-based credit assignment. Specifically, following GRPO's approach of assigning the same response-level advantage to all tokens, including intermediate reasoning tokens, our sequence noisy-only experiment (in the response for Weakness 1) retains only teacher advantages whose signs oppose the update direction implied by response correctness throughout the entire trajectory.

Even under this setting, the student achieves performance comparable to standard OPD later in training. This suggests that the student may be relatively insensitive to teacher supervision during OPD training.

We will ensure that our findings are consistently framed with appropriate qualifications throughout the paper. We sincerely appreciate the reviewer's recognition of our work and constructive feedback throughout the discussion.

Reviewer_uGYWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduce a new method OPSA which require no external teacher, no hint, no verifiable reward and zero external supervision. This paper also investigate the source of performance improvements in OPD and found out that OPD works largely by suppressing low logprobability tokens which requires no teacher. Compare with base method, OPSA significantly improves performance.

Critical strengths
  1. The motivation is clear. This paper first studies what cause the current OPD improvement and they find out the teacher machanism does not needed. Then they proposed the OPSA method. Paper is very easy to follow
  2. method is simple and easy to implement. The method does not need external teacher, no verifiable reward and no external supervision.
  3. method has significant performance improvement. Based on the result, it has 200-300% performance improvement which is significant.
Critical weaknesses
  1. One of the core finding of this paper is the take away message from line 246-249. However based on Figure 4 and 5, the OPD convergence speed is much faster than OPSA. If no teacher is necessary, can auther discuss why OPSA has slower convergence speed?
  2. The improvements in Avg@k and Pass@k may result from longer responses (Fig 5), which provide more inference-time computation rather than necessarily indicating a better policy.
  3. What is the distribution of H among the selected tokens? If most values are concentrated while Hmin​ and Hmax​ are outliers, min-max normalization may compress most normalized scores into a narrow range and provide little meaningful adaptation.
Questions for authors
  1. Auther did not formally define AfixA^{fix} and can auther discuss about how to choose this value? I think this AfixA^{fix} is a significant hyperparameter of the loss function which need to be discussed.
  2. What happens when the correct token initially has low probability? OPSA may suppress rare but useful actions
  3. Why should the sign of the teacher student log probability ratio agree with trajectory level correctness?
5 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

Thank you for your great efforts on the review and constructive comments.


W1: One of the core finding of this paper is the take away message from line 246-249. However based on Figure 4 and 5, the OPD convergence speed is much faster than OPSA. If no teacher is necessary, can auther discuss why OPSA has slower convergence speed?


  1. The advantage ranges and magnitudes differ across OPD, OPSA, and the fixed-negative-advantage setting. As shown in Figure 3(a), the negative advantages in OPD span a wide range from approximately -20 to 0, whereas OPSA and fixed-negative-advantage training use much smaller advantage magnitudes. Since OPSA does not rely on external supervision, we adopt a relatively conservative advantage range. These differences in advantage magnitude may contribute to the observed differences in convergence speed.

  2. With appropriately increased update magnitudes, OPSA, fixed-negative-advantage training, and OPD exhibit similar convergence speeds. To investigate the impact of update strength, we increase the learning rate from 1e-6 to 5e-6. Notably, our original experiments simply followed the training setup adopted in prior OPD work, without carefully tuning the learning rate for our method. As shown in the table below, all three methods achieve comparable convergence speeds under the increased learning rate. These results suggest that the convergence speed difference observed in Figures 4 and 5 may be largely attributable to differences in update magnitudes. Training in the below table is initialized on Qwen3-1.7B, the teacher model is Qwen3-4B-Instruct for OPD.

    AIME24 Avg@420406080100
    OPD25.0034.1732.5034.1731.67
    A=-0.518.3333.3338.3336.6737.50
    OPSA20.0036.6740.8338.3344.17

W2: The improvements in Avg@k and Pass@k may result from longer responses (Fig 5), which provide more inference-time computation rather than necessarily indicating a better policy.


  1. We have already compared performance at similar response lengths via test-time scaling. In Table 11, we apply test-time scaling to GRPO and OPD to match OPSA's response length. Despite generating longer responses, these baselines show no improvement in Avg@32, while OPSA maintains an approximately 16% advantage at comparable lengths.

  2. Further comparison at similar response lengths via early-stopping OPSA training. We additionally select intermediate OPSA checkpoints with response lengths comparable to those of fully trained OPD models. As shown in the table below, even these partially trained OPSA models substantially outperform OPD in both Avg@32 and Pass@32, with further training yielding additional gains.

    MethodsAIME24AIME24response length
    avg@32pass@32token number
    Qwen3-1.7B13.4440.004457
    +GRPO33.9670.0019108
    +OPD32.0873.3315286
    +OPSA(step239)40.8376.6714372
    +OPSA(step259)41.9876.6715891
    +OPSA48.8580.0023205
  3. OPSA improves reasoning ability of the base model. We do not consider performance gains accompanied by longer responses a weakness, as increased response length during RL training can reflect improved reasoning capabilities. By adaptively reshaping token-level distributions, OPSA encourages more effective reasoning behaviors. Our experiments show consistent improvements across hybrid, instruct, and thinking models, enabling stronger performance through more extensive reasoning.


W3: What is the distribution of H among the selected tokens? If most values are concentrated while Hmin and Hmax are outliers, min-max normalization may compress most normalized scores into a narrow range and provide little meaningful adaptation.


To address the reviewer's concern that min-max normalization may lead to an uneven advantage distribution, we directly report the distribution of normalized negative advantages across training steps. Since our loss is computed only on the lowest-20%-logp tokens, we report statistics for these selected tokens, averaged over entire batches. As shown in the table below, the advantages are relatively evenly distributed, suggesting that min-max normalization does not introduce severe imbalance and enables fine-grained token-level adaptation.

Advantage RangeToken CountRatio
−1,−0.9)∣1989.6∣2.35∣[−0.9,−0.8)∣13796.4∣16.29∣[−0.8,−0.7)∣22669.2∣26.76∣[−0.7,−0.6)∣24434.8∣28.85∣[−0.6,−0.5-1,-0.9)|1989.6|2.35%| |[-0.9,-0.8)|13796.4|16.29%| |[-0.8,-0.7)|22669.2|26.76%| |[-0.7,-0.6)|24434.8|28.85%| |[-0.6,-0.5

|21810.0|25.75%|

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

Q1: Auther did not formally define AfixA^{fix} and can auther discuss about how to choose this value? I think this AfixA^{fix} is a significant hyperparameter of the loss function which need to be discussed.


  1. We have already explicitly specified Afix=−0.75A^{fix}=-0.75 in Section 4.2 of our paper. We use this value across all models without model-specific tuning, demonstrating consistent performance improvements.

  2. Ablation on AfixA^{fix}. We further evaluate different values of AfixA^{fix} on Qwen3-1.7B. As shown in the table below, OPSA achieves substantial gains across different negative values, while positive advantages cause training collapse. This suggests that the negative update direction matters more than the precise choice of AfixA^{fix}.

    MethodsAIME24AIME24AIME25AIME25HMMT25HMMT25
    avg@32pass@32avg@32pass@32avg@32pass@32
    Qwen3-1.7B13.4440.009.6930.005.7323.33
    +OPSA(Afix=0.75A^{fix}=0.75)0.103.330.000.000.103.33
    +OPSA(Afix=−0.5A^{fix}=-0.5)45.8373.3335.6366.6725.2153.33
    +OPSA(Afix=−1.0A^{fix}=-1.0)42.9283.3337.7170.0021.0450.00
    +OPSA(Afix=−0.75A^{fix}=-0.75)48.8580.0035.3166.6723.3350.00

Q2: What happens when the correct token initially has low probability? OPSA may suppress rare but useful actions


Sampling a correct low-probability token is extremely rare, and OPSA can effectively handle such cases. Depending on the token distribution, a correct low-probability token may belong to either the head or tail:

  1. Tail tokens: Their extremely low sampling probabilities make such cases rare. Even when sampled, further suppression is unlikely to substantially affect the overall output distribution.

  2. Head tokens: As illustrated in Figure 6, OPSA redistributes probability mass among competing head tokens rather than simply eliminating the sampled token. Thus, a correct head token is unlikely to be pushed into the tail by a single update.

Overall, these cases are rare, and OPSA's distribution reshaping can mitigate their potential negative effects, consistent with our empirical performance improvements.


Q3: Why should the sign of the teacher student log probability ratio agree with trajectory level correctness?


Aligning advantage signs with trajectory-level correctness is desirable for outcome-based optimization. OPD may reinforce incorrect answer tokens or suppress correct ones. For example, even if the student assigns 99% probability to a correct answer token, a teacher probability of 98% still yields a negative advantage, suppressing that correct token. We consider such updates undesirable from an optimization perspective. Moreover, OPD essentially replaces the advantage computation in GRPO, where advantage signs are aligned with response correctness. This motivates our definition of noisy supervision, which is further supported by recent studies, including OPDVR [1] and RA-OPD [2], that emphasize aligning teacher-induced updates with outcome correctness.


References

[1] Lin et al. (2026). On-policy Distillation with Verifiable Reward. arXiv:2608.24696.

[2] Gan et al. (2026). When Teacher Guidance Misleads: Reward-Aligned On-Policy Distillation. arXiv:2608.27960.

AuthorsOpenReview ↗

A Kind Reminder

Dear Reviewer uGYW,

We sincerely appreciate your insightful comments and advice, which have been instrumental in improving the quality and clarity of our paper. We hope that our responses have adequately addressed your concerns. As the discussion period nears its conclusion, If you have any remaining questions or concerns, we would be grateful for the opportunity to clarify them.

If you find that our responses have satisfactorily addressed your concerns, we would sincerely appreciate it if you could consider updating your rating accordingly.

Thank you again for your time and thoughtful feedback throughout the review process.

Best, Authors of Submission 4720

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.