- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper investigates whether OPD improvements depend on teacher-specific supervision and proposes On-Policy Self-Adaptation (OPSA). OPSA applies entropy-adaptive negative advantages to the lowest log-p sampled tokens, without teacher feedback or verifiable rewards during parameter updates. The reported results show substantial improvements across mathematical reasoning, code generation, and multiple model families.
Critical strengths
The idea of self-adaptive is interesting. The teacher-free training recipe is simple and practically attractive, and the reported gains are substantial. Cross-model and cross-domain experiments strengthen the empirical contribution.
Critical weaknesses
The definition of noisy supervision is questionable. A negative advantage on a correct answer means that the teacher assigns it lower probability than the student, not necessarily that the guidance is wrong. Disagreement between advantage signs and answer correctness therefore does not directly establish unreliable supervision.
Matching OPD’s performance without a teacher does not show that OPD itself improves through the same mechanism. Moreover, filtering trajectories using final answer tokens still retains teacher guidance throughout the reasoning trace. These experiments do not sufficiently isolate the contribution of teacher information.
The contribution of entropy weighting needs better controls for advantage magnitude and loss normalization. The mathematical validation split used for checkpoint selection should also be identified, and the main comparisons should include results across independent training seeds.
Questions for authors
Could the observed gains partly reflect the tendency of reverse KL to concentrate probability on particular teacher modes, especially when teacher and student already share dominant modes? This does not exclude knowledge transfer, since the teacher may still determine which modes are reinforced or suppressed. Could the authors compare against a sharpened student snapshot with matched update magnitudes, and examine whether the conclusions also hold under forward KL?
Does entropy weighting still outperform a fixed negative advantage when token selection, loss normalization, and update magnitudes are matched?
8 comments
Official Comment by Authors (2/3)
W3: The contribution of entropy weighting needs better controls for advantage magnitude and loss normalization. & Q2: Does entropy weighting still outperform a fixed negative advantage when token selection, loss normalization, and update magnitudes are matched?
-
OPSA's improvements stem from entropy-adaptive advantages rather than advantage magnitude alone. To isolate the effect of entropy adaptation, we introduce a mean-matched fixed-negative baseline, which replaces OPSA's token-level advantages with their batch mean at each training step. We also evaluate fixed advantages of −0.5 and −1.0 to further ensure comparable update magnitudes across settings. All settings use identical lowest-20%-logp token selection and loss normalization. As shown in the table below, OPSA consistently outperforms these baselines, suggesting that its gains arise from the entropy-dependent assignment of advantages rather than their magnitude alone.
Methods AIME24 AIME24 AIME25 AIME25 HMMT25 HMMT25 avg@32 pass@32 avg@32 pass@32 avg@32 pass@32 Qwen3-1.7B 13.44 40.00 9.69 30.00 5.73 23.33 + 31.35 70.00 20.73 53.33 13.65 40.00 + 33.44 70.00 21.77 56.67 14.69 40.00 +mean-matched fixed-negative-advantage 28.85 73.33 24.58 50.00 18.23 43.33 +OPSA 48.85 80.00 35.31 66.67 23.33 50.00 -
OPSA does not introduce any special loss normalization. Since OPSA computes the loss only on the 20% of tokens with the lowest student log-probabilities, we naturally normalize the loss by the number of selected tokens rather than the total response length. This ensures that the loss reflects the average contribution of the tokens actually used for optimization.
-
The fixed-negative-advantage baseline uses exactly the same token selection strategy as OPSA. As explicitly stated in Line 238 of our paper, the fixed-negative-advantage experiments also compute the loss only on the 20% of tokens with the lowest student log-probabilities. Therefore, both settings use identical token selection strategies, ensuring that their performance differences are not caused by differences in token selection.
W3: The mathematical validation split used for checkpoint selection should also be identified, and the main comparisons should include results across independent training seeds.
-
We follow the standard checkpoint selection strategy in RL post-training and apply it consistently across all methods. Specifically, we select the best-performing checkpoint on AIME24, using the metric of avg@4, following prior OPD and RL studies. All baselines and our method use the same selection strategy to ensure fair comparisons.
-
We further report results using an entirely independent dataset, AMC23, for checkpoint selection. As shown in the table below, selecting checkpoints based on either AIME24 or AMC23 yields comparable results across different methods. This further demonstrates that our conclusions are robust to the choice of checkpoint selection benchmark. We note that the identical results under different checkpoint selection strategies for OPD and OPSA arise because both strategies select the same checkpoint for each method.
Methods AIME24 AIME24 AIME25 AIME25 HMMT25 HMMT25 avg@32 pass@32 avg@32 pass@32 avg@32 pass@32 Qwen3-1.7B 13.44 40.00 9.69 30.00 5.73 23.33 +OPD$_{aime24}$ 32.08 73.33 20.52 50.00 13.85 40.00 +OPD$_{amc23}$ 32.08 73.33 20.52 50.00 13.85 40.00 +GRPO$_{aime24}$ 33.96 70.00 25.31 50.00 15.10 43.33 +GRPO$_{amc23}$ 31.56 70.00 26.35 56.67 14.69 43.33 +OPSA$_{aime24}$ 48.85 80.00 35.31 66.67 23.33 50.00 +OPSA$_{amc23}$ 48.85 80.00 35.31 66.67 23.33 50.00 -
Comparison of different training seeds. Following prior RL and OPD studies, we use a fixed random seed of 42 for training and rollout generation across all methods to ensure fair comparisons. To assess robustness despite the computational cost, we additionally evaluate OPD and OPSA with three seeds (41, 42, and 43). As shown in the table below, despite some variation across seeds, OPSA consistently and substantially outperforms OPD, demonstrating the robustness of our method.
Methods AIME24 AIME24 AIME25 AIME25 HMMT25 HMMT25 avg@32 pass@32 avg@32 pass@32 avg@32 pass@32 Qwen3-1.7B 13.44 40.00 9.69 30.00 5.73 23.33 +OPD$_{41}$ 29.90 70.00 16.35 43.33 11.77 40.00 +OPD$_{42}$ 32.08 73.33 20.52 50.00 13.85 40.00 +OPD$_{43}$ 32.71 70.00 19.69 46.67 13.65 40.00 +OPSA$_{41}$ 44.90 76.67 27.81 60.00 21.15 50.00 +OPSA$_{42}$ 48.85 80.00 35.31 66.67 23.33 50.00 +OPSA$_{43}$ 49.48 80.00 35.94 63.33 22.08 53.33
Official Comment by Authors (3/3)
Q1: Could the observed gains partly reflect the tendency of reverse KL to concentrate probability on particular teacher modes, especially when teacher and student already share dominant modes? This does not exclude knowledge transfer, since the teacher may still determine which modes are reinforced or suppressed. Could the authors compare against a sharpened student snapshot with matched update magnitudes, and examine whether the conclusions also hold under forward KL?
Our paper does not analyze OPD from the perspective of why reverse KL is effective. We are unsure of the intended question, but speculate that the reviewer is suggesting using a sharpened student snapshot as the teacher to examine whether OPD's gains persist when the student and teacher already share dominant modes. In our paper, reverse KL is used only in Section 2.3 to investigate the student's insensitivity to teacher supervision. Nevertheless, we provide additional forward-KL experiments for completeness.
-
A sharpened student snapshot does not improve performance. We construct the teacher by applying temperature scaling (t=0.5) to a student snapshot's logits. Under this setting, approximately 95% of tokens receive non-negative advantages, including 60% with zero advantages. Despite sharing the same dominant modes as the student, this sharpened teacher fails to improve performance, suggesting that shared dominant modes alone are insufficient to explain OPD's gains. This is consistent with our finding that suppressing low-probability tokens through negative advantages is important for OPD's improvements.
AIME24 Avg@4 20 40 60 80 100 Standard OPD 25.00 34.17 32.50 34.17 31.67 Standard OPD (forward kl) 21.67 28.33 34.17 30.83 30.00 Sharpened student as teacher 13.33 11.67 7.50 9.17 6.67 -
Our findings in Section 2.3 remain valid under forward KL. We additionally conduct OPD experiments using forward KL and observe conclusions consistent with those under reverse KL. The corresponding results are reported in the experimental table in our response to W2 (in Official Comment by Authors (1/3)).
Official Comment by Authors (1/3)
Thank you for your great efforts on the review and constructive comments.
Under our definition of noisy supervision, OPD will reinforce incorrect answer tokens or suppress correct ones. For example, even when the student assigns 99% probability to a correct answer token, a teacher probability of 98% still yields a negative advantage, discouraging the student from generating that correct token. We therefore consider such supervision undesirable from an optimization perspective. Moreover, in practice, the training of OPD only replaces the advantage computation in GRPO, where advantage signs are aligned with answer correctness. This motivates our definition of noisy supervision, which is also consistent with recent studies, including OPDVR [1] and RA-OPD [2], that emphasize aligning advantage directions with answer correctness.
References
[1] Lin et al. (2026). On-policy Distillation with Verifiable Reward. arXiv:2608.24696.
[2] Gan et al. (2026). When Teacher Guidance Misleads: Reward-Aligned On-Policy Distillation. arXiv:2608.27960.