acceptodds
Under review as a conference paper at ICLR 2027

Expected Reasoning-Step Return Unifies On-Policy Learning from Rewards and Teachers

Abstract

On-policy reasoning models can learn from task rewards or teacher signals, but these sources differ in form and can favor conflicting updates, leaving unclear which should guide a given reasoning action. We introduce Expected Reasoning-Step Return (ERSR), which treats semantic reasoning steps as macro-actions and uses Monte Carlo student-policy rollouts to estimate the expected final task reward of student-generated and teacher-proposed actions in a common return space for step-level comparison. ERSR analysis reveals an outcome-dependent asymmetry: student actions are more beneficial than teacher replacements on successful trajectories, whereas teacher replacements become more beneficial on failed trajectories. We further show that student answer-probe gains track student-step ERSR utility and distinguish beneficial from harmful reasoning steps. Based on these findings, we propose Return-Referenced On-Policy Learning (ROPL), which reinforces student reasoning on successful trajectories and distills teacher signals on failed ones, while using group success rate for difficulty scaling and student-probe gains for step-level modulation. Experiments across reasoning benchmarks and teacher–student configurations show that ROPL consistently outperforms strong baselines. ERSR training dynamics further show that ROPL jointly exploits substantial utility from both reward- and teacher-side signals, whereas existing hybrids often leave substantial residual utility in one branch.

Primary area:
reinforcement learning
Keywords:
On-Policy Learning, Reasoning Models, Reinforcement Learning, On-Policy Distillation, Step-Level Credit Assignment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_3d37OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Expected Reasoning-Step Return (ERSR), a counterfactual framework that compares student reasoning steps and teacher replacements in a common student-policy expected-return space. ERSR reveals an outcome-dependent asymmetry: on successful trajectories, preserving student steps is more beneficial, whereas on failed trajectories, teacher replacements are substantially more useful. Student answer-probe gains track student-step utility but not teacher-replacement utility. Building on these findings, the authors propose Return-Referenced On-Policy Learning (R²OPL), which routes successful trajectories to reward-based RL and failed ones to teacher-guided on-policy distillation, with difficulty scaling and probe-based step-level modulation. Across eight mathematical, scientific, and logical reasoning benchmarks, R²OPL consistently outperforms reward-only, distillation-only, and hybrid baselines, and ERSR dynamics show that it exploits both reward- and teacher-side utility.

Critical strengths

I think there are three key strengths. First, the analysis is interesting and connects to the method to some extent, even though the mapping isn’t perfect; the insight that correct and incorrect trajectories should be handled differently is genuinely valuable. Second, the method looks quite practical to implement, and the answer probe is a sensible design—they also propose an engineering solution to address scalability and reduce the actual cost. Third, the empirical gains are quite clear.

Critical weaknesses

I don't have clear weakness, only some questions. See below.

Questions for authors

How exactly are the teacher replacement steps generated in the analysis? What are the detailed procedures, and how do they ensure that the teacher’s replacement naturally terminates within a single semantic step?

The fallback seems completely unreasonable: if no step-level information is available, the entire response is treated as a single segment, in which case the analysis no longer makes sense.

Can your method be extended to agent settings?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 3d37

Response to Q1. We thank the reviewer for asking for clarification. For each student reasoning step, we prompt the teacher with the prefix hk−1h_{k-1}, i.e., the student trajectory preceding the current step, and let the teacher continue generating from the same prefix.

We adopt an oversampling-and-segmentation strategy rather than requiring the teacher to naturally terminate at a semantic step boundary. Specifically, we set the maximum generation length to 2,048 tokens and extract the first complete semantic step from the generated response using our step segmentation procedure, discarding any subsequent content. In our experiments, approximately 99% of teacher generations contain an extractable complete step within this token budget. If a complete step cannot be extracted, we resample, allowing up to three retries.

This procedure provides a practical way to obtain complete teacher replacement steps without requiring explicit step-level stopping behavior from the teacher.


Response to Q2. We thank the reviewer for raising this concern. We acknowledge that, in a small number of cases where no reliable structural boundaries can be identified, the entire response is retained as a single step. We would like to clarify its implications for both ERSR evaluation and R²OPL training.

For ERSR evaluation, single-step trajectories are excluded because we do not evaluate the final reasoning step. Therefore, these trajectories do not contribute to intermediate-step utility estimates.

For R²OPL training, a single-step trajectory effectively reduces step-level credit assignment to trajectory-level reward assignment, resembling standard GRPO. In this case, Answer Probe cannot provide meaningful step-level modulation. However, as shown by the ablation in Figure 4 (Page 9), removing Answer Probe mainly affects convergence efficiency: the model reaches peak performance at 500 steps instead of 250, while the final performance decreases only slightly (59.4 vs. 58.8). This suggests that the loss of probe-based modulation primarily slows training rather than causing substantial performance degradation or training collapse.

Thus, although the conservative fallback may reduce the effectiveness of step-level modulation for a small number of trajectories, it does not compromise the overall ERSR analysis or the stability of R²OPL training.


Response to Q3. We thank the reviewer for this interesting question. We believe both ERSR and R²OPL can be extended to agent settings, from two complementary perspectives.

(1) Extending ERSR to agent-level utility analysis.

In multi-turn agent tasks, each interaction turn, including reasoning and tool calls, can be treated as a macro-action. ERSR can compare a student-generated action with a teacher-replacement action from the same interaction history, estimating their respective expected final rewards under subsequent student-policy interactions. When a replacement changes the environment state, subsequent interactions can be re-executed accordingly. This provides a natural extension of ERSR for analyzing the utility of different agent decisions.

(2) Extending R²OPL to agent training.

The outcome-conditioned learning paradigm can also be applied to agent trajectories: successful trajectories are optimized through RL, while failed trajectories receive teacher-guided OPD on student-generated decision tokens. Furthermore, Answer Probe can be adapted to estimate the agent's progress toward the final task objective after each interaction turn.

Notably, this probing idea has already been explored in agent RL. For example, IGPO [1], published at ICLR 2026, introduces turn-level answer-probability probing for multi-turn search agents. It evaluates the probability of producing the ground-truth answer after each interaction turn and uses the changes as information-gain rewards. Its vectorized implementation also employs appended answer probes and customized attention masks, demonstrating the practicality of efficient answer probing in agent settings.

Overall, both the ERSR analysis and the R²OPL optimization paradigm can be naturally extended from reasoning steps to agent interaction turns, with appropriate adaptations to environment transitions and tool interactions.

Reference:

[1] Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agents.

thank you for your rebuttal. I will keep my score.

AuthorsOpenReview ↗

Thank You for Your Consideration (Reviewer 3d37)

Thank you for your follow-up and for taking the time to consider our responses. We sincerely appreciate your time and consideration.

Reviewer_cXTuOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ERSR, which measures the value of a reasoning step by its effect on the student’s expected success rate, and builds R²OPL on that idea: reinforce successful student trajectories and distill from the teacher on failed ones. The results are promising, but there is a gap between the utility ERSR measures and the OPD signal the method actually uses. Overall, the approach has potential, but the current evidence does not yet establish that its claims generalize.

Critical strengths
  • ERSR puts student steps and teacher-replacement steps in a shared expected-return framework, giving the paper a clear motivation.
  • The paper reports gains across several model configurations and math, science, and logic benchmarks, along with ablation studies.
  • Routing learning signals by trajectory outcome is intuitive. The answer probe’s reported training overhead is modest, at 2%–8%.
Critical weaknesses
  • Limited teacher coverage: The Qwen experiments mainly use a 4B teacher with a 1.7B student, or a 4B teacher with a 4B student. Although the paper also includes a Gemma 26B-A4B configuration, it does not systematically test how teacher capability or the teacher–student gap affects results.
  • High cost: The offline ERSR analysis uses about 11.52 million student rollouts. The paper uses cheaper estimates during training, but this does not establish that individual-step estimates are reliable.
  • Unstable step definitions: Segmentation relies mainly on explicit headings, which Gemma follows only 75.4% of the time. Changing the boundaries can change the measured utility of a step.
  • Mismatch between measurement and training signal: ERSR evaluates the benefit of inserting a teacher-generated step, while R²OPL actually applies OPD signals to student-generated tokens. The paper reports that these token-level teacher signals are nearly uncorrelated with teacher-replacement utility.
  • Limited step-level credit assignment: About one-third of student steps on successful trajectories have negative ERSR, yet the method still reinforces successful trajectories overall. The probe’s correlation with step utility is also modest.
Questions for authors
  • Scan teacher capability and scale: Include stronger, weaker, and similarly capable teachers, and report how performance and ERSR vary with the teacher–student capability gap.
  • Validate the link between ERSR and OPD: Compare teacher-step replacement utility with the change in success rate after actual OPD updates. Include a baseline that filters or gates teacher signals using ERSR.
  • Test robustness to step boundaries: Recompute ERSR using different segmentation granularities, segmenters, or a manually annotated subset; check whether the conclusions and step rankings hold.
  • Quantify estimation error and cost: Compare MC@2, MC@8, and MC@128 against an independent high-budget estimate, report confidence intervals, and measure final training performance under different rollout budgets.
  • Replicate across random seeds: Report means and variance or confidence intervals for the main results to establish that the gains are not due to a single run or evaluation sample.

I would be inclined to raise my score if the authors provide the requested experiments and analyses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer cXTu(Q1, Q2, Q3)

Response to Q1 and Q2. We sincerely thank the reviewer for these valuable suggestions. Following the reviewer's recommendations, we conducted additional experiments to investigate how teacher capability and reasoning mode affect ERSR utility, and to examine the relationship between ERSR and actual OPD transfer performance. The results are reported in Tables 11 and 12 (Page 24) of the revised manuscript.

(1) Teacher capability and reasoning mode.

We fix the student as Qwen3-1.7B (non-thinking) and evaluate three 4B teacher configurations with different capabilities and reasoning modes: Qwen3-4B (non-thinking), Qwen3-4B-Instruct-2507, and Qwen3-4B (thinking). As shown in Table 11, their teacher-replacement utilities consistently increase with teacher capability. In particular, on failed trajectories (R = 0), the ERSR utilities increase from +3.37 to +7.76 and +15.03 (× 10⁻²). A similar ranking is observed on successful trajectories (R = 1), with utilities of +1.62, +2.08, and +5.81, respectively. These results suggest that ERSR captures differences in the potential learning value of teachers across different capability levels and reasoning modes.

(2) Relationship between ERSR and actual OPD transfer.

We further conducted standard OPD training with the same student and each teacher configuration. As shown in Table 12, the initial student achieves 43.8 Avg@16, while the students after OPD reach 45.7, 54.0, and 58.1, corresponding to gains of +1.9, +10.2, and +14.3 points, respectively. Notably, this ranking is fully consistent with the teacher-replacement ERSR ranking under both successful and failed trajectory conditions.

These results suggest that ERSR is not intended to precisely predict the magnitude of OPD gains, but can provide a useful coarse-grained assessment for ranking the relative distillation utility of different teachers, even across different reasoning modes.

(3) Clarifying the scope of ERSR and its use for teacher-signal filtering.

We would also like to clarify that ERSR is designed as a global, coarse-grained credit-analysis framework, rather than a precise estimator of individual-step distillation gains. Its purpose is to diagnose the relative learning utility of student and teacher signals under a given policy. Reliable estimation of an individual step's utility requires extensive independent Monte Carlo rollouts, which, in our setting, can be hundreds of times more expensive than directly performing a teacher update. Given this substantial computational cost, applying ERSR to filter teacher signals online would introduce considerable overhead. We therefore focus on using ERSR for coarse-grained utility analysis and teacher comparison, where it can provide useful guidance at a more manageable computational cost.

We hope these additional results clarify the relationship between ERSR and practical OPD transfer, and further demonstrate its value as a diagnostic tool for comparing teacher utility.


Response to Q3. We thank the reviewer for this suggestion. We evaluated ERSR under three segmentation granularities: heading-level, paragraph-level, and sentence-level, as summarized in Tables 14 and 15 (Appendix A.4.2, Page 26).

The key finding remains consistent across all three granularities, particularly on failed trajectories (R = 0): student-step utilities remain negative (−2.52, −1.28, and −0.27), while teacher-replacement utilities remain positive and higher (+7.76, +3.74, and +0.28), respectively (all values in × 10⁻²).

Although utility magnitudes decrease with finer segmentation, these results support the robustness of our main ERSR finding against different step definitions.

AuthorsOpenReview ↗

Response to Reviewer cXTu(Q4, Q5)

Response to Q4. We thank the reviewer for this suggestion. We evaluated low-cost ERSR estimation against an independent MC@128 reference, using non-overlapping rollout samples to avoid sample reuse. The results are reported in Table 16 (Appendix A.4.3, Page 27).

The results show that aggregate ERSR can be reasonably approximated with substantially reduced budgets. In particular, using 500 sampled steps with MC@2 yields typical deviations of 0.0133 and 0.0177 for student-step and teacher-replacement utilities, respectively, with 95% error bounds of 0.0326 and 0.0435. Increasing the budget to MC@8 further reduces these deviations. Notably, at the same nominal sampling budget, 500 steps with MC@2 provides more accurate estimates than 250 steps with MC@4, suggesting that broader step coverage is particularly valuable for aggregate utility estimation. We therefore adopt 500 steps with MC@2 for training-time ERSR tracking.

We would also like to clarify that ERSR is used solely for analysis and training-time monitoring, and does not participate in policy optimization or parameter updates. Therefore, changing its estimation budget only affects the accuracy and computational cost of utility estimation, not the final training performance. Accordingly, we focus on quantifying the estimation accuracy–cost trade-off rather than conducting separate model training under different ERSR estimation budgets.

Response to Q5. We sincerely thank the reviewer for emphasizing the importance of statistical reliability. Due to limited computational resources and the tight rebuttal timeline, we have not yet completed the multi-seed experiments. We plan to provide an updated manuscript around November 16–17, incorporating the additional multi-seed results, and will update our response accordingly. We greatly appreciate the reviewer's patience and understanding.

Thanks for Tables 11 and 12 — the agreement between the ERSR and post-OPD orderings is encouraging, and the response has moved me toward a higher score. My remaining concern is that everything reported so far sits within one model generation at a single student size, so one complete Qwen3.5 configuration — a larger student run end to end through the same ERSR + R²OPL protocol — would settle for me whether the conclusion is robust. However, I will increase my score and waiting for your new results.

AuthorsOpenReview ↗

Additional Experimental Results (Reviewer cXTu)

Thank you for your constructive feedback and for highlighting the importance of cross-generation validation. Following your suggestion, we have conducted additional experiments with the Qwen3.5 family, addressing two complementary questions.

(1) Does ERSR utility correlate with actual performance changes after OPD? (Appendix A.11.1, Page 37, Tables 23 and 24)

To directly investigate this relationship, we fix Qwen3.5-4B as the student and select two teachers with substantially different capabilities: the weaker Qwen3.5-2B and the stronger Qwen3.5-9B. For each teacher, we evaluate ERSR utilities and conduct standard OPD training to examine whether the estimated teacher-replacement utility is consistent with the actual performance changes after distillation.

The results show a clear agreement between the two evaluations. As reported in Table 23, the stronger 9B teacher consistently exhibits higher teacher-replacement utility than the weaker 2B teacher under both successful and failed trajectory conditions. More importantly, Table 24 shows that OPD with the weaker 2B teacher degrades the student's eight-benchmark average from 74.4 to 63.7 (−10.7 points), whereas the stronger 9B teacher improves it to 75.7 (+1.2 points). Thus, the direction of teacher-replacement utility on failed trajectories aligns with the actual performance changes after OPD. These findings provide further evidence that ERSR captures meaningful differences in teacher utility and reflects their downstream distillation effectiveness.

(2) End-to-end evaluation of R²OPL with Qwen3.5 (Appendix A.11.2, Page 38, Table 25 and Figure 16)

We further conduct a complete end-to-end evaluation using Qwen3.5-4B as the student and Qwen3.5-9B as the teacher. We compare R²OPL against GRPO, OPD, and SRPO across eight reasoning benchmarks. As shown in Table 25, R²OPL achieves the highest overall average performance of 79.6, outperforming GRPO (79.2), OPD (75.7), and SRPO (76.2). Figure 16 further tracks the training dynamics and ERSR utilities throughout optimization, showing that R²OPL effectively exploits both student-step and teacher-replacement utilities, in contrast to the predominantly single-source utility consumption observed in the individual RL and OPD baselines.

These results provide further evidence for both the effectiveness of R²OPL and the underlying ERSR analysis under a newer model generation.

(3) Additional multi-seed experiments (Page 8, Table 3).

In addition, we have completed three-seed experiments for both SRPO and R²OPL and incorporated the mean and standard deviation into Table 3. The results continue to demonstrate consistent performance improvements of R²OPL, further establishing the robustness of our findings across different random seeds.

We sincerely thank the reviewer for this valuable suggestion, which has helped us further strengthen the empirical validation and generalizability of our work.

Reviewer_8e1eOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Expected Reasoning-Step Return (ERSR), which treats semantic reasoning steps as macro-actions and compares student-generated steps with teacher replacements by estimating their expected final rewards under the same student continuation policy. Using this analysis, the authors report that student steps are generally more useful on successful trajectories, whereas teacher replacements are more useful on failed trajectories, and that student answer-probe gains provide a lightweight proxy for student-step utility. Based on these observations, they introduce Return-Referenced On-Policy Learning (R²OPL), which routes successful trajectories to reward-based reinforcement and failed trajectories to on-policy distillation, with additional difficulty scaling and step-level probe modulation. Experiments on eight mathematical, scientific, and logical reasoning benchmarks across three teacher–student configurations show that R²OPL outperforms reward-only, distillation-only, and hybrid baselines, supported by ablations and ERSR training-dynamics analyses.

Critical strengths
  1. ERSR provides a useful diagnostic for comparing student and teacher reasoning. By evaluating the original student step and a teacher replacement from the same prefix, while sampling all subsequent continuations from the student policy, ERSR directly measures whether the student can benefit from a proposed teacher action.

  2. The reported improvements are broad and practically relevant. R²OPL outperforms reward-only, distillation-only, and hybrid baselines across multiple mathematical, scientific, and logical reasoning benchmarks, teacher–student configurations, and model scales.

Critical weaknesses

The motivation and empirical contribution of answer-probe modulation remain insufficiently established. Although the paper reports a positive correlation between student-probe gains and student-step ERSR utility, the correlations are relatively modest (0.246 on successful trajectories and 0.211 on failed trajectories), leaving the reliability of this proxy unclear. Moreover, removing answer-probe modulation reduces the average score only from 59.4 to 58.8. Without repeated runs or uncertainty estimates, it is unclear whether this small difference reflects a robust improvement or ordinary experimental variation. Thus, the evidence does not convincingly establish that answer-probe modulation makes a meaningful contribution to R²OPL.

Questions for authors

How robust is the improvement contributed by answer-probe modulation? Given the relatively modest correlations between student-probe gains and ERSR utility and the small performance difference in the ablation study (59.4 vs. 58.8), can the authors provide additional evidence, such as results across multiple random seeds or uncertainty estimates, to establish that this component yields a consistent and meaningful improvement?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 8e1e

We sincerely thank the reviewer for this insightful comment. We agree that the correlations between student-probe gains and ERSR utility are relatively modest (0.246 and 0.211), and that the final performance improvement is limited (0.6 points). Following this suggestion, we have revised the manuscript to better clarify the motivation and contribution of Answer Probe, emphasizing its role in improving training efficiency.

First, we would like to highlight the inherent computational challenges of hybrid RL–OPD training. RL requires complete rollouts to obtain final-answer rewards, but can allocate all available GPUs to the student model. In contrast, OPD requires deploying an additional teacher, but recent work [1] demonstrates that distillation from short reasoning prefixes can achieve performance comparable to full-trajectory distillation, without requiring complete rollouts. Hybrid RL–OPD training inherits both computational costs: it requires generating complete rollouts for reward computation while simultaneously allocating substantial GPU resources to teacher deployment. This makes hybrid training particularly expensive.

Answer Probe is designed to improve the convergence efficiency under this computational constraint. As shown in Figure 2, despite the modest Pearson correlations, probe-based ranking reveals meaningful differences in step utility:

  • Successful trajectories: Student-step utility increases from approximately 2.1×10−22.1\times10^{-2} to 5.8×10−25.8\times10^{-2} from the lowest to the highest probe quintile, indicating that the probe can identify high-utility reasoning steps.
  • Failed trajectories: The lowest probe quintile corresponds to the most negative student-step utility (−4.6×10−2-4.6\times10^{-2}), suggesting that the probe can identify potentially harmful reasoning steps.

Furthermore, as shown in Figure 4 (Page 9), R²OPL with Answer Probe reaches its peak performance at 250 steps, compared with 500 steps without it. We evaluate this ablation every 50 steps, rather than every 100 steps in the main experiments, to capture convergence dynamics more precisely. Meanwhile, our customized attention mask limits the additional computational overhead to only 2%–8%.

We acknowledge that Answer Probe brings only a modest improvement in final performance. Its primary practical benefit is accelerating convergence: it reaches peak performance with approximately 50% fewer training steps while introducing only 2%–8% additional computational overhead. We have revised the manuscript to clarify this distinction.

Regarding statistical reliability, we appreciate the reviewer's suggestion to evaluate Answer Probe across multiple random seeds. Due to limited computational resources and the tight rebuttal schedule, these experiments could not be completed for the current revision. We plan to incorporate the additional results, including the mean and variance across seeds, in an updated manuscript around November 16–17, and will provide a further response accordingly. We sincerely appreciate the reviewer's patience.

[1] Fast and Effective On-Policy Distillation from Reasoning Prefixes.

AuthorsOpenReview ↗

Additional Experimental Results (Reviewer 8e1e)

Thank you for your helpful feedback. We would like to share two additional sets of experiments incorporated into the revised manuscript.

(1) Multi-seed experiments (Page 8, Table 3). We have added three-seed experiments for both SRPO and R²OPL, reporting the mean and standard deviation across independent runs. The results consistently demonstrate that R²OPL outperforms SRPO, further confirming the robustness of its performance improvements across different random seeds.

(2) Experiments with the Qwen3.5 family (Page 37, Appendix A.11). We have additionally conducted experiments using the Qwen3.5 family, with Qwen3.5-4B as the student and Qwen3.5-9B as the teacher. These results provide further support for both our ERSR analysis, confirming the outcome-dependent asymmetry between student-step and teacher-replacement utilities, and the effectiveness of R²OPL in improving reasoning performance.

We hope these additional results further strengthen the empirical evidence and generality of our findings. Thank you again for your time and consideration.

Thanks for the authors' response. I would like to keep my positive scores.

AuthorsOpenReview ↗

Appreciation for the Reviewer’s Positive Assessment

We sincerely thank the reviewer for maintaining their positive assessment. We greatly appreciate your time, consideration, and constructive feedback.

Reviewer_2RDROpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

ERSR uses student-policy MC rollouts to compare, in a common expected-return space, keeping a student reasoning step with replacing it by a teacher step. The analysis reports an outcome-dependent asymmetry and finds that student answer probes track step utility. Based on this, R²OPL applies RL to successful trajectories and OPD to failed ones, with difficulty scaling and probe modulation. It outperforms nine baselines across three teacher–student settings and eight benchmarks.

Critical strengths

1.ERSR is a reusable diagnostic: it compares student and teacher actions from the same prefix in final-reward space, at large scale, with a cheap training-time estimator. 2.Strong, broad gains that hold without AMC23, the benchmark used for tuning (56.8 vs. 53.8 and 59.6 vs. 57.2 on the other seven benchmarks). 3.An efficient packed-probe implementation (2–8% overhead) and informative training-dynamics analysis.

Critical weaknesses

1.The increment over SRPO is limited. SRPO already routes successes to RL and failures to distillation, and probe modulation adds only 0.6 points. 2.The tuning and baseline protocol need clarification. µ and λ were tuned on AMC23. Checkpoint selection is not described. Distillation baselines train for 100 vs. 500 steps. SRPO and RLSD were modified to use an external teacher and given no comparable branch-scale tuning. 3.There is a single seed, and the mechanism–method link is indirect. The utility of a full-step teacher replacement is not the same as the utility of a token-level distillation update, and the analysis uses thinking-disabled, structured prompts.

Questions for authors

1.Can you redo the asymmetry analysis stratified by pre-step quantities, e.g., an independently estimated V(h_{k−1}) or an independent rollout started from h_{k−1}, with confidence intervals? 2.Which benchmarks and checkpoints were used for selection? Can you report results under a frozen configuration chosen on an independent validation set? 3.How much of the gain remains against branch-scale-matched SRPO and extended distillation baselines, across multiple seeds?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 2RDR (Q1–Q2)

We sincerely thank the reviewer for the constructive feedback. We have conducted additional analyses and experiments to address the concerns, as detailed below.


Response to Q1. We thank the reviewer for this suggestion. We independently estimated pre-step values using MC@128, stratified reasoning steps into ten intervals, and recomputed both ERSR utilities within each interval, reporting asymmetric 95% confidence intervals. This analysis is detailed in Appendix A.4.2, fourth paragraph ("ERSR across pre-step value"), with results presented in Table 13 (Page 26).

As shown in Table 13, the original asymmetry persists across pre-step-value strata:

  • Successful trajectories (R=1R=1): Student-step utility is higher in 7/10 intervals, with weighted means of +3.20×10−2+3.20\times 10^{-2} vs. +0.70×10−2+0.70\times 10^{-2}.
  • Failed trajectories (R=0R=0): Teacher-replacement utility is higher in all 10 intervals, with weighted means of −2.10×10−2-2.10\times 10^{-2} vs. +6.40×10−2+6.40\times 10^{-2}.

These results confirm that the observed outcome-dependent asymmetry persists across different pre-step values.


Response to Q2. We clarify the hyperparameter and checkpoint selection protocols as follows.

Hyperparameter and checkpoint selection. Both μ\mu and λ\lambda were selected based on AMC23. All methods used the same benchmark for checkpoint selection, following these evaluation schedules:

  • OPD-based methods (OPD, ExOPD, and EOPD): 100 training steps, evaluated every 20 steps.
  • RL and hybrid methods: 500 training steps, evaluated every 100 steps.

For each method, we selected the checkpoint achieving the best AMC23 performance. AMC23 was the only benchmark used for hyperparameter and checkpoint selection. The remaining seven benchmarks were never involved in selection and were evaluated only once using the selected checkpoints.

We currently retain AMC23 in the reported results for completeness. However, if the reviewer considers it necessary, we are willing to exclude AMC23 and report results exclusively on the remaining seven benchmarks.

Regarding the OPD training budget, following ExOPD [1] and Rethinking OPD [2], OPD-based methods typically converge rapidly, often reaching their best performance within the first 100 steps. To further verify this, we extended OPD, ExOPD, and EOPD training to 500 steps, evaluating AMC23 every 20 steps. As detailed in Appendix A.8.1, Figure 8 (Page 31) shows that all three methods reach their peak performance within the first 100 steps, supporting our original training budget of 100 steps.

We have clarified these protocols and added the extended training results in the revised manuscript.

References:

[1] Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation (ExOPD, 2026).

[2] Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe (Rethinking OPD, 2026).

AuthorsOpenReview ↗

Response to Reviewer 2RDR (Q3)

Response to Q3. We sincerely thank the reviewer for this helpful suggestion. Following this recommendation, we conducted branch-scale-matched experiments for SRPO. As shown in Table 5 (Page 10), appropriate branch scaling improves SRPO to 59.2 Avg@16, achieving performance comparable to R²OPL (59.4 Avg@16).

Importantly, our main contribution is to establish the necessity of asymmetric branch scaling through ERSR-based utility analysis and gradient-dynamics analysis, rather than simply introducing another weighting scheme. We provide three complementary perspectives to support this design.

(1) An interesting connection to KL-regularized RL.

A perhaps unexpected perspective is that reverse-KL OPD can be viewed as a form of KL regularization in RL, with the teacher playing the role of the reference policy.

On Page 6 of SRPO [1], immediately before its Experiments section, the authors state:

The denominator normalizes by the total number of routed tokens, so each branch contributes in proportion to the tokens it covers. This avoids introducing an additional mixing hyperparameter and naturally adapts to the evolving sample composition.

However, adapting branch weights according to token proportions does not necessarily ensure balanced optimization. To see why, consider the following connection.

GRPO [2] (Page 13) with a reference-model constraint can be formulated as:

J_GRPO = J_RL − β KL(πθ || πref)

Meanwhile, a weighted combination of RL and reverse-KL OPD takes the form:

J_Hybrid = μ J_RL + λ J_OPD = μ J_RL − λ KL(πθ || πT)

When the teacher serves as the reference model, the two objectives share the same optimization form, with an effective KL coefficient:

β_eff = λ / μ

In SRPO, token-based normalization implicitly sets:

μ = N_success_tokens / N_total_tokens

λ = N_failure_tokens / N_total_tokens

In our experiments, the average training reward reaches approximately 0.65 in later stages. Assuming comparable average trajectory lengths, this implies μ ≈ 0.65 and λ ≈ 0.35, yielding:

β_eff ≈ 0.35 / 0.65 = 0.54

For comparison, KL coefficients in commonly used GRPO configurations are often around 10⁻³–10⁻² (e.g., the default in verl is 10⁻³); the original DeepSeekMath implementation uses 0.04. Thus, SRPO's implicit teacher-anchoring coefficient is remarkably large by comparison.

In other words, the teacher is not merely teaching the student; it may also be holding the steering wheel. Such a strong teacher constraint can substantially restrict policy updates and suppress reward-driven exploration.

This reveals a fundamental limitation of SRPO's token-based normalization: adapting branch weights to sample composition does not guarantee balanced optimization between reward and teacher signals.

(2) Direct evidence of distillation-dominated optimization in SRPO.

We further examined SRPO under three configurations: self-teacher with μ = λ = 1, external teacher with μ = λ = 1, and external teacher with asymmetric scaling (μ = 2.5, λ = 0.1).

As detailed in Appendix A.10.3 (Page 36), Figure 13 shows that both equal-weight configurations exhibit substantial gradient imbalance, with negative-to-positive gradient norm ratios remaining around 7–10 for the self-teacher and approximately 10 for the external teacher. Asymmetric scaling reduces this ratio to approximately 0.6–0.8.

The training dynamics further reveal two distinct manifestations of distillation-dominated optimization. As shown in Figures 14 and 15 (Appendix A.10.3, Page 36), SRPO with a self-teacher suffers severe late-stage reward and performance collapse, resembling the degradation observed in OPSD. With an external teacher, SRPO instead exhibits rapid early improvement followed by saturation, consistent with the fast-convergence behavior of standard OPD. In contrast, asymmetric scaling enables sustained improvement, reaching approximately 0.70 mean reward and 77.5 AMC23 Avg@16.

These results demonstrate that standard SRPO is dominated by distillation signals, whereas appropriate branch scaling restores effective reward-driven optimization.

(3) ERSR evidence of underutilized reward-side utility.

As shown in Figure 5 (Page 10), SRPO's teacher-replacement utility decreases rapidly during training, while student-step utility remains nearly unchanged from its initial level. This indicates that SRPO primarily exploits teacher-side learning opportunities while leaving substantial student-side utility unrealized, further suggesting insufficient exploitation of reward-driven learning signals.

References:

[1] Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing.

[2] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.

AuthorsOpenReview ↗

Response to Reviewer 2RDR (Additional Clarifications)

Additional clarification on Answer Probe.

We would also like to clarify the motivation and role of Answer Probe from a computational-efficiency perspective. Hybrid RL–OPD training inherits the computational bottlenecks of both paradigms: RL requires complete rollouts to obtain final-answer rewards, while OPD requires deploying an additional teacher, typically allocating half of the available GPUs to it. In contrast, RL can devote all GPUs to the student, whereas recent work has demonstrated that prefix-only OPD can achieve performance comparable to full-trajectory OPD while substantially reducing computational costs [1]. Consequently, hybrid training simultaneously incurs the cost of long rollouts and teacher deployment, making it particularly expensive.

Answer Probe is designed to improve the convergence efficiency of such expensive hybrid training. As shown in Figure 4 (Page 9), R²OPL with Answer Probe reaches its peak performance at 250 steps, whereas the variant without it requires 500 steps. We evaluated this ablation every 50 steps, rather than every 100 steps in the main experiments, to capture the convergence dynamics more precisely. Meanwhile, our customized attention mask limits the additional computational overhead of Answer Probe to only 2%–8%.

We acknowledge that Answer Probe provides only a modest improvement in final performance (0.6 points). Its primary benefit, however, lies in accelerating convergence: it reaches peak performance with approximately 50% fewer training steps while introducing only 2%–8% additional computational overhead. We have clarified this distinction in the revised manuscript.

[1] Fast and Effective On-Policy Distillation from Reasoning Prefixes*.


Regarding multi-seed experiments. We sincerely appreciate the reviewer's concern about statistical reliability. Due to limited experimental resources and computational availability, we were unable to complete the multi-seed experiments within this initial response period. We plan to provide a second revised version around November 16–17, incorporating the multi-seed results into the manuscript, and will update our response accordingly once the revision is available. We sincerely thank the reviewer for their patience and understanding.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Updated Manuscript and Additional Experiments

We sincerely thank all reviewers for their thoughtful and constructive feedback. In response to the comments, we have conducted additional experiments and analyses and revised the manuscript accordingly. The updated manuscript has been uploaded to OpenReview. We appreciate the reviewers' time and valuable suggestions, and hope the revisions address their concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.