- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Expected Reasoning-Step Return (ERSR), a counterfactual framework that compares student reasoning steps and teacher replacements in a common student-policy expected-return space. ERSR reveals an outcome-dependent asymmetry: on successful trajectories, preserving student steps is more beneficial, whereas on failed trajectories, teacher replacements are substantially more useful. Student answer-probe gains track student-step utility but not teacher-replacement utility. Building on these findings, the authors propose Return-Referenced On-Policy Learning (R²OPL), which routes successful trajectories to reward-based RL and failed ones to teacher-guided on-policy distillation, with difficulty scaling and probe-based step-level modulation. Across eight mathematical, scientific, and logical reasoning benchmarks, R²OPL consistently outperforms reward-only, distillation-only, and hybrid baselines, and ERSR dynamics show that it exploits both reward- and teacher-side utility.
Critical strengths
I think there are three key strengths. First, the analysis is interesting and connects to the method to some extent, even though the mapping isn’t perfect; the insight that correct and incorrect trajectories should be handled differently is genuinely valuable. Second, the method looks quite practical to implement, and the answer probe is a sensible design—they also propose an engineering solution to address scalability and reduce the actual cost. Third, the empirical gains are quite clear.
Critical weaknesses
I don't have clear weakness, only some questions. See below.
Questions for authors
How exactly are the teacher replacement steps generated in the analysis? What are the detailed procedures, and how do they ensure that the teacher’s replacement naturally terminates within a single semantic step?
The fallback seems completely unreasonable: if no step-level information is available, the entire response is treated as a single segment, in which case the analysis no longer makes sense.
Can your method be extended to agent settings?
3 comments
thank you for your rebuttal. I will keep my score.
Thank You for Your Consideration (Reviewer 3d37)
Thank you for your follow-up and for taking the time to consider our responses. We sincerely appreciate your time and consideration.
Response to Reviewer 3d37
Response to Q1. We thank the reviewer for asking for clarification. For each student reasoning step, we prompt the teacher with the prefix hk−1, i.e., the student trajectory preceding the current step, and let the teacher continue generating from the same prefix.
We adopt an oversampling-and-segmentation strategy rather than requiring the teacher to naturally terminate at a semantic step boundary. Specifically, we set the maximum generation length to 2,048 tokens and extract the first complete semantic step from the generated response using our step segmentation procedure, discarding any subsequent content. In our experiments, approximately 99% of teacher generations contain an extractable complete step within this token budget. If a complete step cannot be extracted, we resample, allowing up to three retries.
This procedure provides a practical way to obtain complete teacher replacement steps without requiring explicit step-level stopping behavior from the teacher.
Response to Q2. We thank the reviewer for raising this concern. We acknowledge that, in a small number of cases where no reliable structural boundaries can be identified, the entire response is retained as a single step. We would like to clarify its implications for both ERSR evaluation and R²OPL training.
For ERSR evaluation, single-step trajectories are excluded because we do not evaluate the final reasoning step. Therefore, these trajectories do not contribute to intermediate-step utility estimates.
For R²OPL training, a single-step trajectory effectively reduces step-level credit assignment to trajectory-level reward assignment, resembling standard GRPO. In this case, Answer Probe cannot provide meaningful step-level modulation. However, as shown by the ablation in Figure 4 (Page 9), removing Answer Probe mainly affects convergence efficiency: the model reaches peak performance at 500 steps instead of 250, while the final performance decreases only slightly (59.4 vs. 58.8). This suggests that the loss of probe-based modulation primarily slows training rather than causing substantial performance degradation or training collapse.
Thus, although the conservative fallback may reduce the effectiveness of step-level modulation for a small number of trajectories, it does not compromise the overall ERSR analysis or the stability of R²OPL training.
Response to Q3. We thank the reviewer for this interesting question. We believe both ERSR and R²OPL can be extended to agent settings, from two complementary perspectives.
(1) Extending ERSR to agent-level utility analysis.
In multi-turn agent tasks, each interaction turn, including reasoning and tool calls, can be treated as a macro-action. ERSR can compare a student-generated action with a teacher-replacement action from the same interaction history, estimating their respective expected final rewards under subsequent student-policy interactions. When a replacement changes the environment state, subsequent interactions can be re-executed accordingly. This provides a natural extension of ERSR for analyzing the utility of different agent decisions.
(2) Extending R²OPL to agent training.
The outcome-conditioned learning paradigm can also be applied to agent trajectories: successful trajectories are optimized through RL, while failed trajectories receive teacher-guided OPD on student-generated decision tokens. Furthermore, Answer Probe can be adapted to estimate the agent's progress toward the final task objective after each interaction turn.
Notably, this probing idea has already been explored in agent RL. For example, IGPO [1], published at ICLR 2026, introduces turn-level answer-probability probing for multi-turn search agents. It evaluates the probability of producing the ground-truth answer after each interaction turn and uses the changes as information-gain rewards. Its vectorized implementation also employs appended answer probes and customized attention masks, demonstrating the practicality of efficient answer probing in agent settings.
Overall, both the ERSR analysis and the R²OPL optimization paradigm can be naturally extended from reasoning steps to agent interaction turns, with appropriate adaptations to environment transitions and tool interactions.
Reference:
[1] Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agents.