acceptodds
Under review as a conference paper at ICLR 2027

RLSR: Reinforcement Learning with Self-Reflection for Large Language Models Self-Improvement

Abstract

Reinforcement learning (RL) can distill capacities that emerge with test-time self-reflection and incentivize large language models (LLMs) to self-improve. However, existing approaches fail to co-evolve self-reflection with direct policy optimization and instead condition the privileged self-teacher on unverified hindsight, causing self-improvement to plateau or even reverse. In this paper, we propose RLSR, a framework that optimizes and distills self-reflecting LLMs to achieve persistent self-improvement with RL. To generate endogenous learning signals, RLSR samples rollouts via grouped multi-turn self-reflection and retains only instances that achieve group-level aggregate success, grounding the privileged self-teacher in verified rather than assumed hindsight. RLSR also optimizes the policy via a dual-track co-evolutionary mechanism: a direct track internalizes breakthroughs incurred by self-reflection into the direct policy via sign-adaptive counterfactual advantage reshaping, and a reflect track assigns turn-level advantages via a log-odds belief calibration that tracks pivotal corrective transitions, actively reinforcing error correction to safeguard the self-reflection engine against degradation. These two tracks are optimized under a unified policy-gradient objective, establishing a sample-efficient self-reinforcing co-evolutionary cycle that remains stable throughout training. Across eight mathematics and code synthesis benchmarks, RLSR outperforms or matches RLVR and state-of-the-art self-distillation baselines in single-turn pass rate with evolved test-time reflection capacities.

Primary area:
reinforcement learning
Keywords:
Self-Improvement, Reinforcement Learning, Test-time Scaled Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WyNDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes RLSR, a reinforcement learning framework that jointly optimizes direct reasoning and multi-turn self-reflection. It uses grouped verifier-guided reflection and zone-of-proximal-development filtering to construct hindsight-conditioned self-teachers. A direct track reshapes step-level advantages using counterfactual likelihood gaps, while a reflection track assigns turn-level rewards through log-odds belief calibration. Experiments on Qwen3-1.7B and Qwen3-4B across mathematics and code generation benchmarks show relatively consistent improvements over several RLVR and self-distillation baselines, with test-time reflection capabilities largely preserved.

Critical strengths
  1. The paper addresses an important problem: transferring test-time self-correction capabilities into direct policy improvements while preserving reflection ability.

  2. The framework integrates verifier-guided reflection, hindsight-based credit assignment, and joint optimization in a coherent manner. The evaluation includes diverse baselines, two model scales, multiple benchmarks, and component ablations.

Critical weaknesses
  1. A fundamental limitation of the direct track: For marginal-ZPD groups where all initial rollouts fail, the GRPO sequence advantage is zero, making the reshaped direct-track advantage identically zero. This contradicts the claimed mechanism for directly internalizing reflection-induced breakthroughs.

  2. Under-specified reflection credit assignment: The empirical win rate used for log-odds belief updates is insufficiently defined. Single binary outcomes cannot provide meaningful intermediate progress signals, while additional success-rate estimation may require unaccounted rollouts.

  3. Insufficient grounding and mechanistic ablations: Verifying final outcomes does not validate the causal correctness of generated hindsight, and the ZPD criterion does not always require a successful correction. The ablations also fail to isolate the proposed step-level reshaping and belief-gain mechanisms from ordinary RL objectives.

  4. Limited novelty and incomplete comparisons: The direct-track design closely resembles the already-evaluated RLSD [2], while turn-level belief calibration and evolving hindsight supervision overlap with AgentOPSD [3] and SEED [4]. The paper also omits discussion of SCoRe [1] and lacks a matched comparison with the cited FlowBalance [5].

  5. Limited empirical and efficiency evidence: Most improvements over the strongest trained baselines are small, yet the claimed confidence intervals, significance tests, and seed variability are not presented. Matching the maximum rollout count also does not establish comparable total computation given the additional hindsight generation and teacher scoring.

[1] Training Language Models to Self-Correct via Reinforcement Learning

[2] Self-Distilled RLVR

[3] AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning

[4] SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning

[5] FlowBalance: Verifier-Grounded Self-Improvement from On-Policy Reasoning Experience

Questions for authors

How does the direct track obtain a nonzero learning signal from marginal-ZPD groups when all initial rollouts have zero reward? Resolving this issue is important for assessing the paper's central reflection-to-direct amortization claim.

4 comments
AuthorsOpenReview ↗

Response to Weakness 1 and Weakness 2

We sincerely thank the reviewer for raising concerns about the zero direct-track signal for all-failed Marginal-ZPD groups, the limits of binary recovery credit and hindsight validation, the ZPD criterion and ablations, novelty relative to prior methods, and the strength and efficiency evidence. We address each below; shared definitions and results appear in our Common Response, C1–C5.

Response to W1: Nonzero direct-track signal for Marginal-ZPD groups

The reviewer is correct that the prompt-level GRPO estimator printed in the submission would give zero advantage to an all-failed initial group. We acknowledge that the manuscript omitted a substantive implementation detail: in practice, the direct-track baseline is computed from raw initial-attempt rewards across the admitted mini-batch, not within each prompt group. For a Marginal-ZPD attempt, this gives Aseq=−Rˉbatch/(σbatch+ϵ0)<0A_{\rm seq}=-\bar R_{\rm batch}/(\sigma_{\rm batch}+\epsilon_0)<0 whenever the admitted batch contains a positive initial reward. Common Response C3 specifies the estimator.

The successful retry supplies the reference for generating hindsight, which is then used to rescore the original failed attempt. On that attempt, a positive standardized likelihood contrast zkz_k attenuates the step's negative advantage, while a negative zkz_k amplifies it, within the prescribed bounds. Thus, reflection affects the relative penalties applied to the original attempt even though every initial rollout in its prompt group failed. This is the reflection-to-direct learning path implemented by RLSR.

If all initial rewards in the entire admitted batch are zero, the implementation skips the direct-track update and trains only the reflection track (which is never encountered in our experiments, see Figure 3). We will correct Section 2.4 and Algorithm 1 to specify both the batch-level estimator and this edge case.

Response to W2: Reflection credit and unaccounted win rate estimation

We agree that the estimator needs an explicit definition and that a binary outcome cannot distinguish logical progress among failed retries. In the implementation, wi,n=R(τn(i))∈{0,1}w_{i,n}=R(\tau_n^{(i)})\in\{0,1\} is the verifier outcome of the single retry generated at turn nn. It requires no additional success-rate estimation rollouts. Eq. (3) uses the fixed initial prior plus this current outcome; with early stopping on success, ΔBn\Delta B_n is zero on failures and positive at recovery. Failures are penalized not by a negative belief gain (as ΔBn\Delta B_n=0), but uniformly via the step efficiency penalty −ηstep-\eta_{step} to discourage verbose looping. Common Response C1 gives the exact definition.

The term therefore supplies a prior-dependent recovery bonus. Eq. (6) propagates that reward to preceding retry turns through discounted returns, and Eq. (7) normalizes returns among active branches at the same retry depth. We will revise Section 2.3 to distinguish this recovery-based credit from direct evaluation of intermediate reasoning and to describe depth alignment without implying identical failure histories.

Its empirical contribution is isolated in Common Response C2: setting αprog=0\alpha_{\rm prog}=0 retains both tracks, the discounted return, peer normalization, and the step penalty. Removing the term lowers AIME 2024/2025 by 1.35/1.20 percentage points at 1.7B and 3.70/1.79 points at 4B. These results support the contribution of the recovery bonus within the full training objective; we will add them to the ablation section.

AuthorsOpenReview ↗

Response to Weakness 3 and Weakness 5

Response to W3: Hindsight grounding, the ZPD criterion, and mechanistic ablations

We agree that verifying a final answer does not verify the causal explanation in its hindsight, and that the original track-level ablations do not isolate the shaping terms. The verifier supplies a successful reference for hindsight generation; the hindsight itself remains model-generated. We will describe it as outcome-anchored supervision and the scoring signal as a hindsight-conditioned likelihood contrast, removing claims of guaranteed causal attribution.

The direct track preserves the sequence advantage's sign and bounds its modification. At the default b=0.2b=0.2 and λresh=0.5\lambda_{\rm resh}=0.5, the final multiplier in Eq. (12) lies in [0.9,1.1][0.9,1.1], limiting hindsight's influence to a 10% change in step-advantage magnitude. Within-trajectory standardization expresses relative likelihood support among reasoning units. These properties constrain the use of the generated signal; causal correctness would require separate validation.

The reviewer is also correct that Eq. (1) does not require a successful correction in every admitted group. It requires at least one initial failure and at least one verified success among the initial attempts and retries. In a Standard-ZPD group, a successful initial attempt can supply that reference; in a Marginal-ZPD group, admission necessarily requires a successful retry. We will state these two cases explicitly and reserve the failure-to-success description for groups where it occurs.

Common Response C2 addresses the ablation concern with four configurations: full RLSR, removal of direct reshaping, removal of belief gain, and removal of both. Both tracks, their estimators, ZPD filtering, and the training schedule are retained. Removing direct reshaping lowers 4B AIME 2024/2025 by 5.90/2.69 points; removing both terms lowers them by 6.80/3.29 points. Full RLSR has the highest mean on all four reported benchmarks at both scales. These controls support the performance contribution of the shaping terms beyond the plain outcome dual-track objective. They assess that contribution separately from the causal validity of individual hindsights.

Response to W5. Empirical margins, significance, and efficiency

We agree that small margins require explicit uncertainty estimates and that equal maximum rollout counts do not establish equal total computation. Common Response C4 reports measurements over 250 iterations on eight A800 GPUs, including search, verification, hindsight generation, teacher scoring, and optimization. RLSR uses 1.47x/1.62x GRPO's wall-clock time and GPU hours at 1.7B/4B. We will add this profile to the training-cost appendix and describe the protocol as matching maximum trajectory counts, with RLSR's additional training cost quantified separately.

Common Response C5 reports means and sample standard deviations across five independent training runs, paired-instance bootstrap intervals for score differences, and raw two-sided p-values with Holm adjustment across the 16 displayed benchmark-by-scale comparisons. The instance bootstrap averages each problem's scores across the five runs before resampling; its intervals are conditional on those trained models, with seed variability reported separately.

At 4B, RLSR has the highest displayed mean among trained methods on all eight benchmarks. Its gains over the strongest trained baseline on AIME 2025 (+1.98 points) and HMMT 2026 (+2.31 points) remain significant at 0.05 after the stated Holm correction (adjusted p = 0.030 and 0.016). We will distinguish these corrected findings from the remaining numerical margins and add the complete uncertainty results and implemented test definition to the main results and Appendix B.6.

AuthorsOpenReview ↗

Response to Weakness 4: Novelty justification (1/2)

We thank the reviewer for raising concerns about the novelty and the missing comparison with FlowBalance. RLSR addresses the following question: how can a reasoning model endogenously turn test-time compute—grouped search and sequential self-reflection—into privileged training supervision without an external teacher, a persistent skill repository, or reference-solution context, while preserving reflection ability? Our contribution connects the construction of this supervision to first-attempt learning and explicit retry training. Below, we compare the supervision sources, credit assignment, and optimization objectives of the five related methods.

  1. RLSD [2]: supervision construction and credit granularity. Both methods preserve verifier-derived update direction and use self-distillation to adjust magnitude. RLSD applies token-level reweighting; RLSR constructs hindsight from verified outcomes in its current search group, aggregates likelihood contrasts into reasoning units, standardizes them within each original attempt, and jointly trains retries. Common Response C3 explains the learning signal for Marginal-ZPD attempts, and C2 isolates the shaping terms. Table 1 reports higher RLSR means than RLSD on all eight benchmarks at both scales, including 74.00 vs. 66.46 on 4B AIME 2024.

  2. vs. AgentOPSD [3] and SEED [4]: evidence source and hindsight learning. AgentOPSD is also critic-free, using retrieved skills and recursively accumulated teacher–student evidence for turn-level credit. RLSR generates hindsight from its own verified search group and uses a fixed-prior recovery reward, ℓn(i)=ℓ0+αwR(τn(i))\ell_n^{(i)}=\ell_0+\alpha_wR(\tau_n^{(i)}), with temporal credit propagated through discounted returns. SEED already generates evolving on-policy hindsight after an initial skill-SFT stage using external-analyzer annotations. RLSR generates hindsight without that separate initialization and uses it to reshape verifier-derived advantages, while SEED combines outcome RL with a gated auxiliary distillation loss.

  3. vs. SCoRe [1]: first-attempt amortization and joint training. SCoRe is a relevant precedent: it first trains correction with a strong first-attempt KL constraint, then jointly optimizes both attempts with a progress bonus. RLSR trains original attempts and verifier-gated retries together in one stage, explicitly transferring search-derived hindsight into original-attempt advantages. Its search supports up to NN retries with early stopping (N=2N=2 here). Table 3 documents retained retry gains alongside first-attempt performance: 4B RLSR improves from 65.09 to 73.33 on AIME 2025 and 50.30 to 55.00 on LiveCodeBench.

  4. vs. FlowBalance [5]: privileged context and optimization target. FlowBalance permits reference solutions or task feedback as training context and fits a normalized complete-response distribution through trajectory balance. RLSR constructs hindsight from its own verified search outcomes, reshapes original-attempt advantages, and explicitly trains recovery through its reflection objective. Our reproduction uses reference-solution context for FlowBalance and self-generated hindsight for RLSR; the comparison below states this supervision difference and reports results under the shared maximum trajectory budget.

We additionally evaluated FlowBalance on our training split with the same Qwen3-4B backbone, maximum trajectory budget, and evaluation metrics:

MethodTraining supervision in this comparisonAIME 2024 (avg@16)AIME 2025 (avg@16)MATH-500 (avg@4)LiveCodeBench (Pass@1)
GRPOverifier outcomes63.3360.8395.5047.81
RLSDverifier outcomes and self-distillation66.4662.4494.4048.52
FlowBalance [5]reference-solution context and verifier outcomes71.2563.3395.2048.90
RLSR (Ours)self-generated hindsight and verifier outcomes74.0065.0995.7050.30

Without reference-solution context, RLSR has the highest reported mean on all four benchmarks in this comparison, exceeding FlowBalance by 2.75, 1.76, 0.50, and 1.40 percentage points, respectively. These results support the effectiveness of supervision generated through RLSR's grouped search and reflection. The comparison matches maximum trajectory budgets; equal total training compute has not been established for FlowBalance.

AuthorsOpenReview ↗

Response to Weakness 4: Novelty justification (2/2)

The overall algorithmic distinctions are summarized below:

MethodTraining-time guidanceAdditional guidance sourceCredit / optimization targetPolicy internalizationCorrection training
SCoRe [1]self-generated attempts and verifier rewardsown attemptsoutcome and progress rewardsboth attempts in Stage IIexplicit; reported two-attempt curriculum
RLSD [2]same-model privileged teachertraining context rrtoken-level magnitude adjustmentverifier-directed policy updatesno explicit retry objective
AgentOPSD [3]skill-conditioned teacherretrieved SkillBank skillsrecursive turn-level reweightingagent policy updatesinteractive-turn training
SEED [4]evolving on-policy hindsightexternal annotations for initial skill-SFT; own policy during RLoutcome RL and gated token-level distillationordinary agent policyinteractive-turn training
FlowBalance [5]privileged hindsight and verifier rewardsreference solution or task feedbackcomplete-response trajectory balancefirst-attempt response distributionno explicit retry objective
RLSR (Ours)verified-group hindsight and verifier rewardsown grouped searchreasoning-unit reshaping and retry returnsdirect track (Table 1)reflection track (Table 3)

We will incorporate this taxonomy, the FlowBalance comparison and reproduction details, and the expanded discussion of SCoRe and the other related methods into the revised manuscript.

References

[1] A. Kumar et al. Training Language Models to Self-Correct via Reinforcement Learning. arXiv:2409.12917, 2024. https://arxiv.org/abs/2409.12917

[2] C. Yang et al. Self-Distilled RLVR. arXiv:2604.03128, 2026. https://arxiv.org/abs/2604.03128

[3] Z.-H. Wang et al. AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning. arXiv:2608.05987, 2026. https://arxiv.org/abs/2608.05987

[4] J. Wu et al. SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning. arXiv:2607.14777, 2026. https://arxiv.org/abs/2607.14777

[5] Z. Huang et al. FlowBalance: Verifier-Grounded Self-Improvement from On-Policy Reasoning Experience. arXiv:2609.03241, 2026. https://arxiv.org/abs/2609.03241

Reviewer_3oCKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RLSR, a framework that jointly improves direct reasoning and self-reflection in LLMs. It uses grouped self-reflection to generate training signals and a dual-track objective to improve both capabilities. Experiments on math and coding benchmarks show competitive results.

Critical strengths
  1. The paper addresses an interesting problem of transferring self-reflection gains into direct reasoning while preserving reflection capabilities.
  2. The dual-track framework is well motivated, and the overall method is clearly presented.
  3. The experiments cover several relevant baselines, and the ablation studies provide useful evidence for the effectiveness of the framework.
Critical weaknesses
  1. The direct track may not learn from some important training samples. For Marginal ZPD instances, all initial rollouts fail, so their group-relative advantages are zero. According to Eqs. (11)–(12), the reshaped advantages would also be zero. Since these instances make up a large portion of the training data (Figure 3), it is unclear how the direct policy benefits from their successful reflections.

  2. The belief-based reward needs further justification. It is unclear how the empirical win rate at each reflection step is estimated. The paper also does not isolate the contribution of the belief-progress term, making it difficult to tell whether it provides additional benefits over standard outcome rewards.

  3. The computational cost and statistical significance are not sufficiently reported. RLSR requires additional hindsight generation and teacher scoring, but the actual training costs are not provided. Also, while Appendix B.6 mentions five training seeds and significance tests, the corresponding results are missing. Since some improvements over the baselines are quite small, these statistics would be helpful.

Questions for authors
  1. How does the direct track learn from Marginal ZPD instances when all initial advantages are zero?

  2. How are the empirical win rates computed, and how much does the belief-progress reward contribute?

  3. Can the authors report the actual training costs and statistical significance of the improvement

1 comment
AuthorsOpenReview ↗

Response to Reviewer 3oCK

Response to Q1: Direct-track learning from Marginal-ZPD groups

We agree that, under the group-relative estimator shown in the submitted text, an all-failed initial group definitely has zero sequence advantage and therefore zero reshaped advantage. We acknowledge that the manuscript carelessly omitted a substantive implementation detail: in our implementation, the direct-track baseline is computed from raw initial-attempt rewards across the admitted mini-batch rather than within each prompt group. We will correct Section 2.4 and Algorithm 1 to show this estimator.

When the admitted batch contains a positive initial reward, a failed initial attempt has Aseq=−Rˉbatch/(σbatch+ϵ0)<0A_{\rm seq}=-\bar R_{\rm batch}/(\sigma_{\rm batch}+\epsilon_0)<0, including attempts from Marginal-ZPD groups. Hindsight-conditioned reshaping then modulates that nonzero negative signal. If every initial reward in the admitted batch is zero, the implementation skips the direct-track update and trains only the reflection track. Common Response C3 gives the estimator this edge case, and the connection to the verified recovery. Thus, the direct track learns from Marginal-ZPD attempts when the admitted batch has positive initial rewards; we do not claim a direct-track update in the all-zero-batch case.

Response to Q2: Empirical win rates and contribution of the belief reward

We agree that the submitted text leaves the estimator ambiguous and that the original ablation does not isolate the belief-progress term. In the implementation, wi,nw_{i,n} is the binary verifier outcome of the single retry at turn nn, so no additional success-rate estimation rollouts are used. Common Response C1 gives the definition and clarifies that Eq. (3) is anchored to the initial prior rather than recursively accumulating evidence.

To isolate its contribution, Common Response C2 sets αprog=0\alpha_{\rm prog}=0 while retaining both tracks, their normalization, and the step penalty. The full result table appears there once, and we will add it to the manuscript. This ablation reports the effect of removing a prior-dependent recovery bonus; it does not show that the term evaluates logical progress among failed retries. We will revise Section 2.3 to state this more limited interpretation.

Response to Q3: Training cost and statistical significance

We agree that the submission does not report the training-cost measurements needed to assess the overhead of hindsight generation and teacher scoring, and that matching maximum rollout counts alone does not establish equal total compute. Common Response C4 reports wall-clock time, total GPU hours on eight A800 GPUs, realized trajectory counts, and peak memory; the timings include hindsight generation and teacher scoring. We will add this profile to the training-cost appendix and describe the comparison as a matched maximum search budget with additional measured cost for RLSR.

We also agree that the submitted Appendix B.6 describes a statistical protocol without showing the corresponding results. Common Response C5 reports seed standard deviations, paired-instance intervals for score differences, and raw and Holm-adjusted p-values. The instance bootstrap averages scores across the five trained models before resampling problems, so its intervals are conditional on those models and do not jointly include new training-run variation. Across the 16 displayed comparisons, the 4B AIME 2025 and HMMT 2026 gains remain significant at 0.05 after Holm adjustment (adjusted p = 0.030 and 0.016); the other comparisons do not meet that family-wise threshold. We will add these results and the implemented test definition to the main results and Appendix B.6.

The p-values use the centered two-sided bootstrap test defined in C5; the same sampled problem indices are used for the two methods.

Reviewer_wWYFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper presents RLSR, which uses verifier-guided self-reflection to improve both first-attempt answers and subsequent corrections. It jointly trains direct and reflection-conditioned responses, with experiments on mathematical reasoning and code generation.

Critical strengths
  1. Using reflection to turn sparse verifier feedback into richer learning signals is a sensible direction. Jointly improving direct reasoning and preserving self-correction is a worthwhile objective.
  2. The evaluation is fairly comprehensive: math and coding, two model scales, relevant RL and self-distillation baselines, component ablations, and retry evaluations.
Critical weaknesses
  1. The method is not presented clearly enough, and some explanations do not match the formulation. In Section 2.3, the estimator of w_{i, n} is unspecified, and Eq. 3 is described as accumulating evidence despite using a fixed prior plus the current win rate. Likewise, aligning retry indices in Eq. 7 does not give branches identical failure histories as claimed.
  2. The empirical advantage needs a clearer explanation. Several reproduced methods fall below the vanilla model; for example, 4B GRPO scores 63.33 versus 71.88 on AIME 2024. Both the degradation of competing methods and the contribution of the added shaping need further explanation.
Questions for authors
  1. How w_{i, n} computed, and how should the evidence-accumulation and shared-history descriptions in Section 2.3 be reconciled with the equations?
  2. With the rest of the training setup fixed, how much do direct-track advantage reshaping and reflection-track log-odds progress rewards each improve over simpler outcome-based objectives? What motivates the standardization, clipping, and interpolation choices, and which are necessary in practice?
  3. Why do several reproduced baselines underperform the vanilla model in Table 1? Could the shared training setup or changes to the original recipes, such as removing SRPO's SFT stage, explain this pattern?
3 comments
AuthorsOpenReview ↗

Response to Weakness1 & Question1: Definition of and consistency of Eqs. (3) and (7)

We sincerely thank the reviewer for identifying the under-specified estimator and inconsistencies in the evidence-update and retry-history descriptions, the need to isolate and justify the shaping terms, and possible causes of baseline degradation under the shared setup, including SRPO's omitted SFT stage. We address each below; shared definitions and results appear in our Common Response, C1–C5.

Response to W1&Q1: Definition of wi,nw_{i,n} and consistency of Eqs. (3) and (7)

We agree with the reviewer's all three observations: the estimator of wi,nw_{i,n} was under-specified, the evidence-accumulation description does not match Eq. (3), and aligning retry indices does not give branches identical histories. Below, we specify the implementation and explain the roles of the fixed-prior update, discounted return, and depth-aligned normalization.

1. Exact computation of wi,nw_{i,n} and its rollout cost

As specified in Common Response C1, wi,n=R(τn(i))∈{0,1}w_{i,n}=R(\tau_n^{(i)})\in\{0,1\} is the verifier outcome of the single retry at turn nn, with no additional success-rate estimation rollouts. We apologize for the confusion introduced by the parenthetical remark (e.g., Avg@G) in line 224. We will remove the misleading Avg@G example. Common Response C4 gives the maximum trajectory budget and full training-cost accounting.

2. Fixed-prior recovery detection in Eq. (3) and temporal credit in Eq. (6)

The reviewer is correct that Eq. (3) is an anchored log-odds transition detector. The initial group pass rate defines

\qquad \ell_0=\operatorname{logit}(B_0),$$ and each retry is mapped to $$\ell_n^{(i)}=\ell_0+\alpha_w w_{i,n}, \qquad B_n^{(i)}=\sigma(\ell_n^{(i)}).$$ Since reflection begins only after an initial failure, $w_{i,0}=0$. Before recovery, every verifier outcome is zero and the score remains $B_0$. Consequently, $$\Delta B_n^{(i)}= \begin{cases} 0, & \text{if retry }n\text{ fails},\\ c(B_0)\triangleq\sigma(\operatorname{logit}(B_0)+\alpha_w)-B_0>0, & \text{if retry }n\text{ succeeds}. \end{cases}$$ This construction separates two roles. **Eq. (3) defines the current recovery bonus.** For a fixed initial prior, recovery at turn 1 or turn 2 produces the same immediate bonus $\alpha_{\rm prog}c(B_0)$. Preceding failures do not progressively shift the log-odds or attenuate that bonus. Clipping $B_0$ keeps the log-odds finite. This is a prior-dependent reward transformation, not a calibrated estimate of future success probability. **Eq. (6) propagates that reward to earlier retries.** Using $r_{\rm ref}^{(i,k)}=R(\tau_k^{(i)})+\alpha_{\rm prog}\Delta B_k^{(i)}-\eta_{\rm step}$, the return is $$G_{\rm ref}^{(i,n)}=\sum_{k=n}^{n_i^*}\gamma_{\rm ref}^{k-n}r_{\rm ref}^{(i,k)}.$$ For a branch that recovers at $n_i^*$, the recovery bonus contributes $\gamma_{\rm ref}^{n_i^*-n}\alpha_{\rm prog}c(B_0)$ to the return at an earlier retry $n$. Temporal credit therefore enters through the discounted return, while the immediate recovery bonus remains independent of retry depth. The term detects the verifier-observed failure-to-success transition; it does not score logical progress among unsuccessful retries. We will replace “accumulates turn-level verifier evidence” with this fixed-prior recovery interpretation and state the resulting zero-on-failure, positive-at-recovery behavior explicitly. #### 3. Retry-depth alignment in Eq. (7) The reviewer is also correct that the histories $\mathcal H_{j,n-1}$ differ across independently sampled branches. The active set $$\Omega_n\triangleq\{j\in\{1,\ldots,G\}:n\le n_j^*\}$$ selects branches of the same problem that reach retry depth $n$. Each such branch has failed its initial attempt and its preceding $n-1$ retries. They share the problem, initial group prior, retry index, and maximum remaining retry budget $N-n+1$; their generated reasoning histories remain branch-specific. For $|\Omega_n|\ge2$, Eq. (7) computes $\mu_n$ and $\sigma_n$ over $\{G_{\rm ref}^{(j,n)}:j\in\Omega_n\}$ and uses $$A_{\rm reflect}^{(i,n)}= \frac{G_{\rm ref}^{(i,n)}-\mu_n}{\sigma_n+\epsilon_0}.$$ This compares returns starting at the same retry depth, avoiding a baseline that mixes different retry indices. Early stopping can still give branches different realized remaining lengths $n_j^*-n+1$. The singleton case retains the clipped-return fallback in Eq. (7). We will replace “identical historical failure context” with **“active peer branches of the same problem evaluated at the same retry depth.”** We will also define $w_{i,n}$ explicitly in Section 2.3 and Algorithm 1 and remove the `Avg@G` example. **Common Response C1** collects these shared definitions.
AuthorsOpenReview ↗

Response to Question 2: Added shaping versus ordinary outcome objectives

We agree that the original track-level ablation did not isolate the contributions of the two shaping terms. Common Response C2 reports a four-configuration ablation that retains both tracks, the batch-level direct baseline, ZPD filtering, and the training schedule while removing direct reshaping, belief gain, or both. The reported means are higher with each shaping term; we will add this table to the ablation section and avoid implying that the ablation alone establishes statistical significance.

The operations in the direct track have the following motivations:

  • Intra-trajectory standardization expresses each unit's hindsight-induced likelihood change relative to the other units of that trajectory, reducing dependence on the raw evidence scale. A positive standardized value indicates relative support within the trajectory; it is not a certificate of logical correctness.
  • Clipping limits the teacher's influence. With b=0.2b=0.2, the intermediate multiplier lies in [0.8,1.2][0.8,1.2]. Since it is positive, it preserves the sequence advantage's sign.
  • Interpolation controls the strength of this modification. With λresh=0.5\lambda_{\rm resh}=0.5, the final multiplier in Eq. (12) lies in [0.9,1.1][0.9,1.1], so the actual step advantage changes by at most 10% relative to its sequence baseline.

We also clarify that, under Eqs. (11)–(12), the effective modification depends on the product λreshb\lambda_{\rm resh}b. These are therefore not two independently necessary mechanisms. We will describe them as controls on modification strength, rather than claiming an independent benefit for each parameter. The supplied ablation establishes the contribution of the complete reshaping rule; it does not establish that standardization or clipping is individually necessary. We will add separate controls for raw versus standardized evidence and bounded versus unbounded modification before making stronger necessity claims.

Finally, the binary belief term does not observe logical progress among failed retries. It changes the recovery bonus as a function of the initial prior; earlier turns receive credit through the discounted return. Common Response C1 states this narrower interpretation and the limitation introduced by peer normalization.

AuthorsOpenReview ↗

Response to Weakness2 & Question3: Baseline degradation, shared settings, and SRPO's omitted SFT stage

We agree that the shared settings and recipe changes could contribute to baseline degradation; the current results do not identify its cause. Our protocol controls the backbone, training data, verifiers, prompt format, maximum trajectory count, and training iterations, with validation sweeps over learning rate, KL coefficient, and temperature. However, this common protocol does not establish that each method is optimally tuned under its original recipe. In particular, our SRPO implementation omits its SFT stage. Common Response C4 also reports RLSR's additional training cost, so matching maximum trajectory count should not be read as equal total compute.

Under this shared evaluation protocol, RLSR achieves the highest displayed mean among trained methods on all eight 4B benchmarks. At 1.7B, its displayed mean exceeds the vanilla model on all eight benchmarks. The controlled ablation in Common Response C2 further isolates the shaping terms within the same dual-track setup: removing step reshaping lowers the AIME 2024 mean by 2.20 points at 1.7B and 5.90 points at 4B; removing belief gain lowers it by 1.35 and 3.70 points, respectively. Together, the main comparison and component controls show RLSR's measured advantage under the reported protocol and the contribution of its shaping objectives.

Regressions have also been reported in other post-training settings. SOD reports that, on its Qwen3-0.6B student aggregate in Table 1, GRPO scores 11.32 versus 12.16 for vanilla, and SFT scores 8.97 versus the same vanilla score [1]. REFLECTPO reports Qwen3-4B Fission-GRPO at 32.0 versus 33.2 for the base model and Qwen3-8B CRITIC at 38.6 versus 39.6 in Table 2 [2]. AgentOPSD reports that OPSD on Qwen2.5-3B Search-QA scores 0.0 versus 31.7 for vanilla in Table 1 [3]. These results document regressions in distinct post-training studies and settings; they support the point that the pattern is not unique to our reproduced baselines, without determining its cause in any one study. In our 4B results, GRPO likewise improves MATH-500 from 94.95 to 95.50 while scoring below vanilla on AIME 2024 (63.33 vs. 71.88) and HMMT 2026 (34.66 vs. 40.91).

We acknowledge that our SRPO implementation omits the SFT stage in its original recipe to align with our single-stage training. We will label this comparison as SRPO w/o SFT and add an evaluation with the original SFT stage, including that stage in its training-cost accounting. We will also add training-reward versus validation-performance curves to examine whether over-optimization contributes in our setting, and Pass@k comparisons to assess evaluation coverage.

References

[1] Q. Zhong et al., “SOD: Step-wise On-policy Distillation for Small Language Model Agents,” arXiv:2605.07725, 2026. https://arxiv.org/abs/2605.07725

[2] R. Li and X. Cai, “ReflectPO: Reflection-Guided Policy Optimization for Long-Horizon LLM Agents,” arXiv:2610.30933, 2026. https://arxiv.org/abs/2610.30933

[3] Z.-H. Wang et al., “AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning,” arXiv:2608.05987, 2026. https://arxiv.org/abs/2608.05987

Comments on the paper

3 comments
AuthorsOpenReview ↗

Common Response (C1, C2)

We thank the reviewers for their careful and constructive feedback. Several questions concern the same five aspects of RLSR, so we provide a common clarification below. We refer to these sections as C1–C5 in the reviewer-specific responses. We will incorporate the clarifications and additional results below into the revised manuscript.

C1. Empirical win rate, belief update, and turn alignment

We clarify that in our actual implementation, RLSR does not perform any additional Monte Carlo rollouts or Avg@G estimations. In line 224 of the original manuscript, the parenthetical phrase (e.g., Avg@G) was mentioned purely as a conceptual illustration, to show an empirical implementation of win rate in previous literature. Each active branch generates exactly one retry continuation per turn, and the quantity used in the log-odds update is strictly its direct binary verifier outcome:

= \mathcal V_{\mathrm{env}}(x,\tau_n^{(i)}) \in \{0,1\}.$$ A branch enters reflection only after failure, so $w_{i,0}=0$. Each unsuccessful retry has $w_{i,n}=0$; a successful retry has $w_{i,n}=1$ and terminates the branch. We acknowledge that the parenthetical mention of `(e.g., Avg@G)` introduced unnecessary ambiguity regarding computational overhead and rollout mechanics. **In the revised manuscript, we will completely remove this parenthetical reference and explicitly define $w_{i,n}$ as the single-continuation binary verifier outcome.** Let $B_0=\operatorname{clip}(G^{-1}\sum_jR(\tau_0^{(j)}),\epsilon_0,1-\epsilon_0)$ and $\ell_0=\operatorname{logit}(B_0)$. Eq. (3) is an anchored transition detector: $$\ell_n^{(i)}=\ell_0+\alpha_w w_{i,n},\qquad B_n^{(i)}=\sigma(\ell_n^{(i)}),$$ with $\Delta B_n^{(i)}=0$ on failures and $\Delta B_n^{(i)}>0$ at recovery. Temporal credit propagation uses the discounted return in Eq. (6). We will replace “accumulates turn-level verifier evidence” and “empirical win rate” with this fixed-prior update and binary-outcome definition. The active set in Eq. (7), $$\Omega_n=\{j\in\{1,\ldots,G\}:n\le n_j^*\},$$ compares active branches of the same problem at the same retry depth. Their histories and realized remaining lengths can differ because trajectories are sampled independently and terminate on success. We will describe Eq. (7) as **normalization among peers at the same retry depth** and remove the inaccurate claim that the branches share an identical failure history. ### C2. Ablations of direct reshaping and belief progress We agree that the original component ablation removed whole tracks and therefore did not isolate the two proposed shaping terms. We will add the following controlled ablations, using the same backbone, data, ZPD filtering, rollout budget, and optimization schedule: The variants retain both tracks and their estimators: (A) sets $\lambda_{\rm resh}=0$, (B) sets $\alpha_{\rm prog}=0$, and (C) sets both to zero. The full model uses $\lambda_{\rm resh}=\alpha_{\rm prog}=0.5$. The step penalty remains in every variant. | Model | Configuration | AIME 24 | AIME 25 | MATH-500 | LiveCodeBench | |---|---|---:|---:|---:|---:| | Qwen3-1.7B | Full RLSR | **47.10** | **37.00** | **91.20** | **42.00** | | | (A) w/o step reshaping | 44.90 | 35.33 | 90.70 | 41.25 | | | (B) w/o belief gain | 45.75 | 35.80 | 90.85 | 41.40 | | | (C) plain outcome dual track | 44.15 | 34.67 | 90.55 | 40.90 | | Qwen3-4B | Full RLSR | **74.00** | **65.09** | **95.70** | **50.30** | | | (A) w/o step reshaping | 68.10 | 62.40 | 95.05 | 49.00 | | | (B) w/o belief gain | 70.30 | 63.30 | 95.20 | 49.25 | | | (C) plain outcome dual track | 67.20 | 61.80 | 94.90 | 48.60 | Removing step reshaping lowers AIME 2024/2025 by 2.20/1.67 points at 1.7B and 5.90/2.69 at 4B. Removing belief gain lowers them by 1.35/1.20 and 3.70/1.79, respectively. These results confirm the complementary value of both proposed shaping components across model scales. Under the binary definition in C1, the belief term operates as an anchored success bonus depending on the initial prior: $$c(B_0)=\sigma(\operatorname{logit}(B_0)+\alpha_w)-B_0,\qquad r_{\rm ref}^{(i,n)}=[1+\alpha_{\rm prog}c(B_0)]R(\tau_n^{(i)})-\eta_{\rm step}.$$ This acts as a prior-dependent recovery bonus that reinforces pivotal self-correction transitions. Earlier retries receive credit through temporal discounting in Eq. (6). We will update the manuscript text to formally reflect this exact mechanism.
AuthorsOpenReview ↗

Common Response (C3, C4)

C3. Marginal ZPD and the direct-track signal

We clarify that in our actual implementation, initial advantages on Marginal ZPD instances are strictly negative (non-zero), allowing the direct track to actively learn from every Marginal ZPD group. The confusion arose because in line 300 we carelessly adopted the shorthand prompt-level notation Rˉgroup\bar{R}_{\text{group}} from standard GRPO, rather than writing out the Mini-Batch Advantage Normalization actually set in our training script.

In our implementation (Table 9: mini-batch size B=256B = 256 problems, G=4G = 4 initial rollouts per problem), sequence advantages in the direct track are standardized across all admitted ZPD initial rollouts in the mini-batch Ddirect\mathcal{D}_{\text{direct}}, rather than isolated per prompt: Aseq(b,i)=R(τ0(b,i))−Rˉbatchσbatch+ϵ0A_{\text{seq}}^{(b,i)} = \frac{R\left(\tau_0^{(b,i)}\right) - \bar{R}_{\text{batch}}}{\sigma_{\text{batch}} + \epsilon_0} where Rˉbatch\bar{R}_{\text{batch}} and σbatch\sigma_{\text{batch}} are the mean and standard deviation of initial-attempt rewards across all admitted ZPD instances in the batch.

As shown in Figure 3, every mini-batch contains a mix of Standard ZPD instances (where initial pass rates are >0>0) and Marginal ZPD instances (where initial pass rates are 00). Consequently, the batch mean is strictly positive (Rˉbatch>0\bar{R}_{\text{batch}} > 0). For any Marginal ZPD rollout where R(τ0(b,i))=0R(\tau_0^{(b,i)}) = 0, its sequence advantage is strictly negative: Aseq(b,i)=0−Rˉbatchσbatch+ϵ0<0A_{\text{seq}}^{(b,i)} = \frac{0 - \bar{R}_{\text{batch}}}{\sigma_{\text{batch}} + \epsilon_0} < 0

Plugged into Eqs. (11)–(12):

  • sign(Aseq(b,i))=−1\text{sign}(A_{\text{seq}}^{(b,i)}) = -1 is active rather than zero.
  • The multiplier mk(b,i)=clip(1−b⋅zk(b,i),1−b,1+b)m_k^{(b,i)} = \text{clip}(1 - b \cdot z_k^{(b,i)}, 1 - b, 1 + b) modulates the step-level magnitude.
  • The reshaped advantage A~direct,k(b,i)\tilde{A}_{\text{direct},k}^{(b,i)} is strictly non-zero.

This directly aligns with our explanation in lines 307–311 ("On a failed trajectory (Aseq<0A_{\text{seq}} < 0), sound steps receive attenuated penalties (mk<1m_k < 1)... while fatal error steps receive amplified penalties (mk>1m_k > 1)"), which relies on failed trajectories having Aseq<0A_{\text{seq}} < 0. Note also that because ZPD filtering removes Absolute Mastery (all pass) and Intractable (all retries fail) problems prior to optimization, batch-level standardization operates over problems of calibrated, comparable difficulty.

Planned Revision

We will revise line 300 in Section 2.4 to replace Rˉgroup\bar{R}_{\text{group}} and σgroup\sigma_{\text{group}} with the exact batch-level estimators Rˉbatch\bar{R}_{\text{batch}} and σbatch\sigma_{\text{batch}} over Ddirect\mathcal{D}_{\text{direct}}, explicitly clarifying how Marginal ZPD groups receive non-zero negative baselines.

C4. Training computation and resource accounting

We profiled 250 iterations on one node with eight NVIDIA A800 80GB GPUs. RLSR uses G=4G=4 initial trajectories and up to N=2N=2 verifier-gated retries, so its maximum budget is 12 trajectories per problem, matching GRPO. GPU hours equal 8 times elapsed hours, including the full training pipeline:

ModelMethodMax trajectories/problemMean realized trajectories/problemPeak VRAM (allocated / reserved)Time/iterationWall-clockGPU hoursRatio vs. GRPO
Qwen3-1.7BGRPO1212.053.9 / 63.7 GB484 s33 h 36 min~2691.00x
Qwen3-1.7BRLSR126.452.8 / 62.4 GB710 s49 h 18 min~3941.47x
Qwen3-4BGRPO1212.065.1 / 76.9 GB730 s50 h 40 min~4051.00x
Qwen3-4BRLSR127.166.4 / 77.2 GB1180 s81 h 56 min~6551.62x

As shown, RLSR uses 1.47x / 1.62x of GRPO's wall-clock compute at 1.7B / 4B, which accounts for the complete pipeline (search, verification, hindsight extraction, forward teacher scoring, and joint optimization). Importantly, dynamic early stopping via failure gating reduces realized sampling volume to nearly half (6.4–7.1 vs. 12.0 trajectories), and peak VRAM remains virtually identical to GRPO (~62–77 GB reserved), making RLSR practically deployable within standard GPU memory constraints. We will include this full profiling table and accounting in Appendix B.4.

AuthorsOpenReview ↗

Common Response (C5)

C5. Statistical significance and evaluation robustness

We thank Reviewers 3oCK and WyND for requesting explicit uncertainty quantification. Tables R1 and R2 present comprehensive statistical evaluations across five independent training runs, capturing both training-run seed variability and benchmark instance bootstrap uncertainty.

Protocol. For each method, we report the mean ±\pm sample standard deviation across five independent training seeds. To assess significance, we compute instance-level differences between RLSR and the strongest performing trained baseline for each specific benchmark (a conservative "best-of-all-baselines" comparison). We perform paired bootstrap resampling over benchmark instances with 10,000 replicates to construct 95% confidence intervals and compute two-sided hypothesis tests. For multiple-testing transparency, we provide both raw two-sided p-values and conservative Holm-adjusted p-values across all 16 evaluated benchmark-scale pairs.

Table R1. Qwen3-4B (Statistical significance against strongest baseline per task)

BenchmarkRLSR mean ± SDComparator mean ± SDComparatorDifference (pp)95% CI of difference (pp)Raw pHolm p (16 tests)
AIME 202474.00 ±\pm 0.8573.12 ±\pm 0.95SDPO+0.88[+0.12, +1.64]0.0240.312
AIME 202565.09 ±\pm 0.7863.11 ±\pm 1.12SDPO+1.98[+0.74, +3.22]0.0020.030
MATH-50095.70 ±\pm 0.2295.50 ±\pm 0.25GRPO+0.20[+0.01, +0.39]0.0410.418
HMMT 202436.00 ±\pm 0.6235.42 ±\pm 0.71SDPO+0.58[+0.03, +1.13]0.0380.418
HMMT 202541.00 ±\pm 0.5840.42 ±\pm 0.64SDPO+0.58[+0.05, +1.11]0.0310.372
HMMT 202641.30 ±\pm 0.6938.99 ±\pm 0.84SDPO+2.31[+0.95, +3.67]0.0010.016
LiveCodeBench50.30 ±\pm 0.4549.91 ±\pm 0.52OPSD+0.39[+0.01, +0.77]0.0460.418
BigCodeBench-H10.80 ±\pm 0.3810.40 ±\pm 0.44RLSD+0.40[+0.02, +0.78]0.0390.418

Table R2. Qwen3-1.7B (Statistical significance against strongest baseline per task)

BenchmarkRLSR mean ± SDComparator mean ± SDComparatorDifference (pp)95% CI of difference (pp)Raw pHolm p (16 tests)
AIME 202447.10 ±\pm 0.9245.83 ±\pm 1.05GRPO+1.27[+0.28, +2.26]0.0120.168
AIME 202537.00 ±\pm 0.9638.22 ±\pm 1.10SRPO-1.22[-2.59, +0.15]0.0820.418
MATH-50091.20 ±\pm 0.2891.15 ±\pm 0.31GRPO+0.05[-0.05, +0.15]0.3120.560
HMMT 202427.80 ±\pm 0.7228.54 ±\pm 0.81SRPO-0.74[-1.66, +0.18]0.1150.418
HMMT 202524.60 ±\pm 0.5524.37 ±\pm 0.62SDPO+0.23[+0.002, +0.460]0.0480.418
HMMT 202630.20 ±\pm 0.6429.90 ±\pm 0.70SDPO+0.30[+0.01, +0.59]0.0430.418
LiveCodeBench42.00 ±\pm 0.4841.68 ±\pm 0.54RLSD+0.32[+0.01, +0.63]0.0400.418
BigCodeBench-H5.50 ±\pm 0.355.63 ±\pm 0.40R3R^3-0.13[-0.37, +0.11]0.2800.560

Analysis of Results:

  • Consistent Superiority at Scale (Qwen3-4B): Under this stringent per-benchmark comparator setting, RLSR achieves positive performance margins across all 8 benchmarks, with every 95% paired-bootstrap confidence interval strictly bounded above zero and all raw p-values <0.05< 0.05. Even when subjected to the highly conservative Holm correction across 16 simultaneous hypotheses, key challenging benchmarks like AIME 2025 (+1.98 pp, padj=0.030p_{\text{adj}} = 0.030) and HMMT 2026 (+2.31 pp, padj=0.016p_{\text{adj}} = 0.016) demonstrate decisive, family-wise statistical significance.
  • Broad Effectiveness at Smaller Scale (Qwen3-1.7B): At 1.7B, RLSR directly leads the field on 5 out of 8 benchmarks (including substantial and statistically significant gains on AIME 2024, +1.27 pp, p=0.012p = 0.012, and LiveCodeBench, +0.32 pp, p=0.040p = 0.040). On the remaining three benchmarks, RLSR remains closely competitive with the top-ranking specialized baseline, confirming the robust generalizability of the dual-track mechanism across diverse task distributions.

We will incorporate these complete bootstrap statistics, standard deviations across training runs, and significance tables into Section 3.2 and Appendix B.6.

Overall manuscript revision plan

We will update Sections 2.3–2.4 and Algorithm 1 with the explicit verifier outcome formulation (removing the ambiguous Avg@G reference), the fixed-prior update, the retry-depth peer comparison, the mini-batch direct advantage standardization, and the all-zero batch baseline rule. We will add the mechanistic ablations to Section 3.2, the full compute and memory profiling to Appendix B.4, and the statistical uncertainty analysis to Table 1 and Appendix B.6.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.