acceptodds
Under review as a conference paper at ICLR 2027

Learning to Localize and Repair Deviations in Long-Horizon Reasoning

Abstract

Large language model (LLM) agents increasingly tackle complex long-horizon tasks through multi-step reasoning, tool use, and environment interaction. However, local errors can propagate through subsequent steps, derailing reasoning trajectories and wasting computation. Existing approaches often provide scalar supervision with limited corrective guidance or rely on reflection and revision that incur additional inference overhead. We therefore propose **RLOR**, an observation-guided trajectory repair framework. RLOR identifies the earliest deviation as a pivot, injects observation-based natural-language feedback, and repairs only the subsequent continuation while preserving the valid prefix. We use multi-round pivot annotations to guide curriculum supervised fine-tuning (SFT) and pivot-aware reinforcement learning (RL) with suffix-localized credit assignment. This training enables agents to internalize and generalize repair behavior, allowing autonomous correction without external teacher feedback at deployment. Experiments on long-horizon reasoning benchmarks show that RLOR improves accuracy and token efficiency, with ablations confirming the contributions of pivot localization, observation content, and the training design.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-Horizon Reasoning, LLM Agent, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UYuXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes RLOR for repairing errors in long reasoning trajectories. It identifies an early deviation, generates an observation-based explanation, and repairs the continuation while preserving the preceding prefix. Training combines a three-stage SFT curriculum with pivot-aware GSPO updates. Experiments on five benchmarks report improvements in accuracy and token efficiency.

Critical strengths

The experiments report consistent accuracy gains over GSPO across five benchmarks using the same backbone and evaluation harness. The reduction in token usage is also encouraging. Together, these results suggest that the proposed training approach has practical value.

Critical weaknesses
  1. The contribution relative to recent work is insufficiently established. Several closely related methods are missing from both the discussion and comparisons. PivoARL [1] combines error localization, prefix reuse, local retries, and pivotal credit assignment. Search-R2 [2] repairs faulty suffixes and removes its refiner at deployment. ELPO [3] focuses corrective updates on critical errors and their suffixes. The SFT component also overlaps with recent work. InT [4] internalizes targeted interventions through SFT before further RL training. P-BRIDGE [5] distills teacher-guided error diagnosis and prefix-preserving repairs, while masking the prefix loss during SFT. These methods differ from RLOR in important details. However, the paper does not establish what its particular combination adds. The current baselines leave this central question unresolved.

  2. The deployment procedure is underspecified. The teacher locates pivots by inspecting completed trajectories. The trained policy is then claimed to detect deviations during generation. It is unclear how this transition is implemented. In particular, does inference truncate an earlier suffix, or append a correction to the current history? This distinction matters for understanding both the repair mechanism and its reported efficiency.

[1] Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retry

[2] Search-R2: Enhancing Search-Integrated Reasoning via Actor-Refiner Collaboration

[3] Learning from the Irrecoverable: Error-Localized Policy Optimization for Tool-Integrated LLM Reasoning

[4] InT: Self-Proposed Interventions Enable Credit Assignment in LLM Reasoning

[5] Failures are Treasures: Constructing a Pedagogical Bridge for Agentic Strategy Distillation

Questions for authors
  1. What distinguishes RLOR’s repair training from InT and P-BRIDGE, and its policy optimization from PivoARL, Search-R2, and ELPO? What controlled comparisons or ablations establish these differences as meaningful contributions?

  2. Could you provide the deployment algorithm and an example from the trained policy? Please specify when pivots are generated, what context is retained, and whether tool interactions are replayed.

6 comments
AuthorsOpenReview ↗

Response to Reviewer UYuX - Part 1

For W1&Q1:

We thank the reviewer for requesting a clearer explanation of what RLOR adds and whether its design choices matter. RLOR combines staged learning of explicit repair representations with pivot-conditioned update allocation based on terminal rewards. This integration trains the policy to generate its own corrective guidance and continue reasoning within an ongoing trajectory.

1. Differences from the five methods

Related workDistinction in RLOR
PivoARL [1]RLOR applies a shared terminal advantage to pivot-and-suffix updates, with weak prefix KL, rather than cross-episode prefix credit and a separate reflection-return objective.
Search-R2 [2]RLOR learns explicit pivot/reason generation through its curriculum and optimizes masked GSPO with correctness/format rewards, rather than jointly optimizing Actor–Meta-Refiner roles with an evidence-density process reward.
ELPO [3]RLOR uses generated pivots, with an entropy-based fallback for incorrect no-pivot trajectories, to select sequence-level updates, rather than binary-search rollout trees, hierarchical advantages, and error-localized adaptive clipping.
InT [4]Both approaches internalize interventions through training. RLOR uses the progressive supervision sequence, masks the input prefix from SFT loss, and subsequently applies pivot-conditioned policy optimization. InT’s selected recipe instead reinforces the prefix and intervention while excluding the repaired suffix.
P-BRIDGE [5]RLOR separates pivot/reason annotation from continuation generation and combines staged repair supervision with subsequent outcome-driven RL, rather than distilling teacher-repaired trajectories and extracted strategy insights.

2. Evidence from the manuscript and additional controls

Figures 2 and 7 support the utility of pivot localization and semantically aligned reasons through teacher-assisted intervention diagnostics. Tables 4 and 8 show the benefit of progressively learning localization, reasons, and continuation over mixed or continuation-only training; Table 11 shows that the curriculum’s macro-average advantage persists across the evaluated policy–trajectory-source combinations. Figure 3 and Table 2 further support improved accuracy–token-cost trade-offs, rather than gains obtained simply through longer reasoning.

To further test these design choices, we added same-backbone SFT controls and RL controls sharing the same Stage-SFT initialization.

MethodBrowseCompAMO-BenchSealQA₀HLEGAIAAvg.
Mono32.732.027.140.169.240.2
Rejection-Sampling SFT33.134.029.739.871.541.6
Teacher-assisted Success-SFT33.942.030.640.270.643.5
Plain-text Pivot-SFT (Flat-SFT)34.244.031.540.371.844.4
Annotation-only SFT (Stages 1+2)34.648.033.239.870.445.2
Student-Annotation Stage-SFT34.238.030.640.271.843.0
RLOR Mixture-SFT33.334.028.539.670.941.3
RLOR Stage-SFT35.454.032.540.573.147.1

Rejection-Sampling SFT retains correct trajectories from five unassisted rollouts per question on the same pool, matching the curriculum’s total training-example presentations. Success-SFT removes all pivot markers and associated reasons from the same successful student repairs. The plain-text control replaces markers with natural language, retaining localization semantics, reasons, continuations, and stage order. Annotation-only trains only Stages 1–2 with GPT-5.2 annotations; Student-Annotation uses Qwen3.5-35B-A3B annotations. Mixture-SFT mixes tasks instead of training stages sequentially.

Our curriculum-based Stage-SFT achieves the best overall performance, highlighting the benefits of progressive, structured repair learning over alternative training strategies.

MethodBrowseCompAMO-BenchSealQA₀HLEGAIAAvg.
GSPO43.660.040.346.875.453.2
Per-trajectory equal-token random mask42.556.038.745.574.751.5
Random suffix boundary44.758.041.446.676.553.4
RLOR46.264.044.747.378.156.1

Random masking preserves the updated-token count per trajectory; random boundaries select arbitrary assistant-step suffixes. Unselected assistant tokens receive weak KL.

RLOR outperforms GSPO and both controls, supporting pivot-aware updates beyond token-count reduction or arbitrary suffix selection.

Taken together, the SFT comparisons support the value of staged learning of explicit repair representations, while the RL controls support pivot-conditioned update allocation beyond ordinary GSPO or the tested randomized update schemes. These findings support RLOR’s specific training design: learning to generate corrective guidance, then using terminal rewards to optimize pivot-conditioned continuations within an ongoing trajectory.

AuthorsOpenReview ↗

Response to Reviewer UYuX - Part 2

For W2&Q2:

Thank you for pointing out this ambiguity. We clarify that offline suffix replacement is used only for SFT data construction, whereas online deployment follows an append-only correction mechanism. The teacher identifies intervention points retrospectively to construct supervision, but the trained policy autonomously generates pivots and corrective continuations during inference. We distinguish these two procedures below.

(1) SFT data construction

During offline data construction, the teacher examines completed trajectories to identify intervention points. The question (x) and retained trajectory prefix (h) are used as the shared conditioning context, while the original off-track suffix is replaced by the corresponding supervision target.

All stages share the question–prefix input; Stage 3 retains only successful repairs.

StageModel inputSupervised outputSingle-round examplesAdditional later-round examplesMulti-round total
1. Pivot localizationx + hP4,6092,5177,126
2. Pivot-reason generationx + hP + r4,6092,5177,126
3. Continuation repairx + hP + r + s+2,7621,9654,727

Each stage trains for eight epochs (learning rate 10⁻⁵, cosine schedule, warmup ratio 0.05); subsequent stages use the preceding final checkpoint. No validation-based switching or checkpoint selection is used.

This schematic adapts Figure 9 to illustrate the supervision formats; it is neither a serialized training sample nor an autonomous RLOR rollout.

Shared input: question x + retained history h

x:
Which person represented the association acknowledged by the
target early-2020s article on mining and water impacts?

h (abridged):
IMWA has been considered as a candidate association, but the target
article and its acknowledgements have not yet been verified.

P:
<|PIVOT_INTERVENTION|>

r (abridged):
Narrowing the search to IMWA is unsupported. Identify the target
article and use its acknowledgements to determine the association
and its representative.

s+ (abridged):
Assistant: Search for the target article and inspect its acknowledgements.
Tool: ... AWARD ... under the directorship of Sharon Pollard ...
Assistant: Verify the co-author's leadership appointment.
Tool: ... appointed Executive Director of AWARD in August 2023 ...
Assistant: Answer: Sharon Pollard.

Stage 1 target: P
Stage 2 target: P + r
Stage 3 target: P + r + s+

In all three stages, the question and retained prefix are conditioning context and receive no SFT loss. Only target assistant tokens are supervised; tool observations are excluded from the loss. We will add representative serialized training examples for all three stages to the appendix, including their input–target boundaries and token-level loss masks.

(2) Online deployment

Unlike offline data construction, deployment does not involve teacher intervention or retrospective suffix replacement. Instead, the trained policy generates pivot markers as part of its own autoregressive generation. When a pivot is produced, the policy continues with an observation-based reason and corrective actions, conditioned on the accumulated history.

In the testing phase, RLOR generates its own pivot, observation-based reason, and corrective continuation within the current rollout. Correction is append-only: all history is retained, with no teacher, truncation, or automatic tool replay.

H ← question and interaction history
while within 128k tokens and 256 assistant turns:
    Generate the policy’s next output and append it to H.
    If a pivot appears, continue with its reason and corrective actions.
    If a final answer is emitted, return it.
    Execute new tool calls, with at most three running concurrently.
    Append observations to H; they count toward the token limit.

The curriculum progressively supervises P,P+r, andP+r+s⁺ from the same question–prefix input, teaching the policy to generate the intervention rather than receive it externally. Offline suffix replacement constructs training targets; deployment redirects future generation without rewriting history. We will add a policy-generated trace to distinguish this behavior from the offline teacher-intervention example in Figure 9.

Thank you for the detailed clarification and additional controls. The SFT comparisons and randomized RL controls provide stronger evidence that structured repair supervision and pivot-conditioned updates are useful. The explanation that deployment is append-only, retains the full history, and involves no automatic tool replay also resolves my question about the inference procedure.

My central concern remains whether the contribution relative to the closest existing methods is sufficiently established. The comparison table identifies concrete differences from InT, P-BRIDGE, PivoARL, Search-R2, and ELPO. However, the additional experiments primarily establish the usefulness of the proposed components relative to success-only SFT, ordinary GSPO, and randomized update schemes. Their implications for the claimed distinctions from prior repair and credit-assignment methods remain limited.

For example, outperforming random suffix boundaries supports the value of informed update allocation, but does not establish the advantage of RLOR’s particular allocation rule relative to existing localization-aware approaches. Likewise, the curriculum controls support staged repair learning, while leaving the value of its specific differences from closely related intervention-based training insufficiently isolated. A controlled comparison with a representative close method, or an ablation directly testing a claimed distinguishing choice, would provide stronger evidence on this central issue.

The requested example from the deployed policy also remains outstanding. The response explicitly describes the supplied example as an offline supervision schematic and promises a policy-generated trace. The teacher-assisted intervention diagnostics are informative, but do not directly demonstrate how the trained policy autonomously generates and uses a pivot during an actual rollout. The requested trace would help connect the clarified deployment procedure to the claimed behavior.

I acknowledge that the rebuttal improves the experimental support and resolves the procedural ambiguity. However, my main concern about the distinct contribution remains insufficiently addressed, so I maintain my rating.

Reviewer_foceOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RLOR, a framework that improves long-horizon reasoning in LLM agents via observation-guided trajectory repair. Rather than regenerating full trajectories, RLOR identifies the earliest point of error (the pivot), preserves the preceding valid prefix, and repairs the continuation using observation-based feedback. The training pipeline consists of: (1) a multi-round, teacher-annotated dataset; (2) a three-stage curriculum SFT targeting pivot identification, feedback formulation, and trajectory recovery; and (3) a pivot-aware RL stage employing suffix-focused policy updates alongside prefix KL constraints. Experiments on five challenging benchmarks demonstrate consistent accuracy improvements over standard SFT and GSPO baselines. Furthermore, the authors validate the approach through ablation studies and analyses on inference overhead and pivot annotation consistency.

Critical strengths
  • Well-designed framework targeting error propagation. Addressing the critical issue of error cascading in multi-step reasoning, the paper introduces a clean, unified pipeline that couples deviation localization with observation-guided suffix repair. Preserving valid prefixes rather than discarding entire trajectories is both methodologically sound and practically efficient, striking a compelling balance between repair accuracy and computational overhead.

  • Rigorous evaluation with consistent efficiency and accuracy gains. The empirical validation is comprehensive, spanning five established long-horizon benchmarks under strictly controlled baselines and backbones. The method delivers clear improvements over GSPO while effectively reducing average token counts and tool calls. In addition, the ablation studies—covering curriculum training stages, pivot supervision signals, and inference compute budgets—provide robust evidence for the efficacy of each proposed component.

Critical weaknesses
  • Incomplete specification of the curriculum SFT pipeline. Key implementation and training details for the curriculum SFT stages are missing. Specifically, the paper does not specify the exact composition and scale of the data for each stage, nor does it provide representative samples. Furthermore, the paper lacks explicit transition criteria to determine whether a stage has converged before moving to the next, which poses reproducibility challenges.

  • Missing comparison with test-time self-repair / scaling baselines. The proposed mechanism inherently suggests an inference-time repair paradigm: reviewing a completed roll-out, localizing the deviation, and repairing the suffix on-the-fly. However, the evaluation is restricted to token budgets and Best-of-N sampling without investigating such a self-repair inference baseline. Conducting this analysis would better disentangle whether the observed advantages stem strictly from the specialized training recipe or could be approximated via test-time scaling alone.

Questions for authors
  • Could the authors clarify the precise scale, data composition, and qualitative examples for each stage in the curriculum SFT? Furthermore, was the progression across stages determined by a predefined training schedule or dynamic validation convergence metrics?

  • To help isolate the source of improvement, could the authors benchmark RLOR against an inference-time self-review baseline that detects pivots and regenerates continuations on-the-fly? Demonstrating RLOR's superiority over such a baseline under controlled inference budgets would substantially strengthen the evidence for the proposed training pipeline.

6 comments
AuthorsOpenReview ↗

Response to Reviewer foce - Part 1

For W1&Q1:

Thank you for requesting these details. All stages share the question–prefix input; Stage 3 retains only successful repairs.

StageModel inputSupervised outputSingle-round examplesAdditional later-round examplesMulti-round total
1. Pivot localizationx + hP4,6092,5177,126
2. Pivot-reason generationx + hP + r4,6092,5177,126
3. Continuation repairx + hP + r + s+2,7621,9654,727

Each stage trains for eight epochs (learning rate 10⁻⁵, cosine schedule, warmup ratio 0.05); subsequent stages use the preceding final checkpoint. No validation-based switching or checkpoint selection is used.

This schematic adapts Figure 9 to illustrate the supervision formats; it is neither a serialized training sample nor an autonomous RLOR rollout.

Shared input: question x + retained history h

x:
Which person represented the association acknowledged by the
target early-2020s article on mining and water impacts?

h (abridged):
IMWA has been considered as a candidate association, but the target
article and its acknowledgements have not yet been verified.

P:
<|PIVOT_INTERVENTION|>

r (abridged):
Narrowing the search to IMWA is unsupported. Identify the target
article and use its acknowledgements to determine the association
and its representative.

s+ (abridged):
Assistant: Search for the target article and inspect its acknowledgements.
Tool: ... AWARD ... under the directorship of Sharon Pollard ...
Assistant: Verify the co-author's leadership appointment.
Tool: ... appointed Executive Director of AWARD in August 2023 ...
Assistant: Answer: Sharon Pollard.

Stage 1 target: P
Stage 2 target: P + r
Stage 3 target: P + r + s+

In all three stages, the question and retained prefix are conditioning context and receive no SFT loss. Only target assistant tokens are supervised; tool observations are excluded from the loss. We will add representative serialized training examples for all three stages to the appendix, including their input–target boundaries and token-level loss masks.

AuthorsOpenReview ↗

Response to Reviewer foce - Part 2

For W2&Q2:

We thank the reviewer for suggesting a test-time repair comparison. We added completed-answer self-review and self-localized suffix regeneration across Mono, Stage-SFT, GSPO, and RLOR. All questions follow the prescribed workflow; ground truth is used only for final scoring.

ModelStandard rolloutCompleted-answer self-reviewSelf-localization + suffix regeneration
Mono40.240.843.1
Stage-SFT47.146.547.9
GSPO53.252.655.2
RLOR56.154.558.4

Mono with suffix repair remains below Stage-SFT, while RLOR outperforms GSPO under the same repair workflow and benefits further from suffix regeneration. These results support gains from repair training beyond test-time revision alone, while demonstrating their complementarity.

All methods use a 128k retained-trajectory limit, including observations, review, and regeneration, with 256 assistant turns and three concurrent tool calls. Discarded suffixes are excluded. Unlike post-hoc suffix regeneration, standard RLOR corrects without truncating history.

Thank you for adding the self-repair comparisons. Please report the actual cumulative token consumption of each method, including the initial rollout, review/localization, regeneration, and any subsequently discarded suffixes. A shared retained-trajectory limit does not necessarily ensure comparable total token usage.

Please also include a token-budget-matched majority-voting (self-consistency) baseline, which selects the most frequent final answer among NN independently sampled rollouts. The number of samples NN should be chosen based on measured token usage so that the cumulative token consumption across all samples approximately matches that of RLOR. This would help assess whether RLOR retains its advantage over sampling and voting under a comparable inference token budget.

Reviewer_e4SeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RLOR, a training recipe for agents doing long-horizon tool-augmented reasoning. A teacher model (GPT-5.2) marks the earliest step where a trajectory goes wrong (the “pivot”) and writes a short natural-language reason. The trajectory is then cut there, the reason is inserted, and a repaired continuation is generated, possibly over several rounds.

Critical strengths

The idea of keeping the valid prefix and repairing only from the first real mistake is intuitive, and it’s nice to see it carried all the way from data construction through SFT into the RL credit assignment. The same-backbone comparison is the right setup. GSPO starts from the same Stage-SFT checkpoint, so the RLOR vs. GSPO gap isolates the masking scheme fairly cleanly. I also appreciated that the authors report annotation agreement (Table 3), a trajectory-source ablation (Table 11), and say openly where results are not monotonic.

Critical weaknesses
  1. The gains may come mostly from distilling GPT-5.2 rather than from learning to repair. The pivot reasons can amount to near-complete solution plans (e.g., Figure 9), and the SFT data keeps only teacher-guided continuations that succeed. No baseline controls for this: rejection-sampling SFT, or SFT on teacher-assisted successful trajectories without the pivot structure, is missing. The intervention results in Figure 2 and Table 9 also rely on teacher feedback at test time, and in Table 9 later rounds only revisit questions that are still wrong, which requires the gold answer. These are oracle-assisted upper bounds and should be labeled as such.

  2. The evidence for the RL stage is thin. All results are single runs on small benchmarks: on AMO-Bench, RLOR vs. GSPO is 64 vs. 60, about two problems. In Figure 6, the advantage over GSPO fades or reverses at Bo3/Bo5 on GAIA and HLE, and on GAIA at Bo3 even Mono beats RLOR (90.6 vs. 88.1). The latency comparison in Table 7 is an approximation rather than a measurement, and it’s unclear whether the Reflexion/CRITIC judge sees gold answers. Prior work on RL-trained self-correction (e.g., SCoRe) is not discussed.

Overall, I like the idea and the Bo1 numbers hold up against GSPO, so I’m mildly in favor. The distillation baseline and seed variance are what would actually convince me.

Questions for authors
  1. Can you add a rejection-sampling SFT baseline, or SFT on teacher-assisted successful trajectories without pivot markers or reasons, at the same data size? This would separate the effect of the repair structure from teacher distillation.

  2. Where do the SFT and RL training questions come from, and how many are there? How was disjointness from the five test sets checked, especially for BrowseComp-style questions?

  3. What exactly does the GPT-5.2 judge or teacher see in the Reflexion/CRITIC/Self-Refine baselines? In Table 9, is the decision to run another round based on the gold answer?

  4. Can you report variance over seeds or bootstrap intervals for Table 1, at least for RLOR vs. GSPO?

3 comments
AuthorsOpenReview ↗

Response to Reviewer e4Se - Part 1

For W1 & Q1:

Thank you for highlighting the distillation concern. We added SFT controls to separate teacher supervision from structured repair learning.

Rejection-Sampling SFT retains correct trajectories from five unassisted rollouts per question on the same pool, matching the curriculum’s total training-example presentations. Success-SFT removes all pivot markers and associated reasons from the same successful student repairs. The plain-text control replaces markers with natural language, retaining localization semantics, reasons, continuations, and stage order. Annotation-only trains only Stages 1–2 with GPT-5.2 annotations; Student-Annotation uses Qwen3.5-35B-A3B annotations. Mixture-SFT mixes tasks instead of training stages sequentially.

MethodBrowseCompAMO-BenchSealQA₀HLEGAIAAvg.
Mono32.732.027.140.169.240.2
Rejection-Sampling SFT33.134.029.739.871.541.6
Teacher-assisted Success-SFT33.942.030.640.270.643.5
Plain-text Curriculum-SFT34.244.031.540.371.844.4
Annotation-only SFT (Stages 1+2)34.648.033.239.870.445.2
Student-Annotation Stage-SFT34.238.030.640.271.843.0
RLOR Mixture-SFT33.334.028.539.670.941.3
RLOR Stage-SFT35.454.032.540.573.147.1

Teacher annotations help, but Stage-SFT outperforms Success-SFT and the plain-text control, which retains the same reasons and continuations. Thus, teacher supervision alone does not explain the gains.

 

For W2:

We thank the reviewer for highlighting the need to establish the contribution of the RL objective and clarify its evaluation scope. We added two randomized controls, both sharing the Stage-SFT initialization and common training settings. Random masking selects the same number of eligible assistant tokens per trajectory without replacement; random boundaries uniformly select valid assistant-step boundaries for suffix updates. Unselected assistant tokens receive weak KL, and correct no-pivot trajectories retain full updates.

MethodBrowseCompAMO-BenchSealQA₀HLEGAIAAvg.
GSPO43.660.040.346.875.453.2
Per-trajectory equal-token random mask42.556.038.745.574.751.5
Random suffix boundary44.758.041.446.676.553.4
RLOR46.264.044.747.378.156.1

RLOR outperforms GSPO and both controls, supporting pivot-aware updates beyond token-count reduction or arbitrary suffix selection.

Higher sampling budgets. RLOR aims to internalize correction within a single rollout, improving pass@1 reliability rather than requiring repeated sampling. We acknowledge GAIA/HLE reversals at larger NN and do not claim a higher sampling-based capability ceiling. “BoN” will be renamed pass@N (any-correct).

Feedback access and latency. Reflexion is gold-verifier-gated, unlike standard RLOR (details under Question 3). CRITIC performs search-based verification even for initially correct answers, followed by revision and judging. Self-Refine sequentially performs feedback, refinement, and judging without additional search (Appendix B.2). Standard RLOR deployment does not use reference answers or reference-derived correctness labels to trigger correction.

Relation to SCoRe. SCoRe first trains second-attempt correction while constraining the first response, then jointly optimizes both attempts with an improvement bonus. RLOR corrects within one trajectory, applying reward-driven pivot-and-suffix updates with weak prefix KL and no inter-attempt bonus. Neither uses reference-answer feedback at deployment. We will include a discussion of related work.

AuthorsOpenReview ↗

Response to Reviewer e4Se - Part 2

For Q2:

We thank the reviewer for requesting clearer data provenance and contamination checks.

SFT contains 4,609 / 4,609 / 2,762 single-round stage examples, plus 2,517 / 2,517 / 1,965 later-round examples; these are not independent-question totals. RL uses 17,672 examples from in-house BrowseComp-style queries and subsets of DAPO-Math, NuminaMath-CoT, and HotpotQA training data. SFT and RL question pools are disjoint.

In-house queries are built answer-first from knowledge-graph-style constraints with verified answer uniqueness. Seeds are separate from evaluation targets; test questions, rewrites, and test-directed sources are excluded. We screen all training queries against all five test sets by semantic similarity and manually inspect high-similarity pairs, finding no question-level overlap under this procedure.

 

For Q3:

Test-time comparisons. We distinguish correctness judging from feedback generation. For Reflexion, the correctness judge compares the candidate answer with the gold/reference answer, and the resulting decision determines whether a reflection-and-retry round is run. Gold information therefore affects the inference control flow, rather than being used only for final scoring. We will explicitly label this implementation as reference-answer-verifier-gated. CRITIC and Self-Refine follow the verification, feedback, and revision workflows described in Appendix B.2. In contrast, standard RLOR generates its own pivot, observation-based reason, and continuation from the ongoing interaction history, without reference-answer feedback.

Offline analysis: GPT-5.2 sees the completed trajectory and its incorrect status, but not the answer text, we will label these as teacher-assisted diagnostics. However, Table 9 additionally uses gold answer verification to revisit incorrect questions, explicitly ground-truth-verifier-gated, rather than autonomous deployment. Pass@N only scores whether any generated candidate is correct; it does not imply a gold-blind selector.

 

For Q4:

GSPO/RLOR results in Table 1 average 20 repeated evaluations per fixed checkpoint. Variances and approximate 95% confidence intervals are below.

BenchmarkGSPO mean (variance)RLOR mean (variance)Improvement (variance)Approx. 95% CI of improvement
BrowseComp43.6 (0.43)46.2 (0.77)+2.6 (0.84)[2.17, 3.03]
AMO-Bench60.0 (10.53)64.0 (4.21)+4.0 (13.05)[2.31, 5.69]
SealQA₀40.3 (4.77)44.7 (4.48)+4.4 (5.83)[3.27, 5.53]
HLE46.8 (0.98)47.3 (1.14)+0.5 (0.52)[0.16, 0.84]
GAIA75.4 (4.39)78.1 (3.62)+2.7 (3.83)[1.78, 3.62]
Macro-average53.22 (0.57)56.06 (0.36)+2.84 (0.76)[2.43, 3.25]

RLOR achieves a +2.84-percentage-point average gain, with an approximate 95% CI of [2.43, 3.25], supporting a positive aggregate improvement under repeated evaluation.

Reviewer_e4SeOpenReview ↗

Feedback

Thanks for the response, it cleared up most of my questions. So I’m keeping my score as it is.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.