acceptodds
Under review as a conference paper at ICLR 2027

Focal Credit Assignment for Ultra Long-Horizon Agentic Reinforcement Learning

14 views

Abstract

Training long-horizon large language model (LLM) agents with outcome-based reinforcement learning poses a credit-assignment challenge. GRPO assigns the same trajectory-level advantage to all actions, despite their different contributions to task success. Through a policy-gradient analysis, we identify two sources of mismatch between this trajectory-level advantage and the ideal action-specific credit: value shifts inherited from preceding history and stochasticity in subsequent interactions. Empirical analysis further reveals that sharp value changes are concentrated at a small fraction of actions. Motivated by these observations, we propose FocalRL, which tackles long-horizon credit assignment through local action repair. Specifically, an LLM-based locator first identifies consequential errors in failed trajectories and generates rubrics specifying the desired corrections. Then, we restore the interaction state before each identified error and sample bounded local continuations, which are evaluated against the repair rubrics. We combine these rubric-scored local groups with outcome-scored full-trajectory groups for policy optimization, providing targeted supervision that mitigates history value offsets and reduces the influence of downstream stochasticity. Experimental results show that FocalRL outperforms GRPO and most related methods under same rollout budget on representative agentic search and coding benchmarks, including BrowseComp, GAIA, and SWE-bench Verified. Notably, our deep research agent built on Qwen3.5-9B achieves 57.9% accuracy on BrowseComp, setting a new state-of-the-art among sub-10B models. Our code is available at https://anonymous.4open.science/r/FocalRL-5DE0/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
credit assignment, agent, large language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rzGqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper analyzes two major sources of credit assignment errors in long-horizon agentic RL—history offset and rollout noise. Motivated by the idea of shortening the credit assignment horizon, it proposes FocalRL, which leverages a strong LLM to identify critical errors and generate repair rubrics, then performs short-horizon local rollouts with rubric-based rewards for targeted policy optimization. Experiments on search and coding agent benchmarks demonstrate superior performance over baselines such as GRPO and Tree-GRPO.

Critical strengths
  1. The paper studies the critical credit assignment problem in long-horizon agentic RL, provides a clear theoretical decomposition of credit assignment noise into history offset and rollout noise, and offers empirical evidence to motivate its method.
  2. FocalRL introduces a relatively straightforward framework based on LLM-guided error localization, short-horizon local rollouts, and rubric-based rewards. It achieves substantial improvements over GRPO and competitive credit assignment baselines across multiple agentic benchmarks.
  3. The paper provides extensive evaluations across diverse benchmarks, along with ablation studies and mechanism analyses that investigate the effectiveness of error localization, local rewards, and different locator models.
Critical weaknesses

The proposed method relies on additional LLM inference for online error localization, rubric generation, and local rollout evaluation. In particular, using a powerful model such as GPT-5.6-Sol as the locator may introduce substantial computational overhead and latency, potentially limiting the scalability of RL training. Although the authors adopt asynchronous training and demonstrate that a smaller locator can also be effective, a detailed analysis of wall-clock training time and total computational cost is missing.

Questions for authors
  1. Could the authors report the wall-clock training time and total computational cost, including the overhead of LLM-based error localization and reward evaluation?

  2. How much of the improvement comes from better credit assignment versus additional LLM supervision? Could the authors evaluate random branching with the same rubric-based rewards?

4 comments
AuthorsOpenReview ↗

Q1: Wall-clock training time and total cost including auxiliary-model overhead

Q1: Could the authors report the wall-clock training time and total computational cost, including the overhead of LLM-based error localization and reward evaluation?

Response:

We thank the reviewer for requesting this clarification. Over 80 training steps, FocalRL with Qwen3.5-9B as both locator and PRM takes 40 hours and costs an estimated USD 7,387.20, compared with 32 hours and USD 6,902.63 for GRPO. These estimates include GPU, search, webpage summarization, outcome evaluation, error localization, and process reward evaluation costs.

1. Complete training time and cost breakdown. The following table summarizes the full 80-step training run:

ConfigurationWall-clock training timeGPU costSearch API costWebpage summarization costOutcome evaluation costLocator/PRM costEstimated total cost
GRPO32 h$1,326.08$5,286.20$284.36$5.99$0.00$6,902.63
FocalRL (Qwen3.5-9B locator and PRM)40 h$1,657.60$5,132.36$404.25$2.99$190.00$7,387.20

GPU costs are estimated using 16 AMD MI300X GPUs at USD 2.59 per GPU-hour. Search costs are estimated at USD 7 per 1,000 requests. Webpage summarization and outcome evaluation costs use GPT-5.6-Luna rates of USD 0.20 per million input tokens and USD 1.20 per million output tokens. Outcome evaluation costs are estimated using 128 calls per step for FocalRL and 256 for GRPO. Jina Reader is deployed locally, so no external Jina API fees are incurred.

2. Asynchronous execution limits the additional wall-clock time. FocalRL adds error localization, local resampling, and PRM evaluation. Our fully asynchronous RL implementation allows policy rollout workers to continue sampling while locator and PRM requests are processed, so the latencies of these operations do not simply add up. The measured wall-clock training time increases from 32 to 40 hours, a 25% increase.

3. Bounded local rollouts limit search costs and overall overhead. Search API usage accounts for the largest share of training costs. With short local continuations, FocalRL incurs slightly lower search costs than GRPO: USD 5,132.36 versus USD 5,286.20, a reduction of 2.9%. This partially offsets the additional GPU, summarization, and locator/PRM costs.

Including all listed components, FocalRL with Qwen3.5-9B as both locator and PRM increases the estimated total training cost by 7.0%, while improving macro-average accuracy by 4.76 percentage points across five benchmarks (Table 3). We therefore recommend the Qwen3.5-9B configuration for large-scale training. We will include this complete accounting of training time and costs in the revised paper.

AuthorsOpenReview ↗

Q2: Isolating the value of error localization with a random-branching ablation

Q2: How much of the improvement comes from better credit assignment versus additional LLM supervision? Could the authors evaluate random branching with the same rubric-based rewards?

Response:

We thank the reviewer for this question. We add a random-branching ablation that changes only the intervention-point selection, keeping the rubric-generation procedure, GPT-5.6-Sol PRM, bounded local rollout horizon, and other training settings unchanged relative to the GPT-5.6-Sol variant. FocalRL with GPT-5.6-Sol and Qwen3.5-9B as both locator and PRM outperforms this random-branching baseline by 7.40 and 3.74 percentage points in macro-average accuracy, respectively.

MethodBrowseComp (200)BrowseComp-ZHxbench-2505xbench-2510GAIA-text
SFT + GRPO50.547.180.047.070.9
SFT + FocalRL (random pivot; GPT-5.6-Sol PRM)52.049.879.046.073.8
SFT + FocalRL (Qwen3.5-9B locator and PRM)58.554.081.051.074.8
SFT + FocalRL (GPT-5.6-Sol locator and PRM)66.056.182.049.084.5

1. Random branching with rubric-based rewards provides modest gains. This variant outperforms GRPO on three of five benchmarks, improving macro-average accuracy by 1.02 percentage points. Thus, random branching with rubric-based local rewards provides a modest overall benefit, although the gains are inconsistent across benchmarks.

2. Error-based localization provides substantial additional gains under the same reward supervision. With the rubric-generation procedure and GPT-5.6-Sol PRM held fixed, replacing random pivot selection with error-based localization improves performance on all five benchmarks, increasing macro-average accuracy by 7.40 percentage points. This controlled comparison isolates the benefit of selecting consequential errors as intervention points, beyond the benefit of providing LLM-based rubric rewards.

3. The advantage persists with a smaller locator and PRM. FocalRL using Qwen3.5-9B for both roles also outperforms the random-branching variant with the stronger GPT-5.6-Sol PRM on all five benchmarks, with a macro-average gain of 3.74 percentage points. These results show that a stronger reward evaluator alone does not substitute for targeted error localization.

We will include this ablation and clarify its configuration in the revised paper.

Thank you for your response. I will maintain my current score and lean toward acceptance.

Thank you for your thoughtful review and support for our work. We would be happy to address any further questions or concerns.

Reviewer_gkGhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses credit assignment for long-horizon LLM agents trained with outcome-based GRPO. It decomposes the trajectory-level advantage into action-specific credit, a history offset, and rollout noise, and argues that value drops concentrate on a few actions. FocalRL uses an LLM locator (GPT-5.6-Sol, given the ground-truth answer and a successful trajectory) to select a pivotal error turn in a failed trajectory and write a two-criterion repair rubric. It restores the pre-error context, samples scored by an LLM PRM against the rubric, and mixes these local groups with standard full-rollout groups in GRPO. Validation covers five deep-research benchmarks and SWE-bench Verified/Lite with Qwen3.5-4B/9B, including 57.9 on BrowseComp for the 9B model, comparisons with GRPO, Tree-GRPO and ARPO, and locator/PRM analyses.

Critical strengths
  1. Controlled empirical gains in a realistic setting. With the same backbone and data, FocalRL improves over GRPO by +8.4 points on average across five deep-research benchmarks and by +5.0 on SWE-bench Verified, with 256K-token contexts and asynchronous RL. This practical value is the main reason I lean toward acceptance.

  2. A pragmatic design for long horizons. Restarting from a shared prefix removes the history offset from the group baseline, and bounded continuations with rubric rewards make branch-based credit assignment affordable at a scale where full MC value estimation is infeasible.

Critical weaknesses
  1. Incremental novelty, with the closest prior work missing. Each component has strong precedents: branching from intermediate states (VinePPO, SPO, TreeRL, Tree-GRPO, ARPO), choosing where to branch (FR3E, AttnRL, GPO), locating erroneous steps (OmegaPRM, pivotal token search), and rubric-based process rewards. Several closely related agent works are not discussed: CSO (arXiv 2602.03412), which locates critical steps in failed agent trajectories and branches from them on the same GAIA-Text and xbench benchmarks; PivoARL (arXiv 2607.03702), which restarts RL from the earliest erroneous turn while reusing the correct prefix. The contribution therefore reads mainly as system integration, and the paper offers limited insight beyond the leaderboard results (e.g., which errors are locally repairable, or how the error distribution evolves during RL).

  2. The source of the gains is unclear. (a) Dependence on external models: the locator (GPT-5.6-Sol) sees the ground truth and a successful trajectory, the PRM is GPT-5.6-Luna, and GPT-5.6-Luna also summarizes pages in the visit tool. With a self-locator, the average gain over GRPO drops from +8.4 to +4.8, and the PRM appears to remain external, so a fully self-contained variant is never tested. (b) Untested mechanism: the best locator hits the pivot exactly only 35.7% of the time, and there is no random-pivot + rubric ablation showing that accurate localization matters. (c) Evaluation: the section 4.3 comparison with Tree-GRPO and ARPO reports training reward rather than held-out performance, results come from single runs on 100–200-item test sets with no variance, and locator/PRM costs are excluded from the matched token budget.

Questions for authors
  1. Algorithm 1 requires a same-task successful reference. How are all-fail groups handled, given that zero-variance groups motivate the method?
  2. What are the locator and PRM calls, tokens and costs per training step?
  3. A fully self-contained variant (the policy acting as both locator and PRM).
  4. A random-pivot (or entropy-pivot) + rubric ablation, to isolate the value of accurate localization.
10 comments
AuthorsOpenReview ↗

W1(a): Methodological novelty and supporting ablations

W1 (a): Incremental novelty, with the closest prior work missing. The contribution therefore reads mainly as system integration.

Response:

We sincerely thank the reviewer for highlighting these related works. We would like to highlight that building effective PRMs for RL is widely recognized challenging, as discussed in DeepSeek-R1 [1] and PURE [2]. Therefore, most existing branching methods rely on outcome rewards, requiring continuations from intermediate states to reach termination before receiving a reward. In contrast, FocalRL uses the error diagnosis itself to define a locally assessable success criterion, decoupling reward assignment from long-horizon task completion. A bounded continuation can therefore receive targeted supervision for correcting the identified error without completing the remaining task.

The FocalRL-Full ablation directly tests the importance of this local supervision design. FocalRL-Full retains the same locator and localization procedure, but replaces bounded local continuations and repair-based process rewards with full continuations and outcome rewards. This reduces performance by 6.6 points. Since error localization and branching are retained, the improvement cannot be attributed to these shared components alone: the result demonstrates the additional benefit of targeted rewarding within a bounded local horizon.

Fig. 5 further examines what makes the local reward informative. Without the repair rubric, the evaluator tends to assign scores of 0.5, providing limited discrimination among continuations. Adding the rubric increases reward–outcome alignment from 0.60 to 0.71 AUC. The rubric is sample-specific: it is generated from a particular error diagnosis and defines what constitutes a valid correction at that point in the trajectory. This turns the diagnosis into an explicit evaluation criterion, enabling the evaluator to distinguish effective repairs before task completion.

Note that although PivotRL [7] also doesn't need outcome rewards. However, it evaluates local actions using predefined matching rules, whereas FocalRL generates sample-specific repair rubrics from error diagnoses to assess corrective progress. For example, its SWE-bench verifier assigns rewards by matching tool-call names (Appendix A.2 of [7]). In our mini-swe-agent setup, all environment interactions use a single Bash tool. A tool-name-only verifier would therefore assign the same reward to all valid Bash calls, providing no signal to distinguish effective corrections from erroneous commands.

CSO [8], highlighted by the reviewer, relies on terminal-success verification to construct training preferences. PivoARL [9] similarly uses discounted outcome rewards for credit assignment.

AuthorsOpenReview ↗

W1(a) (continued): Comparison with related methods and references

W1(a) (continued): Comparison with related methods and references

The following table compares where related methods intervene, how far they continue, and whether scoring those continuations requires terminal rewards.

MethodIntervention pointContinuation horizonNo terminal reward needed to score continuations
VinePPO [3]Each reasoning stepFull rollout✗
GPO [4]MC-based step selectionFull rollout✗
Tree-GRPO [5]Random step selectionFull rollout✗
ARPO [6]Entropy-based step selectionFull rollout✗
PivotRL [7]MC-based step selectionSingle-turn rollout✓
CSO [8]PRM-based step selectionFull rollout✗
PivoARL [9]LLM-based error localizationFull rollout✗
Agent-R [10]LLM-based error localizationFull revision trajectory✗
SPO [11]Segment boundariesFull rollout✗
TreeRL [12]Entropy-based selectionFull rollout✗
FR3E [13]Entropy-based selectionFull rollout✗
AttnRL [14]Attention-based selectionFull rollout✗
OmegaPRM [15]MC-based binary searchFull rollout✗
Pivotal Token Search [16]MC-based binary searchFull rollout✗
FocalRLLLM-based error localizationBounded local rollout✓

References

[1] Guo, D., et al. "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning." arXiv:2501.12948, 2025.

[2] Cheng, J., et al. "Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning." NeurIPS 2025.

[3] Kazemnejad, A., et al. "VinePPO: Refining Credit Assignment in RL Training of LLMs." ICML 2025.

[4] Yu, J., et al. "GPO: Learning from Critical Steps to Improve LLM Reasoning." NeurIPS 2025.

[5] Ji, Y., et al. "Tree Search for LLM Agent Reinforcement Learning." ICLR 2026.

[6] Dong, G., et al. "Agentic Reinforced Policy Optimization." ICLR 2026.

[7] Yi, J., et al. "PivotRL: High Accuracy Agentic Post-Training at Low Compute Cost." arXiv:2603.21383v1, 2026.

[8] Li, M., et al. "Verified Critical Step Optimization for LLM Agents." Findings of ACL 2026.

[9] Guo, W., et al. "Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retry." arXiv:2607.03702, 2026.

[10] Yuan, S., et al. "Agent-R: Training Language Model Agents to Reflect via Iterative Self-Training." arXiv:2501.11425, 2025.

[11] Guo, Y., et al. "Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models." arXiv:2505.23564, 2025.

[12] Hou, Z., et al. "TreeRL: LLM Reinforcement Learning with On-Policy Tree Search." ACL 2025.

[13] Zheng, T., et al. "First Return, Entropy-Eliciting Explore." arXiv:2507.07017, 2025.

[14] Liu, R., et al. "Attention as a Compass: Efficient Exploration for Process-Supervised RL in Reasoning Models." ICLR 2026.

[15] Luo, L., et al. "Improve Mathematical Reasoning in Language Models by Automated Process Supervision." arXiv:2406.06592, 2024.

[16] Abdin, M., et al. "Phi-4 Technical Report." arXiv:2412.08905, 2024.

AuthorsOpenReview ↗

W1(b): Locally repairable error types and shifts in error locations during training

W1(b): The paper offers limited insight beyond the leaderboard results, such as which errors are locally repairable or how the error distribution evolves during RL.

Response:

We thank the reviewer for this suggestion. We supplement the performance results with two analyses of the errors targeted by FocalRL.

1. Error types reveal concrete opportunities for local repair. We inspect 200 diagnosed errors in Deep Research and Software Engineering, with the following distributions:

Deep Research

Error typePercentage
Ignoring counterevidence or useful clues33%
Premature commitment to an unverified candidate or hypothesis28%
Misinterpretation of evidence or factual relationships20%
Misunderstanding of task objectives or constraints19%

Software Engineering

Error typePercentage
Overly broad changes causing regressions41%
Implementation errors or incomplete constraint coverage31%
Incorrect diagnosis or fault localization21%
Disregard of failed validation or weakened validation criteria7%

Across both domains, these error types offer opportunities for local repair: revisiting overlooked evidence, verifying a candidate, correcting an interpretation, rechecking task constraints, narrowing a patch, fixing an implementation defect, revising a diagnosis, or responding appropriately to failed validation. The key observation is that progress toward correcting these errors can be assessed locally, before the entire task is completed. FocalRL turns each error diagnosis into a sample-specific repair rubric that defines what such corrective progress should accomplish.

2. Diagnosed consequential errors shift later in the trajectory during training. We track the selected pivot positions with FocalRL training:

Training stepsMean pivot positionMean trajectory lengthRelative pivot position
0–199.4019.2148.9%
20–3911.6722.9350.9%
40–5921.1435.6259.4%
60–7933.9749.0069.3%

As shown in the table above, the mean pivot position increases from 9.40 to 33.97, while its relative position increases from 48.9% to 69.3%. Thus, the shift is not explained solely by longer trajectories: the diagnosed errors also occur proportionally later. This pattern is consistent with improved early-stage decisions, with the remaining consequential errors shifting toward later stages of task execution.

We will incorporate these analyses into the revised paper to clarify both the local behaviors targeted by FocalRL and how the locations of the targeted errors evolve during training.

Reviewer_cA7wOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FocalRL enhances outcome-oriented GRPO with brief repair sessions that concentrate on the errors made by long-term agents. A locator picks an intervention state using a failed episode, its correct solution, and a good example. This process is followed by crafting a two-part repair rubric. The policy collects continuations from the chosen state and a judge evaluates them based on the absence of the detected mistake and taking a proper corrective step. Group relative rewards are allocated independently for complete episodes and local continuations and both are optimized simultaneously. The work demonstrates superior performance of FocalRL over GRPO in experiments with search and software engineering tasks.

Critical strengths

This research has made considerable engineering efforts towards building a feasible and effective training framework for agentic RL with long horizon, which deserves recognition. In terms of experiment evaluation, the paper covers a variety of search tasks and software engineering tasks. Using the default locator, FocalRL has achieved an average improvement of 8.42 percentage points compared with GRPO with respect to five searching benchmarks. Using the 9B policy backbone model directly as the locator, it has reached a gain of 4.76 points. Moreover, by analyzing the error localization and rubric-based rewards, it has been proved that focal search can benefit reinforcement learning. All of these make this paper convincing with rich empirical validations.

Critical weaknesses
  1. The methodological novelty over closely related approaches is insufficiently established, and the experiments do not adequately isolate the contributions of the proposed components. The basic procedure of FocalRL, consisting of finding critical errors with the help of an LLM, reverting to the respective intermediate state, sampling local continuations, and awarding process-oriented rubric rewards based on an LLM's judgment, seems to be mostly a combination of existing methods without clearly fundamental innovations in credit assignment. Particularly, PivotRL combines pivot selection, local on-policy rollouts, and rewards based on verifiers; Agent-R utilizes error localization via a model in order to create corrective trajectories.

  2. It is yet not clear whether the benefits seen by FocalRL are due to local credit assignment or the use of privileged strong model supervision since it makes use of GPT-5.6-Sol (provides ground truth answers and successful trajectories) as the locator and GPT-5.6-Luna as PRM and web page summarization.
    Whereas the Qwen3.5-9B locator in table 3 outperforms GRPO, what is yet to be determined is whether GPT-5.6-Luna is still being used as the PRM; more so when there is an increase of 7.5 in performance of 58.5 to 66% in BrowseComp-200.

Questions for authors
  1. Figure 4 compares FocalRL with Tree-GRPO and ARPO using training rewards from a 4B model over 32 training steps. However, no held-out benchmark results are provided for these methods. Since improvements in training reward do not necessarily translate into better generalization, could the authors report downstream evaluation results, ideally on BrowseComp or GAIA, under matched training budgets? This would provide stronger evidence that FocalRL improves over existing credit-assignment methods beyond the training distribution.

  2. Table 1 compares FocalRL against results reported by prior work, but the agents may differ substantially in their tool scaffolds, search backends, webpage processing, and inference budgets. Particularly, FocalRL uses GPT-5.6-Luna as the model for webpage summarization, which indicates that the whole system relies on a more powerful model than the baseline despite the 9B policy backbone. Can the authors clarify which baselines, if any, were reproduced in the same setting? What is the interpretation of the sub-10B SOTA when considering the difference in the whole system?

  3. Figure 1(b) reports a score of 66.0% on BrowseComp-200, whereas Table 1 reports 57.9% on the full BrowseComp benchmark. Could the authors clarify how the 200-question subset was selected and whether these results use the same checkpoint and evaluation configuration? Given this substantial difference, could the authors also report the GRPO baseline on the full BrowseComp benchmark to enable a more direct comparison?

9 comments
AuthorsOpenReview ↗

W1: Methodological novelty and supporting ablations

W1: The novelty of FocalRL compared with PivotRL and Agent-R.

Response:

We sincerely thank the reviewer for highlighting these related works. We would like to highlight that building effective PRMs for RL is widely recognized challenging, as discussed in DeepSeek-R1 [1] and PURE [2]. Therefore, most existing branching methods rely on outcome rewards, requiring continuations from intermediate states to reach termination before receiving a reward. In contrast, FocalRL uses the error diagnosis itself to define a locally assessable success criterion, decoupling reward assignment from long-horizon task completion. A bounded continuation can therefore receive targeted supervision for correcting the identified error without completing the remaining task.

The FocalRL-Full ablation directly tests the importance of this local supervision design. FocalRL-Full retains the same locator and localization procedure, but replaces bounded local continuations and repair-based process rewards with full continuations and outcome rewards. This reduces performance by 6.6 points. Since error localization and branching are retained, the improvement cannot be attributed to these shared components alone: the result demonstrates the additional benefit of targeted rewarding within a bounded local horizon.

Fig. 5 further examines what makes the local reward informative. Without the repair rubric, the evaluator tends to assign scores of 0.5, providing limited discrimination among continuations. Adding the rubric increases reward–outcome alignment from 0.60 to 0.71 AUC. The rubric is sample-specific: it is generated from a particular error diagnosis and defines what constitutes a valid correction at that point in the trajectory. This turns the diagnosis into an explicit evaluation criterion, enabling the evaluator to distinguish effective repairs before task completion.

Compared with PivotRL [7], which requires additional MC sampling to select informative pivots, FocalRL uses an LLM-based comparison of successful and failed trajectories to locate consequential errors, providing a more lightweight and scalable alternative. Moreover, PivotRL evaluates local actions using predefined matching rules, whereas FocalRL generates sample-specific repair rubrics from error diagnoses to assess corrective progress. For example, its SWE-bench verifier assigns rewards by matching tool-call names (Appendix A.2 of [7]). In our mini-swe-agent setup, all environment interactions use a single Bash tool. A tool-name-only verifier would therefore assign the same reward to all valid Bash calls, providing no signal to distinguish effective corrections from erroneous commands.

Agent-R uses outcome-based filtering of MCTS-generated trajectories to construct revision examples for SFT only. FocalRL instead turns error diagnoses into repair criteria that provide targeted process rewards for RL, allowing bounded local continuations to receive supervision for corrective progress before task completion.

We provide a broader comparison with related methods and the complete references in the accompanying response, “W1 (continued): Comparison with related methods.”

AuthorsOpenReview ↗

W1 (continued): Comparison with related methods and references

W1 (continued): Comparison with related methods

To complement our response on methodological novelty, the following table compares intervention-point selection, continuation horizons, and whether scoring continuations requires terminal rewards.

MethodIntervention pointContinuation horizonNo terminal reward needed to score continuations
VinePPO [3]Each reasoning stepFull rollout✗
GPO [4]MC-based step selectionFull rollout✗
Tree-GRPO [5]Random step selectionFull rollout✗
ARPO [6]Entropy-based step selectionFull rollout✗
PivotRL [7]MC-based step selectionSingle-turn rollout✓
CSO [8]PRM-based step selectionFull rollout✗
PivoARL [9]LLM-based error localizationFull rollout✗
Agent-R [10]LLM-based error localizationFull rollout✗
SPO [11]Segment boundariesFull rollout✗
TreeRL [12]Entropy-based selectionFull rollout✗
FR3E [13]Entropy-based selectionFull rollout✗
AttnRL [14]Attention-based selectionFull rollout✗
OmegaPRM [15]MC-based binary searchFull rollout✗
Pivotal Token Search [16]MC-based binary searchFull rollout✗
FocalRLLLM-based error localizationBounded local rollout✓

References

[1] Guo, D., et al. "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning." arXiv:2501.12948, 2025.

[2] Cheng, J., et al. "Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning." NeurIPS 2025.

[3] Kazemnejad, A., et al. "VinePPO: Refining Credit Assignment in RL Training of LLMs." ICML 2025.

[4] Yu, J., et al. "GPO: Learning from Critical Steps to Improve LLM Reasoning." NeurIPS 2025.

[5] Ji, Y., et al. "Tree Search for LLM Agent Reinforcement Learning." ICLR 2026.

[6] Dong, G., et al. "Agentic Reinforced Policy Optimization." ICLR 2026.

[7] Yi, J., et al. "PivotRL: High Accuracy Agentic Post-Training at Low Compute Cost." arXiv:2603.21383v1, 2026.

[8] Li, M., et al. "Verified Critical Step Optimization for LLM Agents." Findings of ACL 2026.

[9] Guo, W., et al. "Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retry." arXiv:2607.03702, 2026.

[10] Yuan, S., et al. "Agent-R: Training Language Model Agents to Reflect via Iterative Self-Training." arXiv:2501.11425, 2025.

[11] Guo, Y., et al. "Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models." arXiv:2505.23564, 2025.

[12] Hou, Z., et al. "TreeRL: LLM Reinforcement Learning with On-Policy Tree Search." ACL 2025.

[13] Zheng, T., et al. "First Return, Entropy-Eliciting Explore." arXiv:2507.07017, 2025.

[14] Liu, R., et al. "Attention as a Compass: Efficient Exploration for Process-Supervised RL in Reasoning Models." ICLR 2026.

[15] Luo, L., et al. "Improve Mathematical Reasoning in Language Models by Automated Process Supervision." arXiv:2406.06592, 2024.

[16] Abdin, M., et al. "Phi-4 Technical Report." arXiv:2412.08905, 2024.

AuthorsOpenReview ↗

W2: Separating the benefits of error localization from stronger model supervision

W2: It is yet not clear whether the benefits seen by FocalRL are due to local credit assignment or the use of privileged strong model supervision by GPT-5.6-sol or GPT-5.6-Luna. Whether GPT-5.6-Luna is used as the PRM?

Response:

We thank the reviewer for raising this question. We clarify four points below.

1. GPT-5.6-Luna is used as the PRM only in the rubric analysis in Fig. 5. For the RL training runs in Table 3, FocalRL uses Qwen3.5-9B as both the locator and PRM, whereas FocalRL† uses GPT-5.6-Sol for both. The increase from 58.5 to 66.0 on BrowseComp-200 therefore corresponds to replacing both components, rather than the locator alone. We will make these configurations explicit in the revised paper.

2. Error localization with reference information enables supervision from the 9B model itself. As discussed in Section 4.4 and supported by Table 4, access to a successful reference and the ground-truth answer makes error localization more tractable than generating a successful trajectory from scratch. This enables Qwen3.5-9B to identify consequential errors and provide useful supervision for its own improvement. In Table 3, FocalRL uses Qwen3.5-9B as both the locator and PRM, and consistently outperforms GRPO across all five benchmarks, with an average improvement of 4.76 percentage points. These results demonstrate that FocalRL's gains do not require a stronger external locator or PRM.

3. Error-based localization provides additional gains under matched reward supervision. Table 4 shows that LLM-based locators identify more consequential errors than random selection. To test whether this translates into better training outcomes, we add a random-branching ablation that changes only the intervention-point selection, keeping the rubric-generation procedure, GPT-5.6-Sol PRM, bounded local rollout horizon, and other training settings unchanged relative to the GPT-5.6-Sol variant.

MethodBrowseComp (200)BrowseComp-ZHxbench-2505xbench-2510GAIA-text
SFT + GRPO50.547.180.047.070.9
SFT + FocalRL (random pivot; GPT-5.6-Sol PRM)52.049.879.046.073.8
SFT + FocalRL (Qwen3.5-9B locator and PRM)58.554.081.051.074.8
SFT + FocalRL (GPT-5.6-Sol locator and PRM)66.056.182.049.084.5

With the same GPT-5.6-Sol PRM and rubric-generation procedure, error-based localization outperforms random branching on all five benchmarks, with a macro-average gain of 7.40 percentage points. Moreover, using Qwen3.5-9B for both the locator and PRM still outperforms random branching with the stronger GPT-5.6-Sol PRM by 3.74 percentage points. These results show that the gains cannot be explained by access to a stronger reward evaluator alone: selecting consequential errors makes this supervision more effective.

4. The locator and PRM provide training-time supervision, without participating in inference. At evaluation time, the trained 9B policy can perform the task without assistance from locators or PRMs (We also show in Q2 that Qwen3.5-9B is effective as the summary model). Their role is analogous to other training-time supervision sources, such as ground-truth answers in RLVR or teacher models in building SFT data. Most baselines in Table 1 also use teacher models to construct training data. For example, MiroThinker-v1.0 uses GPT-OSS and DeepSeek-V3.1 for trajectory synthesis, while Marco-DeepResearch uses Qwen3.5-Plus, GLM-5, and Kimi-K2. External model supervision is also used during RL: WebExplorer employs DeepSeek-V3 as an outcome judge, and Marco-DeepResearch uses a two-stage reward evaluation pipeline with Qwen-Turbo-Latest and GPT-4.1. The key methodological distinction lies in how this supervision is used. FocalRL uses it to locate consequential errors and define sample-specific repair rubric, while the policy itself generates the local continuations used for on-policy RL.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Reviewers,

Thank you for your thoughtful feedback and constructive discussion. We believe our responses and additional experiments have addressed all the concerns raised, including methodological novelty, the role of stronger-model supervision, baseline comparability, component contributions, and training costs. We have added held-out comparisons with Tree-GRPO and ARPO, a random-pivot ablation, a 9B-only locator/PRM configuration, controlled scaffold evaluations, and complete cost accounting. We appreciate your acknowledgment of these additions and your positive assessments.

The distinctive contribution of FocalRL is turning error diagnoses into sample-specific repair rubrics, enabling PRMs to provide targeted credit assignment for long-horizon agentic RL. This allows short local continuations to receive meaningful rewards for corrective progress before the full task is completed. Beyond this methodological contribution, training on publicly available data yields 57.9% accuracy on the full BrowseComp benchmark, setting a new state of the art among agents with sub-10B policy models, alongside strong results on other search and coding benchmarks.

As the discussion period concludes, we would be grateful if you would consider raising your scores in light of the additional evidence and the concerns addressed. Thank you again for the time and care you have devoted to reviewing our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.