- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper analyzes two major sources of credit assignment errors in long-horizon agentic RL—history offset and rollout noise. Motivated by the idea of shortening the credit assignment horizon, it proposes FocalRL, which leverages a strong LLM to identify critical errors and generate repair rubrics, then performs short-horizon local rollouts with rubric-based rewards for targeted policy optimization. Experiments on search and coding agent benchmarks demonstrate superior performance over baselines such as GRPO and Tree-GRPO.
Critical strengths
- The paper studies the critical credit assignment problem in long-horizon agentic RL, provides a clear theoretical decomposition of credit assignment noise into history offset and rollout noise, and offers empirical evidence to motivate its method.
- FocalRL introduces a relatively straightforward framework based on LLM-guided error localization, short-horizon local rollouts, and rubric-based rewards. It achieves substantial improvements over GRPO and competitive credit assignment baselines across multiple agentic benchmarks.
- The paper provides extensive evaluations across diverse benchmarks, along with ablation studies and mechanism analyses that investigate the effectiveness of error localization, local rewards, and different locator models.
Critical weaknesses
The proposed method relies on additional LLM inference for online error localization, rubric generation, and local rollout evaluation. In particular, using a powerful model such as GPT-5.6-Sol as the locator may introduce substantial computational overhead and latency, potentially limiting the scalability of RL training. Although the authors adopt asynchronous training and demonstrate that a smaller locator can also be effective, a detailed analysis of wall-clock training time and total computational cost is missing.
Questions for authors
-
Could the authors report the wall-clock training time and total computational cost, including the overhead of LLM-based error localization and reward evaluation?
-
How much of the improvement comes from better credit assignment versus additional LLM supervision? Could the authors evaluate random branching with the same rubric-based rewards?
4 comments
Q2: Isolating the value of error localization with a random-branching ablation
Q2: How much of the improvement comes from better credit assignment versus additional LLM supervision? Could the authors evaluate random branching with the same rubric-based rewards?
Response:
We thank the reviewer for this question. We add a random-branching ablation that changes only the intervention-point selection, keeping the rubric-generation procedure, GPT-5.6-Sol PRM, bounded local rollout horizon, and other training settings unchanged relative to the GPT-5.6-Sol variant. FocalRL with GPT-5.6-Sol and Qwen3.5-9B as both locator and PRM outperforms this random-branching baseline by 7.40 and 3.74 percentage points in macro-average accuracy, respectively.
| Method | BrowseComp (200) | BrowseComp-ZH | xbench-2505 | xbench-2510 | GAIA-text |
|---|---|---|---|---|---|
| SFT + GRPO | 50.5 | 47.1 | 80.0 | 47.0 | 70.9 |
| SFT + FocalRL (random pivot; GPT-5.6-Sol PRM) | 52.0 | 49.8 | 79.0 | 46.0 | 73.8 |
| SFT + FocalRL (Qwen3.5-9B locator and PRM) | 58.5 | 54.0 | 81.0 | 51.0 | 74.8 |
| SFT + FocalRL (GPT-5.6-Sol locator and PRM) | 66.0 | 56.1 | 82.0 | 49.0 | 84.5 |
1. Random branching with rubric-based rewards provides modest gains. This variant outperforms GRPO on three of five benchmarks, improving macro-average accuracy by 1.02 percentage points. Thus, random branching with rubric-based local rewards provides a modest overall benefit, although the gains are inconsistent across benchmarks.
2. Error-based localization provides substantial additional gains under the same reward supervision. With the rubric-generation procedure and GPT-5.6-Sol PRM held fixed, replacing random pivot selection with error-based localization improves performance on all five benchmarks, increasing macro-average accuracy by 7.40 percentage points. This controlled comparison isolates the benefit of selecting consequential errors as intervention points, beyond the benefit of providing LLM-based rubric rewards.
3. The advantage persists with a smaller locator and PRM. FocalRL using Qwen3.5-9B for both roles also outperforms the random-branching variant with the stronger GPT-5.6-Sol PRM on all five benchmarks, with a macro-average gain of 3.74 percentage points. These results show that a stronger reward evaluator alone does not substitute for targeted error localization.
We will include this ablation and clarify its configuration in the revised paper.
Thank you for your response. I will maintain my current score and lean toward acceptance.
Thank you for your thoughtful review and support for our work. We would be happy to address any further questions or concerns.
Q1: Wall-clock training time and total cost including auxiliary-model overhead
Q1: Could the authors report the wall-clock training time and total computational cost, including the overhead of LLM-based error localization and reward evaluation?
Response:
We thank the reviewer for requesting this clarification. Over 80 training steps, FocalRL with Qwen3.5-9B as both locator and PRM takes 40 hours and costs an estimated USD 7,387.20, compared with 32 hours and USD 6,902.63 for GRPO. These estimates include GPU, search, webpage summarization, outcome evaluation, error localization, and process reward evaluation costs.
1. Complete training time and cost breakdown. The following table summarizes the full 80-step training run:
GPU costs are estimated using 16 AMD MI300X GPUs at USD 2.59 per GPU-hour. Search costs are estimated at USD 7 per 1,000 requests. Webpage summarization and outcome evaluation costs use GPT-5.6-Luna rates of USD 0.20 per million input tokens and USD 1.20 per million output tokens. Outcome evaluation costs are estimated using 128 calls per step for FocalRL and 256 for GRPO. Jina Reader is deployed locally, so no external Jina API fees are incurred.
2. Asynchronous execution limits the additional wall-clock time. FocalRL adds error localization, local resampling, and PRM evaluation. Our fully asynchronous RL implementation allows policy rollout workers to continue sampling while locator and PRM requests are processed, so the latencies of these operations do not simply add up. The measured wall-clock training time increases from 32 to 40 hours, a 25% increase.
3. Bounded local rollouts limit search costs and overall overhead. Search API usage accounts for the largest share of training costs. With short local continuations, FocalRL incurs slightly lower search costs than GRPO: USD 5,132.36 versus USD 5,286.20, a reduction of 2.9%. This partially offsets the additional GPU, summarization, and locator/PRM costs.
Including all listed components, FocalRL with Qwen3.5-9B as both locator and PRM increases the estimated total training cost by 7.0%, while improving macro-average accuracy by 4.76 percentage points across five benchmarks (Table 3). We therefore recommend the Qwen3.5-9B configuration for large-scale training. We will include this complete accounting of training time and costs in the revised paper.