- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
To better evaluate step-level values of the whole trajectory in long-horizon LLM agent tasks, this paper introduced a method, Beyond-Local Policy Optimization (BLPO), which estimates task-progress advantages from experience shared across tasks and training updates. Considering the differences in task difficulty, this paper also utilized a context-balanced global advantage estimator to generate more reliable estimation results. Experiments on ALFWorld and WebShop with 1.5B and 7B show that BLPO outperforms other baselines, proving its effectiveness.
Critical strengths
- This paper proposes a novel method BLPO to achieve finer-grained evaluation for long-horizon LLM agent tasks. The idea is interesting and well-motivated.
- This paper conducts extensive experiments on two benchmarks under different model scales, which clearly demonstrate its effectiveness.
Critical weaknesses
- It would be better if the authors added some figures to illustrate the details in Section 4, which would make the paper easier to follow.
- The gains may largely stem from domain priors in the hand-crafted abstraction, and no baseline uses the same priors.
- In Section 6, the authors mentioned they used environment-specific rules. This makes it challenging to extend BLPO to other long-horizon LLM agent tasks, thereby limiting its generalizability.
- Tables 2 and 4 do not report the number of independent training runs or variability, leaving the robustness of the ablation results unclear. In particular, component removal in Table 2 causes substantially larger drops in Out-Success than in In-Success. Please verify this pattern across independent training runs.
Questions for authors
As mentioned in Section 6, how would you improve the generalizability of BLPO? Also, how can the quality of environment-specific rules be evaluated? In particular, how sensitive is BLPO's performance to different rule designs?
3 comments
Response to Reviewer acFn (Part2)
To W3 and Q1: Generalizability, rule quality, and sensitivity to rule design
Thank you for your constructive comments. BLPO is suited to long-horizon tasks where interaction feedback helps identify local progress and decisions across tasks serve similar functions. Action feedback in ALFWorld and public page states in WebShop provide this information. Within each environment, the same rules apply automatically across task instances; transfer requires adapting the progress and action representations while retaining the credit estimator.
Potential transfer settings include embodied navigation and manipulation, web and desktop tasks, workflows involving multiple tools, code debugging, and data processing, among others. BLPO does not require predefined complete solution paths: the global branch can use reliably aligned local decisions, while other decisions retain trajectory and local credit. Future work could explore using LLMs to extract progress and action functions from histories available before each decision, or training representation models for experience alignment, extending the framework to more open-ended tasks.
For rule quality, Figure 3(b) compares preferences estimated from other tasks with those in held-out tasks across five folds. The highest-confidence group has higher agreement than the lowest in all five. We further inspected 200 randomly sampled pairs of decisions from different tasks assigned to the same group: 96% had corresponding goal-relative action functions.
For the effect of rule design on performance, Table 2 replaces action functions with concrete action labels while retaining progress states and global memory. We also measured coverage across tasks: the fraction of decisions whose groups contain at least two tasks within an update.
| Environment and metric | Full BLPO | w/o Act. abstr. | Coverage with functional grouping | Coverage after subdivision by concrete action |
|---|---|---|---|---|
| ALFWorld Out-Success | 91.54% | 82.94% | 74.84% | 21.50% |
| WebShop Task Success | 74.02% | 71.22% | 98.33% | 45.15% |
Concrete action labels reduce both coverage and success rates, supporting functional grouping for experience reuse.
For sensitivity to rule refinement, we retained the original progress states and action functions, then split existing groups by the number of prior occurrences of the current page observation within the same trajectory. We recomputed credit on the same WebShop records:
| Added condition: prior occurrences of the current page observation | Global credit sign agreement | Fused step credit sign agreement | Global credit magnitude change | Fused step credit magnitude change |
|---|---|---|---|---|
| 0 / at least 1 | 90.75% | 90.83% | 51.42% | 33.78% |
| 0 / 1 / at least 2 | 82.24% | 83.03% | 70.11% | 46.80% |
Statistics use decisions with originally nonzero global credit. Sign agreement requires nonzero credit in both settings; magnitude change is the summed absolute change divided by the original summed absolute credit. Most credit directions remain unchanged under both refinements, indicating some directional stability, while fusion reduces magnitude changes.
Response to Reviewer acFn (Part3)
To W4: Reliability of the ablations and the larger drops in Out-Success
Thank you for your helpful comments. Long-horizon agentic RL requires repeated trajectory generation, multiple rounds of environment interaction, and policy updates, making repeated full training costly. Related work such as GiGPO and HGPO therefore uses three-seed evaluation after one training run, without three independent training runs. We follow the same protocol: each configuration in Tables 2 and 4 is trained once, and the final policy is evaluated with three seeds. We allocate the training budget to comparisons across two environments, two model scales, and key mechanisms, covering 18 training configurations.
For evaluation variability, we supplement the ALFWorld ablations in Table 2 with sample standard deviations across three evaluation seeds, preserving the original means:
| Setting | In-Success | Out-Success |
|---|---|---|
| Full BLPO | 94.01 ± 0.23 | 91.54 ± 0.60 |
| w/o Act. pool. | 91.34 ± 0.11 | 83.14 ± 0.23 |
| w/o Task-Progress | 92.25 ± 1.11 | 83.53 ± 2.22 |
| w/o Act. abstr. | 88.80 ± 1.11 | 82.94 ± 0.30 |
| w/o Residual | 88.41 ± 2.22 | 81.97 ± 1.66 |
| w/o EMA | 90.89 ± 2.00 | 84.11 ± 1.76 |
| w/o Conf. | 89.97 ± 0.74 | 81.05 ± 2.77 |
For the larger Out-Success drops, a plausible explanation is that unfamiliar situations rely more on transferable experience and reliable credit estimation. In Table 2, removing residualization or confidence adjustment causes larger Out-Success drops, supporting the roles of context adjustment and noise control. Table 4 provides additional evidence: restricting experience to the current task reduces Out-Success by 9.57 percentage points, compared with 6.38 for In-Success, further supporting the value of experience from other tasks in unfamiliar situations.
Response to Reviewer acFn (Part1)
To W1: Illustrations and explanations in Section 4
Thank you for your helpful suggestions. Figure 1 illustrates BLPO’s overall workflow, from grouping experience across tasks to credit estimation and policy updates. For example, the mug and apple tasks show how decisions involving different objects can share statistics when their progress states and action functions correspond. We will refine the caption and add visual panels for key modules to clarify how experience is grouped, aggregated, and converted into credit, and how these steps correspond to Section 4.
To W2: Whether the gains primarily come from domain priors in the abstraction
Thank you for your valuable feedback. Local comparison groups in existing agentic RL provide limited experience to support reliable step-level credit estimation. BLPO therefore reuses experience across tasks and training updates to improve credit assignment. We first designed a grouping method based on task-progress states and action functions to align comparable decisions across tasks. We then estimate credit from this grouped experience through context adjustment, balanced aggregation, cross-update accumulation, and reliability scaling. The compared baselines do not use this form of experience alignment and reuse across tasks; grouping is thus an integral part of our method.
To distinguish the contributions of the abstraction rules and the mechanisms for using the grouped experience, Tables 2 and 4 keep the rules fixed while varying either the scope of experience sharing or the credit estimation mechanism. The mean success rates with Qwen2.5-1.5B are shown below:
Restricting sharing alone reduces the two success rates by 9.57 and 7.03 percentage points, supporting the additional value of experience from other tasks. Removing residualization, EMA, or confidence adjustment while retaining the same rules also lowers success in both environments. These controls with fixed priors support the contributions of both experience reuse across tasks and the mechanisms used to estimate credit.
Moreover, these gains come with little additional computational cost. In Table 3, BLPO’s advantage computation accounts for only 0.61% of total training update time. These computations are confined to training, so BLPO introduces no additional inference overhead.