acceptodds
Under review as a conference paper at ICLR 2027

BLPO: Beyond-Local Policy Optimization for Long-Horizon LLM Agents

Abstract

Long-horizon tasks require large language model (LLM) agents to coordinate interdependent actions, a capability central to automating complex workflows. Stepwise group-based reinforcement learning provides finer credit than trajectory-level supervision, but reliable comparisons must account for differences in interaction history. Enforcing this consistency within the limited rollouts of each task can leave insufficient support for estimating action preferences. We propose Beyond-Local Policy Optimization (BLPO), which estimates task-progress advantages from experience shared across tasks and training updates. We summarize trajectory prefixes into task-progress states and abstract actions by their function, allowing corresponding decisions from different tasks to contribute to shared estimates. To control bias and noise from heterogeneous tasks, we accumulate context-relative returns in a global memory with balanced aggregation and reliability weighting. These global advantages complement local comparisons over normalized actions and trajectory-level credit without a learned critic or additional rollouts. Experiments on ALFWorld and WebShop with 1.5B and 7B policies show improved task success over the evaluated baselines under matched training budgets. Further analyses show faster learning and shorter trajectories with little computational overhead.

Primary area:
reinforcement learning
Keywords:
agentic reinforcement learning, LLM agents, long-horizon tasks, credit assignment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_acFnOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

To better evaluate step-level values of the whole trajectory in long-horizon LLM agent tasks, this paper introduced a method, Beyond-Local Policy Optimization (BLPO), which estimates task-progress advantages from experience shared across tasks and training updates. Considering the differences in task difficulty, this paper also utilized a context-balanced global advantage estimator to generate more reliable estimation results. Experiments on ALFWorld and WebShop with 1.5B and 7B show that BLPO outperforms other baselines, proving its effectiveness.

Critical strengths
  1. This paper proposes a novel method BLPO to achieve finer-grained evaluation for long-horizon LLM agent tasks. The idea is interesting and well-motivated.
  2. This paper conducts extensive experiments on two benchmarks under different model scales, which clearly demonstrate its effectiveness.
Critical weaknesses
  1. It would be better if the authors added some figures to illustrate the details in Section 4, which would make the paper easier to follow.
  2. The gains may largely stem from domain priors in the hand-crafted abstraction, and no baseline uses the same priors.
  3. In Section 6, the authors mentioned they used environment-specific rules. This makes it challenging to extend BLPO to other long-horizon LLM agent tasks, thereby limiting its generalizability.
  4. Tables 2 and 4 do not report the number of independent training runs or variability, leaving the robustness of the ablation results unclear. In particular, component removal in Table 2 causes substantially larger drops in Out-Success than in In-Success. Please verify this pattern across independent training runs.
Questions for authors

As mentioned in Section 6, how would you improve the generalizability of BLPO? Also, how can the quality of environment-specific rules be evaluated? In particular, how sensitive is BLPO's performance to different rule designs?

3 comments
AuthorsOpenReview ↗

Response to Reviewer acFn (Part1)

To W1: Illustrations and explanations in Section 4

Thank you for your helpful suggestions. Figure 1 illustrates BLPO’s overall workflow, from grouping experience across tasks to credit estimation and policy updates. For example, the mug and apple tasks show how decisions involving different objects can share statistics when their progress states and action functions correspond. We will refine the caption and add visual panels for key modules to clarify how experience is grouped, aggregated, and converted into credit, and how these steps correspond to Section 4.

To W2: Whether the gains primarily come from domain priors in the abstraction

Thank you for your valuable feedback. Local comparison groups in existing agentic RL provide limited experience to support reliable step-level credit estimation. BLPO therefore reuses experience across tasks and training updates to improve credit assignment. We first designed a grouping method based on task-progress states and action functions to align comparable decisions across tasks. We then estimate credit from this grouped experience through context adjustment, balanced aggregation, cross-update accumulation, and reliability scaling. The compared baselines do not use this form of experience alignment and reuse across tasks; grouping is thus an integral part of our method.

To distinguish the contributions of the abstraction rules and the mechanisms for using the grouped experience, Tables 2 and 4 keep the rules fixed while varying either the scope of experience sharing or the credit estimation mechanism. The mean success rates with Qwen2.5-1.5B are shown below:

SettingModificationALFWorld Out-SuccessWebShop Task Success
Full BLPOComplete sharing and estimation mechanisms91.5474.02
Same-task-onlyRestrict experience to the current task81.9766.99
w/o ResidualRemove residualization from global estimation81.9773.24
w/o EMAUse only experience from the current batch84.1169.92
w/o Conf.Remove confidence adjustment81.0571.09

Restricting sharing alone reduces the two success rates by 9.57 and 7.03 percentage points, supporting the additional value of experience from other tasks. Removing residualization, EMA, or confidence adjustment while retaining the same rules also lowers success in both environments. These controls with fixed priors support the contributions of both experience reuse across tasks and the mechanisms used to estimate credit.

Moreover, these gains come with little additional computational cost. In Table 3, BLPO’s advantage computation accounts for only 0.61% of total training update time. These computations are confined to training, so BLPO introduces no additional inference overhead.

AuthorsOpenReview ↗

Response to Reviewer acFn (Part2)

To W3 and Q1: Generalizability, rule quality, and sensitivity to rule design

Thank you for your constructive comments. BLPO is suited to long-horizon tasks where interaction feedback helps identify local progress and decisions across tasks serve similar functions. Action feedback in ALFWorld and public page states in WebShop provide this information. Within each environment, the same rules apply automatically across task instances; transfer requires adapting the progress and action representations while retaining the credit estimator.

Potential transfer settings include embodied navigation and manipulation, web and desktop tasks, workflows involving multiple tools, code debugging, and data processing, among others. BLPO does not require predefined complete solution paths: the global branch can use reliably aligned local decisions, while other decisions retain trajectory and local credit. Future work could explore using LLMs to extract progress and action functions from histories available before each decision, or training representation models for experience alignment, extending the framework to more open-ended tasks.

For rule quality, Figure 3(b) compares preferences estimated from other tasks with those in held-out tasks across five folds. The highest-confidence group has higher agreement than the lowest in all five. We further inspected 200 randomly sampled pairs of decisions from different tasks assigned to the same group: 96% had corresponding goal-relative action functions.

For the effect of rule design on performance, Table 2 replaces action functions with concrete action labels while retaining progress states and global memory. We also measured coverage across tasks: the fraction of decisions whose groups contain at least two tasks within an update.

Environment and metricFull BLPOw/o Act. abstr.Coverage with functional groupingCoverage after subdivision by concrete action
ALFWorld Out-Success91.54%82.94%74.84%21.50%
WebShop Task Success74.02%71.22%98.33%45.15%

Concrete action labels reduce both coverage and success rates, supporting functional grouping for experience reuse.

For sensitivity to rule refinement, we retained the original progress states and action functions, then split existing groups by the number of prior occurrences of the current page observation within the same trajectory. We recomputed credit on the same WebShop records:

Added condition: prior occurrences of the current page observationGlobal credit sign agreementFused step credit sign agreementGlobal credit magnitude changeFused step credit magnitude change
0 / at least 190.75%90.83%51.42%33.78%
0 / 1 / at least 282.24%83.03%70.11%46.80%

Statistics use decisions with originally nonzero global credit. Sign agreement requires nonzero credit in both settings; magnitude change is the summed absolute change divided by the original summed absolute credit. Most credit directions remain unchanged under both refinements, indicating some directional stability, while fusion reduces magnitude changes.

AuthorsOpenReview ↗

Response to Reviewer acFn (Part3)

To W4: Reliability of the ablations and the larger drops in Out-Success

Thank you for your helpful comments. Long-horizon agentic RL requires repeated trajectory generation, multiple rounds of environment interaction, and policy updates, making repeated full training costly. Related work such as GiGPO and HGPO therefore uses three-seed evaluation after one training run, without three independent training runs. We follow the same protocol: each configuration in Tables 2 and 4 is trained once, and the final policy is evaluated with three seeds. We allocate the training budget to comparisons across two environments, two model scales, and key mechanisms, covering 18 training configurations.

For evaluation variability, we supplement the ALFWorld ablations in Table 2 with sample standard deviations across three evaluation seeds, preserving the original means:

SettingIn-SuccessOut-Success
Full BLPO94.01 ± 0.2391.54 ± 0.60
w/o Act. pool.91.34 ± 0.1183.14 ± 0.23
w/o Task-Progress92.25 ± 1.1183.53 ± 2.22
w/o Act. abstr.88.80 ± 1.1182.94 ± 0.30
w/o Residual88.41 ± 2.2281.97 ± 1.66
w/o EMA90.89 ± 2.0084.11 ± 1.76
w/o Conf.89.97 ± 0.7481.05 ± 2.77

For the larger Out-Success drops, a plausible explanation is that unfamiliar situations rely more on transferable experience and reliable credit estimation. In Table 2, removing residualization or confidence adjustment causes larger Out-Success drops, supporting the roles of context adjustment and noise control. Table 4 provides additional evidence: restricting experience to the current task reduces Out-Success by 9.57 percentage points, compared with 6.38 for In-Success, further supporting the value of experience from other tasks in unfamiliar situations.

Reviewer_LvzhOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Beyond-Local Policy Optimization (BLPO) to address the problem of limited local rollouts and unreliable action-level credit assignment in RL for long-horizon LLM agents. BLPO uses predefined environment-specific rules to map interaction histories and concrete actions from different tasks into shared task-progress states and abstract actions, allowing experience to be shared across tasks and training updates. The authors propose a global advantage estimation method. First, returns are centered within each context to reduce bias caused by differences in task difficulty. Then, context-balanced aggregation and EMA are used to maintain a global statistical memory. Finally, the global advantages are adjusted based on effective sample support and estimation uncertainty. Experiments are conducted on two long-horizon agent benchmarks ALFWorld and WebShop using Qwen2.5-1.5B and 7B models.

Critical strengths
  1. This paper extends credit assignment from within-task comparisons to shared experience across tasks and training updates. By aligning task-progress states and action functions, decisions from different tasks can provide useful supervision for each other. This approach goes beyond the limitations of forming comparison groups based only on concrete observations or local interaction histories.

  2. BLPO considers several challenges in sharing experience across tasks, including differences in task difficulty, imbalanced context visit frequencies, and estimation uncertainty. To address these issues, the authors design residual centering, context-balanced aggregation, and reliability-based weighting mechanisms.

Critical weaknesses
  1. A key part of BLPO is mapping concrete interaction histories and actions into shared task-progress states (zz) and action functions (uu). However, both mappings rely on predefined environment-specific rules rather than being learned automatically. ALFWorld and WebShop have relatively clear task structures and action semantics, making it easier to design such abstractions. It remains unclear whether this method can work well in more open-ended real-world agent scenarios, where task structures are less clear and subgoals are harder to define.

  2. Although BLPO achieves good overall results in Table 1, the improvements over the strongest baseline, HGPO, are relatively small on some key metrics.

Questions for authors

Could the authors provide more details on how the abstraction rules for task-progress states and action functions are designed, and how the correctness of cross-task alignment is verified? In particular, can the execution state (BB), which is used to construct the abstract state (zz), be fully obtained from the observable interaction history before the current decision? Or does it rely on the environment's internal ground-truth state or additional task completion information? Also, were these rules fixed before the experiments, or were they adjusted based on the evaluation tasks or experimental results?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Lvzh (Part1)

To W1: Environment-level abstraction and applicability to open-ended tasks

Thank you for your valuable comments. BLPO is suited to long-horizon tasks where interaction feedback helps identify local progress and decisions across tasks serve similar functions. Within each environment, the same rules are applied automatically across task instances. Context adjustment, balanced aggregation, and reliability scaling then estimate credit from the shared return statistics.

These rules provide a basis for aligning decisions across tasks. To test the additional benefit of sharing experience across those tasks, Table 4 keeps the rules and estimator unchanged and restricts sharing to the current task:

SettingALFWorld Out-SuccessWebShop Task Success
Full BLPO91.5474.02
Same-task-only81.9766.99

Sharing experience across tasks improves the two metrics by 9.57 and 7.03 percentage points, respectively. With identical grouping rules, these gains directly support the contribution of cross-task experience reuse.

For applicability to more open-ended tasks, BLPO does not require predefined complete solution paths. The global branch can use reliably aligned local decisions, while other decisions retain trajectory and local credit. Potential transfer settings therefore include embodied navigation and manipulation, web and desktop tasks, workflows involving multiple tools, code debugging, and data processing, among others. Sensory and execution feedback, interface changes, tool outputs, and test or validation results can provide a basis for aligning local experience in these settings. The same credit estimator can be retained by adapting the progress and action representations. Future work could also explore using LLMs to extract progress and action functions from interaction histories available before each decision, or learning representations for experience alignment, extending shared credit estimation to more open-ended tasks.

AuthorsOpenReview ↗

Response to Reviewer Lvzh (Part2)

To W2: Improvements over strong baselines and their practical value

Thank you for your helpful feedback. On 7B WebShop, HGPO improves over GiGPO by 0.91 percentage points, while BLPO further improves over HGPO by 0.92 percentage points, yielding comparable gains. These successive modest improvements may also reflect a bottleneck to further progress on WebShop at this level of performance.

Beyond comparisons with existing methods, controlled experiments further examine the contribution of cross-task experience reuse. In the Table 4 comparison above, restricting sharing while keeping the rules and estimator unchanged reduces ALFWorld Out-Success and WebShop Task Success by 9.57 and 7.03 percentage points, respectively. Removing task-progress credit in Table 2 also reduces these metrics by 8.01 and 5.53 percentage points. Together, these results support the practical value of using experience across tasks for credit assignment.

These gains also come with little additional computational cost. In Table 3, advantage computation accounts for only 0.61% of BLPO’s total training update time. This computation is confined to training and adds no inference overhead.

AuthorsOpenReview ↗

Response to Reviewer Lvzh (Part3)

To Q1: Rule construction, alignment verification, and information sources

Thank you for your constructive suggestions. Section 4.3 describes how progress states and action functions are constructed: given the task requirements, we track the completion of relevant conditions using interaction history and execution feedback available before the decision, then identify each action’s function relative to the current progress. Decisions share statistics only when both progress states and action functions correspond. The following WebShop examples illustrate this grouping:

Task requirementCurrent progressActionAction function
Buy a blue shirtItem page; required option unmetSelect blueSatisfy a required option
Buy a size-L coatItem page; required option unmetSelect size LSatisfy a required option
Buy a blue shirtBlue already selectedClick blue againRepeat a selection

The first two decisions involve different products and options but both satisfy an unmet purchase condition, allowing them to share statistics. The third repeats an existing selection and is treated separately.

For alignment quality, Figure 3(b) examines preference consistency across tasks through a five-fold held-out-task test: preferences estimated from four folds are compared with those in the remaining fold. The highest-confidence group achieves greater directional agreement than the lowest-confidence group in all five folds. Building on this analysis, we further examined the semantic quality of the grouping by inspecting 200 randomly sampled pairs of decisions from different tasks assigned to the same group. In 96% of these pairs, the goal-relative action functions correspond.

For the information sources of B_t, we use no privileged or hidden environment information unavailable to the baseline agents. All inputs are available before the current decision: task goals, prior actions, and execution feedback in ALFWorld; task instructions, observations, and public page states such as selected options in WebShop. Final success labels or purchase scores are not used for grouping. Trajectory returns support subsequent credit estimation, not the construction of earlier progress states.

For rule development, the rules were defined from task semantics before the experiments and remained fixed throughout. Evaluation tasks and results were not used to select or adjust them.

Reviewer_uiePOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Beyond-Local Policy Optimization (BLPO) for credit assignment in long-horizon LLM agents. The method uses environment-specific rules to summarize interaction histories into task-progress states and map concrete actions to their goal-relative functions. It then aggregates context-centered returns across tasks and policy updates in a global memory, with support and uncertainty adjustments. The resulting task-progress advantages are combined with trajectory-level advantages and local advantages over normalized actions, without a learned critic or additional rollouts. Experiments on ALFWorld and WebShop with Qwen2.5-1.5B and Qwen2.5-7B report improved task success, supported by component ablations, cross-task analyses, and computational measurements.

Critical strengths
  1. BLPO uses experience shared across tasks and training updates to improve step-level credit assignment without a learned critic or additional rollouts.
  2. Comprehensive experimental results on ALFWorld and WebShop demonstrate the effectiveness of the proposed method.
Critical weaknesses
  1. Since Table 1 uses previously reported baselines, the authors should clarify evaluation consistency and whether the improvements are stable across training runs.
  2. Eq. (17) ignores the covariance introduced by shared context baselines, which may underestimate uncertainty. The authors should clarify this approximation and its effect on advantage estimation.
Questions for authors
  1. What do the seeds represent for each RL method in Table 1, and how do the imported baseline evaluation protocols correspond to Appendix A.1? Are results across independent training runs available for the main comparisons and ablations?
  2. Is omitting covariance in Eq. (17) an intentional approximation? Does accounting for the shared-context dependence materially change the estimated uncertainty or the resulting advantage weights?
2 comments
AuthorsOpenReview ↗

Response to Reviewer uieP (Part1)

To W1 and Q1: Evaluation consistency, training runs, and controlled experiments

Thank you for your constructive comments. A complete training run in long-horizon agentic RL involves repeated trajectory generation, multiple rounds of environment interaction, and policy updates, making repeated full training across multiple configurations computationally expensive. Accordingly, related work, including GiGPO and HGPO, uses one complete training run followed by evaluation with three seeds, rather than three independent training runs. We follow the same setup and allocate our training budget to main experiments across two environments and two model scales, together with component ablations and controls that restrict sharing to the same task, covering 18 distinct training configurations.

For the correspondence between Table 1 and Appendix A.1, we ran GiGPO and HGPO ourselves and obtained performance comparable to their reported results. To facilitate direct comparison with published results, Table 1 uses the baseline results reported in the HGPO paper and retains their statistical convention. BLPO evaluates its final policies under aligned settings and reports the mean and standard deviation across three evaluation seeds, as detailed in Appendix A.1.

Beyond the main comparison, Table 4 examines the benefits by controlling the scope of experience sharing: it retains the abstraction rules and all other components while restricting experience to the current task. The corresponding mean success rates are shown below:

SettingALFWorld Out-SuccessWebShop Task Success
Full BLPO91.5474.02
Same-task-only81.9766.99

Restricting sharing reduces the two success rates by 9.57 and 7.03 percentage points, respectively, directly supporting the value of cross-task experience reuse for credit assignment. The component ablations in Table 2 further support the mechanisms used to exploit this experience, providing evidence from both performance comparisons and controlled experiments.

AuthorsOpenReview ↗

Response to Reviewer uieP (Part2)

To W2 and Q2: Covariance approximation and its effect on credit estimation

Thank you for your helpful suggestions. Equation (17) intentionally uses a diagonal covariance approximation, allowing uncertainty to be computed directly from statistics maintained online. This avoids maintaining additional pairwise joint statistics between actions and simplifies the accumulation and decay of historical information across updates. EVE likewise uses a diagonal Fisher approximation for efficient value uncertainty estimation [1]. Shared contexts can introduce correlations, so we further examined the practical effect of accounting for them.

For the effect on uncertainty estimates and advantage weights, we used fixed interaction records from both environments, keeping action means, marginal variances, and other settings unchanged. We introduced only action correlations estimated from shared contexts and recomputed uncertainty and credit:

Change after accounting for covarianceALFWorldWebShop
Mean absolute relative change in standard errors5.77%7.35%
Mean absolute relative change in credit scaling weights1.72%1.27%
Overall normalized absolute change in final task-progress advantages1.54%1.17%

The last row divides the summed absolute changes in advantages by their original summed absolute magnitudes. Accounting for correlations changes the final advantages by approximately 1%–2% overall. Equation (11) combines the dispersion of action means, uncertainty, and a stabilizing term in the scaling denominator, so changes in standard errors do not translate proportionally into changes in final advantages.

For the practical role of the current confidence adjustment, Table 2 provides further training results: removing confidence adjustment while retaining the other components reduces both success rates.

SettingALFWorld Out-SuccessWebShop Task Success
Full BLPO91.5474.02
w/o Conf.81.0571.09

This ablation supports the role of the current confidence adjustment, while the covariance comparison shows that accounting for correlations has a small effect on final credit in the examined records. Together, these results support a design that balances online computational efficiency with effective credit estimation.

[1] Simon Schmitt, John Shawe-Taylor, and Hado van Hasselt. Exploration via Epistemic Value Estimation. AAAI, 2023.

Reviewer_j2MpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BLPO estimates step-level credit for long-horizon language-model agents by sharing experience across tasks and policy updates. It maps interaction histories and actions to task-progress states and action functions, centers returns within concrete contexts, and maintains a memory of residual returns. Support and uncertainty estimates scale the resulting global credit, which is combined with local and trajectory-level advantages. Experiments use ALFWorld and WebShop with 1.5B and 7B backbones.

Critical strengths
  1. The method addresses the tension between conditioning on enough history to make comparisons meaningful and retaining enough examples to estimate credit. Sharing task progress and action function provides a concrete, inexpensive way to expand support without a learned critic.

  2. The same-task-only control preserves the abstraction and accumulation mechanism while reducing WebShop success from 74.02 to 66.99. Together with the held-out-task preference analysis, this provides targeted evidence for the usefulness of cross-task sharing.

Critical weaknesses
  1. Residual centering does not remove differences in action composition across contexts. Equations 8–11 compare residual means for actions that may have different context support. For example, suppose two contexts share a progress state, with expected returns for (A, B, C) of (0.9, 0.8, 1.0) and (0.2, 0.1, 0.0). If the first context contains one observation each of A and C, and the second contains B and C, centering produces residuals of −0.05 for A and +0.05 for B, even though A is better in both contexts. Replicating these patterns across distinct contexts satisfies the support threshold without resolving the reversal. This does not invalidate the reported gains, but it limits the interpretation of global credit. Held-out-task agreement may preserve the same composition effect. The existing logs could help establish whether important action functions actually have overlapping context support.

  2. The main error bars do not establish training robustness. Appendix A.1 reports three evaluation seeds for the final checkpoint. The HGPO comparison likewise uses test seeds, so this is not evidence of a mismatched seed convention. Nevertheless, the 7B improvements over HGPO are only 0.92 percentage points for both WebShop success and ALFWorld out-of-distribution success. Their stability across training runs is unknown. Since Table 1 imports baseline results while the learning curves use the authors' runs, the correspondence between checkpoints, evaluation task lists, and baseline versions should be explicit. The larger ablation differences provide useful directional evidence, but their uncertainty is also incompletely reported.

Questions for authors
  1. How much concrete-context overlap exists between the principal action functions sharing a progress state, and do their preference signs remain stable on common-support contexts?

  2. How many independently trained checkpoints underlie Tables 1, 2, and 4, and how do the imported baseline results align with BLPO's evaluation protocol?

3 comments
AuthorsOpenReview ↗

Response to Reviewer j2Mp (Part1)

To W1 and Q1: Context composition, common support, and preference stability

Thank you for your helpful comments. Residual centering adjusts for differences in return baselines across contexts. To examine composition effects that may remain after centering, we conducted an analysis using common contexts: recomputing preferences using only contexts where both actions were observed yielded 92.14% directional agreement with the original estimates.

For the role of residual centering, Table 2 provides a controlled comparison that removes residualization while retaining the other components. The relevant ALFWorld results with Qwen2.5-1.5B are reproduced below:

SettingIn-SuccessOut-Success
Full BLPO94.0191.54
w/o Residual88.4181.97
Drop after removal (percentage points)5.609.57

Both success rates decline without residualization, supporting the practical role of adjusting context baselines in shared credit estimation. To address the more specific concern about context composition, we further examined common support and its effect on action preferences.

For common support and preference stability, we measured context overlap between action pairs within shared progress states and recomputed preferences using only contexts where both actions were observed. The common baseline cancels when comparing actions within the same context. For decisions associated with disagreements between these preferences and the original estimates, we also examined whether reliability scaling reduced their global credit magnitudes.

DiagnosticResult
Comparisons with at least two effective common contexts, weighted by query frequency92.58%
Mean overlap of context contributions, measured by intersection over union with EMA weights28.24%
Agreement in preference direction between estimates from common contexts and the original estimates92.14%
Decisions involved in disagreements with reduced global credit magnitude64.28%
Decisions involved in disagreements with global credit magnitude reduced by at least half32.36%

BLPO pools experience across different concrete contexts without requiring identical context support across actions. Most comparisons have common support, and most preference directions in the examined sample remain unchanged when recomputed using only common contexts. Reliability scaling further reduces the magnitude of global credit for many decisions involved in disagreeing comparisons.

AuthorsOpenReview ↗

Response to Reviewer j2Mp (Part2)

To W2 and Q2: Training and evaluation protocol, result correspondence, and experimental reliability

Thank you for your valuable suggestions. Long-horizon agentic RL requires repeated generation of long trajectories, multiple rounds of environment interaction, and policy optimization, making repeated full training across multiple configurations computationally expensive. Accordingly, closely related work, including GiGPO and HGPO, uses one complete training run followed by evaluation with three seeds, rather than three independent training runs. We follow the same practice and allocate our training budget to broader experimental validation: main experiments across two environments and two model scales, together with component ablations and controls that restrict sharing to the same task, covering 18 distinct training configurations.

For the correspondence between the main table and learning curves, we ran GiGPO and HGPO ourselves and obtained performance comparable to their reported results. To facilitate direct comparison with published results, Table 1 uses the baseline results reported in the HGPO paper, with BLPO following the same evaluation protocol as detailed in Appendix A.1. Figure 2 shows our own training runs, with all methods evaluated on the same fixed evaluation set to compare learning speed, success rate, and interaction length.

For the magnitude of the improvements, BLPO gains 0.92 percentage points over HGPO on 7B WebShop, comparable to HGPO’s 0.91-point gain over GiGPO. Beyond this comparison, Table 4 directly examines the contribution of cross-task experience reuse. With Qwen2.5-1.5B, restricting experience to the current task while keeping all other components unchanged gives the following results:

SettingALFWorld In-SuccessALFWorld Out-SuccessWebShop Task ScoreWebShop Task Success
Full BLPO94.0191.5487.4374.02
Same-task-only87.6381.9787.1966.99

Restricting sharing reduces ALFWorld Out-Success and WebShop Task Success by 9.57 and 7.03 percentage points, respectively, supporting our central contribution of improving credit assignment through cross-task experience reuse. The ablations in Table 2 further support the mechanisms used to estimate credit from this experience.

For evaluation variability in these controlled comparisons, we additionally report the standard deviations across three evaluation seeds for the final policies in both environments:

SettingALFWorld In-SuccessALFWorld Out-SuccessWebShop Task ScoreWebShop Task Success
Full BLPO94.01 ± 0.2391.54 ± 0.6087.43 ± 0.5674.02 ± 1.37
w/o Act. pool.91.34 ± 0.1183.14 ± 0.2382.79 ± 1.1566.21 ± 1.67
w/o Task-Progress92.25 ± 1.1183.53 ± 2.2282.75 ± 1.1668.49 ± 3.56
w/o Act. abstr.88.80 ± 1.1182.94 ± 0.3086.83 ± 1.4371.22 ± 1.75
w/o Residual88.41 ± 2.2281.97 ± 1.6685.14 ± 1.1273.24 ± 0.20
w/o EMA90.89 ± 2.0084.11 ± 1.7686.09 ± 1.7369.92 ± 1.41
w/o Conf.89.97 ± 0.7481.05 ± 2.7785.00 ± 1.1471.09 ± 2.25
Same-task-only87.63 ± 0.6381.97 ± 0.4187.19 ± 0.4366.99 ± 1.22

Across both environments, all six component ablations reduce mean task success relative to full BLPO. Together with the same-task-only control, these consistent trends support the value of experience reuse across tasks and the mechanisms used to estimate credit.

I have no further questions.

Comments on the paper

1 comment
Area_Chair_CzEFOpenReview ↗

Please engage with authors

Dear Reviewers,

The author-reviewer discussion will be closed on November 18 (AoE). If you haven’t yet replied to the authors’ latest response, please do so on OpenReview as soon as possible so they have time to follow up. A brief reply indicating which of your concerns have been addressed and what questions remain would be helpful. Thank you for your time and for engaging with the authors.

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.