acceptodds
Under review as a conference paper at ICLR 2027

ProAS: A Progress-Aware Advantage Shaping Framework for Multi-Turn Agentic Reinforcement Learning

13 views

Abstract

Reinforcement learning (RL) has emerged as the primary paradigm for optimizing Large Language Model (LLM) agents, yet efficient critic-free algorithms (e.g., GRPO) suffer from a fundamental uniform credit assignment dilemma that broadcasts trajectory-level advantages equally to all actions. Our analysis reveals that this uniformity inherently reinforces structurally invalid actions in preferred trajectories and ignores the downstream cascading influence of early pivotal deviations in unpreferred ones. To resolve this without introducing critic overhead, we propose Progress-Aware Advantage Shaping (ProAS), a plug-and-play framework that leverages a dynamically evolving Agentic Reference Graph (ARG) to achieve fine-grained credit assignment via Masking Unreasonable Incentives (MUI) and Asymmetric Penalty Redistribution (APR). Extensive experiments across Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct demonstrate that ProAS universally elevates mainstream group-based baselines to new state-of-the-art performance, achieving peak success rates of 91.9% and 94.3% on ALFWorld, alongside 66.7% and 70.8% on WebShop, respectively. Furthermore, it accelerates training convergence by up to while maintaining strong robustness against imperfect or distilled ARGs.

Primary area:
reinforcement learning
Keywords:
RL, agentic training, GRPO

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KrydOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ProAS, a progress-aware advantage shaping framework for multi-turn agentic reinforcement learning. It constructs a dynamically evolving Agentic Reference Graph (ARG) to identify reasonable and progress-advancing actions. Based on these signals, ProAS masks unreasonable positive advantages and redistributes negative advantages using Future-Drift. Experiments on ALFWorld and WebShop demonstrate improvements across several group-based RL algorithms.

Critical strengths
  1. The motivation is reasonable, particularly the asymmetric treatment of positive and negative advantages.

  2. The method is easy to integrate with existing RL algorithms and demonstrates consistent empirical improvements across different baselines and model scales.

Critical weaknesses

1. Insufficient discussion of closely related work.

The paper does not discuss Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learning (GraphGPO, ICML 2026). Both graph-based progress estimation and fine-grained credit assignment have already been explored in this work. Although ProAS introduces a different advantage shaping mechanism, the authors should better clarify its novelty and distinguish its contributions from GraphGPO.

2. Strong dependence on expert-annotated reference graphs.

ProAS requires expert annotations or successful reference trajectories to initialize ARG, which may significantly limit its applicability to more open-ended agentic tasks. Moreover, expert trajectories can contain redundant or unnecessary actions, potentially affecting the reliability of the reference graph. The paper does not sufficiently investigate how the quality and availability of expert guidance influence the method.

3. Potentially excessive penalization of early actions.

The Future-Drift mechanism naturally assigns larger negative weights to earlier steps because they have longer remaining trajectories. However, these steps are not necessarily more responsible for the eventual failure. In many agentic tasks, early decisions may involve relatively constrained action spaces, while later stages require more complex exploration and recovery from mistakes. Excessively penalizing early steps while weakening penalties at later stages may discourage useful exploration or reduce the agent's ability to learn from late-stage errors.

Questions for authors
  1. How does ProAS differ from GraphGPO (ICML 2026), particularly regarding graph-based progress estimation and step-level credit assignment?

  2. How dependent is ProAS on expert annotations and reference graph quality? Can it work effectively without expert initialization, and how does it handle redundant actions in expert trajectories?

  3. Since Future-Drift is naturally larger for earlier steps, how can the authors distinguish genuine downstream influence from a simple temporal weighting effect? Have the authors compared APR with a position-based weighting baseline?

  4. Could the stronger penalization of early steps and relatively weaker penalization of later steps limit the agent's ability to learn recovery behaviors after making mistakes? Can the authors provide experiments or analyses addressing this concern?

1 comment

Thank you to the authors for the detailed rebuttal and additional experiments. I appreciate the clarifications regarding the differences between ProAS and GraphGPO, as well as the new comparisons with GraphGPO and G2PO.

While these responses address several of my original questions, my main concerns remain:

  1. Novelty and empirical comparison. Although ProAS introduces a different advantage shaping mechanism, its core motivation of graph-guided step-level credit assignment substantially overlaps with prior work. The newly added comparisons show only modest improvements over GraphGPO (+0.8%) and G2PO (+0.3%). Moreover, ProAS introduces additional expert-derived reference information, making it difficult to isolate the contribution of the proposed shaping mechanism under a matched supervision budget.

  2. Effectiveness of Future-Drift for credit assignment. I appreciate the position-based weighting ablation, which demonstrates the benefits of Future-Drift over a simple temporal heuristic. However, Future-Drift measures the number of subsequent non-progressing steps rather than the actual causal contribution of an action to task failure. Therefore, I remain unconvinced that it provides sufficiently reliable step-level credit assignment, particularly for early actions followed by downstream mistakes.

Overall, I acknowledge the empirical improvements and the effort made in the rebuttal. Nevertheless, I do not believe the additional evidence sufficiently resolves my concerns regarding the methodological novelty and the underlying credit assignment mechanism. I therefore maintain my original score of 2 (Weak Rejection).

Reviewer_tKgmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies the uniform credit assignment problem in critic-free agentic reinforcement learning, where trajectory-level advantages may reinforce unreasonable actions in successful trajectories while overlooking the downstream influence of individual actions in unsuccessful ones. To address this issue, the authors propose Progress-Aware Advantage Shaping (ProAS), a plug-and-play framework that leverages a dynamically evolving Agentic Reference Graph (ARG) to provide fine-grained credit assignment through Masking Unreasonable Incentives (MUI) and Asymmetric Penalty Redistribution (APR). Extensive experiments on ALFWorld and WebShop demonstrate that ProAS consistently boosts mainstream group-based RL baselines and accelerates training convergence.

Critical strengths
  1. The paper addresses the important challenge of uniform credit assignment in multi-turn agentic RL. It clearly identifies the limitations of assigning identical advantages to actions within a trajectory and provides intuitive motivation through action-intervention analysis.

  2. ProAS introduces an interesting asymmetric advantage shaping strategy through MUI and APR, supported by a dynamically evolving ARG. The framework is lightweight, does not require an additional critic, and can be integrated into existing group-based RL algorithms.

  3. The paper evaluates ProAS across two benchmarks, two model scales, and multiple RL baselines. The results demonstrate consistent improvements in most settings, while ablation studies provide evidence for the contributions of MUI and APR. Additional experiments with incomplete and distilled ARGs further strengthen the empirical evaluation.

Critical weaknesses
  1. Although the paper includes reasonable baselines (GRPO, DAPO, GiGPO, and MiGPO), it lacks comparisons with GraphGPO (ICML 2026) and G2PO (NeurIPS 2026), two closely related graph-based credit assignment methods. Both use Qwen2.5 models and evaluate on ALFWorld and WebShop. Such comparisons would better support the paper's SOTA and efficiency claims.

  2. ProAS initializes its ARG using official annotations or officially annotated sub-goals, introducing substantial task-specific prior knowledge. Although the distilled ARG experiment shows promising results, it remains unclear how sub-goals and state equivalence classes are constructed without predefined structures. This limits the evidence for generalizability and makes it difficult to isolate the contributions of advantage shaping from those of additional reference information.

Questions for authors
  1. The paper claims PA_t=1⇒Rea_t=1, but the definitions do not appear to guarantee this. PA only requires reaching a previously unvisited ARG state, whereas Rea requires the complete transition to belong to the ARG. Could a rollout reach an unvisited ARG state through an unrecorded transition, resulting in PA_t=1 but Rea_t=0? Given the online evolution procedure illustrated in Appendix D.3 (Table 9), could the authors clarify whether additional transition-validity checks are applied when computing PA and Rea before newly discovered successful paths are incorporated into the ARG?

  2. Could the authors compare ProAS with these methods under matched training and evaluation protocols, including model, tasks, rewards, interaction horizons, and training budgets? Comparisons of final performance, convergence speed, and computational overhead would strengthen the empirical evidence. In particular, since GraphGPO constructs its graph from sampled rollouts without requiring the same official reference annotations used by ProAS, a comparison of their dependence on additional supervision, ideally under matched information budgets, would be informative.

  3. Could the authors clarify how teacher-generated rollouts are collected and filtered, and how sub-goals, state equivalence classes, and valid transitions are extracted? Does this process still require official annotations or task-specific rules? These details are important for reproducibility and generalizability.

  4. The Static variant achieves a slightly higher WebShop SR (65.9%) than standard ProAS (64.6%), despite the claimed benefits of online evolution. Although the difference is small relative to the reported variability, could the authors explain why online evolution does not consistently improve performance?

Reviewer_omD8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Progress-Aware Advantage Shaping (ProAS), a framework that can be integrated into existing group-based policy optimization algorithms. It addresses the credit-assignment problem arising when GRPO applies the same trajectory-level advantage to all actions within a trajectory. ProAS constructs an Agentic Reference Graph (ARG) from successful references and updates it with new successful paths discovered during training. The ARG is used to annotate each action's reasonableness (Rea) and progress advancement (PA), which then guide the modification of the existing advantages.

Masking Unreasonable Incentives (MUI) sets positive advantages to zero for actions whose transitions are unsupported by the ARG. Asymmetric Penalty Redistribution (APR) reweights negative advantages at the action level using mean-normalized Future-Drift, the number of non-advancing steps from the current step through the end of the trajectory. The paper evaluates ProAS with GRPO, DAPO, and GiGPO using Qwen2.5-1.5B/7B-Instruct on ALFWorld and WebShop, reporting improvements in success rates and efficiency in reaching target success rates.

Critical strengths
  • The paper is clearly written, and the analysis in Section 4.2 provides useful motivation for the proposed metrics.   The progression from the uniform-credit problem through the action-intervention analysis to the introduction of MUI and APR is clear. In particular, the rationale for limiting the reinforcement of inappropriate actions in successful trajectories and redistributing suppression across actions in failed trajectories is easy to follow.
  • The additional operations are simple and require no separate critic.   ProAS applies masks and nonnegative weights to existing advantages, and the paper evaluates its integration with GRPO, DAPO, and GiGPO. In environments where an ARG can be constructed, it can adjust action-level learning signals while retaining the existing training loop.
Critical weaknesses

1. Insufficient detail on ARG construction and the ProAS-specific implementation makes reproducibility and applicability difficult to assess.

The ARG provides the key supervision used to determine Rea and PA, but its construction procedure is not described in sufficient detail. The paper gives some information about the initial annotation sources and examples in the appendices. More detail is needed on how raw observations are represented as states, how subgoals are defined and state equivalence is determined, and how rollout transitions are matched to the ARG and pruned. Distinguishing the information supplied by each environment from the components that must be separately designed would also help readers assess the work required to apply the method elsewhere.

Appendix B.2 provides basic training settings, including the optimizer, learning rate, number of rollouts, and horizon. However, the exact code revision, ARG processing procedures, and implementation details and configurations for integrating ProAS with DAPO and GiGPO need to be specified more precisely.

2. Spd@K alone provides limited evidence of faster learning over the full training run.

The learning curves cross in some comparisons, making it difficult to conclude that learning is generally faster based solely on the first time a particular success-rate threshold is reached, as measured by Spd@70 or Spd@50. Reporting per-seed threshold-crossing points and their uncertainty, any interpolation or smoothing between evaluations, the aggregation procedure across seeds, and sensitivity to the chosen threshold would make the reported speedups easier to interpret. The rule for determining when a threshold is reached is particularly important because evaluation occurs only every 10 RL updates.

Questions for authors

1. Is the full ProAS framework necessary for short tasks such as those in WebShop?

  • On WebShop, Static ARG achieves a higher mean success rate than Full ProAS, while GRPO performs similarly to Incomplete & Static. Could the short 15-step horizon and a potentially limited variety of successful paths explain why the initial ARG appears sufficient?
  • Please discuss whether successful-path diversity explains this result and what additional benefits the full ProAS framework provides for such tasks.

2. How are the distribution and scale of advantages managed after shaping?

  • According to Eq. (7), MUI removes some positive signals, while APR reweights negative advantages by dividing FD by its mean. This can change the overall mean and standard deviation of the advantages, as well as the balance between positive and negative signals. Does the implementation apply any additional normalization after shaping?
  • Reporting the advantage distributions before and after shaping, the proportion masked to zero, and the range of FD weights would help clarify how the learning signal changes and how its scale affects training stability. Please also explain how this scaling is handled when advantages differ across steps, as in GiGPO.

3. How does the ARG represent useful repetition and information gathering?

The ARG is defined as a DAG, and its construction removes invalid transitions and redundant self-loops. How does the method distinguish necessary repetition from redundant loops in tasks that require repeated visits to the same locations, such as resource gathering?

Base → resource-gathering site → base → resource-gathering site → base

In this example, including resource quantities or cumulative progress in the nodes could distinguish the visits. Please explain how the current ARG state representation incorporates such information and what additional design would be required in a new environment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.