- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ProAS, a progress-aware advantage shaping framework for multi-turn agentic reinforcement learning. It constructs a dynamically evolving Agentic Reference Graph (ARG) to identify reasonable and progress-advancing actions. Based on these signals, ProAS masks unreasonable positive advantages and redistributes negative advantages using Future-Drift. Experiments on ALFWorld and WebShop demonstrate improvements across several group-based RL algorithms.
Critical strengths
-
The motivation is reasonable, particularly the asymmetric treatment of positive and negative advantages.
-
The method is easy to integrate with existing RL algorithms and demonstrates consistent empirical improvements across different baselines and model scales.
Critical weaknesses
1. Insufficient discussion of closely related work.
The paper does not discuss Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learning (GraphGPO, ICML 2026). Both graph-based progress estimation and fine-grained credit assignment have already been explored in this work. Although ProAS introduces a different advantage shaping mechanism, the authors should better clarify its novelty and distinguish its contributions from GraphGPO.
2. Strong dependence on expert-annotated reference graphs.
ProAS requires expert annotations or successful reference trajectories to initialize ARG, which may significantly limit its applicability to more open-ended agentic tasks. Moreover, expert trajectories can contain redundant or unnecessary actions, potentially affecting the reliability of the reference graph. The paper does not sufficiently investigate how the quality and availability of expert guidance influence the method.
3. Potentially excessive penalization of early actions.
The Future-Drift mechanism naturally assigns larger negative weights to earlier steps because they have longer remaining trajectories. However, these steps are not necessarily more responsible for the eventual failure. In many agentic tasks, early decisions may involve relatively constrained action spaces, while later stages require more complex exploration and recovery from mistakes. Excessively penalizing early steps while weakening penalties at later stages may discourage useful exploration or reduce the agent's ability to learn from late-stage errors.
Questions for authors
-
How does ProAS differ from GraphGPO (ICML 2026), particularly regarding graph-based progress estimation and step-level credit assignment?
-
How dependent is ProAS on expert annotations and reference graph quality? Can it work effectively without expert initialization, and how does it handle redundant actions in expert trajectories?
-
Since Future-Drift is naturally larger for earlier steps, how can the authors distinguish genuine downstream influence from a simple temporal weighting effect? Have the authors compared APR with a position-based weighting baseline?
-
Could the stronger penalization of early steps and relatively weaker penalization of later steps limit the agent's ability to learn recovery behaviors after making mistakes? Can the authors provide experiments or analyses addressing this concern?
Thank you to the authors for the detailed rebuttal and additional experiments. I appreciate the clarifications regarding the differences between ProAS and GraphGPO, as well as the new comparisons with GraphGPO and G2PO.
While these responses address several of my original questions, my main concerns remain:
Novelty and empirical comparison. Although ProAS introduces a different advantage shaping mechanism, its core motivation of graph-guided step-level credit assignment substantially overlaps with prior work. The newly added comparisons show only modest improvements over GraphGPO (+0.8%) and G2PO (+0.3%). Moreover, ProAS introduces additional expert-derived reference information, making it difficult to isolate the contribution of the proposed shaping mechanism under a matched supervision budget.
Effectiveness of Future-Drift for credit assignment. I appreciate the position-based weighting ablation, which demonstrates the benefits of Future-Drift over a simple temporal heuristic. However, Future-Drift measures the number of subsequent non-progressing steps rather than the actual causal contribution of an action to task failure. Therefore, I remain unconvinced that it provides sufficiently reliable step-level credit assignment, particularly for early actions followed by downstream mistakes.
Overall, I acknowledge the empirical improvements and the effort made in the rebuttal. Nevertheless, I do not believe the additional evidence sufficiently resolves my concerns regarding the methodological novelty and the underlying credit assignment mechanism. I therefore maintain my original score of 2 (Weak Rejection).