- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies step-level credit assignment in group-based RL for multi-turn agentic tasks with sparse terminal rewards. Building on GiGPO's anchor-state grouping, the authors note that step-level advantages are computed from each trajectory's own return, so failed actions in the same anchor-state group receive identical credit. They propose PGPO, which defines an empirical state potential as the mean step return within each anchor-state group. It then forms a potential-guided advantage from the one-step potential difference between successor and current states, and adds it to the GiGPO step-level advantage with a task-level success-adaptive weight. The method requires no critic, extra inference, or additional rollouts.
PGPO is evaluated on ALFWorld and WebShop with Qwen2.5-1.5B and 7B, where it reports the best overall results among group-based baselines. The paper also provides ablations of the advantage components, hyperparameter sensitivity analyses, failure-side dispersion and preference diagnostics, and training overhead measurements.
Critical strengths
-
The paper is clearly written and well organized, and the motivating issue is illustrated with a concrete ALFWorld example (Fig. 1) supported by logged advantage values (Table 4).
-
The failure-side credit issue is formalized explicitly (Eq. 4), with its assumptions clearly stated.
-
The method is simple and lightweight. It requires no critic, extra inference, or additional rollouts, and adds negligible training overhead.
-
PGPO achieves the best overall results on both ALFWorld and WebShop at both the 1.5B and 7B scales.
-
The appendix provides useful supporting analyses, including a leave-one-out check on potential estimation, group-size statistics, and a text-similarity grouping variant.
Critical weaknesses
-
The paper attributes failure-side credit misassignment to actions inheriting only their own trajectory's outcome. In my view, using a trajectory's own return is a trade-off that GiGPO and similar methods make to keep bias low, not necessarily a flaw, and the paper does not discuss this trade-off.
-
The examples in Fig. 1 and Table 4 show the need for finer-grained credit assignment, but they do not show why a potential function is needed.
-
The main justification given for using potentials is that they require no critic or extra inference. The paper does not argue why potentials are necessary compared with other fine-grained credit assignment approaches.
-
The paper lacks comparisons with other fine-grained credit assignment methods, such as learned value functions, PRMs, SPA-RL, and the potential-based TIPS and Step Potential Advantage Estimation.
-
PGPO's grouping rule, return computation, and step-level advantage are largely the same as GiGPO's. Because within-group normalization cancels the current-state potential, for non-terminal steps Aᵠ is equivalent to the normalized mean return of the successor state's group. The main increment over GiGPO is therefore the one-step lookahead and the weight w(x).
-
The potential Φ groups visits by the current observation only, so visits under different interaction histories are averaged together. In partially observable environments like ALFWorld, the same observation can correspond to different task progress (e.g., whether the object has already been picked up and heated), which may distort Φ as an estimate of state value.
Questions for authors
-
With only 8 trajectories per task, how reliable is the empirical potential Φ? What fraction of successor states fall into singleton groups, where Aᵠ reduces to the trajectory's own outcome?
-
Could you compare PGPO against other fine-grained credit assignment approaches under the same setup, such as a learned value function, PRM-based methods, and the potential-based TIPS and Step Potential Advantage Estimation?