acceptodds
Under review as a conference paper at ICLR 2027

PGPO: Potential-Guided Policy Optimization for Multi-Turn Agentic Tasks

Abstract

Group-based reinforcement learning (RL) has become an effective paradigm for LLM post-training, but in multi-turn agentic tasks with sparse terminal rewards, it often provides coarse credit for intermediate actions. To obtain more fine-grained credit assignment, recent work such as GiGPO introduces step-level advantages for intermediate actions. However, these step-level signals still rely on the final outcome of each individual trajectory. As a result, actions within failed trajectories can remain poorly differentiated, so effective actions can receive the same unfavorable credit as erroneous ones. In this work, we propose Potential-Guided Policy Optimization (PGPO) for multi-turn agentic tasks. PGPO estimates empirical state potentials from anchor-state-group return statistics within each rollout group. It then derives action advantages from potential differences between adjacent states, enabling cross-trajectory credit propagation. This provides finer-grained step-level credit assignment, especially within failed trajectories. Experiments on ALFWorld and WebShop show strong overall performance relative to recent group-based RL methods. Further analysis provides evidence that PGPO yields more informative failure-side credit signals with negligible training overhead.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, LLM agents, Policy optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_t9TkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies step-level credit assignment in group-based RL for multi-turn agentic tasks with sparse terminal rewards. Building on GiGPO's anchor-state grouping, the authors note that step-level advantages are computed from each trajectory's own return, so failed actions in the same anchor-state group receive identical credit. They propose PGPO, which defines an empirical state potential as the mean step return within each anchor-state group. It then forms a potential-guided advantage from the one-step potential difference between successor and current states, and adds it to the GiGPO step-level advantage with a task-level success-adaptive weight. The method requires no critic, extra inference, or additional rollouts.

PGPO is evaluated on ALFWorld and WebShop with Qwen2.5-1.5B and 7B, where it reports the best overall results among group-based baselines. The paper also provides ablations of the advantage components, hyperparameter sensitivity analyses, failure-side dispersion and preference diagnostics, and training overhead measurements.

Critical strengths
  1. The paper is clearly written and well organized, and the motivating issue is illustrated with a concrete ALFWorld example (Fig. 1) supported by logged advantage values (Table 4).

  2. The failure-side credit issue is formalized explicitly (Eq. 4), with its assumptions clearly stated.

  3. The method is simple and lightweight. It requires no critic, extra inference, or additional rollouts, and adds negligible training overhead.

  4. PGPO achieves the best overall results on both ALFWorld and WebShop at both the 1.5B and 7B scales.

  5. The appendix provides useful supporting analyses, including a leave-one-out check on potential estimation, group-size statistics, and a text-similarity grouping variant.

Critical weaknesses
  1. The paper attributes failure-side credit misassignment to actions inheriting only their own trajectory's outcome. In my view, using a trajectory's own return is a trade-off that GiGPO and similar methods make to keep bias low, not necessarily a flaw, and the paper does not discuss this trade-off.

  2. The examples in Fig. 1 and Table 4 show the need for finer-grained credit assignment, but they do not show why a potential function is needed.

  3. The main justification given for using potentials is that they require no critic or extra inference. The paper does not argue why potentials are necessary compared with other fine-grained credit assignment approaches.

  4. The paper lacks comparisons with other fine-grained credit assignment methods, such as learned value functions, PRMs, SPA-RL, and the potential-based TIPS and Step Potential Advantage Estimation.

  5. PGPO's grouping rule, return computation, and step-level advantage are largely the same as GiGPO's. Because within-group normalization cancels the current-state potential, for non-terminal steps Aᵠ is equivalent to the normalized mean return of the successor state's group. The main increment over GiGPO is therefore the one-step lookahead and the weight w(x).

  6. The potential Φ groups visits by the current observation only, so visits under different interaction histories are averaged together. In partially observable environments like ALFWorld, the same observation can correspond to different task progress (e.g., whether the object has already been picked up and heated), which may distort Φ as an estimate of state value.

Questions for authors
  1. With only 8 trajectories per task, how reliable is the empirical potential Φ? What fraction of successor states fall into singleton groups, where Aᵠ reduces to the trajectory's own outcome?

  2. Could you compare PGPO against other fine-grained credit assignment approaches under the same setup, such as a learned value function, PRM-based methods, and the potential-based TIPS and Step Potential Advantage Estimation?

Reviewer_SuB9OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the credit assignment problem for multi-turn, sparse-reward agentic tasks. Building on the anchor-state-based step-level advantage introduced by GiGPO, the paper identifies a failure-side credit degeneracy and proposes Potential Guided Policy Optimization (PGPO) to mitigate the degeneracy. Specifically, PGPO constructs the advantage of each step by combining a potential-guided advantage, calculated from empirical potential of its successor state, with the GiGPO step-level advantage, then doing adaptive weighting. The paper conducted experiments comparing PGPO primarily with GiGPO and HGPO on ALFWorld and WebShop, with Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct as base models.

Critical strengths

The paper conducted relatively thorough ablation study and hyperparameter analysis on the studied method and tasks, including the ablation studies on AE,AϕA^E, A^\phi, ASA^S, and adaptive weighting, and hyperparameter analysis on adaptive weighting parameters α,β\alpha, \beta and potential discount factor γ\gamma, making it a solid and practical credit-assignment heuristic study.

Critical weaknesses
  1. Limited Novelty and Insights. The step level advantage constructed by PGPO is a combination of GiGPO step-advantage and a proposed potential-guided advantage. The proposed advantage is based on cross-trajectory average state return, which is similar to the idea proposed in G2PO [1] Equations (4) and (5). Therefore the novelty of the proposed advantage estimation appears limited. Moreover, although ablation studies are extensive, the method design lacks principled justification. For instance, it is unclear why the potential guided advantage needs to be combined with GiGPO's step advantage, and why this choice leads to better empirical performance.

  2. Small and Saturated Environments. The evaluations only use ALFWorld and WebShop as task environments. Both environments are relatively simple, and the agent execution traces can easily produce overlapping states for empirical value/potential estimations. The paper does not evaluate PGPO on complex environments where states are more diverse. Moreover, the improvements of PGPO over baselines appear marginal, e.g. only 0.59 percentage point improvement over HGPO on WebShop.

(minor) missing discussion of a few related papers: G2PO [1], GEPO [2], HiPER [3]

[1] Wang, Yunan, et al. "Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learning." arXiv preprint arXiv:2606.22995 (2026). [2] Yuan, Jiazhen, et al. "Graph-Enhanced Policy Optimization in LLM Agent Training." arXiv preprint arXiv:2510.26270 (2025). [3] Peng, Jiangweizhi, et al. "Hiper: Hierarchical reinforcement learning with explicit credit assignment for large language model agents." arXiv preprint arXiv:2602.16165 (2026).

Questions for authors
  1. Could the authors explain the essential differences between the potential guided advantage proposed in PGPO and the idea employed in G2PO?
  2. Why the potential guided advantage should be combined with GiGPO's step-level advantage? What does this indicate?
  3. Would PGPO be able to generalize to more complex environments, where states are less overlapped?
Reviewer_jsF6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a new algorithm for multi-step RL. It follows the line of GiGPO using anchor state grouping for better estimation of step-level advantages. The main idea builds on the state potential and the discounted difference of the state potential. The normalzed state potential difference is then defined as the potential-guided advantage, and added on top of the outcome reward based advantage from GRPO. The algorithm is showed to have better performance compared to GiGPO on ALFWorld and WebShop.

Critical strengths
  • The derivation of the algorithm holds up and it is a drop-in change for advantage computation so it's easy to adopt.
  • The limitation of GiGPO is true and the method does work around that, so the motivation is sound.
  • The method shows consistent gains.
Critical weaknesses
  • Similar to GiGPO, the proposed method PGPO relies on grouping states. This is possible for simple environments as in ALFWorld and WebShop, but much harder to do in realistic settings like coding agents or search agents where is state space is large.
  • The potential difference term cancels out (since the actions start from the same anchor state), so A^φ= z-score ofΦ(s_{t+1}).
  • The design of the weight term w(x) = β(1 − p_succ(x))^α is not jusitified or tested in an ablation.
Questions for authors
  • Since Φ(s̃_m) and γ cancel under Eq. 8, what role does the potential-difference / PBRS framing play?
  • What does the ablation A^S·(1 + w(x)) with no potentials give?
  • How does PGPO behave on a setting with a larger state space?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.