- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Addressing the limitations of existing methods in long-horizon agentic reinforcement learning, this study proposes a state-graph policy optimization approach SGPO. SGPO aggregates transitions observed across a group of related tasks into an empirical state graph. By solving the deterministic equivalence (CE) Bellman equation on this graph, it derives the expected discounted return—serving as a shared successor value—thereby enabling the sharing of future evidence across different trajectories that revisit the same successor states. SGPO combines the CE values computed from the graph with the sampled rewards from each actual trajectory using backward . It explicitly balances bias and variance by introducing an interpolation parameter, . Across two benchmarks—ALFWorld (long-horizon embodied tasks) and WebShop (web navigation)—SGPO has demonstrated strong average overall performance and task success rates, consistently achieving these results across various Qwen model scales.
Critical strengths
- SGPO retains the outer training loop, rollout sampling, anchor-point partitioning, and episode-based advantage estimation, replacing only the step-wise return estimator. It requires neither a learned critic network nor any additional environment interactions or model evaluations.
Critical weaknesses
1.There is a logical gap between the value derived from the empirical anchor-point model and the true value of the policy. The certainty-equivalent (CE) Bellman system shown in the diagram is solved by averaging over observed transition frequencies, rather than by maximizing over policy actions.
2.In reinforcement learning, the unbiasedness of policy gradients typically serves as a crucial cornerstone for the theoretical convergence of algorithms. In Section 6 (Limitations), the paper explicitly notes that "same-batch value fitting" and "state reweighting through local standardization" preclude the direct application of guarantees for unbiased policy gradients. Because SGPO utilizes empirical values—computed within groups—to normalize and bootstrap trajectories, the resulting "certainty-equivalent" values (derived on-the-fly from the current batch of samples) do not constitute globally unbiased estimates; this represents a clear logical compromise in the theoretical derivation.
Questions for authors
1.In Appendix D, the authors state that anchor keys rely strictly on exact-string matching of environment-defined anchor_obs, which trades coverage against aliasing. Given that text-based or web environments (like ALFWorld and WebShop) often exhibit historical context divergence under the same observation string, how severe is the impact of state aliasing on the certainty-equivalent (CE) Bellman system? Have the authors explored soft or semantic-based matching to mitigate this?
2.In Section 5.3 and Appendix C.4, the offline evaluation demonstrates that Real CE achieves lower MSE. However, the authors appropriately note that a reduction in pooled MSE does not directly establish a reduction in policy-gradient variance. Do the authors have empirical or theoretical evidence from online training to confirm that SGPO actually reduces the variance of the policy gradient compared to the MC endpoint ()?
3.The default trace parameter is fixed on a development split (Section 4.2). How sensitive is SGPO's performance to when transferring to environments with significantly different stochasticity or longer horizons? Is there a heuristic rule for choosing without extensive validation splits?