- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces GRAFT, which builds a trajectory graph from rollouts, estimates node values via Bellman iteration, and derives step-level advantages from value differences plus a Graph GAE variant, evaluated on ALFWorld, WebShop, and SearchQA against GRPO, GiGPO, and GraphGPO. However, due to various shortcomings of this article, my suggestion is a weak rejection. Below are my thoughts on the strength, weakness, and my question of the paper.
Critical strengths
- The proposed approach constructs trajectory graphs from sampled rollouts and estimates step-level advantages. These are achieved through Bellman value iteration and Graph GAE. The algorithmic contribution is demonstrated through many experiments, where GRAFT achieves improvements over existing baselines.
- The experiment is quite comprehensive. The article is relatively complete.
- The paper is well-written with clear exposition and reasonable experiments.
Critical weaknesses
- The core issue has been resolved by GraphGPO, and this article only criticizes its mechanism without proving the existence of residual unresolved problems. GraphGPO has already done "cross trajectory aggregation of states and step level credit allocation using graphs". The difference between the two articles is that GraphGPO uses the shortest path heuristic reward, while this article replaces it with Bellman value iteration+Graph GAE. But the authors have never proven that there is a residual problem after GraphGPO that cannot be solved by GraphGPO and must be solved by Bellman. The author only said that the reward of GraphGPO is heuristic, which is a criticism of the baseline mechanism, not a proof that the problem has not been solved.
- Weak theoretical justification. Proposition 1 assumes access to the true value function V^π in a deterministic environment, making the TD residual equal to the true advantage. This oracle result does not establish the accuracy of GRAFT's practical graph-based estimator.
- Insufficiently justified Graph GAE. The proposed estimator uniformly averages reachable TD residuals without accounting for policy probabilities. Its claimed bias reduction and interpolation toward Monte Carlo advantages are not rigorously established.
- The actual advantage estimator combines Graph GAE with normalization, yet the authors first present the one-step TD estimate as theoretically aligned with the standard advantage definition, before introducing Graph GAE to address its limitations.
- In Sec 3.3, there might be some imprecise parts: a) The proposed Graph GAE averages reachable edges without policy-probability weighting, potentially introducing systematic bias; b) Setting μ=0 and σ=|A_t^GAE | effectively yields A≈sign(A_t^GAE), discarding magnitude information without justification; c) Eq. (11) does not clarify whether normalization uses s_t or ϕ(s_t).
Questions for authors
See above.