- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the problem that how to assign credit in long horizion agentic tasks. The formulation generally follows GraphGPO (https://arxiv.org/abs/2605.26684), while the authors propose free-energy as the creteria for the credit assignment. Empirically, the proposed method has good performance in ALFWorld and Webshop, illustrating the superior of the proposed credit assignment method.
Critical strengths
- The proposed method shows very well empirical performance. The proposed method shows very good performance in ALFWorld and Webshop.
- The proposed structure aware credit refinment is interesting. It shows performance improvement even if it is somehow heuristic.
Critical weaknesses
-
The presentation need to be improved. I highly recommend the author clarify more about the meaning of figure 1 and figure2, it is confusing when I first read the figure. Meanwhile, I would recommend add some illustrative figure in section 4.2 to understand your method.
-
Lack of comparison with standard RL advantage estimation. Since the authors formulate the problem using an empirical state-transition graph, the proposed free-energy credit can be directly interpreted from a standard RL perspective. Specifically, under unit transition costs, the partition function in Eq. (3) is equivalent to the discounted value function , where is the number of remaining transitions until termination. At the same time, Eq. (4) corresponds to the standard Bellman evaluation equation. I would encourage the authors to clarify this connection and include comparisons.
Questions for authors
-
Could the author clarify the experimental setup, if I understand correctly, the authors are doing eval on the 128 tasks (might not unique) in seen env in ALFWorld, I am curious about the performance on unseen env, could the authors also report the number in unseen env?
-
The proposed method didn't show too much improvement in Webshop, is it because of it doesn't have too much same state? Can the author provide some statistics that shows how many state merged in training for webshop if available? More general question here is how to define same state, I believe this question is easy for ALFWorld since the state is clear, but for Webshop and harder tasks, how can we define state in this context, for example, for a general tool calling task, how can we apply this method to the setting?
6 comments
W2: The connection to standard value functions and Bellman evaluation is insufficiently explained, and comparisons with standard RL advantage estimation methods are missing.
We thank the reviewer for the feedback. We address the value-function interpretation, how our method constructs credit, and the additional comparison with a standard return estimator. In the final manuscript, we will clarify how the partition function relates to standard value functions and Bellman evaluation and include the Monte Carlo (MC) comparison to connect the mathematical interpretation and credit construction to the experimental evidence.
(1) Under unit transition costs, the partition function can be interpreted as the discounted success value of an empirical transition process.
As the reviewer points out, under unit transition costs, Eq. (3) has the following value-function interpretation.
\qquad \gamma=e^{-\beta}.$$ Here, $q_x$ is the empirical transition process constructed from a group of sampled trajectories for the same task. Equation (4) already gives the Bellman recursion for computing this quantity. We will clarify its correspondence to standard value evaluation in reinforcement learning. **Our contribution is to use continuation evidence from the sampled trajectories to construct relative action credit that accounts for both the probabilities and costs of successful continuations.** **(2) FE constructs action credit by jointly accounting for the probabilities and costs of successful continuations across trajectories.** GiGPO uses the return observed after each action, and GraphGPO uses the shortest distance to success in the merged graph. **FE aggregates the transition probabilities and path costs of successful continuations supported by the empirical graph while retaining the effect of failed branches on the success probability.** Compared with per-trajectory returns, shared states allow continuation evidence to propagate across trajectories. Compared with shortest-path scoring, the empirical probabilities and excess costs of the paths also affect action credit. The free-energy formulation follows the framework of Kivimäki et al., already cited in the submission. Its variational view of path distributions provides a theoretical basis for jointly evaluating probabilities and costs. For our empirical success process, **Proposition 1, Eq. (5), and Appendix C.2 of the submission give the following decomposition and its full proof.** $$F_\beta(s)=d_{\mathrm{SP}}(s)-\frac{\log p_+(s)}{\beta}-\frac{\log\eta_\beta(s)}{\beta}.$$ Here, $p_+(s)$ is the empirical success probability, and $\eta_\beta(s)$ captures the discounted contribution of the costs that successful paths incur beyond the minimum successful path cost. This decomposition applies to states from which a successful terminal state is reachable. **It identifies two factors omitted by shortest-path valuation and provides a testable mathematical explanation of how FE changes action credit.**(3) To assess the effect of this valuation relative to standard sampled returns, we added an ablation that replaces FE with per-trajectory discounted Monte Carlo (MC) returns.
We replaced the FE step-level scores with discounted returns computed from each trajectory, keeping source-state grouping and normalization, episode-level advantages, structure-aware refinement rules, the discount factor, and the training budget fixed. The table also includes the shortest-path ablation reported in Table 2 of the submission.
| Step-level valuation (other components held fixed) | ALFWorld success rate (%) | WebShop success rate (%) |
|---|---|---|
| Shortest-path valuation (Table 2 of the submission) | 94.53 ± 0.78 | 78.13 ± 3.58 |
| Per-trajectory discounted MC (additional ablation) | 95.57 ± 3.16 | 76.56 ± 3.13 |
| FE valuation (full method) | 99.22 ± 0.78 | 80.21 ± 3.93 |
All results use Qwen2.5-1.5B trained for 150 updates. We report the mean and sample standard deviation across the same three evaluation seeds used in the paper. We ran the additional MC ablation on both ALFWorld and WebShop.
On ALFWorld, FE improved the success rate by 3.65 percentage points over the MC ablation. On WebShop, the matched MC ablation achieved a success rate of 76.56 ± 3.13%, compared with 80.21 ± 3.93% for FE. Compared with shortest-path valuation, FE improved success rates by 4.69 and 2.08 percentage points on ALFWorld and WebShop, respectively. These comparisons show that aggregating successful continuations improves task success rates when the other components of the credit assignment framework are held fixed.
We thank the reviewer for the suggestion. In the revised manuscript, we updated the descriptions and captions for Figures 1 and 2 as follows.
Figure 1. Two limitations of current credit assignment. (a) Shortest-distance scoring gives equally distant successor states the same score despite different observed success rates. (b) Trajectory-level credit can reward repeated loop closures in successful rollouts and assigns them zero advantage in all-failure groups with identical rewards.
Figure 2. Overview of FESGPO. Free-energy credit scores successor states using successful-path probabilities and costs, then standardizes these scores among transitions sharing the same source state. Structure-aware refinement adjusts episode credit using repeated loop closures, persistent exits, and rollout outcomes. The two components are combined through a weighted sum for token-level PPO updates.
We also added an illustration to Section 4.2 of the revised manuscript showing repeated loop closures, persistent exit actions, and the corresponding credit adjustments. The figure uses illustrative numerical examples to show how credit changes in each case. The new illustration is available in the anonymous repository.