- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
The paper proposes Grounded Relative-Value Advantage Weighting (GRAW), a plug-in for step-level credit assignment in long-horizon LLM agents. Its motivation is an empirical asymmetry: non-parametric advantages retain informative signs despite noisy magnitudes, while learned value movements provide relative ranking information despite unreliable direct credit estimates. GRAW trains a lightweight value head on detached policy representations using rollout-derived targets, then uses predicted value movements to reweight existing advantages within each trajectory. Positive and negative credits are redistributed separately, preserving every original advantage sign and the total absolute credit in each group. The method is evaluated with GiGPO and HGPO on WebShop, ScienceWorld, and ALFWorld using Qwen2.5 models from 1.5B to 7B. Validation includes Monte Carlo credit diagnostics, ranking agreement with environment-progress proxies, ablations, and computational-cost measurements. The reported results show performance improvements with limited additional computation and no extra rollouts.
Critical strengths
-
The authors present a diagnostic finding that the signs of credit estimates are more reliable than their magnitudes, which I find reasonably well supported.
-
The authors conduct extensive evaluations, including multiple-seed experiments and ablation studies. Overall, the conclusions are supported to some extent. I particularly appreciate the method’s control over credit magnitudes, as some studies suggest that these magnitudes can substantially affect performance, with even randomized credit assignment outperforming standard GRPO.
Critical weaknesses
I still recommend rejecting this paper. I believe the authors’ understanding of credit assignment may be somewhat misguided, and the paper’s central insight—that credit magnitudes are unreliable—has already been explored in closely related work. The experimental scope is also relatively narrow and does not adequately establish the method’s general applicability. Please see the Questions for Authors for further details.
Questions for authors
Please address the following questions:
-
In lines 157–158, you state that the reference advantages in Figure 1 are estimated using 32 continuation rollouts. This number may be insufficient; please consider using more samples, such as 64, to assess the robustness of the findings. Figure 1 is useful and directly illustrates your central idea. However, GiGPO and HGPO are closely related methods from the same family, which limits the breadth and generalizability of the evidence. I accept GiGPO as a representative non-parametric credit assignment method, but analyzing a substantially different approach, such as TreeGRPO, could considerably strengthen the paper.
-
In Figure 2, the PPO critic’s MAE appears to increase during training, suggesting that its prediction accuracy deteriorates. However, the PPO results in Table 1 do not appear to differ substantially from those of GRPO. Could you explain this relationship? One possible explanation is that success rates are initially low, making many reference values close to zero and consequently yielding a lower MAE early in training.
-
I am concerned that the paper’s interpretation of credit assignment may be somewhat misguided. Assigning a higher advantage to an action should indicate that it is relatively better than other actions at the same state, rather than merely producing greater absolute progress. For example, under the current policy , an action at state may make only modest progress but still deserve a large advantage if the other available actions perform much worse. This relative comparison is central to methods such as GiGPO. In Figure 3, however, you evaluate the ordering of redistribution multipliers among actions within a trajectory using within-trajectory progress as the reference. I do not find this evaluation sufficiently convincing: credit assignment concerns an action’s advantage relative to alternative actions in group-based methods, or relative to the state-value baseline in PPO-style methods, rather than the amount of progress alone. Could you justify this evaluation criterion and directly assess whether redistribution improves agreement with reference advantages?
-
Regarding the method itself, the central insight—that credit estimates produced by existing credit assignment algorithms can be unreliable—appears closely related to prior work [1, 2]. I did not find direct comparisons with these methods in the paper. Nevertheless, the observation that advantage signs can be more reliable than their magnitudes provides an additional insight that I find intuitively plausible. Could you clarify the contribution beyond these works and provide appropriate comparisons?
[2] StepOPSD: Step-Aware Online Preference Distillation for Agent Reinforcement Learning.
6 comments
Response to Q2
Q2. Critic Value Error and PPO Performance
As noted in Sec. 3.2, increasing MAE partly reflects a broader distribution of MC reference values as the policy improves. We provide further analysis to clarify this trend and its relationship to PPO's performance.
(1) Interpreting the Increase in Critic MAE
We analyzed the PPO critic on WebShop (seed 0) across training checkpoints.
Table A. PPO critic evaluation against MC reference state values.
| Step | Success Rate | MC Value (Mean ± Std.) | Critic MAE | Relative Skill | Pearson Corr. |
|---|---|---|---|---|---|
| 25 | 10% | 0.10 ± 0.16 | 0.101 | −0.02 | 0.23 |
| 50 | 31% | 0.23 ± 0.28 | 0.193 | +0.15 | 0.53 |
| 100 | 60% | 0.42 ± 0.33 | 0.247 | +0.17 | 0.69 |
| 150 | 67% | 0.52 ± 0.37 | 0.192 | +0.39 | 0.77 |
Relative skill measures the critic's MAE improvement over a baseline that predicts the same constant value for every state:
The best constant is the median MC reference value, which minimizes MAE among constant predictions. Thus, a skill of zero indicates performance equal to this baseline, while positive values indicate better predictions. Pearson correlation measures the correlation between critic predictions and MC reference state values.
At step 25, the success rate is only 10%, and MC reference values are concentrated near zero (0.10 ± 0.16). Predicting zero for every state yields an MAE of 0.100, nearly identical to the critic's MAE of 0.101. As training progresses, the MC value mean increases from 0.10 to 0.52, while its standard deviation grows from 0.16 to 0.37. Thus, reference values become larger and more dispersed, contributing to higher absolute prediction errors.
Despite the initial increase in MAE, the critic's relative skill improves from −0.02 to +0.39, indicating 39% lower MAE than the best constant baseline at step 150. Its Pearson correlation also increases from 0.23 to 0.77. These results suggest that the higher MAE reflects changes in the reference-value distribution rather than deteriorating state-value prediction. Nevertheless, despite improvements in relative skill and correlation, the critic still exhibits substantial absolute prediction errors, and the resulting PPO advantages frequently disagree in sign with MC reference advantages (Fig. 2).
(2) Explaining the Performance Similarity Between PPO and GRPO
Despite substantial critic prediction errors and frequent advantage sign disagreements, PPO performs similarly to GRPO. To understand this, we examine how PPO advantages are computed and how they depend on realized trajectory outcomes.
In our PPO configuration, advantages are computed using generalized advantage estimation (GAE), which aggregates temporal-difference (TD) residuals:
\qquad A_t^{\mathrm{PPO}} = \sum_{\ell=0}^{T-t}(\gamma\lambda)^\ell\delta_{t+\ell}.$$ Here, $V_\phi$ is the learned critic, and $\lambda$ controls the contribution of subsequent residuals. In sparse-reward environments, where rewards are given only at the terminal step $T$, the terminal reward $R$ contributes to preceding advantages through $(\gamma\lambda)^{T-t}R$. Thus, although GAE incorporates state-dependent value predictions, its advantage estimates remain strongly influenced by the realized trajectory outcome. **Consequently, critic prediction errors do not necessarily lead to proportional performance degradation, as realized trajectory outcomes still provide a substantial learning signal.** To verify this dependence, we compared two predictors of PPO advantage signs: an outcome-based predictor that assigns positive signs to actions in successful trajectories and negative signs to those in failed trajectories, and the MC reference advantage signs. Both were evaluated on the same actions with nonzero MC reference advantages. Across six checkpoints, the outcome-based predictor agrees with PPO advantage signs in 86.6–100% of cases, compared with only 52.8–60.1% for MC reference signs. The difference is 33.8–41.1 percentage points, with trajectory-bootstrap 95% CIs excluding zero at every checkpoint. These results show that PPO's credit directions are much more closely associated with realized trajectory outcomes than with the relative contributions of individual actions. **These findings help explain why PPO can achieve performance comparable to GRPO despite imperfect critic predictions.** PPO can still learn from trajectory-level outcomes even when its critic is inaccurate. However, as shown in Fig. 2(b), this does not necessarily provide reliable action-level credit, since learned advantages can assign incorrect update directions. Meanwhile, Fig. 2(c) shows that learned value movements retain useful relative ranking information. This motivates GRAW to preserve the update directions of non-parametric advantages while using learned value movements to refine their relative allocation.Response to Q3
Q3. Justification and Validation of Credit Redistribution
We thank the reviewer for this point. We agree that policy advantage measures an action's value relative to alternatives at the same state, rather than its absolute environment progress. We clarify how GRAW relates to this definition and why Fig. 3 provides a meaningful evaluation of credit redistribution.
(1) Policy Advantage and Empirical Credit Estimation
In RL, the action-value function measures the expected return after taking action at state , while the state-value function averages returns over actions sampled from the same policy. The true advantage is defined as
GiGPO and HGPO approximate this relative comparison by contrasting a single realized return for each action against empirical baselines constructed from matching-state visits. In contrast, the true action value averages over possible future continuations. Although these empirical contrasts can provide informative update directions, as shown in Fig. 1(a), their magnitudes remain influenced by downstream decisions in the sampled continuation. Consequently, non-parametric advantages can provide useful directional guidance while misallocating credit magnitudes across actions.
To mitigate this limitation, GRAW separates update direction from credit magnitude. The base non-parametric advantage determines whether an action is reinforced or suppressed, while learned value movements determine how much credit is assigned to each action. GRAW redistributes credit within each trajectory and update-direction group, preserving both the original advantage signs and the total credit magnitude of each group.
Under deterministic transitions and zero intermediate rewards, the true advantage at an intermediate step satisfies
This motivates GRAW's learned value movement , which serves as a relative reweighting signal rather than a direct estimate of advantage.
(2) Justification of the Progress-Based Evaluation in Fig. 3
Although environment progress is not equivalent to policy advantage, Fig. 3 evaluates a specific aspect of GRAW's redistribution mechanism. GiGPO and HGPO already incorporate relative comparisons against sampled alternatives at matching states when constructing their base advantages. GRAW preserves the resulting update directions and only redistributes credit magnitudes among actions sharing the same direction.
Accordingly, Fig. 3 examines whether GRAW assigns larger multipliers to actions associated with greater environment progress in positive-credit groups, or larger decreases in progress in negative-credit groups. This evaluation focuses on how learned values guide the relative allocation of existing credit, rather than determining whether an action is better or worse than alternatives at the same state.
We acknowledge that progress-based evaluation does not directly establish agreement with true policy advantages, as actions with similar environment progress may have different advantages depending on the alternatives available at their respective states. We therefore complement Fig. 3 with a direct comparison against Monte Carlo reference advantages, as presented below.
(3) Direct Validation Against Monte Carlo Reference Advantages
To directly address the reviewer's concern, we evaluate whether GRAW's redistribution improves agreement with MC reference advantages rather than merely environment progress. We conducted this evaluation on GiGPO+GRAW with 64 MC continuations at six checkpoints. Pairwise ranking accuracy and Spearman correlation are measured within each trajectory and base advantage direction group, matching the scope of GRAW's credit redistribution.
The final GRAW advantages achieve better ranking agreement with MC references than the base advantages at all six checkpoints. Average pairwise ranking accuracy increases from 0.594 to 0.624, while within-trajectory Spearman correlation increases from 0.161 to 0.287.
These improvements arise from the complementary roles of the two signals. The base advantages provide more reliable update directions than the learned head movements, with average sign agreement of 0.678 vs. 0.504. In contrast, the learned head provides better relative ranking information, achieving pairwise accuracy of 0.634 vs. 0.594 for the base advantages. GRAW preserves base advantage signs and uses learned value movements to redistribute credit magnitudes.
These results show that GRAW achieves better ranking agreement with MC reference advantages, complementing the progress-based analysis in Fig. 3. Detailed results, including trajectory-bootstrap confidence intervals, are provided in our response to Reviewer qcG2 (W1).
Response to Q1
We thank the reviewer for suggesting a more robust and broader evaluation of non-parametric credit reliability. To address these concerns, we extended the analysis in Fig. 1 in two ways: (1) increasing the number of MC continuations from 32 to 64, and (2) evaluating Tree-GRPO [1], a structurally different non-parametric credit assignment method. We examine both directional reliability and the dependence of empirical credit on sampled outcomes.
(1) Directional Reliability with 64 MC Continuations
We evaluate the sign agreement between non-parametric advantages and MC reference advantages across training checkpoints for GiGPO and Tree-GRPO.
Table A. Sign agreement with MC reference advantages using 64 continuations (95% CIs).
GiGPO's sign agreement increases from 0.552 at step 25 to 0.826 at step 150, with the lower bounds of all 95% CIs exceeding 0.5. Tree-GRPO also achieves sign agreement above 0.5 at all six checkpoints, with confidence intervals excluding 0.5 at five of them. Although its sign agreement is generally lower than GiGPO's, these results suggest that Tree-GRPO also provides useful directional information, consistent with the observation in Fig. 1(a).
(2) Dependence of Empirical Credit on Sampled Outcomes
We further quantify the outcome dependence illustrated in Fig. 1(b). We fit linear regressions with the empirical advantage AB as the dependent variable and examine how much additional variance is explained by the realized trajectory outcome after accounting for the MC reference advantage.
Specifically, R2 denotes the coefficient of determination, measuring the proportion of variance in AB explained by a regression model. We define
R2(MC):=R2(AB∼A^π), R2(MC+Outcome):=R2(AB∼A^π,Y), K:=R2(MC+Outcome)−R2(MC).
Here, A^π is the MC reference advantage, and Y is the binary trajectory outcome (success or failure). R2(MC) measures the variance in empirical credit explained by MC reference advantages alone, while R2(MC+Outcome) measures the variance explained when trajectory outcomes are additionally included. Their difference, K, quantifies the additional explanatory power of realized outcomes beyond MC reference advantages.
Table B. Outcome dependence of GiGPO and Tree-GRPO credit using 64 MC continuations.
For GiGPO, MC reference advantages alone explain relatively little variation in empirical credit (R2(MC)=0.083–0.286), while adding trajectory outcomes increases explained variance by K=0.169–0.622. Some outcome dependence is expected because GiGPO explicitly incorporates trajectory-level credit. Nevertheless, realized outcomes explain substantial additional variation even after accounting for MC reference advantages, consistent with Fig. 1(b).
Tree-GRPO exhibits a similar pattern. MC reference advantages alone explain only 0.054–0.185 of the variation in empirical credit, while trajectory outcomes provide an additional explained variance of K=0.226–0.632. Thus, the dependence on sampled outcomes persists in a structurally different non-parametric estimator, suggesting that this phenomenon is not specific to GiGPO's credit construction.
These results support the two observations in Fig. 1: non-parametric advantages provide useful directional information, but their estimated credit values remain associated with the outcomes of individual sampled trajectories. Importantly, these patterns are observed with 64 MC continuations and extend beyond the GiGPO/HGPO family to Tree-GRPO. We will include these additional analyses in the revised manuscript.
[1] Ji, Yuxiang, et al. "Tree search for llm agent reinforcement learning." International Conference on Learning Representations. 2026.