- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper claims that GiGPO's state-matched grouping shows which action is better at a state, not why the same action works better under some histories. Following this, a new method is proposed to match identical actions across trajectories, take each one’s deviation from a reference return, and regress out episode and state advantages.
Critical strengths
The authors perform a relatively wide range of major evaluations on multiple benchmarks, including the agentic benchmarks and QA benchmarks.
Critical weaknesses
First, the choice of baseline methods is very limited; only GiGPO is compared among step-level methods. Moreover, the central claim is not well discussed or supported: matched actions can differ in return because of state, later decisions or noise, not only history, yet the gap is credited backward to earlier steps. Second, the step of residualization looks ad hoc. A batch-level regression on the episode and state advantages makes each step's credit depend on the whole batch, and it is not clear if the residual is signal or noise.
Questions for authors
The authors may consider more comprehensive ablation studies, instead of using only WebShop 1.5B for component analysis.
3 comments
A broader replay analysis supports this observation. For 138 candidate decision pairs with comparable outcome differences, we measured directional agreement before and after adding action-axis credit:
| Diagnostic | Base advantage | With action-axis credit |
|---|---|---|
| Decision pairs agreeing with the controlled intervention | 93 / 138 | 126 / 138 |
| Directional agreement | 67.4% | 91.3% |
Adding action-axis credit increases the number of intervention-consistent pairs from 93 to 126 and improves directional agreement by 23.9 percentage points. This provides direct empirical support that action-axis credit can improve assignment to historical decisions and correct some errors in the base advantage, beyond simply adding another signal.
Together with the random-matching ablation, these results support action-conditioned comparisons at both the individual-decision and policy-performance levels. We have added these experiments to the revised manuscript and welcome further suggestions on controlled replay and the evaluation of historical credit assignment.
Q3: Is batch-level residual regression justified?
Thank you for this question. We clarify the relationship between action-axis credit and existing advantages through the projection's mathematical properties, numerical checks, and sensitivity to batch composition.
First, residualization removes linear overlap with existing advantages. The mixed action credit T may contain components already explained by the episode advantage E and state advantage S. Including those components again can double-count credit. On the valid support Ω, we therefore project T onto X = [1, E, S] by least squares:
The residual R is used as the additional credit. By construction, R is linearly orthogonal to E and S in the fitted space and represents the component they cannot explain linearly. Numerical checks support the implementation: the design matrices in six actual training batches are all full rank, with normalized orthogonality error below 4.03×10⁻¹⁵.
Second, the residual estimates are stable under changes in batch composition. We fixed the target task's trajectories, raw credit, and training weights, varying only the other tasks used to fit the regression coefficients. The median cosine similarity between residual directions remains 0.9926–0.9995, and the strict ranking-flip rate for the total advantage is only 1.09%–1.14%. Thus, credit directions remain consistent for the vast majority of targets across batch compositions.
Third, residualization allows complementary action information to contribute without double-counting. We provide two forms of evidence concerning signal and noise. The projection systematically removes only components in the linear span of E and S. Separately, the random-matching ablation shows a substantial performance loss when action correspondence is destroyed, providing behavioral evidence that the propagated comparisons carry discriminative information.
These analyses establish the rationale and numerical correctness of residualization and show stability for most targets across batches. The results are included in Appendix H.7. The component ablations in Q4 below provide additional behavioral comparisons of structured credit and noise.
Q4: Why were component ablations conducted only on WebShop 1.5B?
Thank you for pointing out this limitation. Component contributions can vary across tasks and model sizes, so a single setting is insufficient to support a broader conclusion. We have extended the ablations to ALFWorld and WebShop with 1.5B and 7B models, as well as Search 3B, to examine action matching, historical credit propagation, and residualization.
Using GiGPO as the baseline, we remove or replace individual DCGPO components while keeping training and evaluation settings fixed within each task.
Table S4a: Component ablations on ALFWorld and WebShop 1.5B.
| Configuration | ALFWorld success rate (%) | WebShop Score | WebShop success rate (%) |
|---|---|---|---|
| GiGPO | 80.10 ± 5.00 | 82.80 ± 4.60 | 68.00 ± 1.10 |
| DCGPO | 89.80 ± 2.20 | 85.70 ± 0.10 | 71.70 ± 2.50 |
| Without future credit propagation | 86.92 ± 2.73 | 84.15 ± 0.07 | 69.92 ± 2.21 |
| Without residualization | 87.18 ± 2.66 | 85.12 ± 0.51 | 70.81 ± 2.36 |
| Random noise | 80.76 ± 4.82 | 82.96 ± 1.12 | 67.89 ± 2.86 |
| Random action matching | 77.46 ± 3.17 | 78.92 ± 1.24 | 65.83 ± 2.96 |
We applied the same ablation design to ALFWorld and WebShop 7B and Search 3B to test whether the component contributions generalize across model sizes and tasks.
Table S4b: Component ablations on ALFWorld 7B, WebShop 7B, and Search 3B.
| Configuration | ALFWorld 7B success rate (%) | WebShop 7B Score | WebShop 7B success rate (%) | Search 3B Overall (%) |
|---|---|---|---|---|
| GiGPO | 89.8 ± 0.0 | 85.90 ± 0.94 | 77.80 ± 1.73 | 45.50 ± 0.68 |
| DCGPO | 93.40 ± 0.63 | 87.00 ± 0.82 | 78.70 ± 0.86 | 46.50 ± 0.61 |
| Without future credit propagation | 91.57 ± 1.24 | 86.31 ± 0.91 | 78.09 ± 1.32 | 45.83 ± 0.74 |
| Without residualization | 92.13 ± 1.06 | 86.64 ± 0.73 | 78.12 ± 1.18 | 46.02 ± 0.69 |
| Random noise | 90.14 ± 1.32 | 85.98 ± 0.88 | 77.51 ± 1.46 | 45.46 ± 0.77 |
| Random action matching | 83.21 ± 2.18 | 82.36 ± 1.43 | 72.86 ± 2.07 | 43.91 ± 0.92 |
The ablations address three questions.
First, does reliable action matching matter? DCGPO improves over GiGPO across all tested tasks and model sizes. Random matching performs worse than both DCGPO and GiGPO in every setting, showing consistent negative transfer. For example, ALFWorld 1.5B success falls to 77.46, below GiGPO's 80.10. This indicates that the gains depend on semantically comparable action references across histories, rather than arbitrary cross-trajectory information transfer.
Second, does backward credit propagation help? Removing future credit propagation reduces performance in every setting. This is the no-propagation configuration in Table H.4, and the results agree across the two tables. The finding supports our motivation: later actions provide evidence about the quality of historical contexts, and propagating it to earlier decisions complements their local advantage estimates.
Third, what does residualization contribute? The no-residualization variant uses the unprojected mixed action credit, while the random-noise control tests unstructured perturbations. DCGPO outperforms both controls in every setting, and random noise remains close to the GiGPO baseline. Random perturbations do not reproduce DCGPO's gains, and removing the projection loses part of the benefit. This supports the value of residual action information beyond the existing advantages and addresses the signal-versus-noise question in Q3.
These ablations distinguish the contributions of action comparison, historical propagation, and residualization across tasks and model sizes. Together with Tables H.3–H.4, they provide broader empirical support for the DCGPO components.
Dear Reviewer 5DXg,
Thank you for your detailed comments on baseline coverage, credit assignment, and residualization. We apologize that the original manuscript did not adequately cover related baselines or validate the proposed mechanisms. Your questions prompted us to examine whether action-axis credit adds value to a stronger history-aware baseline, whether later comparison evidence informs the relevant earlier decisions, and whether residualization preserves useful information. We address these questions below, with additional diagnostics and component ablations across tasks and model sizes.
Q1: Is comparison with GiGPO alone sufficient to establish effectiveness among step-level methods?
GiGPO is a representative step-level credit-assignment method and a direct baseline for testing our incremental contribution. It combines episode- and state-level advantages to provide fine-grained credit without extra rollouts or a learned evaluator. DCGPO builds on this framework by adding action-conditioned comparisons and backward credit propagation, so a controlled comparison with GiGPO directly tests the added mechanisms.
To assess whether action-axis credit also helps a stronger history-aware method, we added an HGPO comparison. HGPO refines state reference groups using historical context. DCGPO instead compares the subsequent outcomes of the same action across histories and propagates that evidence to the decisions that established those conditions. We combined the two mechanisms and compared them under the same model, training budget, rollout count, and evaluation settings.
DCGPO with a history window improves the three metrics over HGPO by 1.79 percentage points, 2.24 points, and 2.67 percentage points, respectively. This supports the complementarity of the mechanisms. History-aware grouping refines the state references to improve comparability; the action axis holds action semantics fixed across histories and propagates residualized credit to the decisions that formed those histories. Their combination improves performance within the original rollout budget and without an additional learned evaluator. The incremental gains over both GiGPO and HGPO support the core contribution of DCGPO: an action-based comparison dimension that can complement different advantage estimators for long-range agent credit assignment.
Q2: Later outcomes also depend on states, subsequent decisions, and randomness. Why propagate credit to earlier actions?
We agree that later decisions and environmental randomness also affect outcomes, and apologize that the original manuscript did not provide direct experimental evidence on this point. We therefore conducted controlled replay using actual training trajectories to test whether action-axis credit reflects the contribution of earlier decisions.
The motivation is that an early action can establish conditions whose effects become apparent only in a later operation. For example, two trajectories may both end with a purchase, but differ in whether the correct model was selected beforehand. The purchase action cannot determine that earlier selection. Comparing the same operation across histories can therefore provide additional learning evidence for earlier decisions.
To test this mechanism, we replayed fixed execution plans from the same initial conditions, replacing or deleting only a candidate prerequisite action. We fixed the remaining plan and environmental randomness while allowing downstream states to change naturally in response to the intervention. We then checked whether the outcome difference agreed with the originally assigned credit.
A WebShop example illustrates the mechanism. For a task requiring a large-sized garment, we performed single-step deletion tests on a successful trajectory:
The necessary size-selection action initially had negative local action-axis evidence, but received positive additional credit after later evidence was incorporated. Deleting it caused failure. This example shows how later outcomes can help identify prerequisites that matter for task completion. Some redundant operations also received positive credit, leaving room to improve attribution.