acceptodds
Under review as a conference paper at ICLR 2027

DCGPO: Dual-Conditioned Group Policy Optimization for Long-Horizon LLM Agents

Abstract

Group-based reinforcement learning (RL) has become an important paradigm for training long-horizon LLM agents, enabling critic-free advantage estimation through relative comparisons within rollout groups. Recent methods improve credit granularity through step-level grouping, comparing actions taken from recurring environment states. However, state-conditioned grouping answers only which action is better under a matched state; it cannot address the complementary action-conditioned question: why does the same action yield different utility across historical contexts? We call this missing comparison dimension the cross-context incomparability of action utility. We propose DCGPO (Dual-Conditioned Group Policy Optimization), which integrates state-conditioned action comparison with action-conditioned cross-context comparison in a unified advantage estimator. For each action occurrence with cross-trajectory matches in a rollout group, DCGPO constructs a weighted reference return from matching occurrences in other trajectories and defines its action-axis relative advantage as the deviation of the current return from this reference. Because this advantage reflects the relative continuation outcome of the same action under an already-formed context, DCGPO propagates it backward, with distance-based discounting, to the preceding decisions that shaped that context rather than assigning it directly to the current action. It then removes components already explained by episode- and state-level advantages and combines episode-level trajectory quality, state-conditioned action advantage, and residual action-axis credit into a three-axis step-level relative advantage. Experiments on ALFWorld, WebShop, search-augmented question answering, and agentic safety show consistent gains across tasks and model scales, highlighting dual-conditioned comparison as an efficient complement to state-axis credit across diverse long-horizon agent settings.

Primary area:
reinforcement learning
Keywords:
Agentic RL, Reinforcement Learning, Credit Assignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5DXgOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper claims that GiGPO's state-matched grouping shows which action is better at a state, not why the same action works better under some histories. Following this, a new method is proposed to match identical actions across trajectories, take each one’s deviation from a reference return, and regress out episode and state advantages.

Critical strengths

The authors perform a relatively wide range of major evaluations on multiple benchmarks, including the agentic benchmarks and QA benchmarks.

Critical weaknesses

First, the choice of baseline methods is very limited; only GiGPO is compared among step-level methods. Moreover, the central claim is not well discussed or supported: matched actions can differ in return because of state, later decisions or noise, not only history, yet the gap is credited backward to earlier steps. Second, the step of residualization looks ad hoc. A batch-level regression on the episode and state advantages makes each step's credit depend on the whole batch, and it is not clear if the residual is signal or noise.

Questions for authors

The authors may consider more comprehensive ablation studies, instead of using only WebShop 1.5B for component analysis.

3 comments

Dear Reviewer 5DXg,

Thank you for your detailed comments on baseline coverage, credit assignment, and residualization. We apologize that the original manuscript did not adequately cover related baselines or validate the proposed mechanisms. Your questions prompted us to examine whether action-axis credit adds value to a stronger history-aware baseline, whether later comparison evidence informs the relevant earlier decisions, and whether residualization preserves useful information. We address these questions below, with additional diagnostics and component ablations across tasks and model sizes.

Q1: Is comparison with GiGPO alone sufficient to establish effectiveness among step-level methods?

GiGPO is a representative step-level credit-assignment method and a direct baseline for testing our incremental contribution. It combines episode- and state-level advantages to provide fine-grained credit without extra rollouts or a learned evaluator. DCGPO builds on this framework by adding action-conditioned comparisons and backward credit propagation, so a controlled comparison with GiGPO directly tests the added mechanisms.

To assess whether action-axis credit also helps a stronger history-aware method, we added an HGPO comparison. HGPO refines state reference groups using historical context. DCGPO instead compares the subsequent outcomes of the same action across histories and propagates that evidence to the decisions that established those conditions. We combined the two mechanisms and compared them under the same model, training budget, rollout count, and evaluation settings.

ConfigurationALFWorld success rate (%)WebShop ScoreWebShop success rate (%)
HGPO95.35 ± 1.1989.62 ± 1.8478.95 ± 0.90
DCGPO with a history window97.14 ± 0.9691.86 ± 1.3881.62 ± 1.12

DCGPO with a history window improves the three metrics over HGPO by 1.79 percentage points, 2.24 points, and 2.67 percentage points, respectively. This supports the complementarity of the mechanisms. History-aware grouping refines the state references to improve comparability; the action axis holds action semantics fixed across histories and propagates residualized credit to the decisions that formed those histories. Their combination improves performance within the original rollout budget and without an additional learned evaluator. The incremental gains over both GiGPO and HGPO support the core contribution of DCGPO: an action-based comparison dimension that can complement different advantage estimators for long-range agent credit assignment.

Q2: Later outcomes also depend on states, subsequent decisions, and randomness. Why propagate credit to earlier actions?

We agree that later decisions and environmental randomness also affect outcomes, and apologize that the original manuscript did not provide direct experimental evidence on this point. We therefore conducted controlled replay using actual training trajectories to test whether action-axis credit reflects the contribution of earlier decisions.

The motivation is that an early action can establish conditions whose effects become apparent only in a later operation. For example, two trajectories may both end with a purchase, but differ in whether the correct model was selected beforehand. The purchase action cannot determine that earlier selection. Comparing the same operation across histories can therefore provide additional learning evidence for earlier decisions.

To test this mechanism, we replayed fixed execution plans from the same initial conditions, replacing or deleting only a candidate prerequisite action. We fixed the remaining plan and environmental randomness while allowing downstream states to change naturally in response to the intervention. We then checked whether the outcome difference agreed with the originally assigned credit.

A WebShop example illustrates the mechanism. For a task requiring a large-sized garment, we performed single-step deletion tests on a successful trajectory:

Intervention in the successful trajectorySuccessTask ScoreOriginally assigned additional credit
Original successful trajectoryYes1.0—
Delete the action selecting size largeNo0.9+0.046265
Delete a later repeated color selectionYes1.0+0.027802
Delete a later repeated target-demographic selectionYes1.0+0.034932

The necessary size-selection action initially had negative local action-axis evidence, but received positive additional credit after later evidence was incorporated. Deleting it caused failure. This example shows how later outcomes can help identify prerequisites that matter for task completion. Some redundant operations also received positive credit, leaving room to improve attribution.

A broader replay analysis supports this observation. For 138 candidate decision pairs with comparable outcome differences, we measured directional agreement before and after adding action-axis credit:

DiagnosticBase advantageWith action-axis credit
Decision pairs agreeing with the controlled intervention93 / 138126 / 138
Directional agreement67.4%91.3%

Adding action-axis credit increases the number of intervention-consistent pairs from 93 to 126 and improves directional agreement by 23.9 percentage points. This provides direct empirical support that action-axis credit can improve assignment to historical decisions and correct some errors in the base advantage, beyond simply adding another signal.

Together with the random-matching ablation, these results support action-conditioned comparisons at both the individual-decision and policy-performance levels. We have added these experiments to the revised manuscript and welcome further suggestions on controlled replay and the evaluation of historical credit assignment.

Q3: Is batch-level residual regression justified?

Thank you for this question. We clarify the relationship between action-axis credit and existing advantages through the projection's mathematical properties, numerical checks, and sensitivity to batch composition.

First, residualization removes linear overlap with existing advantages. The mixed action credit T may contain components already explained by the episode advantage E and state advantage S. Including those components again can double-count credit. On the valid support Ω, we therefore project T onto X = [1, E, S] by least squares:

β^=arg⁡min⁡β∥TΩ−XΩβ∥22.\hat{\beta} = \arg\min_{\beta}\|T_{\Omega} - X_{\Omega}\beta\|_2^2.

The residual R is used as the additional credit. By construction, R is linearly orthogonal to E and S in the fitted space and represents the component they cannot explain linearly. Numerical checks support the implementation: the design matrices in six actual training batches are all full rank, with normalized orthogonality error below 4.03×10⁻¹⁵.

Second, the residual estimates are stable under changes in batch composition. We fixed the target task's trajectories, raw credit, and training weights, varying only the other tasks used to fit the regression coefficients. The median cosine similarity between residual directions remains 0.9926–0.9995, and the strict ranking-flip rate for the total advantage is only 1.09%–1.14%. Thus, credit directions remain consistent for the vast majority of targets across batch compositions.

Third, residualization allows complementary action information to contribute without double-counting. We provide two forms of evidence concerning signal and noise. The projection systematically removes only components in the linear span of E and S. Separately, the random-matching ablation shows a substantial performance loss when action correspondence is destroyed, providing behavioral evidence that the propagated comparisons carry discriminative information.

These analyses establish the rationale and numerical correctness of residualization and show stability for most targets across batches. The results are included in Appendix H.7. The component ablations in Q4 below provide additional behavioral comparisons of structured credit and noise.

Q4: Why were component ablations conducted only on WebShop 1.5B?

Thank you for pointing out this limitation. Component contributions can vary across tasks and model sizes, so a single setting is insufficient to support a broader conclusion. We have extended the ablations to ALFWorld and WebShop with 1.5B and 7B models, as well as Search 3B, to examine action matching, historical credit propagation, and residualization.

Using GiGPO as the baseline, we remove or replace individual DCGPO components while keeping training and evaluation settings fixed within each task.

Table S4a: Component ablations on ALFWorld and WebShop 1.5B.

ConfigurationALFWorld success rate (%)WebShop ScoreWebShop success rate (%)
GiGPO80.10 ± 5.0082.80 ± 4.6068.00 ± 1.10
DCGPO89.80 ± 2.2085.70 ± 0.1071.70 ± 2.50
Without future credit propagation86.92 ± 2.7384.15 ± 0.0769.92 ± 2.21
Without residualization87.18 ± 2.6685.12 ± 0.5170.81 ± 2.36
Random noise80.76 ± 4.8282.96 ± 1.1267.89 ± 2.86
Random action matching77.46 ± 3.1778.92 ± 1.2465.83 ± 2.96

We applied the same ablation design to ALFWorld and WebShop 7B and Search 3B to test whether the component contributions generalize across model sizes and tasks.

Table S4b: Component ablations on ALFWorld 7B, WebShop 7B, and Search 3B.

ConfigurationALFWorld 7B success rate (%)WebShop 7B ScoreWebShop 7B success rate (%)Search 3B Overall (%)
GiGPO89.8 ± 0.085.90 ± 0.9477.80 ± 1.7345.50 ± 0.68
DCGPO93.40 ± 0.6387.00 ± 0.8278.70 ± 0.8646.50 ± 0.61
Without future credit propagation91.57 ± 1.2486.31 ± 0.9178.09 ± 1.3245.83 ± 0.74
Without residualization92.13 ± 1.0686.64 ± 0.7378.12 ± 1.1846.02 ± 0.69
Random noise90.14 ± 1.3285.98 ± 0.8877.51 ± 1.4645.46 ± 0.77
Random action matching83.21 ± 2.1882.36 ± 1.4372.86 ± 2.0743.91 ± 0.92

The ablations address three questions.

First, does reliable action matching matter? DCGPO improves over GiGPO across all tested tasks and model sizes. Random matching performs worse than both DCGPO and GiGPO in every setting, showing consistent negative transfer. For example, ALFWorld 1.5B success falls to 77.46, below GiGPO's 80.10. This indicates that the gains depend on semantically comparable action references across histories, rather than arbitrary cross-trajectory information transfer.

Second, does backward credit propagation help? Removing future credit propagation reduces performance in every setting. This is the no-propagation configuration in Table H.4, and the results agree across the two tables. The finding supports our motivation: later actions provide evidence about the quality of historical contexts, and propagating it to earlier decisions complements their local advantage estimates.

Third, what does residualization contribute? The no-residualization variant uses the unprojected mixed action credit, while the random-noise control tests unstructured perturbations. DCGPO outperforms both controls in every setting, and random noise remains close to the GiGPO baseline. Random perturbations do not reproduce DCGPO's gains, and removing the projection loses part of the benefit. This supports the value of residual action information beyond the existing advantages and addresses the signal-versus-noise question in Q3.

These ablations distinguish the contributions of action comparison, historical propagation, and residualization across tasks and model sizes. Together with Tables H.3–H.4, they provide broader empirical support for the DCGPO components.

Reviewer_8emNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes DCGPO, a critic-free credit assignment method for long-horizon LLM agents that complements GiGPO’s state-conditioned action comparison with an action-conditioned comparison across histories. For each action that has semantic matches in other trajectories of the same rollout group, DCGPO computes the deviation of its continuation return from a weighted reference built from the matched occurrences. It propagates this signal backward to preceding decisions with distance-based discounting, and removes the components already explained by episode- and state-level advantages via linear residualization. The resulting residual credit is added to the GiGPO advantage. DCGPO is evaluated on ALFWorld, WebShop, search-augmented QA, and an agentic safety benchmark with Qwen2.5 models from 1.5B to 7B, reporting gains over GiGPO and other RL baselines.

Critical strengths
  1. The idea of complementing state-conditioned comparison with an action-conditioned view is intuitive and clearly illustrated (Fig. 1).
  2. The method reuses existing rollouts and requires no critic, extra model calls, or additional environment interactions.
  3. The evaluation covers four diverse agent settings and multiple model scales.
Critical weaknesses
  1. The method assumes that differences in the continuation returns of a matched action reflect the quality of the preceding history, and propagates this signal to earlier decisions. However, the continuation return also depends on later decisions and environment stochasticity. The authors acknowledge this attribution is not a causal estimate, but provide no validation of it.

  2. The action-axis term is capped at 10% of the RMS of the episode-plus-state advantage and further weighted by 0.5 (Table A.1, Appendix D), yet the reported gains are large (e.g., 80.1 to 89.8 on ALFWorld at 1.5B). The paper does not explain how such a small additional signal produces these improvements.

  3. The Prompting, PPO, RLOO, and GRPO results in Table 1 are identical to those reported in the GiGPO paper, whereas the GiGPO results are not (e.g., 80.1±5.0 on ALFWorld at 1.5B). This raises concerns about whether the GiGPO baseline was reproduced under a comparable setup.

  4. The number of training seeds is not reported. Several entries have a standard deviation of exactly 0.0 (e.g., GiGPO 7B ALFWorld 89.8±0.0), and the Search experiments use a single seed with gains of 1.0 point.

  5. The ablation results in Fig. 4(a) are inconsistent with Table 1. The full method (C3) reaches 74.2 success / 86.5 score on WebShop 1.5B in Fig. 4(a), versus 71.7 / 85.7 in Table 1. The text also reports 91.3% coverage for C2, whereas the figure shows 91.8% for C2 and 91.3% for C3.

  6. In the training-cost comparison, GRPO takes 157.8 s per step, more than twice GiGPO's 70.5 s, although GiGPO only adds grouping computation on top of GRPO. Hyperparameters such as the maximum number of environment steps and rollout length should be reported to ensure the comparison is fair.

Questions for authors
  1. Were the GiGPO results in Table 1 obtained by rerunning GiGPO? If so, why do they differ from the original paper while the other baselines are copied from it, and were GiGPO and DCGPO trained under identical configurations?

  2. Could you report the hyperparameters relevant to the training-cost comparison, such as the maximum number of environment steps, rollout length, group size, and batch size?

  3. How many training seeds were used for each result, and why do several entries have a standard deviation of exactly 0.0?

  4. Could you reconcile the WebShop 1.5B results and coverage values in Fig. 4(a) with those in Table 1 and the main text?

  5. Is there any empirical evidence that the propagated action-axis credit is attributed to the correct earlier decisions, for example through controlled counterfactual rollouts?

  6. How sensitive is DCGPO to the propagation horizon K, the mixing weight α, and the action weight ω_A?

5 comments

Dear Reviewer 8emN,

Thank you for your careful review and for recognizing the dual-conditioned comparison, efficient rollout reuse, and evaluation across tasks. Your questions about attribution, baseline comparability, and reporting are central to assessing our conclusions. We revisited the manuscript, result provenance, and implementation, and provide the additional experiments and clarifications below.

Q1: Were the GiGPO results in Table 1 rerun? Why do they differ from the original paper, and were GiGPO and DCGPO evaluated under the same settings?

Thank you for examining the fairness of the comparison. We apologize that the original manuscript did not clearly identify the source of the GiGPO results.

The GiGPO results in Table 1 are from our local reruns, not copied from the original paper. Using the training configuration reported in that paper, our reruns achieved stronger ALFWorld and WebShop results than those originally reported and also substantially outperformed the other baselines in the table. We report these stronger reruns to avoid overstating DCGPO's gains against a weaker baseline. In contrast, the Prompting, PPO, RLOO, and GRPO results are taken from the GiGPO paper. We will clarify these sources and the corresponding experimental settings in the next revision.

Our local GiGPO and DCGPO runs use the same model, training and evaluation settings, learning rate, training budget, and other shared hyperparameters. DCGPO adds action-conditioned comparisons and backward credit propagation to GiGPO's episode- and state-level advantages. This controlled comparison isolates the additional contribution of action-axis credit assignment.

We will explicitly identify GiGPO as a local rerun and provide the configuration details in the next revision to make the comparison transparent.

Q2: What hyperparameters were used for the training-cost comparison? Why is GRPO much slower than GiGPO?

Thank you for raising this issue. We rechecked the original configurations and per-step timing logs. The original manuscript did not adequately explain how differences in realized interaction workload affect training time, which could lead readers to interpret the step-time gap as a difference in algorithmic computational efficiency. We apologize for this omission.

Training configuration. GRPO and GiGPO share the same model, sampling and optimization hyperparameters, and GPU allocation:

ParameterGRPO and GiGPO settings
Policy model / taskQwen2.5-3B-Instruct / Agentic Safety
Training examples2,000
Query batch size / rollouts per query16 / 5 (80 trajectories per batch)
Maximum environment interaction steps10
History window (GiGPO)4
Maximum prompt / per-turn response length8,192 / 512 tokens
PPO mini-batch / per-GPU micro-batch16 / 4
Learning rate / PPO epochs / precision1e-6 / 1 / bf16
Sampling parameterstemperature=1, top-p=1, do_sample=true
GPU allocation4 policy GPUs + 4 safety-scoring GPUs

Measured training time. We averaged timings over training updates 1–124, excluding periodic evaluation, and decomposed step time into the main training stages:

Metric (mean per update)GRPOGiGPODCGPO
Training step time (s)157.8070.5281.45
Generation and environment interaction, including terminal scoring (s)137.9064.6370.79
Actor update (s)12.483.016.40
Advantage construction (s)0.0230.0640.079
Tokens per training batch222,143.98104,686.65111,818.14
Mean environment interaction rounds2.0811.1171.170

The GRPO–GiGPO step-time gap primarily reflects interaction workload, rather than advantage computation. Although both methods use the same rollout budget and interaction limit, GiGPO's policy increasingly answers directly and terminates earlier during training, reducing subsequent tool calls and environment interactions. Its mean interaction count is 1.117 rounds, compared with 2.081 for GRPO, and it processes fewer tokens per batch. The shorter GiGPO step time is therefore mainly associated with the interaction behavior of the learned policy, rather than acceleration from grouped advantage computation itself.

The stage-level timings support this explanation. Of the 87.28-second mean step-time gap, 73.27 seconds (approximately 84%) comes from generation and environment interaction. Advantage construction takes only 0.064 seconds for GiGPO and 0.079 seconds for DCGPO, less than 0.1% of their respective step times. DCGPO thus adds historical credit using action information from existing rollouts at a small additional advantage-computation cost, without extra trajectory sampling. We have added the stage-level timings and workload statistics to the revised manuscript to distinguish measured training time from advantage-estimation overhead.

Q3: How many training seeds were used? Why are some standard deviations reported as 0.0?

Thank you for pointing out that the seed counts and zero standard deviations were insufficiently explained. We apologize for the ambiguity.

Our local GiGPO and DCGPO results in Table 1 each use three independent training seeds and report the mean and standard deviation. We will make the result sources and seed counts explicit in the next revision.

The 0.0 standard deviations mainly reflect ALFWorld's discrete success-rate metric and agreement across the three seeds. Table 1 contains 6 entries with a reported standard deviation of 0.0, all on ALFWorld: 5 category-level success rates and 1 overall success rate.

For example, DCGPO 1.5B achieves 100% success in the Pick category for all three seeds, yielding zero standard deviation. ALFWorld computes success rates from a finite set of binary task outcomes, so different trajectories and per-task outcomes can also yield the same aggregate rate. For instance, all three GiGPO 7B seeds achieve approximately 89.84% overall success, despite differences across task categories.

A reported standard deviation of 0.0 therefore means that no difference was observed in the aggregate success rate across the three seeds, or that the difference was below the reporting precision. It does not imply identical model behavior. We have clarified the standard-deviation calculation and numerical precision and provided per-seed results for verification in the revised manuscript.

Q4: How do you explain the discrepancies in performance and coverage between Figure 4(a), Table 1, and the text?

1. Performance discrepancies between Figure 4(a) and Table 1. Thank you for identifying the inconsistency. We have updated Figure 4(a) using the means over three independent training seeds reported in Appendix Table H.1, while retaining the original definitions and names of C1, C2, and C3. We have also updated the caption and corresponding text. C3 now has a WebShop Score of 85.70 and success rate of 71.70%, consistent with the main results table. The figure displays means only; the appendix table reports the full means and sample standard deviations. The truncated vertical axes are marked on the axes and noted in the caption.

2. Coverage discrepancies between Figure 4(a) and the text. We have corrected C2's coverage from 91.8% to 91.3%, retained C3's coverage at 91.3%, and updated the caption and text accordingly. Figure 4 retains the original panel structure: (a) component results and coverage, and (b) training time.

Q5: Is there empirical evidence that action-axis credit is assigned appropriately to earlier decisions?

Thank you for raising this question. We agree that subsequent outcomes also depend on later decisions and environmental randomness; action-axis credit is therefore not equivalent to strict causal attribution. Your comment motivated controlled replay using actual training trajectories to test whether propagated credit reflects the effects of earlier decisions on final outcomes.

The motivation for backward credit propagation is that the effects of conditions established by earlier decisions may become apparent only when later actions are executed. A current action cannot determine its preceding history. Comparing the same semantic action across histories can therefore provide additional credit information for the earlier decisions that created those contexts.

Controlled replay. We first reproduced the original trajectories, then replaced or deleted only a candidate prerequisite action while fixing the initial conditions and all other executed actions. We checked whether the outcome change agreed with the credit assigned during training. The experiment uses intermediate trajectories from three training seeds of a DCGPO variant that retains action matching, backward credit propagation, and residualization, with an additional filter for distinct credit-recipient positions. Replay is used only for post hoc diagnosis; it adds no policy sampling during training and does not enter advantage recomputation.

A WebShop example illustrates the value of later evidence. For a task requiring a large-sized garment, we deleted individual candidate actions while fixing the remaining operations:

InterventionSuccessTask ScoreOriginally assigned additional credit
Original successful trajectoryYes1.0—
Delete the action selecting size largeNo0.9+0.046265
Delete a later repeated color selectionYes1.0+0.027802
Delete a later repeated target-demographic selectionYes1.0+0.034932

The necessary size-selection action initially had negative local action-axis evidence, but received positive additional credit after incorporating later evidence. Deleting it caused failure. This is a concrete example in which later evidence helps identify a necessary prerequisite decision. However, the two deletions that preserved success also received positive credit, showing that the estimator can still assign credit to redundant steps. We inspected all single-step deletions in two control trajectories, rather than selecting only favorable interventions.

We also conducted a broader controlled test using actual training trajectories. Across 768 successfully reproduced trajectories, we completed 2,343 fixed-plan cross-replay pairs. Of these, 138 pairs produced nonzero outcome differences, spanning 10 tasks.

DiagnosticBase advantageWith action-axis credit
Decision pairs agreeing with the controlled intervention93 / 138126 / 138
Directional agreement67.4%91.3%

Adding action-axis credit increases directional agreement with the controlled interventions from 67.4% to 91.3%, a gain of 23.9 percentage points. It also yields intervention-consistent preferences for 8 of the 9 decision pairs that the base advantage could not distinguish. Together with the WebShop example, these results directly support the value of later comparison evidence for assigning credit to earlier decisions.

The random-matching ablation, which reduces ALFWorld success from 89.80% to 77.46%, provides complementary evidence at the policy-performance level. The revised supplementary material includes the complete replay statistics. We welcome further discussion of the replay design and evaluation of historical credit assignment.

Q6: How sensitive is DCGPO to propagation horizon K, mixing weight α, and action weight ω_A? How can a small action-credit contribution yield substantial gains?

These parameters control the propagation range, the balance between local and later evidence, and the strength of the added residual credit, respectively. We report one-variable-at-a-time sweeps around the reference configuration K=8, α=0.25, and ω_A=0.5, keeping all other settings fixed.

1. Propagation horizon K.

SettingALFWorld success rate (%)WebShop ScoreWebShop success rate (%)
K=889.80 ± 2.2085.70 ± 0.1071.70 ± 2.50
K=489.27 ± 2.1685.48 ± 0.3671.39 ± 2.31
K=284.47 ± 2.8981.96 ± 0.8266.61 ± 2.74

Propagation range affects performance, but results are relatively stable once the window covers enough history. Increasing K from 2 to 4 improves ALFWorld success, WebShop Score, and WebShop success by 4.80 percentage points, 3.52 points, and 4.78 percentage points, respectively. Increasing it further to K=8 adds only 0.53 percentage points, 0.22 points, and 0.31 percentage points. This is consistent with our motivation: later comparison evidence must reach the historical decisions it can inform. Short windows can miss important prerequisites, while the marginal benefit decreases as more relevant decisions are covered. The results therefore suggest robustness over the tested range with sufficient propagation, rather than dependence on one precise K value.

2. Mixing weight α. α weights the local action-axis evidence, and 1−α weights evidence propagated from later matched actions. Here, K=8 and ω_A=0.5 are fixed.

SettingALFWorld success rate (%)WebShop ScoreWebShop success rate (%)
α=089.52 ± 2.6185.24 ± 0.4371.86 ± 2.77
α=0.2589.80 ± 2.2085.70 ± 0.1071.70 ± 2.50
α=0.589.36 ± 2.3785.82 ± 0.2871.48 ± 2.42
α=0.7588.21 ± 2.6884.87 ± 0.5570.43 ± 2.81
α=186.84 ± 2.9184.29 ± 0.6469.58 ± 2.93

All three metrics remain close over α = 0–0.5, then decline as α increases to 0.75 and 1. This supports the value of historical credit from later matched actions: placing too much weight on the current action weakens long-range credit assignment. DCGPO performs well across a broad mixing range without requiring a precisely tuned α.

Reviewer_1BQnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DCGPO, which compares matched actions across rollout trajectories and passes their relative continuation outcomes back to earlier decisions. It removes overlap with episode- and state-level advantages and reports improvements across several agent tasks without extra rollouts or a learned critic.

Critical strengths
  1. The proposed comparison of the same action across different histories offers an interesting complement to state-based grouping.
  2. The method reuses existing rollouts, with clear ALFWorld gains and task-level ablations supporting its main components.
Critical weaknesses
  1. HGPO is a particularly relevant missing baseline, since it also addresses history-dependent advantage estimation without extra models or rollouts.
  2. The ablations do not fully clarify how much the gains depend on the quality of action matching.
Questions for authors
  1. Do the gains survive shuffled action matches, and how sensitive are they to matching granularity and propagation distance?
  2. How does DCGPO compare with HGPO under the same setup, and are the smaller QA and WebShop gains consistent across seeds?
2 comments

Dear Reviewer 1BQn,

Thank you for your careful review and constructive feedback. We appreciate your recognition that comparing the same action across histories complements state-based grouping and efficiently reuses existing rollouts. Your comments on the missing HGPO baseline and the dependence on action-matching quality motivated controlled matching ablations and a comparison with HGPO under the same settings. We address both questions below.

Q1: Do the gains persist with random action matching? How sensitive is DCGPO to matching granularity and propagation distance?

We conducted controlled experiments with three training seeds, varying action correspondence, matching thresholds, and propagation distance while holding the remaining training and evaluation settings fixed.

1. The role of action correspondence. Random action matching preserves the cross-trajectory comparison budget, propagation, residualization, and exclusion of the target trajectory. It randomizes only the action correspondence.

ConfigurationALFWorld success rate (%)WebShop ScoreWebShop success rate (%)
DCGPO89.80 ± 2.2085.70 ± 0.1071.70 ± 2.50
DCGPO with random action matching77.46 ± 3.1778.92 ± 1.2465.83 ± 2.96

Random matching reduces ALFWorld success by 12.34 percentage points, WebShop Score by 6.78 points, and WebShop success by 5.87 percentage points. Both WebShop metrics also fall below the no-propagation variant. These results indicate that the gains depend on semantically comparable actions, rather than merely on additional cross-trajectory comparisons or credit propagation. DCGPO compares the subsequent outcomes of the same action across histories to inform earlier decisions. Random matching confounds the effects of history with differences in what the actions do. Propagating these unreliable comparisons can then interfere with the optimization of earlier decisions.

2. Sensitivity to matching granularity. By default, DCGPO exactly matches normalized action types and complete arguments. For relaxed matching, we retain the action-type constraint and use SequenceMatcher to measure argument-string similarity, with thresholds τ = 0.95 and 0.85. Propagation settings remain unchanged.

ConfigurationALFWorld success rate (%)WebShop ScoreWebShop success rate (%)
Default exact matching89.80 ± 2.2085.70 ± 0.1071.70 ± 2.50
String matching: τ = 0.9589.63 ± 2.1485.76 ± 0.3271.92 ± 2.27
String matching: τ = 0.8587.94 ± 2.7685.03 ± 0.8370.46 ± 2.93

The sweep shows a trade-off between matching coverage and reference comparability. At τ = 0.95, all three metrics are close to exact matching, suggesting that near-identical actions can expand coverage while largely preserving correspondence quality. At τ = 0.85, all three metrics decline moderately. Broader matching can group actions with different execution targets, confounding historical effects with functional differences between actions. DCGPO therefore tolerates small differences in string expression, but useful credit propagation still requires comparable behavior as specified by the action arguments.

3. The role of propagation distance. We set the maximum propagation distance to 100%, 50%, 25%, or 0 of the default range, with all other settings fixed. Zero corresponds to no backward propagation.

ConfigurationALFWorld success rate (%)WebShop ScoreWebShop success rate (%)
Full propagation (manuscript results)89.80 ± 2.2085.70 ± 0.1071.70 ± 2.50
50% propagation distance89.27 ± 2.1685.48 ± 0.3671.39 ± 2.31
25% propagation distance84.47 ± 2.8981.96 ± 0.8266.61 ± 2.74
No propagation86.92 ± 2.7384.15 ± 0.0769.92 ± 2.21

Half the default range retains performance close to full propagation. Reducing the range to 25%, however, substantially lowers all three metrics, even below the no-propagation variant. This nonmonotonic pattern suggests that a shorter window is not necessarily better: useful propagation depends on reaching the relevant historical decisions. An overly short window may miss prerequisites that affect later outcomes, whereas a broader range can capture more of the relevant history. This also motivates future work on selecting credit recipients adaptively using action importance and context, while retaining sufficient historical coverage.

Q2: How does DCGPO compare with HGPO under the same settings? Are the smaller QA and WebShop gains consistent across seeds?

Thank you for highlighting HGPO. Its history-aware advantage estimation requires neither an additional model nor extra rollouts, making it an important baseline. HGPO uses state and historical context to construct comparison groups for the current decision. DCGPO compares subsequent outcomes of the same action across histories and propagates that evidence to the earlier decisions that formed those histories. The mechanisms address reference construction and historical credit assignment, respectively, and are naturally complementary. We therefore compared HGPO with DCGPO augmented with a history window under the same training and evaluation settings.

ConfigurationALFWorld success rate (%)WebShop ScoreWebShop success rate (%)
HGPO95.35 ± 1.1989.62 ± 1.8478.95 ± 0.90
DCGPO with a history window97.14 ± 0.9691.86 ± 1.3881.62 ± 1.12

DCGPO with a history window improves ALFWorld success, WebShop Score, and WebShop success over HGPO by 1.79 percentage points, 2.24 points, and 2.67 percentage points, respectively. These results support the complementarity of history-aware grouping and action-axis credit: the former improves the comparison references, while the latter uses outcome differences across histories to inform the decisions that created them.

Consistency across seeds on QA and WebShop. We report paired DCGPO–GiGPO differences for each training seed, together with the mean difference after excluding the seed with the largest gain.

Metric: DCGPO − GiGPOSeed 0Seed 1Seed 2Mean difference ± standard deviationSeeds with positive gainsMean excluding the largest gain
QA Overall (percentage points)+1.38+1.09+1.53+1.33 ± 0.223/3+1.24
WebShop 7B Score+0.44+1.18+1.68+1.10 ± 0.623/3+0.81
WebShop 7B success rate+0.15+0.65+1.90+0.90 ± 0.903/3+0.40

In the QA retest, the mean gain is 1.33 percentage points and remains 1.24 percentage points after excluding the largest gain. On WebShop 7B, the mean improvements in Score and success rate are 1.10 points and 0.90 percentage points, respectively; excluding the largest gain leaves 0.81 points and 0.40 percentage points. Gains are positive for every seed on all three metrics, supporting consistency across seeds.

Comments on the paper

1 comment
Area_Chair_ZxELOpenReview ↗

Please engage with authors

Dear Reviewers,

The author-reviewer discussion will be closed on November 18 (AoE). If you haven’t yet replied to the authors’ latest response, please do so on OpenReview as soon as possible so they have time to follow up. A brief reply indicating which of your concerns have been addressed and what questions remain would be helpful. Thank you for your time and for engaging with the authors.

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.