acceptodds
Under review as a conference paper at ICLR 2027

DCGPO: Distributional Critic Guided Policy Optimization for Multi-Turn LLM Agents

Abstract

Agentic Reinforcement Learning (Agentic RL) for large language model agents faces challenges in credit assignment and exploration efficiency. Stochastic generation does not necessarily yield effective environment exploration: repeated trajectories and limited state coverage can reduce the effective use of interaction budgets. We propose Distributional Critic Guided Policy Optimization (DCGPO), an actor–critic framework that combines turn-level distributional value learning with distribution-guided exploration. The critic supports turn-level advantage estimation and provides action-conditioned return means and dispersion to guide training-time sampling. By ranking policy-generated candidate turns before execution, DCGPO directs environment interactions toward promising behaviors while allowing exploration beyond mean-greedy selection. Our fixed-policy analysis shows that categorical critic learning preserves the form and statistical order of the scalar turn-level error bound for mean-induced advantage estimation. Experiments on embodied and web-based interactive tasks across multiple model scales demonstrate the effectiveness of DCGPO, while component ablations and visitation analyses examine its effects on task performance and exploration.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model Agents, Agentic Reinforcement Learning, Distributional Reinforcement Learning, Exploration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_SdbQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors argue that token-level randomness does not translate into real exploration for LLM agents, and propose DCGPO to fix this. A turn-level distributional critic scores several candidate turns at each step during training, and the candidate with the best mean plus a dispersion bonus is executed. The policy alone is used at evaluation. Results on ALFWorld and WebShop across four Qwen models show consistent gains over PPO, GRPO, GSPO and GiGPO.

Critical strengths

The experiments are carefully run, with three training seeds, four base models, settings shared with GiGPO, and code provided. The preliminary study showing that entropy bonuses do not reduce state revisits is a clear and useful observation.

Critical weaknesses
  1. The core claim, that the distributional critic and dispersion-guided exploration drive the gains, is not supported. On Qwen3-4B WebShop, replacing the scalar critic with the distributional one changes success from 77.60 to 77.87. Adding the dispersion term changes it from 79.27 (β=0) to 80.60 (β=1), within one standard deviation, and β=0.5 and β=2 give 79.27 and 79.80. The paper itself notes these differences are modest relative to seed variation. What remains is training-time selection among three candidates, which does not need a distributional critic at all.

  2. The ablations are only on WebShop, while the largest gains are on ALFWorld (e.g. 87.1 to 93.6 overall and 75.5 to 90.7 on Pick2 for Qwen3-4B). We cannot tell which component produces the results the paper relies on most.

  3. DCGPO generates three candidate turns per step and baselines generate one. No baseline is given the same generation budget, so part of the gain may simply come from best-of-3 sampling with a learned scorer.

Questions for authors

If the ALFWorld ablation and the matched-budget baseline show that the distributional critic and dispersion term account for a clear share of the gains, I will raise my score.

  1. Please run the component ablation (TurnPPO, K=1, K=3 with β=0, full method) on ALFWorld for at least one model.

  2. Please add a baseline with matched generation budget, for example TurnPPO or GiGPO with best-of-3 selection by a scalar critic, or a larger group size.

  3. Can you show a setting where the dispersion term gives a gain clearly beyond seed variation?

7 comments
AuthorsOpenReview ↗

Response to Reviewer SdbQ (I)

Dear Reviewer,

Thank you for your valuable and constructive comments. We have performed additional experiments and analysis to address your concerns, and we believe these additional experiments and analyses have strengthened our manuscript.

W1, W2, Q1 and Q3: ALFWorld component ablations and gains relative to training-seed variation

A for W1, W2, Q1 and Q3: Thank you for the opportunity to provide more evidence. We conducted the requested four-way component ablation on ALFWorld using Qwen3-4B. TurnPPO uses scalar V/Q critics that estimate values at the turn level. The results in Table 1 support two main findings.

Contribution of the distributional critic. With candidate selection disabled (K=1K=1), the distributional variant substantially outperforms TurnPPO on both seen and unseen tasks, showing that distributional critic learning contributes.

Contribution of dispersion beyond mean-only selection. With the candidate count fixed at K=3K=3, full DCGPO outperforms the mean-only variant (β=0\beta=0) on both splits. Both gains exceed three times the reported standard deviation of the corresponding mean-only results, directly addressing the concern about gains relative to seed variation. This descriptive comparison supports the contribution of dispersion-guided selection on ALFWorld.

Together, these comparisons demonstrate that both the distributional critic and dispersion-guided active sampling contribute to DCGPO’s performance on ALFWorld.

Training methodSeen PickSeen LookSeen CleanSeen HeatSeen CoolSeen Pick2Seen overallUnseen overall
TurnPPO (K=1K=1)90.2 ± 1.570.9 ± 8.260.5 ± 3.375.7 ± 8.764.0 ± 12.060.2 ± 5.371.2 ± 3.767.1 ± 2.7
DCGPO (K=1K=1)97.14 ± 0.0079.49 ± 11.7595.06 ± 2.1487.50 ± 0.0081.33 ± 2.3184.72 ± 2.4189.05 ± 1.0983.33 ± 1.14
DCGPO (K=3K=3, β=0\beta=0)92.38 ± 4.3684.62 ± 0.0090.12 ± 4.2879.17 ± 3.6184.00 ± 4.0079.17 ± 4.1785.95 ± 2.5184.08 ± 1.55
DCGPO (K=3K=3, β=1\beta=1)100.0 ± 0.089.7 ± 6.896.3 ± 0.092.4 ± 6.087.1 ± 5.690.7 ± 2.193.6 ± 0.089.6 ± 1.4

Table 1. Task-wise and overall success rates (%) for the four-way component ablation on ALFWorld with Qwen3-4B. Results are reported as mean ± standard deviation across three training seeds.

AuthorsOpenReview ↗

Response to Reviewer SdbQ (II)

W3 and Q2: Three-candidate generation and stronger sampling-budget baselines

A for W3 and Q2: Following your suggestion, to examine the effect of the sampling budget, we added both requested controls: TurnPPO with best-of-3 selection, and GiGPO with rollout group size increased from 8 to 24 on ALFWorld.

1. TurnPPO with best-of-3 selection. We equip TurnPPO with best-of-3 selection (K=3K=3). Table 2 compares this baseline with standard TurnPPO, mean-only DCGPO, and full DCGPO. TurnPPO with best-of-3 selection improves success on both ALFWorld splits and WebShop, but remains below both DCGPO variants, with full DCGPO achieving the highest success rates. Thus, candidate selection helps TurnPPO, but a learned scalar scorer with three candidates does not recover the gains from distributional learning and dispersion-guided selection.

Training methodCandidates per decisionALFWorld seen success (%)ALFWorld unseen success (%)WebShop success (%)WebShop score (%)
TurnPPO (K=1K=1)171.2 ± 3.767.1 ± 2.770.80 ± 11.9879.49 ± 14.21
TurnPPO, best-of-3 (K=3K=3)379.9 ± 6.174.0 ± 5.073.93 ± 2.2385.51 ± 2.44
DCGPO (K=3K=3, β=0\beta=0)385.95 ± 2.5184.08 ± 1.5579.40 ± 0.8789.00 ± 1.92
DCGPO (K=3K=3, β=1\beta=1)393.6 ± 0.089.6 ± 1.483.60 ± 1.7092.40 ± 1.56

Table 2. Comparison of standard TurnPPO, TurnPPO with best-of-3 selection, and DCGPO on ALFWorld and WebShop. All methods use Qwen3-4B for ALFWorld and Qwen2.5-7B-Instruct for WebShop. Results are reported as mean ± standard deviation across three training seeds.

2. Larger-group GiGPO. As shown in Table 3, increasing GiGPO’s rollout group size improves success to a level comparable to DCGPO, but triples the number of independently executed trajectories per task. At training step 200, the larger-group baseline requires nearly four times as many environment interactions as DCGPO. Under matched interaction or training GPU-hour budgets, DCGPO achieves the highest validation success (Table 4), highlighting its greater training efficiency.

Training methodTask batch × rollout groupCandidates per decisionALFWorld seen success (%)ALFWorld unseen success (%)
GiGPO (n=8n=8)16 × 8185.87 ± 1.5580.93 ± 2.88
GiGPO (n=24n=24)16 × 24194.52 ± 0.8691.71 ± 1.12
DCGPO16 × 8393.6 ± 0.089.6 ± 1.4

Table 3. Comparison of GiGPO rollout group sizes and DCGPO on ALFWorld with Qwen3-4B. Results are reported as mean ± standard deviation across three training seeds.

Training methodSuccess at 0.4M interactions (%)Success at 100 training GPU-hours (%)
GiGPO (n=8n=8)70.02 ± 5.0884.07 ± 5.39
GiGPO (n=24n=24)48.48 ± 8.7781.32 ± 5.61
DCGPO93.97 ± 1.9790.28 ± 1.95

Table 4. Validation success under matched interaction and compute budgets on ALFWorld with Qwen3-4B. Results are reported as mean ± standard deviation across three training seeds.

We have updated Appendix F.5 with Figure 7, which plots success rate against cumulative training environment interactions (a) and training GPU-hours (b) over the first 200 updates.

We appreciate your constructive guidance and hope these additional controls address your concerns.

Thanks for the new experiments. The ALFWorld ablation and the scalar best-of-3 baseline are exactly what I asked for, and taken at face value they meet the condition I set for raising my score. Before I change it, I'd like to check two things about how the new numbers were computed.

Appendix E.3 says each ALFWorld game is evaluated with three evaluation seeds, and the main-table rows match that. For example, full DCGPO's Heat score of 92.4% is 133/144, and its category counts add up to the overall 93.6%. But in the three new rows (distributional K=1, mean-only K=3, and Moments), every category count is a multiple of three. That would be very unlikely with three evaluation seeds, so these rows look like they used a single evaluation seed. If so, the key comparison between mean-only (85.95) and full DCGPO (93.6) mixes two different evaluation setups.

Second, the per-category numbers for TurnPPO best-of-3 (92.2, 90.1, 80.4, 70.9, 78.1) can't be produced by either setup. For example, 92.2% on Pick would need between 290.27 and 290.59 successes out of 315 episodes. Its overall and unseen numbers do match, so maybe some episodes were dropped. Could you explain how these numbers were computed?

If the new rows are re-evaluated with the same protocol as the main table and the results hold, I'm happy to raise my score. For now I'll keep it at 2.

Reviewer_7fCROpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To address the turn-level credit assignment and inefficient exploration for LLM agents in long-horizon tasks, the paper proposes a categorical critic to predict return distributions for states and state–action pairs, based on which candidate actions are ranked and selected during training. Thus, the main contribution is to combine turn-level distributional value learning with return-guided training data collection. Experimental results show gains on ALFWorld and WebShop compared to multiple baselines, with mixed results using smaller models on WebShop.

Critical strengths

The examination of the exploration bottleneck in LLM agents is insightful, but why DCGPO is better at exploration than PPO with entropy regularization is underspecified.

Critical weaknesses
  1. The experiments do not isolate the paper’s central algorithmic contribution.
  • Moving from TurnPPO to DCGPO changes both the return representation and the critic’s training, so a scalar V/QV/Q critic with the same candidate-selection procedure should be compared in the ablation study.
  • Lack of comparison against a critic predicting only the return mean and second moment, using the same Q+βσQ+\beta\sigma selector. This tests whether learning the entire categorical distribution provides benefits beyond the two statistics actually used.
  1. The evidence does not establish the claimed exploration-efficiency advantage under comparable training budgets and longer-horizon environments.
  • Main comparisons use a fixed number of updates, but equal update counts do not necessarily imply equal cumulative environment interactions. Since DCGPO performs additional candidate generation and critic scoring, methods at the same compute budgets should be compared, such as performance curves against actual interaction counts and GPU hours.
  • Test a more realistic browser or tool-use environment[1-2] to demonstrate the method's generalization, especially in open-ended state-action spaces. Both ALFWorld and WebShop are finite and closed state-action spaces, which may be simple for current LLMs. Also, recent closed or open-scource frontier models (e.g., GPT-6/Opus-5/DeepSeek-v4.1) should be added as baselines to show the latest performance on these environments.

References:

  1. https://openai.com/index/introducing-swe-bench-verified/
  2. Trivedi, Harsh, et al. "Appworld: A controllable world of apps and people for benchmarking interactive coding agents." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.
Questions for authors

In Figure 1, how to see the benefits of DCGPO compared to PPO with entropy regularization? Any experiments to compare these two methods?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 7fCR (I)

Dear Reviewer,

We appreciate your questions and suggestions. Below, we present new baseline comparisons and clarify the exploration results.

W1.1: TurnPPO with the same candidate-selection procedure

A for W1.1: Thank you for pointing out this confound. TurnPPO uses scalar V/Q critics that estimate values at the turn level. We added a TurnPPO baseline with best-of-3 selection that generates K=3K=3 candidate turns and executes the one with the highest predicted mean return. We compare it with mean-only DCGPO (K=3K=3, β=0\beta=0), which uses the same candidate count and mean-return ranking rule but learns a categorical return distribution.

As shown in Table 1, TurnPPO with best-of-3 selection improves over TurnPPO without selection, but mean-only DCGPO achieves higher success on both ALFWorld splits and WebShop. This supports the value of categorical distributional critic learning beyond scalar candidate ranking, without relying on a dispersion bonus. Full DCGPO further improves performance, supporting the additional contribution of dispersion-guided selection.

Critic and training-time selectorCandidates per decisionALFWorld seen success (%)ALFWorld unseen success (%)WebShop success (%)WebShop score (%)
TurnPPO (K=1K=1)171.2 ± 3.767.1 ± 2.770.80 ± 11.9879.49 ± 14.21
TurnPPO, best-of-3 (K=3K=3)379.1 ± 2.174.0 ± 5.073.93 ± 2.2385.51 ± 2.44
DCGPO (K=3K=3, β=0\beta=0)385.9 ± 2.884.6 ± 2.479.40 ± 0.8789.00 ± 1.92
DCGPO (K=3K=3, β=1\beta=1)393.6 ± 0.089.6 ± 1.483.60 ± 1.7092.40 ± 1.56

Table 1. Comparison of scalar and distributional critics under different training-time selection strategies on ALFWorld and WebShop. All methods use Qwen3-4B for ALFWorld and Qwen2.5-7B-Instruct for WebShop. Results are reported as mean ± standard deviation across three training seeds.

W1.2: A critic predicting only the return mean and second moment

A for W1.2: To test the benefit of learning the full return distribution, we added a moment-based baseline that predicts the return mean and second raw moment at the response level:

\sigma(s,a)=\sqrt{\max\{m_2(s,a)-\mu(s,a)^2,0\}}.$$ The critic is updated by response-weighted MSE regression to the full Monte Carlo targets $(G,G^2)$, with equal weights for the two moment losses. During training-time Q-selection, we sample three candidate responses and select the one maximizing $\mu(s,a_i)+\beta\sigma(s,a_i)$, with $K=3$ and $\beta=1$, matching full DCGPO's selector. Table 2 compares standard TurnPPO, TurnPPO with best-of-3 selection, Moments with Q-selection, and full DCGPO. Moments improves overall success over both TurnPPO baselines on seen and unseen tasks, consistent with the value of incorporating return dispersion into candidate selection. Full DCGPO achieves the highest overall success on both splits. Directly predicting the two statistics used by the selector therefore does not recover its performance, supporting the benefit of categorical return-distribution learning in this setting. | Method | Seen: Pick | Seen: Look | Seen: Clean | Seen: Heat | Seen: Cool | Seen: Pick2 | Seen: All | Unseen: All | |---|---:|---:|---:|---:|---:|---:|---:|---:| | TurnPPO ($K=1$) | 90.2 ± 1.5 | 70.9 ± 8.2 | 60.5 ± 3.3 | 75.7 ± 8.7 | 64.0 ± 12.0 | 60.2 ± 5.3 | 71.2 ± 3.7 | 67.1 ± 2.7 | | TurnPPO, best-of-3 ($K=3$) | 90.5 ± 1.0 | 66.7 ± 0.0 | 82.7 ± 3.3 | 70.1 ± 4.3 | 74.2 ± 5.4 | 76.4 ± 5.0 | 79.1 ± 2.1 | 74.0 ± 5.0 | | Moments ($K=3$, $\beta=1$) | 94.3 ± 2.9 | 84.6 ± 9.2 | 79.0 ± 7.7 | 84.0 ± 5.2 | 85.8 ± 3.4 | 76.9 ± 3.5 | 84.8 ± 0.4 | 82.5 ± 3.0 | | DCGPO | **100.0 ± 0.0** | **89.7 ± 6.8** | **96.3 ± 0.0** | **92.4 ± 6.0** | **87.1 ± 5.6** | **90.7 ± 2.1** | **93.6 ± 0.0** | **89.6 ± 1.4** | Table 2. Task-wise and overall success rates (%) for scalar, moment-based, and distributional critics on ALFWorld with Qwen3-4B. Results are reported as mean ± standard deviation across three training seeds.
AuthorsOpenReview ↗

Response to Reviewer 7fCR (II)

W2.1: Equal update counts versus actual interaction and compute budgets

A for W2.1: We agree that equal update counts alone do not ensure comparable training budgets. We therefore compare success rates under matched training GPU-hour and executed environment interaction budgets. Training GPU-hours include candidate generation, critic scoring, and model updates, but exclude evaluation and checkpointing overhead, thereby accounting for DCGPO’s additional computational costs.

As shown in Table 3, DCGPO achieves the highest validation success under both matched budgets. We have also updated Appendix F.5 with Figure 7, which plots success rates against cumulative training environment interactions (a) and training GPU-hours (b), further demonstrating DCGPO’s interaction and compute efficiency over baselines.

MethodSuccess at 100 training GPU-hours (%)Success at 0.4M training interactions (%)
TurnPPO (K=1K=1)60.75 ± 3.7255.85 ± 1.46
TurnPPO, best-of-3 (K=3K=3)67.92 ± 4.9270.20 ± 4.31
GiGPO, group=884.07 ± 5.3970.02 ± 5.08
GiGPO, group=2481.32 ± 5.6148.48 ± 8.77
DCGPO90.28 ± 1.9593.97 ± 1.97

Table 3. Validation success under matched training GPU-hour and executed environment interaction budgets on ALFWorld with Qwen3-4B. Results are reported as mean ± standard deviation across three training seeds.

W2.2: Generalization to more realistic environments and frontier-model baselines

A for W2.2:

1. Generalization to more realistic environments.

We agree that evaluating a more realistic tool-use environment is important for testing the generality of our approach. AppWorld RL is considerably more expensive than our current ALFWorld/WebShop experiments. For example, recent work uses a Qwen3-14B model with 50-turn/32K-token training trajectories and reports approximately 2,690 GPU-hours for the main AppWorld run [1]. During the rebuttal period, we attempted AppWorld training using the verl-agent integration [2] and the publicly described settings of G2PO [3]. Our corresponding Qwen2.5-14B setup uses 16 H100 GPUs with 80 GB memory each and takes approximately 40 minutes per training step, making repeated debugging and multi-seed retraining difficult during the rebuttal period. We will include validated AppWorld results in the revised manuscript.

2. Frontier-model baselines on the existing benchmarks.

We thank the reviewer for this meaningful suggestion. We have evaluated DeepSeek-v4.1-flash(1017) and GPT-6.1-sol as API-based frontier-model baselines on ALFWorld and WebShop. Both models interact with ALFWorld and WebShop through API calls. For DeepSeek-v4.1-flash, we strictly follow the standard prompting format used in the paper's evaluations. In our tests, GPT-6.1-sol refused the standard prompt format when explicitly asked to output <think> tags, due to anti-distillation safeguards. We therefore use an action-only prompt requesting only <action>...</action> and validate only its content, with native reasoning enabled.

As shown in Table 4, GPT-6.1-sol achieves results comparable to DCGPO with Qwen2.5-7B-Instruct on ALFWorld but is substantially weaker under the WebShop setting, while DeepSeek-v4.1-flash falls well behind on both tasks. These results indicate that neither benchmark is fully saturated by the tested frontier models. Given the potentially greater model scale and inference costs of frontier APIs, achieving comparable or stronger performance with our 4B/7B models highlights the value of DCGPO.

Model / methodALFWorld seen success (%)ALFWorld unseen success (%)WebShop success (%)WebShop score (%)
DeepSeek-v4.1-flash64.7666.6715.0019.68
GPT-6.1-sol95.4892.7956.8063.69
DCGPO (Qwen3-4B)93.6 ± 0.089.6 ± 1.480.60 ± 1.3189.68 ± 0.95
DCGPO (Qwen2.5-7B-Instruct)96.0 ± 1.791.4 ± 3.083.60 ± 1.7092.40 ± 1.56

Table 4. Comparison of frontier-model baselines and DCGPO on ALFWorld and WebShop. DCGPO results are reported as mean ± standard deviation across three training seeds.

References

[1] Pu, Liming, et al. "Explore More, Drift Less: Outcome-Only Reinforcement Learning Can Suffice for Long-Horizon Interactive Agents." arXiv preprint arXiv:2609.01245 (2026).

[2] Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-Group Policy Optimization for LLM Agent Training. In Advances in Neural Information Processing Systems (NeurIPS), 2025.

[3] Yunan Wang, Minghui Song, Zihan Zhang, Shaohan Huang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang. Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learning. In Advances in Neural Information Processing Systems (NeurIPS), 2026.

AuthorsOpenReview ↗

Response to Reviewer 7fCR (III)

Q1: Empirical comparison with entropy-regularized PPO and the role of Figure 1

A for Q1: Thank you for the opportunity to clarify this point. Figure 1 illustrates the exploration limitation observed in our preliminary study and how DCGPO addresses it. Two sets of experiments in the main paper provide the quantitative comparison with entropy-regularized PPO.

Preliminary study (main paper, Section 3, Figure 2, and Table 3). On WebShop/Qwen3-4B at training step 150, PPO with entropy coefficient 0.001 achieves a new-state visitation rate (NR) of 0.107, an episode revisit rate (RR) of 57.4%, and a validation success rate of 67.2%. Without entropy regularization, the corresponding results are 0.124, 56.5%, and 67.2%. Thus, the tested entropy bonus does not improve state exploration or task success. Under the same evaluation protocol, DCGPO achieves NR = 0.234, RR = 43.7%, and success = 79.9%, showing more new-state visits, fewer revisits, and higher task success.

Task performance (main paper, Table 1). The PPO baseline uses entropy regularization. With Qwen3-4B, DCGPO improves ALFWorld seen/unseen success from 67.8%/58.7% to 93.6%/89.6%, and WebShop success from 69.53% to 80.60%. Qwen2.5-7B-Instruct shows the same pattern: ALFWorld seen/unseen success increases from 82.5%/76.4% to 96.0%/91.4%, and WebShop success from 76.40% to 83.60%. These results show that the advantage also holds in the main task evaluations.

These results support Figure 1's motivation: the entropy bonus alone did not improve environment exploration, while DCGPO's return-guided training produces broader state exploration and stronger learned policies in the tested settings.

Thank you for helping us strengthen the empirical comparisons and clarify the scope of our findings.

Reviewer_i645OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes DCGPO, an actor–critic framework for multi-turn LLM agents that combines turn-level distributional value learning with training-time candidate selection. It ranks policy-generated actions using predicted return means and standard deviations to improve exploration and credit assignment. Experiments on ALFWorld and WebShop across four model sizes show generally improved standalone policy performance, with ablations and visitation analyses supporting the benefits of candidate selection and dispersion guidance. A fixed-policy analysis also establishes preservation of scalar turn-level advantage-estimation bounds under idealized assumptions.

Critical strengths
  • Unified credit assignment and exploration: The distributional critic uses return means for turn-level advantage estimation and return dispersion to guide exploration.
  • No additional inference overhead: Training gains transfer to the standalone policy without a critic or candidate selection at evaluation.
  • Strong empirical performance: DCGPO outperforms competitive baselines across multiple settings, with particularly large gains on long-horizon ALFWorld tasks.
Critical weaknesses
  • Reliability of the dispersion signal: The paper does not directly validate whether predicted return dispersion matches empirical return variability. Such an analysis would strengthen the explanation for why the dispersion bonus improves exploration.

  • Candidate generation and critic scoring introduce additional training costs. Comparisons under matched compute or wall-clock budgets would clarify whether DCGPO’s gains justify this overhead.

  • Since trajectories and advantages reflect the selection-guided behavior policy, how large is its mismatch with the standalone actor, and how does this discrepancy affect learning stability and final performance?

Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer i645 (I)

Dear Reviewer,

Thank you for your careful review. We address your concerns below with additional experiments and analysis.

W1: Reliability of predicted return dispersion

A for W1: We thank you for this insightful suggestion. To assess the reliability of the dispersion signal, we directly checked whether predicted return dispersion reflects empirical return variability.

We evaluated Qwen2.5-1.5B-Instruct in ALFWorld at training step 100, when dispersion estimation is challenging and relevant to exploration. We compared the frozen Q critic’s predicted return distributions with empirical distributions from 64 independent rollouts per state–action pair. Each rollout follows the frozen Q-selection policy until termination or the original 50-step episode limit.

We evaluate prediction–MC agreement at two complementary levels, using all 75 state–action pairs collected from 5 trajectories:

  1. Distributional agreement. We measure the first Wasserstein distance (W1W_1) between the predicted and empirical return distributions. Lower values indicate closer agreement. The critic achieves a mean W1W_1 of 1.412, compared with 3.212 for a uniform distribution with the same support atoms. We also measure the mean absolute error of the estimated standard deviation against MC: 0.865 for the critic versus 2.347 for the same uniform baseline, a 63.1% reduction.

  2. Coarse dispersion ranking. Since DCGPO uses dispersion primarily as a relative exploration signal, we group predicted and empirical standard deviations into their respective low, medium, and high tertiles to assess relative variability rather than exact calibration. Agreement reaches 62.7% versus a 33.3% chance baseline, with weighted κ = 0.660 and only 2.7% opposite-extreme mismatches. Across all 75 pairs, Spearman’s ρ is 0.733 and Pearson’s r is 0.758.

These results support predicted dispersion as an informative relative exploration signal, although they do not establish exact calibration.

W2: Additional training cost and comparisons under matched budgets

A for W2: Following your suggestion, to account for the additional training cost, we compare DCGPO with TurnPPO and GiGPO baselines on ALFWorld/Qwen3-4B under matched executed environment interaction and training GPU-hour budgets. TurnPPO uses scalar V/Q critics that estimate values at the turn level. Training GPU-hours include candidate generation, critic scoring, and model updates, but exclude evaluation/checkpoint overhead.

As shown in Table 1, DCGPO achieves the highest validation success under the matched interaction budget. TurnPPO with best-of-3 selection improves over TurnPPO with K=1K=1, but remains well below DCGPO, indicating that candidate sampling alone does not recover its interaction efficiency. In the separate final evaluation after 200 training steps, DCGPO achieves comparable success to GiGPO group=24 with 74.2% fewer environment interactions.

DCGPO also achieves the highest validation success under the matched compute budget, outperforming all baselines even after accounting for its candidate-generation and critic-scoring costs.

We have added Figure 7 to Appendix F.5, showing success rate against training interactions and GPU-hours over 200 steps.

MethodSuccess at 100 training GPU-hours (%)Success at 0.4M training interactions (%)
TurnPPO (K=1K=1)60.75 ± 3.7255.85 ± 1.46
TurnPPO, best-of-3 (K=3K=3)67.92 ± 4.9270.20 ± 4.31
GiGPO, group=884.07 ± 5.3970.02 ± 5.08
GiGPO, group=2481.32 ± 5.6148.48 ± 8.77
DCGPO90.28 ± 1.9593.97 ± 1.97

Table 1. Validation success under matched training GPU-hour and executed environment interaction budgets on ALFWorld with Qwen3-4B. Results are reported as mean ± standard deviation across three training seeds.

AuthorsOpenReview ↗

Response to Reviewer i645 (II)

W3: Mismatch between the selection-guided behavior policy and the standalone actor

A for W3: Following your suggestion, we quantify the mismatch between the selection-guided behavior policy and the standalone actor policy and examine whether it coincides with numerical instability.

Policy mismatch. Using the ALFWorld Qwen2.5-1.5B-Instruct checkpoint at training step 100, we generated 5 actor-only trajectory and froze their states. At each state, we compared 64 actor-only samples with 64 Q-selection outputs, following DCGPO. An equal-sized actor-vs-actor control estimates sampling variability using unselected first candidates. As shown in Table 2, DCGPO produces a measurable, state-dependent redistribution of action probabilities beyond the observed sampling-noise control.

Action-distribution diagnosticResult
Actor-only vs. DCGPO TVD0.208
Actor-vs-actor control TVD0.148
Mean Jensen–Shannon distance0.281

Table 2. Action-distribution mismatch diagnostics on ALFWorld with Qwen2.5-1.5B-Instruct at training step 100.

Numerical stability. We measured the actor gradient norm and the 95th percentile (P95) of absolute approximate PPO-KL between the updated and pre-update actor policies. As shown in Table 3, both are lower for DCGPO than for the no-selection baseline, indicating no amplification of average gradients or upper-tail policy updates. These diagnostics provide evidence of numerical stability.

Training configurationMean actor gradient normP95 of absolute approximate PPO-KL
DCGPO (K=1K=1)0.8520.001801
DCGPO (K=3K=3, β=1\beta=1)0.8290.001571

Table 3. Numerical stability diagnostics over the first 200 training updates across three seeds on ALFWorld with Qwen3-4B.

Benefit to the standalone actor. We additionally conducted ablation experiments on ALFWorld. As shown in Table 4, full DCGPO outperforms the no-selection variant on both seen and unseen tasks, with selection disabled at evaluation. These additional ALFWorld results and the WebShop ablations in the main paper (Figure 4) jointly support the performance advantage of our dispersion-guided sampling strategy, with gains retained by the standalone actor at evaluation.

Training configurationSeen success (%)Unseen success (%)
DCGPO (K=1K=1)89.1 ± 1.383.1 ± 0.4
DCGPO (K=3K=3, β=1\beta=1)93.6 ± 0.089.6 ± 1.4

Table 4. Standalone-actor success rates on ALFWorld with Qwen3-4B. Results are reported as mean ± standard deviation across three training seeds.

Thank you again for your valuable comments. We sincerely hope our additional experimental results and explanations have addressed your concerns.

AuthorsOpenReview ↗

Explanation of Data Modification

We sincerely thank Reviewer SdbQ for identifying the issue with our evaluation seeds. In response, we have corrected the results for DCGPO (K=1K=1) in the first row of Table 4. The updated results are now based on evaluations across three training seeds, with three evaluation seeds for each training seed, ensuring consistency with the experimental protocol described in the main text. We apologize for any confusion this issue may have caused and hope that this correction adequately addresses your concerns.

AuthorsOpenReview ↗

Looking forward to further discussions!

Dear reviewer,

We were wondering if our response and revision have cleared all your concerns. In the previous response, we have tried to address all the points you have raised. We would appreciate it if you could kindly let us know whether you have any remaining questions or suggestions, so that we have sufficient time to address them. We welcome further discussion and feedback that could help strengthen our manuscript. Thanks!

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal Discussion

We sincerely thank all reviewers for their careful evaluations and constructive feedback. Reviewers 7fCR and SdbQ appreciated our analysis of exploration bottlenecks in LLM agents, particularly the observation that entropy bonuses do not necessarily reduce state revisits. Reviewer i645 recognized our unified treatment of credit assignment and exploration, strong empirical performance, and absence of additional inference overhead. Reviewer SdbQ also recognized the carefully conducted experiments. We are pleased that all reviewers have confirmed that their concerns have been addressed during the rebuttal. Below, we summarize the main concerns and the additional evidence provided in response.

  • Do the gains hold under comparable generation, interaction, and compute budgets? (Reviewers i645, 7fCR, and SdbQ)
    We added scalar-critic best-of-3 selection and larger-group GiGPO baselines, together with comparisons under matched environment-interaction and training GPU-hour budgets. DCGPO achieves the highest validation success under both matched interaction and compute budgets, supporting its training-efficiency advantages even after accounting for the additional training costs.

  • Do distributional critic learning and dispersion-guided exploration contribute beyond candidate sampling alone? (Reviewers 7fCR and SdbQ)
    We added component ablations on ALFWorld, a scalar-critic best-of-3 baseline, and a moment-based critic using the same dispersion-guided selection rule. Under the consistent final-evaluation protocol, these comparisons support distinct contributions from distributional critic learning and dispersion-guided exploration beyond candidate sampling alone.

  • Is predicted dispersion an informative exploration signal, and does selection-guided training remain stable? (Reviewer i645)
    We compared predicted dispersion with empirical return variability and examined policy mismatch and optimization stability. The results support predicted dispersion as a useful relative exploration signal and indicate stable training. Additional ALFWorld ablations confirm that the standalone policy retains the gains without extra inference overhead.

  • Are the empirical comparisons sufficiently broad? (Reviewer 7fCR)
    We added frontier-model baselines on ALFWorld and Webshop. The results show that these benchmarks remain challenging for the evaluated frontier models, with DCGPO achieving comparable or stronger performance using 4B/7B models. Following Reviewer 7fCR’s suggestion, validated AppWorld results will also be incorporated into the final manuscript.

  • Is the evaluation protocol consistent and clearly documented? (Reviewer SdbQ)
    We completed the missing evaluation-seed runs for the affected ablations and baseline, corrected the TurnPPO best-of-3 seen-task results, and provided detailed per-seed success counts for verification. Under the consistent final-evaluation protocol, the conclusions regarding distributional learning and dispersion-guided exploration remain supported. Reviewer SdbQ verified the corrected results and confirmed that they addressed the concern. The final manuscript will explicitly distinguish training-time validation from final evaluation.

We have added training-time efficiency curves to the revised manuscript to help reviewers assess DCGPO’s computational efficiency. The remaining rebuttal results and clarifications, together with validated AppWorld results, will be incorporated into the final manuscript. All reviewers have confirmed that the responses and additional evidence address their concerns. We sincerely thank the reviewers for their time, rigorous scrutiny, and constructive suggestions, which have substantially strengthened the empirical evidence and clarity of our paper.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.