- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors argue that token-level randomness does not translate into real exploration for LLM agents, and propose DCGPO to fix this. A turn-level distributional critic scores several candidate turns at each step during training, and the candidate with the best mean plus a dispersion bonus is executed. The policy alone is used at evaluation. Results on ALFWorld and WebShop across four Qwen models show consistent gains over PPO, GRPO, GSPO and GiGPO.
Critical strengths
The experiments are carefully run, with three training seeds, four base models, settings shared with GiGPO, and code provided. The preliminary study showing that entropy bonuses do not reduce state revisits is a clear and useful observation.
Critical weaknesses
-
The core claim, that the distributional critic and dispersion-guided exploration drive the gains, is not supported. On Qwen3-4B WebShop, replacing the scalar critic with the distributional one changes success from 77.60 to 77.87. Adding the dispersion term changes it from 79.27 (β=0) to 80.60 (β=1), within one standard deviation, and β=0.5 and β=2 give 79.27 and 79.80. The paper itself notes these differences are modest relative to seed variation. What remains is training-time selection among three candidates, which does not need a distributional critic at all.
-
The ablations are only on WebShop, while the largest gains are on ALFWorld (e.g. 87.1 to 93.6 overall and 75.5 to 90.7 on Pick2 for Qwen3-4B). We cannot tell which component produces the results the paper relies on most.
-
DCGPO generates three candidate turns per step and baselines generate one. No baseline is given the same generation budget, so part of the gain may simply come from best-of-3 sampling with a learned scorer.
Questions for authors
If the ALFWorld ablation and the matched-budget baseline show that the distributional critic and dispersion term account for a clear share of the gains, I will raise my score.
-
Please run the component ablation (TurnPPO, K=1, K=3 with β=0, full method) on ALFWorld for at least one model.
-
Please add a baseline with matched generation budget, for example TurnPPO or GiGPO with best-of-3 selection by a scalar critic, or a larger group size.
-
Can you show a setting where the dispersion term gives a gain clearly beyond seed variation?
7 comments
Response to Reviewer SdbQ (II)
W3 and Q2: Three-candidate generation and stronger sampling-budget baselines
A for W3 and Q2: Following your suggestion, to examine the effect of the sampling budget, we added both requested controls: TurnPPO with best-of-3 selection, and GiGPO with rollout group size increased from 8 to 24 on ALFWorld.
1. TurnPPO with best-of-3 selection. We equip TurnPPO with best-of-3 selection (). Table 2 compares this baseline with standard TurnPPO, mean-only DCGPO, and full DCGPO. TurnPPO with best-of-3 selection improves success on both ALFWorld splits and WebShop, but remains below both DCGPO variants, with full DCGPO achieving the highest success rates. Thus, candidate selection helps TurnPPO, but a learned scalar scorer with three candidates does not recover the gains from distributional learning and dispersion-guided selection.
| Training method | Candidates per decision | ALFWorld seen success (%) | ALFWorld unseen success (%) | WebShop success (%) | WebShop score (%) |
|---|---|---|---|---|---|
| TurnPPO () | 1 | 71.2 ± 3.7 | 67.1 ± 2.7 | 70.80 ± 11.98 | 79.49 ± 14.21 |
| TurnPPO, best-of-3 () | 3 | 79.9 ± 6.1 | 74.0 ± 5.0 | 73.93 ± 2.23 | 85.51 ± 2.44 |
| DCGPO (, ) | 3 | 85.95 ± 2.51 | 84.08 ± 1.55 | 79.40 ± 0.87 | 89.00 ± 1.92 |
| DCGPO (, ) | 3 | 93.6 ± 0.0 | 89.6 ± 1.4 | 83.60 ± 1.70 | 92.40 ± 1.56 |
Table 2. Comparison of standard TurnPPO, TurnPPO with best-of-3 selection, and DCGPO on ALFWorld and WebShop. All methods use Qwen3-4B for ALFWorld and Qwen2.5-7B-Instruct for WebShop. Results are reported as mean ± standard deviation across three training seeds.
2. Larger-group GiGPO. As shown in Table 3, increasing GiGPO’s rollout group size improves success to a level comparable to DCGPO, but triples the number of independently executed trajectories per task. At training step 200, the larger-group baseline requires nearly four times as many environment interactions as DCGPO. Under matched interaction or training GPU-hour budgets, DCGPO achieves the highest validation success (Table 4), highlighting its greater training efficiency.
| Training method | Task batch × rollout group | Candidates per decision | ALFWorld seen success (%) | ALFWorld unseen success (%) |
|---|---|---|---|---|
| GiGPO () | 16 × 8 | 1 | 85.87 ± 1.55 | 80.93 ± 2.88 |
| GiGPO () | 16 × 24 | 1 | 94.52 ± 0.86 | 91.71 ± 1.12 |
| DCGPO | 16 × 8 | 3 | 93.6 ± 0.0 | 89.6 ± 1.4 |
Table 3. Comparison of GiGPO rollout group sizes and DCGPO on ALFWorld with Qwen3-4B. Results are reported as mean ± standard deviation across three training seeds.
| Training method | Success at 0.4M interactions (%) | Success at 100 training GPU-hours (%) |
|---|---|---|
| GiGPO () | 70.02 ± 5.08 | 84.07 ± 5.39 |
| GiGPO () | 48.48 ± 8.77 | 81.32 ± 5.61 |
| DCGPO | 93.97 ± 1.97 | 90.28 ± 1.95 |
Table 4. Validation success under matched interaction and compute budgets on ALFWorld with Qwen3-4B. Results are reported as mean ± standard deviation across three training seeds.
We have updated Appendix F.5 with Figure 7, which plots success rate against cumulative training environment interactions (a) and training GPU-hours (b) over the first 200 updates.
We appreciate your constructive guidance and hope these additional controls address your concerns.
Thanks for the new experiments. The ALFWorld ablation and the scalar best-of-3 baseline are exactly what I asked for, and taken at face value they meet the condition I set for raising my score. Before I change it, I'd like to check two things about how the new numbers were computed.
Appendix E.3 says each ALFWorld game is evaluated with three evaluation seeds, and the main-table rows match that. For example, full DCGPO's Heat score of 92.4% is 133/144, and its category counts add up to the overall 93.6%. But in the three new rows (distributional K=1, mean-only K=3, and Moments), every category count is a multiple of three. That would be very unlikely with three evaluation seeds, so these rows look like they used a single evaluation seed. If so, the key comparison between mean-only (85.95) and full DCGPO (93.6) mixes two different evaluation setups.
Second, the per-category numbers for TurnPPO best-of-3 (92.2, 90.1, 80.4, 70.9, 78.1) can't be produced by either setup. For example, 92.2% on Pick would need between 290.27 and 290.59 successes out of 315 episodes. Its overall and unseen numbers do match, so maybe some episodes were dropped. Could you explain how these numbers were computed?
If the new rows are re-evaluated with the same protocol as the main table and the results hold, I'm happy to raise my score. For now I'll keep it at 2.
Response to Reviewer SdbQ (I)
Dear Reviewer,
Thank you for your valuable and constructive comments. We have performed additional experiments and analysis to address your concerns, and we believe these additional experiments and analyses have strengthened our manuscript.
W1, W2, Q1 and Q3: ALFWorld component ablations and gains relative to training-seed variation
A for W1, W2, Q1 and Q3: Thank you for the opportunity to provide more evidence. We conducted the requested four-way component ablation on ALFWorld using Qwen3-4B. TurnPPO uses scalar V/Q critics that estimate values at the turn level. The results in Table 1 support two main findings.
Contribution of the distributional critic. With candidate selection disabled (K=1), the distributional variant substantially outperforms TurnPPO on both seen and unseen tasks, showing that distributional critic learning contributes.
Contribution of dispersion beyond mean-only selection. With the candidate count fixed at K=3, full DCGPO outperforms the mean-only variant (β=0) on both splits. Both gains exceed three times the reported standard deviation of the corresponding mean-only results, directly addressing the concern about gains relative to seed variation. This descriptive comparison supports the contribution of dispersion-guided selection on ALFWorld.
Together, these comparisons demonstrate that both the distributional critic and dispersion-guided active sampling contribute to DCGPO’s performance on ALFWorld.
Table 1. Task-wise and overall success rates (%) for the four-way component ablation on ALFWorld with Qwen3-4B. Results are reported as mean ± standard deviation across three training seeds.