- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes VETTA, which combines turn-level advantages with centered token-level advantages for multi-turn LLM agents. It uses a lightweight shared critic with separate value heads and evaluates the method on ALFWorld and WebShop.
Critical strengths
The method is easy to follow. The separation between response-level credit and within-response token variation is intuitive, and the paper includes ablations on advantage composition, critic sharing, and critic depth.
Critical weaknesses
Overall, this is a reasonable study, but I am not convinced that it demonstrates a sufficiently substantial advance over existing work to warrant acceptance at ICLR
-
Comparison with the closest prior work is missing: HyGAE already combines turn- and token-level advantages, and the paper explicitly identifies it as the closest related method. However, there is no direct comparison. The internal ablations are useful, but they do not establish whether centered residual composition and separate value supervision provide a meaningful improvement over HyGAE.
-
Limited gains at the larger model scale: At 7B, the improvements over the strongest baseline are only 1.2 percentage points on ALFWorld and 0.8 on WebShop. The main component analyses focus on 1.5B, leaving it unclear why the gains become smaller and whether the proposed components remain consistently beneficial at the larger scale.
-
Attribution of the gains is not fully clear: The main comparison uses a two-block critic for VETTA and full-depth critics for PPO and Turn-PPO. The paper’s own depth analysis suggests that reducing critic depth can improve performance. Although the internal ablations provide some evidence, it remains unclear how much of the overall benefit comes from the credit assignment method and how much comes from critic size, or whether existing methods would benefit similarly from critic reduction..
Questions for authors
-
How does VETTA compare with HyGAE under comparable training conditions and critic budgets? This would help establish the benefit of the proposed credit composition and value-learning design beyond existing joint turn/token methods.
-
Can the authors explain why the gains become much smaller at 7B and provide evidence that the main components remain beneficial at that scale?
4 comments
Follow-up on HyGAE comparisons and larger-model evaluation
Dear Reviewer 9f32,
Thank you for your helpful suggestions on establishing VETTA's contribution relative to existing methods and understanding its performance at larger model scales.
We have added direct HyGAE comparisons at 1.5B and 7B, comparisons with two-block baselines, and controlled credit-composition ablations. We have also included ScienceWorld results at 7B and Sokoban results at 3B to examine applicability beyond the original benchmarks. These additions appear in Tables 1–4 and Sections 4.2–4.4 of the revised manuscript, with the corresponding settings in the appendices.
As the discussion period draws to a close, we would appreciate your feedback on whether the additional comparisons and analysis clarify the contribution and scope of our method. We would welcome the opportunity to address any remaining questions.
Thank you again for your time and consideration.
I would like to clarify that I read and evaluated the paper myself. I used ChatGPT to look up related information and help organize and phrase my comments in English. The review contains substantial human input, and the concerns below reflect my own assessment of the paper.
My original concerns are as follows:
-
Comparison with HyGAE. HyGAE is the closest related method and also combines turn- and token-level advantages. Without a direct comparison, it was unclear whether VETTA's centered composition and separate value supervision provide a meaningful improvement over HyGAE.
-
Limited gains at 7B. The improvements over the strongest baseline were only 1.2 percentage points on ALFWorld and 0.8 on WebShop. The component analyses mainly focused on 1.5B, leaving it unclear why the gains become smaller and whether the proposed components remain consistently beneficial at 7B.
-
Attribution of the gains. VETTA used a two-block critic, while PPO and Turn-PPO used full-depth critics. Since the paper also showed that reducing critic depth can improve performance, it was unclear how much of the benefit came from the credit assignment method and how much came from critic reduction.
Additional Comments After Rebuttal
-
Comparison with HyGAE. The additional comparison under the same training budget and critic depth addresses this concern. I appreciate the authors providing direct comparisons at both 1.5B and 7B.
-
Limited gains at 7B. This concern remains. The ceiling effect may explain the small gain on ALFWorld, but it does not explain the small gain on WebShop. The additional ScienceWorld evaluation is useful, but the 7B success rate improves from 50.7% to 52.2%, a gain of 1.5 percentage points. The rebuttal also does not sufficiently explain why the gains become smaller or how much each component contributes at 7B.
-
Attribution of the gains. The comparisons with matched critic depth and the controlled credit-composition experiments address this concern. These results provide evidence that critic reduction alone does not explain the improvement.
Overall, the additional experiments address two of my main concerns. However, the gains at the larger model scale remain modest, and I am still not convinced that the paper demonstrates a sufficiently substantial advance over existing work. I will therefore maintain my original score.
Thank you for confirming that the direct HyGAE comparisons and the controlled ablations resolve two of your three original concerns. We would like to clarify one point that bears on your remaining concern about the magnitude of the 7B gains.
The 1.2-point margin on ALFWorld is relative to Turn-PPO, whereas the 0.8-point margin on WebShop is relative to GiGPO. Table 1 shows why the choice of comparator matters.
| 7B method | ALFWorld success (%) | WebShop success (%) |
|---|---|---|
| Turn-PPO | 94.3 | 54.6 |
| GiGPO | 90.2 | 75.2 |
| VETTA | 95.5 | 76.0 |
Turn-PPO is a particularly relevant comparator because it already uses a critic to assign turn-level credit. Against this same method, VETTA gains 1.2 percentage points on ALFWorld and 21.4 points on WebShop at 7B. The WebShop gain is almost identical to the 21.6 points observed at 1.5B. With the comparator held fixed, the 7B advantage ranges from modest on ALFWorld to substantial on WebShop. Turn-PPO and GiGPO each come closest on a different task, while VETTA achieves the highest mean success on both tasks at both model sizes.
The 1.5B ablations show that centering improves success over direct addition on both tasks when critic depth and PPO settings are held fixed. The consistent direction of the full-method comparisons at 1.5B and 7B suggests that coordinating turn and token credit remains useful at the larger scale. We will extend the controlled ablations to 7B to quantify the individual contributions of centering and separate value supervision.
On ScienceWorld, the 1.5-point gain over Turn-PPO means that VETTA successfully completed 27 more test instances on the same 1,819-instance evaluation: 949 successes versus 922 (52.2% versus 50.7%). This is approximately 3% more successful completions. Mean task score also rose from 24.34 to 25.89, a 6.4% relative improvement. We consider this a meaningful gain over the strongest tested baseline on both reported metrics.
Given that the revision resolves two of your three concerns, and that the WebShop comparison shows a substantial gain over Turn-PPO at both model sizes, we respectfully ask you to reconsider whether the revised evidence warrants a higher overall score.
Response to Reviewer 9f32: HyGAE, credit attribution, and 7B
Thank you for recognizing the clarity of the credit decomposition and for the helpful suggestions on comparison with prior work, credit attribution, and larger-model evaluation.
Q1/W1. Comparison with HyGAE and the contribution of credit coordination.
Following your suggestion, we additionally compare VETTA with HyGAE under the same training budget. We first match critic depth using Qwen2.5-1.5B-Instruct and two-block critics for both methods:
At the same critic depth, VETTA improves success by 13.6 percentage points on ALFWorld and 4.6 on WebShop, supporting the benefit of its credit-coordination design over HyGAE.
We further compare the methods on both benchmarks at 1.5B and 7B in the main experiments:
VETTA outperforms HyGAE in all four settings. HyGAE mixes value targets and linearly combines advantages; VETTA supervises turn/token values separately and centers token credit, preserving the turn advantage as the response mean while refining credit within the response.
We have added the same-depth comparison to Section 4.3 (Table 3) and the broader comparison to Section 4.2 (Table 1), with training and evaluation settings in Appendix A.6.
W3. Separating credit composition from critic depth.
To test whether critic reduction explains the main-table gains, we additionally evaluate PPO* and Turn-PPO with two-block critics, matching VETTA's depth on both tasks:
VETTA exceeds two-block Turn-PPO by 3.6 percentage points on ALFWorld and 5.0 on WebShop. These gains remain when critic depth is matched, showing that critic reduction alone does not explain VETTA's advantage and supporting the contribution of coordinating turn/token credit. We have added these baseline comparisons to Section 4.3 (Table 3).
To isolate the contribution of credit composition, our WebShop ablation fixes the shared two-block backbone, both value heads and their supervision, and token-wise PPO. Only the credit supplied to the actor changes:
VETTA exceeds the strongest single-granularity credit rule by 5.4 percentage points. Direct addition and centering both use α=3; their 17.8-point difference supports the centering benefit with the critic and PPO settings fixed. Together, these experiments show that coordinating turn/token credit improves task success beyond the benefit of reducing critic depth.
The direct-addition control is included in Section 4.3 (Table 3), and the actor-credit ablation is analyzed in Section 4.4 (Table 4), with the controlled settings specified in Appendix A.6.
W2/Q2. Performance at 7B and broader applicability.
The larger gains at 1.5B suggest that turn/token coordination may be particularly helpful with limited model capacity. At 7B, VETTA improves ALFWorld success from the strongest baseline's 94.3% to 95.5% and maintains the highest mean success on both original benchmarks.
To test whether smaller gains are a general limitation at larger scales, we evaluated ScienceWorld with Qwen2.5-7B-Instruct and Sokoban with Qwen2.5-VL-3B-Instruct. Turn-PPO outperforms PPO*, GiGPO, and GRPO on both benchmarks, including both Sokoban reward settings:
ScienceWorld demonstrates continued gains at 7B, while Sokoban extends the benefit to a 3B vision-language model under both reward settings. The smaller ALFWorld gain is consistent with diminishing returns near its performance ceiling.
We have incorporated ScienceWorld and Sokoban into the main experiments in Section 4.2 (Table 2), with task descriptions, training settings, and evaluation protocols in Appendices B.1-B.3.