acceptodds
Under review as a conference paper at ICLR 2027

VETTA: Coordinating Turn- and Token-Level Credit Assignment for Multi-Turn LLM Agents

Abstract

Multi-turn LLM agents often receive sparse task feedback across several interactions, while generating each response token by token. This creates two related credit-assignment questions: which responses helped achieve the outcome, and which generation decisions mattered within each response? Existing methods typically focus on only one level: turn-level methods evaluate complete responses but do not distinguish the decisions within them; token-level methods can propagate feedback across turns but do not explicitly model credit for each response. These complementary limitations motivate learning credit at both levels and coordinating it in a single policy update. We introduce VETTA, a credit assignment method that jointly learns turn- and token-level values through separate heads on a shared lightweight critic. VETTA computes advantages along both temporal sequences and combines each turn advantage with a within-response-centered token residual for PPO updates. Furthermore, to reduce value-learning cost, the critic retains only early Transformer blocks from the pretrained checkpoint used to initialize the actor. On two challenging agent benchmarks, ALFWorld and WebShop, VETTA improves success rates over PPO by 37.5% and 22.3%, respectively, with Qwen2.5-1.5B-Instruct and achieves success rates of 95.5% and 76.0%, respectively, with Qwen2.5-7B-Instruct. Critic-depth comparisons further show strong task performance with substantially lower critic-side computation. These results suggest that a compact shared critic can coordinate turn- and token-level credit to improve agent performance while keeping value estimation efficient.

Primary area:
reinforcement learning
Keywords:
Large Language Model Agents, Reinforcement Learning, Multi-Turn Credit Assignment, Value Function Estimation, Efficient Critic Design

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_9f32OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes VETTA, which combines turn-level advantages with centered token-level advantages for multi-turn LLM agents. It uses a lightweight shared critic with separate value heads and evaluates the method on ALFWorld and WebShop.

Critical strengths

The method is easy to follow. The separation between response-level credit and within-response token variation is intuitive, and the paper includes ablations on advantage composition, critic sharing, and critic depth.

Critical weaknesses

Overall, this is a reasonable study, but I am not convinced that it demonstrates a sufficiently substantial advance over existing work to warrant acceptance at ICLR

  • Comparison with the closest prior work is missing: HyGAE already combines turn- and token-level advantages, and the paper explicitly identifies it as the closest related method. However, there is no direct comparison. The internal ablations are useful, but they do not establish whether centered residual composition and separate value supervision provide a meaningful improvement over HyGAE.

  • Limited gains at the larger model scale: At 7B, the improvements over the strongest baseline are only 1.2 percentage points on ALFWorld and 0.8 on WebShop. The main component analyses focus on 1.5B, leaving it unclear why the gains become smaller and whether the proposed components remain consistently beneficial at the larger scale.

  • Attribution of the gains is not fully clear: The main comparison uses a two-block critic for VETTA and full-depth critics for PPO and Turn-PPO. The paper’s own depth analysis suggests that reducing critic depth can improve performance. Although the internal ablations provide some evidence, it remains unclear how much of the overall benefit comes from the credit assignment method and how much comes from critic size, or whether existing methods would benefit similarly from critic reduction..

Questions for authors
  • How does VETTA compare with HyGAE under comparable training conditions and critic budgets? This would help establish the benefit of the proposed credit composition and value-learning design beyond existing joint turn/token methods.

  • Can the authors explain why the gains become much smaller at 7B and provide evidence that the main components remain beneficial at that scale?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 9f32: HyGAE, credit attribution, and 7B

Thank you for recognizing the clarity of the credit decomposition and for the helpful suggestions on comparison with prior work, credit attribution, and larger-model evaluation.

Q1/W1. Comparison with HyGAE and the contribution of credit coordination.

Following your suggestion, we additionally compare VETTA with HyGAE under the same training budget. We first match critic depth using Qwen2.5-1.5B-Instruct and two-block critics for both methods:

MethodALFWorld success (%)WebShop success (%)
HyGAE (2-block)79.368.2
VETTA (2-block)92.972.8

At the same critic depth, VETTA improves success by 13.6 percentage points on ALFWorld and 4.6 on WebShop, supporting the benefit of its credit-coordination design over HyGAE.

We further compare the methods on both benchmarks at 1.5B and 7B in the main experiments:

ModelMethodALFWorld success (%)WebShop success (%)
1.5BHyGAE85.0 ± 0.765.5 ± 0.6
1.5BVETTA91.9 ± 1.173.8 ± 1.1
7BHyGAE93.8 ± 0.473.3 ± 0.6
7BVETTA95.5 ± 0.476.0 ± 0.6

VETTA outperforms HyGAE in all four settings. HyGAE mixes value targets and linearly combines advantages; VETTA supervises turn/token values separately and centers token credit, preserving the turn advantage as the response mean while refining credit within the response.

We have added the same-depth comparison to Section 4.3 (Table 3) and the broader comparison to Section 4.2 (Table 1), with training and evaluation settings in Appendix A.6.

W3. Separating credit composition from critic depth.

To test whether critic reduction explains the main-table gains, we additionally evaluate PPO* and Turn-PPO with two-block critics, matching VETTA's depth on both tasks:

Method (2-block critic)ALFWorld success (%)WebShop success (%)
PPO*70.056.4
Turn-PPO89.367.8
VETTA92.972.8

VETTA exceeds two-block Turn-PPO by 3.6 percentage points on ALFWorld and 5.0 on WebShop. These gains remain when critic depth is matched, showing that critic reduction alone does not explain VETTA's advantage and supporting the contribution of coordinating turn/token credit. We have added these baseline comparisons to Section 4.3 (Table 3).

To isolate the contribution of credit composition, our WebShop ablation fixes the shared two-block backbone, both value heads and their supervision, and token-wise PPO. Only the credit supplied to the actor changes:

Actor credit with the same dual-head criticWebShop success (%)
Token only67.4
Turn only59.8
Direct addition55.0
VETTA centered composition72.8

VETTA exceeds the strongest single-granularity credit rule by 5.4 percentage points. Direct addition and centering both use α=3\alpha=3; their 17.8-point difference supports the centering benefit with the critic and PPO settings fixed. Together, these experiments show that coordinating turn/token credit improves task success beyond the benefit of reducing critic depth.

The direct-addition control is included in Section 4.3 (Table 3), and the actor-credit ablation is analyzed in Section 4.4 (Table 4), with the controlled settings specified in Appendix A.6.

W2/Q2. Performance at 7B and broader applicability.

The larger gains at 1.5B suggest that turn/token coordination may be particularly helpful with limited model capacity. At 7B, VETTA improves ALFWorld success from the strongest baseline's 94.3% to 95.5% and maintains the highest mean success on both original benchmarks.

To test whether smaller gains are a general limitation at larger scales, we evaluated ScienceWorld with Qwen2.5-7B-Instruct and Sokoban with Qwen2.5-VL-3B-Instruct. Turn-PPO outperforms PPO*, GiGPO, and GRPO on both benchmarks, including both Sokoban reward settings:

MethodScienceWorld success (%)ScienceWorld task scoreSokoban sparse success (%)Sokoban dense success (%)
Turn-PPO50.724.378.1 ± 1.678.6 ± 2.0
VETTA52.225.984.9 ± 1.280.7 ± 1.2

ScienceWorld demonstrates continued gains at 7B, while Sokoban extends the benefit to a 3B vision-language model under both reward settings. The smaller ALFWorld gain is consistent with diminishing returns near its performance ceiling.

We have incorporated ScienceWorld and Sokoban into the main experiments in Section 4.2 (Table 2), with task descriptions, training settings, and evaluation protocols in Appendices B.1-B.3.

AuthorsOpenReview ↗

Follow-up on HyGAE comparisons and larger-model evaluation

Dear Reviewer 9f32,

Thank you for your helpful suggestions on establishing VETTA's contribution relative to existing methods and understanding its performance at larger model scales.

We have added direct HyGAE comparisons at 1.5B and 7B, comparisons with two-block baselines, and controlled credit-composition ablations. We have also included ScienceWorld results at 7B and Sokoban results at 3B to examine applicability beyond the original benchmarks. These additions appear in Tables 1–4 and Sections 4.2–4.4 of the revised manuscript, with the corresponding settings in the appendices.

As the discussion period draws to a close, we would appreciate your feedback on whether the additional comparisons and analysis clarify the contribution and scope of our method. We would welcome the opportunity to address any remaining questions.

Thank you again for your time and consideration.

I would like to clarify that I read and evaluated the paper myself. I used ChatGPT to look up related information and help organize and phrase my comments in English. The review contains substantial human input, and the concerns below reflect my own assessment of the paper.

My original concerns are as follows:

  1. Comparison with HyGAE. HyGAE is the closest related method and also combines turn- and token-level advantages. Without a direct comparison, it was unclear whether VETTA's centered composition and separate value supervision provide a meaningful improvement over HyGAE.

  2. Limited gains at 7B. The improvements over the strongest baseline were only 1.2 percentage points on ALFWorld and 0.8 on WebShop. The component analyses mainly focused on 1.5B, leaving it unclear why the gains become smaller and whether the proposed components remain consistently beneficial at 7B.

  3. Attribution of the gains. VETTA used a two-block critic, while PPO and Turn-PPO used full-depth critics. Since the paper also showed that reducing critic depth can improve performance, it was unclear how much of the benefit came from the credit assignment method and how much came from critic reduction.

Additional Comments After Rebuttal

  1. Comparison with HyGAE. The additional comparison under the same training budget and critic depth addresses this concern. I appreciate the authors providing direct comparisons at both 1.5B and 7B.

  2. Limited gains at 7B. This concern remains. The ceiling effect may explain the small gain on ALFWorld, but it does not explain the small gain on WebShop. The additional ScienceWorld evaluation is useful, but the 7B success rate improves from 50.7% to 52.2%, a gain of 1.5 percentage points. The rebuttal also does not sufficiently explain why the gains become smaller or how much each component contributes at 7B.

  3. Attribution of the gains. The comparisons with matched critic depth and the controlled credit-composition experiments address this concern. These results provide evidence that critic reduction alone does not explain the improvement.

Overall, the additional experiments address two of my main concerns. However, the gains at the larger model scale remain modest, and I am still not convinced that the paper demonstrates a sufficiently substantial advance over existing work. I will therefore maintain my original score.

Thank you for confirming that the direct HyGAE comparisons and the controlled ablations resolve two of your three original concerns. We would like to clarify one point that bears on your remaining concern about the magnitude of the 7B gains.

The 1.2-point margin on ALFWorld is relative to Turn-PPO, whereas the 0.8-point margin on WebShop is relative to GiGPO. Table 1 shows why the choice of comparator matters.

7B methodALFWorld success (%)WebShop success (%)
Turn-PPO94.354.6
GiGPO90.275.2
VETTA95.576.0

Turn-PPO is a particularly relevant comparator because it already uses a critic to assign turn-level credit. Against this same method, VETTA gains 1.2 percentage points on ALFWorld and 21.4 points on WebShop at 7B. The WebShop gain is almost identical to the 21.6 points observed at 1.5B. With the comparator held fixed, the 7B advantage ranges from modest on ALFWorld to substantial on WebShop. Turn-PPO and GiGPO each come closest on a different task, while VETTA achieves the highest mean success on both tasks at both model sizes.

The 1.5B ablations show that centering improves success over direct addition on both tasks when critic depth and PPO settings are held fixed. The consistent direction of the full-method comparisons at 1.5B and 7B suggests that coordinating turn and token credit remains useful at the larger scale. We will extend the controlled ablations to 7B to quantify the individual contributions of centering and separate value supervision.

On ScienceWorld, the 1.5-point gain over Turn-PPO means that VETTA successfully completed 27 more test instances on the same 1,819-instance evaluation: 949 successes versus 922 (52.2% versus 50.7%). This is approximately 3% more successful completions. Mean task score also rose from 24.34 to 25.89, a 6.4% relative improvement. We consider this a meaningful gain over the strongest tested baseline on both reported metrics.

Given that the revision resolves two of your three concerns, and that the WebShop comparison shows a substantial gain over Turn-PPO at both model sizes, we respectfully ask you to reconsider whether the revised evidence warrants a higher overall score.

Reviewer_93LVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a credit assignment method named VETTA, which uses a shared critic with separate value heads and combines the turn advantage with a token residual centered within each response. VETTA retains only a few early pretrained Transformer blocks to reduce critic computation. Experiments shows the effectiveness of the proposed VETTA on ALFWorld and WebShop datasets.

Critical strengths

This paper proposes a practical method which can save training cost via a shallow shared critic. In the same-host ALFWorld depth comparison, median value-inference-plus-update time decreases from about 152.0 seconds with 28 blocks to 19.7 seconds with two blocks, while task success remains strong.

Critical weaknesses

The contribution of the proposed credit-assignment rule is not convincingly established. The main comparisons also change critic depth and, for Turn-PPO, the actor loss, so the gains cannot be attributed to the coordination rule alone. The matched ablation covers only one model–task setting and one evaluation run, and its higher success rate comes with a lower task score than token-only credit. Without a direct comparison with HyGAE, which the paper identifies as its closest related method, the advantage over existing joint estimation methods remains unclear.

Questions for authors

Q1. How were the default residual weights selected, and would new tasks require retuning, given the strong and opposite effects of increasing α\alpha across the two environments?

Q2. The experiments do not establish a reliable performance advantage. The reported three seeds evaluate the same trained checkpoint with different decoding seeds, so the error bars in experiments do not capture variation across training runs.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 93LV: Credit attribution, robustness, and alpha

Thank you for the thoughtful suggestions on credit attribution, evaluation reliability, and residual-weight selection.

Q2/W1. Credit attribution and reliability of the experimental results.

To control critic depth, we compare PPO*, Turn-PPO, and VETTA with two-block critics using Qwen2.5-1.5B-Instruct:

Method (2-block critic)ALFWorld success (%)WebShop success (%)
PPO*70.056.4
Turn-PPO89.367.8
VETTA92.972.8

VETTA's reported success exceeds Turn-PPO by 3.6 percentage points on ALFWorld and 5.0 points on WebShop at the same critic depth.

To separately address the actor-loss confound, our WebShop ablation fixes the shared two-block backbone, both value heads and their supervision, and token-wise PPO ratios and clipping. Only the advantage supplied to the actor changes:

Actor credit with the same dual-head criticWebShop success (%)Task Score
Token only67.484.9
Turn only59.874.5
Direct addition55.079.0
VETTA centered composition72.883.6

Direct addition and VETTA both use α=3\alpha=3. Centering raises success from 55.0% to 72.8%, and VETTA exceeds the strongest single-granularity credit rule by 5.4 points. These controls support a credit-composition benefit with critic architecture and PPO settings fixed.

The success/Task Score trade-off reflects different objectives: Task Score measures partial satisfaction, while our positive task reward requires complete success. VETTA improves complete-task success; token-only credit achieves the higher graded score.

We have added the two-block baseline comparisons to Section 4.3 (Table 3) and clarified the credit-composition analysis in Section 4.4 (Table 4).

Following your suggestion, we compare VETTA with HyGAE using Qwen2.5-1.5B-Instruct and two-block critics:

MethodALFWorld success (%)WebShop success (%)
HyGAE (2-block)79.368.2
VETTA (2-block)92.972.8

With critic depth matched, VETTA improves success by 13.6 percentage points on ALFWorld and 4.6 on WebShop, supporting its separate value supervision and centered credit composition.

We further compare the two methods on ALFWorld and WebShop at both 1.5B and 7B under the same training budget:

ModelMethodALFWorld success (%)WebShop success (%)
1.5BHyGAE85.0 ± 0.765.5 ± 0.6
1.5BVETTA91.9 ± 1.173.8 ± 1.1
7BHyGAE93.8 ± 0.473.3 ± 0.6
7BVETTA95.5 ± 0.476.0 ± 0.6

VETTA outperforms HyGAE in all four settings. HyGAE mixes value targets and linearly combines advantages; VETTA supervises turn/token values separately and centers token credit, preserving the turn advantage as the response mean while refining credit within the response.

We have added the same-depth comparison to Section 4.3 (Table 3) and the broader comparison to Table 1, with experiment settings in Appendix A.6.

Independent training runs (Q2).

To assess training stability, we trained VETTA with three different training seeds on ALFWorld using Qwen2.5-1.5B-Instruct and a two-block critic, and evaluated all three policies under fixed decoding settings. The three runs achieved 91.4%, 88.6%, and 89.3% success, giving a mean ± sample SD of 89.8 ± 1.5% and indicating modest observed variation across independently trained policies.

We have added the independent-training results and their evaluation settings to Appendix A.3 (Table 5).

Q1. Residual-weight selection and adaptation to new tasks.

To select the residual weight and examine its task dependence, we compared the manually specified candidates α∈{1,2,3}\alpha\in\{1,2,3\} on ALFWorld and WebShop. The sensitivity analysis reports mean success over three decoding seeds:

α\alphaALFWorld success (%)WebShop success (%)
191.959.7
289.165.7
379.873.8

We selected α=1\alpha=1 for ALFWorld and α=3\alpha=3 for WebShop because they yield the highest observed success on their respective tasks. The opposite trends indicate that the appropriate residual strength depends on the environment.

A plausible interpretation is that wording-sensitive actions, such as WebShop search queries, benefit from a stronger token residual, whereas ALFWorld's whole-command actions favor greater emphasis on turn-level credit. For a new task, we therefore recommend a small-grid comparison, using these action characteristics as a tuning heuristic.

We have clarified the candidate range and task-dependent choice of α\alpha in Appendix A.1 (Figure 5), and specified the evaluation and error-bar definitions in Appendix B.3.

AuthorsOpenReview ↗

Follow-up on credit attribution and evaluation reliability

Dear Reviewer 93LV,

Thank you for your thoughtful comments on credit attribution, evaluation reliability, and residual-weight selection.

We have added comparisons with two-block PPO* and Turn-PPO, controlled credit-composition ablations, and direct comparisons with HyGAE. These experiments help distinguish the contribution of credit coordination from critic depth and PPO settings. We have also reported results from independently trained VETTA policies and clarified the task-dependent choice of the residual weight in the revised manuscript.

As the discussion period draws to a close, we would appreciate your feedback on whether these controls and clarifications address your main questions about attribution and reliability. We would be happy to discuss any points that remain unclear.

Thank you again for your time and consideration.

Reviewer_vqjhOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

VETTA combines turn-level advantages with token-level advantages centered within each response for multi-turn LLM agent training. It estimates the two value functions through separate heads on a shared critic, retains only two Transformer blocks in its main experiments, and uses the combined advantages for token-wise PPO updates. On ALFWorld and WebShop, the paper reports success rates of 91.9% and 73.8% with Qwen2.5-1.5B-Instruct, and 95.5% and 76.0% with Qwen2.5-7B-Instruct, respectively.

Critical strengths
  1. Within-response centering preserves the turn-level advantage as the response mean while allowing token-specific adjustments.
  2. A controlled WebShop ablation reports 72.8% success with centering versus 55.0% with direct addition.
  3. The two-block critic reports approximately 87% and 91% lower critic-side computation time on ALFWorld and WebShop compared with the 28-block critic.
Critical weaknesses
  1. The reported error bars capture decoding variability for fixed checkpoints, leaving robustness across independent training seeds unestablished.
  2. Main-table comparisons vary critic depth and PPO settings, limiting attribution of the reported gains to the proposed advantage fusion.
  3. The theoretical analysis establishes an algebraic projection property but provides no guarantees of unbiasedness, variance reduction, or policy improvement for the fused estimator.
  4. The experiments omit a direct comparison with HyGAE, which the authors identify as the closest related approach.
  5. VETTA shares HiPER’s use of a shared two-head critic and credit assignment across temporal scales, but the incremental value of its turn/token formulation remains insufficiently established without a direct comparison [1].

[1] HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents, 2026.

Questions for authors
  1. Can you report results across independent training seeds and distinguish training variability from decoding variability?
  2. Do the gains persist across both benchmarks and model sizes when critic depth and PPO settings are matched between VETTA and the baselines?
  3. Can you characterize how within-response centering affects policy-gradient bias and variance beyond the algebraic projection property established in the appendix?
  4. How does VETTA compare with HyGAE under matched training budgets and evaluation protocols?
  5. How does VETTA compare with HiPER under a common evaluation protocol, and what evidence establishes the incremental benefit of its turn/token credit formulation?
5 comments
AuthorsOpenReview ↗

Response to Reviewer vqjh - Part 1/2: W1-W3 and Q1-Q3

Thank you for the detailed review and the helpful suggestions on training robustness, credit attribution, and comparisons with related methods.

W1/Q1. Results across independent training runs.

To assess training stability, we trained VETTA with three different training seeds on ALFWorld using Qwen2.5-1.5B-Instruct and a two-block critic, and evaluated all three policies under fixed decoding settings. The three runs achieved 91.4%, 88.6%, and 89.3% success, giving a mean ± sample SD of 89.8 ± 1.5% and indicating modest observed variation across independently trained policies.

We have added the independent-training results and their evaluation settings to Appendix A.3 (Table 5).

W2/Q2. Separating credit composition from critic depth and PPO settings.

To control critic depth, we compare PPO*, Turn-PPO, and VETTA with two-block critics using Qwen2.5-1.5B-Instruct:

Method (2-block critic)ALFWorld success (%)WebShop success (%)
PPO*70.056.4
Turn-PPO89.367.8
VETTA92.972.8

VETTA exceeds Turn-PPO by 3.6 points on ALFWorld and 5.0 on WebShop at the same depth. Each method retains its own value supervision and PPO objective.

To isolate credit composition, the WebShop ablation fixes the shared two-block backbone, both value heads and their supervision, and token-wise PPO ratios and clipping. Only the actor advantage changes:

Actor credit with the same dual-head criticWebShop success (%)
Token only67.4
Turn only59.8
Direct addition55.0
VETTA centered composition72.8

VETTA exceeds the strongest single-granularity credit rule by 5.4 points. Direct addition and centering both use α=3\alpha=3; their 17.8-point difference supports the benefit of centering with the critic and PPO objective held fixed.

These results support the credit-composition benefit with critic and PPO settings fixed, alongside VETTA's advantage over both baselines at matched depth.

We have added the matched-depth and centering comparisons to Section 4.3 (Table 3). Section 4.4 (Table 4) distinguishes value supervision from actor credit.

W3/Q3. Gradient bias, variance, and policy performance.

Centering gives the two credit signals distinct roles in the policy update: the turn advantage provides response-wide credit, while the centered token advantage differentiates decisions within that response. At the gradient level, centering removes the token mean's additional whole-response contribution while preserving the turn-level term.

Let aja_j be the token advantage, sj=∇θlog⁡πθ(yj∣H,y<j)s_j=\nabla_\theta\log\pi_\theta(y_j\mid H,y_{<j}), and bars denote within-response averages. At the rollout policy, before advantage normalization and PPO clipping, the per-response mean gradients satisfy

gc=gd−c,c=αaˉ sˉ.g_c=g_d-c,\qquad c=\alpha\bar a\,\bar s.

Here gcg_c and gdg_d denote centered composition and direct addition.

Bias. Centering changes the expected gradient by removing the expected response-wide contribution of the token mean. For a common target policy gradient g⋆g^\star, define b(g)=E[g]−g⋆b(g)=\mathbb{E}[g]-g^\star. Then

b(gc)−b(gd)=−E[c].b(g_c)-b(g_d)=-\mathbb{E}[c].

The expected correction therefore determines the bias change introduced by centering.

Variance. Centering changes gradient variance through both the variability of the removed component and its correlation with the original gradient. For any fixed parameter-space direction uu,

\operatorname{Var}(u^\top g_c)-\operatorname{Var}(u^\top g_d) &=\operatorname{Var}(u^\top c)\\ &\quad-2\operatorname{Cov}(u^\top g_d,u^\top c). \end{aligned}$$ Variance decreases in that direction when twice the covariance exceeds the correction's variance, explicitly accounting for its dependence on the original gradient. To evaluate the resulting policy performance, we compare direct addition with centering while fixing the shared two-block dual-head critic, its supervision, token-wise PPO, and the residual weight within each task ($\alpha=1$ on ALFWorld and $\alpha=3$ on WebShop): | Credit composition | ALFWorld success (%) | WebShop success (%) | | --- | ---: | ---: | | Direct addition | 81.4 | 55.0 | | VETTA centered composition | **92.9** | **72.8** | Centering improves success by **11.4 percentage points on ALFWorld** and **17.8 on WebShop**. These controlled gains demonstrate that **within-response centering makes turn/token credit composition more effective for policy learning on both benchmarks**. We have expanded **Appendix C.4 (Eqs. 13–15)** with this gradient analysis and summarized its implication in **Section 3.2**. The paired centering ablations are reported in **Section 4.3 (Table 3)**.
AuthorsOpenReview ↗

Response to Reviewer vqjh - Part 2/2: HyGAE and HiPER

W4/Q4. Direct comparison with HyGAE.

Our further comparisons demonstrate VETTA's higher success than HyGAE. We reproduced HyGAE at 1.5B and 7B under the same training budget:

ModelMethodALFWorld success (%)WebShop success (%)
1.5BHyGAE85.0 ± 0.765.5 ± 0.6
1.5BVETTA91.9 ± 1.173.8 ± 1.1
7BHyGAE93.8 ± 0.473.3 ± 0.6
7BVETTA95.5 ± 0.476.0 ± 0.6

HyGAE mixes value targets and linearly combines advantages; VETTA supervises turn/token values separately and centers token credit, preserving the turn advantage as the response mean while refining credit within the response.

We also compare both methods with two-block critics at 1.5B under the same evaluation protocol:

MethodALFWorld success (%)WebShop success (%)
HyGAE (2-block)79.368.2
VETTA (2-block)92.972.8

With critic depth matched, VETTA improves success by 13.6 percentage points on ALFWorld and 4.6 on WebShop. These results demonstrate VETTA's advantage over HyGAE and support its separate value supervision and centered credit composition. We have added these comparisons and analyses to Section 4.2 (Table 1) and Section 4.3 (Table 3), with settings in Appendix A.6.

W5/Q5. HiPER comparison and the incremental value of turn/token credit.

VETTA's contribution beyond a shared critic is the explicit coordination of response-level and within-response credit. HiPER assigns credit to subgoal selection, switching, and execution within a plan-execute hierarchy. VETTA separately supervises turn and token values, uses the turn advantage to determine a response's mean credit, and uses the centered token residual to distinguish individual token decisions within that response. This specifies the additional credit-composition design beyond sharing a critic across temporal scales.

To evaluate this design, we compare VETTA with HiPER on ALFWorld and WebShop. Both methods use Qwen2.5-1.5B-Instruct and a training budget of 150 updates × 128 trajectories per update. We reproduced HiPER using its official implementation:

MethodALFWorld success (%)WebShop success (%)WebShop task score
HiPER80.760.477.8
VETTA92.972.883.6

In this comparison, VETTA achieves higher success rates on both benchmarks and a higher WebShop task score while using a two-block critic.

We have added the direct HiPER comparison, including the ALFWorld task breakdown, to Appendix A.7 (Table 8). Section 5 also clarifies the distinction between hierarchical planning/execution credit and VETTA's turn/token coordination.

AuthorsOpenReview ↗

Follow-up on the additional analyses and comparisons

Dear Reviewer vqjh,

Thank you for your detailed review and suggestions for strengthening the evidence supporting VETTA.

In our responses, we have added independent-training results, matched-depth baseline comparisons, controlled credit-composition ablations, and an explicit analysis of how centering changes policy-gradient bias and variance. We have also provided direct comparisons with HyGAE and HiPER and clarified the contribution of VETTA's turn/token credit formulation. These additions are included in the revised manuscript, with their locations identified in our responses to W1–W5 and Q1–Q5.

As the discussion period draws to a close, we would greatly appreciate your feedback on these additions, particularly which of your main concerns, if any, require further clarification. We would welcome the opportunity to address them during the remaining discussion.

Thank you again for your time and consideration.

Reviewer_mk23OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors consider a credit assignment problem in multi-turn settings. It is a well-known problem: feedback is available at the end of extended interactions, and one seeks to assign credit to individual decisions. Two distinct scales exist here: turn-level and token-level. Turn-level scale is natural in multi-turn settings, yet most LLMs operate on the token-level scale. The authors propose a credit-assignment algorithm that operates on the two scales and provide some empirical evidence that such an approach leads to improved performance.

Critical strengths

Combining the two scales is a viable idea that appeared in the literature in different contexts. The analysis in Section 4.3 is interesting (cf. Table 2). It indicates that the turn-level scale on its own is more “relevant” that the token-level scale on its own. However, it also indicates that the question on how to effectively combine the two scales is nuanced and warrants more investigation.

Critical weaknesses

The authors report that a simplified critic architecture leads to substantially improved policy performance in their experiments. While this is an interesting finding, it is not clear if it is due a particular implementation, the size of the training set, inability to train a complex architecture, or some other reason. For this reason, it is hard to interpret this observation.

Questions for authors

None.

2 comments
AuthorsOpenReview ↗

Response to Reviewer mk23: Critic depth and training budget

Thank you for highlighting the need to explain the critic-depth results. Our leading interpretation is that the large WebShop gap reflects differences in policy-learning progress within the original training budget: the two-block configuration reaches strong performance within 150 updates, while the full-depth configuration benefits substantially from further optimization.

To examine this explanation, we continued the original 28-block WebShop model from 150 to 200 updates and evaluated it on the same 500 tasks with the same three decoding seeds:

Critic blocksTraining updatesSuccess (%)Task score
215073.8 ± 1.183.9 ± 0.6
2815062.9 ± 1.181.4 ± 1.2
2820071.9 ± 1.484.4 ± 0.6

Values are mean ± standard deviation over three decoding seeds.

The additional training reduces the success-rate gap from 10.9 to 1.9 percentage points, closing approximately 82% of the original gap; the continued full-depth model also achieves a slightly higher task score.

We also considered the three alternative explanations raised in your review:

  • Implementation. Both depths use the same VETTA implementation, advantage computation, and actor/critic losses. This controls for differences in the learning algorithm or loss functions that could otherwise explain the shallow model’s advantage.
  • Training-set size. The original experiments use pools of 3,553 ALFWorld tasks and 6,410 WebShop goals, with the same sampling procedure and batch sizes at both depths. Moreover, the full-depth model recovers most of the WebShop gap without expanding the task pool. We therefore consider limited training progress a better-supported explanation than a shortage of available training tasks.
  • Difficulty training a complex architecture. The original full-depth model improves substantially with further training, without changing its architecture or learning objectives. This indicates that it can be trained effectively but requires more training to approach the two-block model’s performance in this experiment.

Taken together, the shared implementation, matched data settings, and successful full-depth continuation favor our training-budget explanation for the WebShop gap: the two-block critic reaches competitive performance earlier in this experiment, with substantially lower critic cost.

We have added the continuation results and the discussion of these factors to Appendix A.5 (Table 7). We have also revised the Introduction and Section 4.4 (RQ4) to emphasize the two-block critic’s balance between task performance and computational cost at the evaluated budget, and the role of learning progress in the observed depth gap.

AuthorsOpenReview ↗

Follow-up on the critic-depth analysis

Dear Reviewer mk23,

Thank you for your thoughtful feedback on the interpretation of the critic-depth results.

Following your suggestions, we continued training the full-depth WebShop model and found that it recovered most of the original performance gap. We have added these results to Appendix A.5 (Table 7), together with a discussion of the implementation, training-data, and optimization factors you raised. We have also refined our interpretation of the two-block critic's performance and computational benefits.

As the discussion period draws to a close, we would appreciate your feedback on whether this additional evidence addresses your concern about the depth comparison. We would be happy to clarify any remaining points.

Thank you again for your time and consideration.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to Reviewer e9pr - Part 1/2: Credit attribution

Thank you for the constructive suggestions on isolating the credit-assignment contribution and strengthening the robustness and efficiency evaluation.

Q1/W1. Matched-depth comparisons and credit-composition ablations.

To isolate the contribution of advantage fusion, our WebShop ablation fixes the shared two-block backbone, both value heads and their supervision, and token-wise PPO ratios and clipping. Only the advantage supplied to the actor changes:

Actor credit with the same dual-head criticWebShop success (%)
Token only67.4
Turn only59.8
VETTA centered composition72.8

Centered composition improves success by 5.4 percentage points over the strongest single-granularity rule, supporting the benefit of coordinating both credit signals with the critic and PPO objective fixed.

To isolate centering on both benchmarks, we compare direct addition with VETTA while holding the dual-head two-block critic, token-wise PPO, and the within-task residual weight fixed (α=1\alpha=1 on ALFWorld and α=3\alpha=3 on WebShop). To address the depth difference in the main table, we also evaluated two-block PPO* and Turn-PPO; all results below use Qwen2.5-1.5B-Instruct:

Method (2-block critic)ALFWorld seen success (%)ALFWorld unseen success (%)WebShop success (%)
PPO*70.064.256.4
Turn-PPO89.384.367.8
Direct addition81.479.955.0
VETTA92.989.672.8

Centering improves success over direct addition by 11.4 points on seen ALFWorld, 9.7 points on unseen ALFWorld, and 17.8 points on WebShop. At the same two-block depth, VETTA also exceeds Turn-PPO by 3.6 points on seen ALFWorld and 5.0 points on WebShop.

The actor-credit ablations in Section 4.4 (Table 4) demonstrate the benefit of coordinating both advantages with the critic and PPO objective fixed. We have added the matched-depth baseline comparisons and the centering controls on both benchmarks to Section 4.3 (Table 3).

AuthorsOpenReview ↗

Response to Reviewer e9pr - Part 2/2: Generalization and efficiency

Q2/W2. Seen/unseen generalization and independent training runs.

To evaluate generalization beyond the seen ALFWorld tasks, we additionally evaluated PPO*, Turn-PPO, and VETTA at both 1.5B and 7B on the 134-task unseen split. The table pairs these new results with the seen-split means from Table 1, using the main-table configurations: full-depth baseline critics and VETTA's two-block dual-head critic.

ModelMethodSeen success, mean (%)Unseen success (%)
1.5BPPO*64.866.4
1.5BTurn-PPO76.483.6
1.5BVETTA91.989.6
7BPPO*72.676.9
7BTurn-PPO94.385.1
7BVETTA95.586.6

On unseen ALFWorld tasks, VETTA achieves higher success than PPO* and Turn-PPO at both model sizes, exceeding Turn-PPO by 6.0 percentage points at 1.5B and 1.5 points at 7B. These results extend VETTA's observed advantage beyond the seen split.

To assess training stability, we trained VETTA with three different training seeds on ALFWorld using Qwen2.5-1.5B-Instruct and a two-block critic, evaluating each policy on both splits under fixed decoding settings. Success rates were 91.4%, 88.6%, and 89.3% on seen tasks and 86.6%, 87.3%, and 83.6% on unseen tasks. The corresponding means ± sample SDs are 89.8 ± 1.5% and 85.8 ± 2.0%. These results indicate modest variation across independently trained policies on both splits.

We have added the independent-training results to Appendix A.3 (Table 5) and the seen/unseen comparisons to Appendix A.4 (Table 6), including their evaluation settings.

Q3/W2. Training time, memory, and residual-weight sensitivity.

To quantify practical savings beyond critic-side timing, we measured the cumulative wall-clock time of 150 training steps on eight A100 80GB GPUs, including rollout, actor/critic updates and in-step validation/checkpointing. Time decreases from 37.8 to 32.7 hours on ALFWorld (13.5%) and 18.5 to 10.4 hours on WebShop (43.9%), while peak per-GPU PyTorch allocated memory decreases by 4.55% and 4.59%, respectively.

Critic value-inference and update time decreases by 87.7% and 91.0%, respectively, leaving critic computation at approximately 4% of an ordinary training step. Rollout and actor updates account for most of the remaining time, explaining why the total training-time reduction is smaller than the critic-side reduction; the reported totals reflect the observed complete training-step workloads.

To clarify the sensitivity of the residual weight, Appendix A.1 (Figure 5) compares the manually specified candidates α∈{1,2,3}\alpha\in\{1,2,3\}, reporting mean success over three decoding seeds:

α\alphaALFWorld success (%)WebShop success (%)
191.959.7
289.165.7
379.873.8

The best observed values are α=1\alpha=1 for ALFWorld and α=3\alpha=3 for WebShop. A plausible interpretation is that wording-sensitive search queries benefit from stronger token-level differentiation, whereas whole-command actions favor greater emphasis on turn-level credit. The opposite trends motivate task-specific selection from a small grid when applying VETTA to a new environment.

We have added the training-time and memory measurements to Appendix B.4 (Table 11).

AuthorsOpenReview ↗

Follow-up on controlled comparisons, generalization, and efficiency

Dear Reviewer e9pr,

Thank you for your constructive suggestions on isolating the credit-assignment contribution and broadening the evaluation.

We have added matched-depth baseline comparisons and controlled centering experiments on both benchmarks, alongside independent-training results and seen/unseen ALFWorld evaluations. We have also reported total training-step time and peak memory measurements, distinguishing these from critic-side savings. The revised manuscript includes the controlled comparisons in Sections 4.3–4.4, the additional robustness and generalization results in Appendices A.3–A.4, and the resource measurements in Appendix B.4.

As the discussion period draws to a close, we would appreciate your feedback on whether these additions address your main concerns. Please let us know if any aspects would benefit from further clarification.

Thank you again for your time and helpful feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.