acceptodds
Under review as a conference paper at ICLR 2027

Learning What the Group Cannot Tell: Decomposed Critics for LLM Reinforcement Learning

Abstract

Reinforcement learning (RL) has become an important approach for post-training large language models (LLMs). In many LLM RL tasks, feedback is sparse or available only at the end of a trajectory, making credit assignment across intermediate states important for effective policy optimization. Proximal Policy Optimization (PPO) addresses this problem with a learned value function, which estimates the expected return of intermediate states and provides state-dependent advantages for policy updates. However, modern LLM-based RL algorithms typically collect multiple rollouts for each prompt, thereby exposing group-level information that conventional critics do not explicitly exploit. In this work, we revisit critic learning in such grouped settings.Instead of learning the full state value directly from individual terminal outcomes, We decompose value estimation into two complementary components: a prompt-level success prior estimated from grouped rollouts and history-dependent evidence learned by the critic. Based on this decomposition, we introduce **G**roup-**A**nchored **V**alue **E**stimation (GAVE), which trains the critic with a group-conditional objective and reconstructs standard state values by combining the learned correction with an empirical prompt prior. We further prove that estimation errors in the empirical prior are not amplified in expectation during reconstruction, yielding a finite-sample error bound. Experiments on LLM agent tasks show that GAVE consistently improves both overall and within-prompt value estimation compared with conventional critics and information-matched baselines, while also leading to improved policy optimization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, reinforcement learning, value estimation, credit assignment

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_bddgOpenReview ↗
Rating
3(+1)
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GAVE, a PPO critic that exploits grouped rollouts. For binary terminal rewards, it decomposes value log-odds into a prompt-level success prior, estimated from sibling rollouts, and a history-dependent evidence term, learned by the critic via a group-count-offset BCE objective. Experiments on agent tasks and math with Qwen3-4B/8B show gains over PPO, GRPO, and GiGPO, as well as more accurate value estimates than standard critics.

Critical strengths
  1. Principled, simple idea: the Bayesian decomposition removes the prompt prior from the critic's target via a group-count offset (Prop. 1). The method is a drop-in change to the critic loss and value reconstruction within the standard PPO pipeline.
  2. Direct critic-level evidence: on a frozen-policy ALFWorld evaluation against Monte Carlo reference values, GAVE achieves lower overall and within-prompt error than information-matched baselines, and the gains are broadly distributed across prefixes.
Critical weaknesses
  1. Credit-assignment claim is not supported by the actual setup. The paper motivates GAVE as improving credit assignment over intermediate decisions. However, the agent experiments use γ=λ=1, so GAE reduces to A^t=Ri−V^(st)\hat A_t = R_i - \hat V(s_t) with no bootstrapping. In this regime, the critic acts only as a state-dependent baseline for variance reduction; it does not propagate value information across steps. This matters because it changes what the experiments actually demonstrate: better baselining, not better credit assignment. The authors should report agent-task results with λ<1 (e.g., 0.9/0.95) for both PPO and GAVE, and show whether GAVE's advantage persists or grows when the critic is used for bootstrapping.

  2. The source of the policy gains is not isolated. With λ=1, GAVE's advantage at the first step is essentially Ri−p~−iR_i - \tilde p_{-i}, i.e., an RLOO-style leave-one-out baseline. The information-matched baselines (MSE/BCE critic + group prior) appear only in the frozen-policy critic evaluation (Table 3), not in RL training (Tables 1–2). The reported gains over PPO, GRPO, and GiGPO may therefore come largely from the leave-one-out prior rather than from the group-conditional evidence critic, which is the paper's main contribution. This is the main reason for my recommendation. In the Table 1/2 settings, the authors should add RL-training ablations: (i) a standard PPO critic combined with the same leave-one-out prior during training (Eq. 18 applied online); (ii) prior only, i.e., GAVE with fϕ≡0f_\phi \equiv 0; (iii) RLOO, and RLOO plus a standard critic baseline.

  3. The baselines are incomplete and not compute-matched. Math compares only against PPO and GRPO. It omits VinePPO, the most direct competitor for value estimation in LLM RL, as well as VC-PPO/VAPO (value pretraining, decoupled GAE), RLOO, and DAPO. GAVE also trains a full Qwen3-4B critic, while GRPO is critic-free, yet no wall-clock time, GPU memory, or compute-matched comparison is reported. The authors should add these baselines and report training cost, ideally with a compute-matched GRPO variant (e.g., more rollouts per prompt).

Questions for authors
  1. What fraction of training groups are uninformative, i.e., K=0K=0 or K=GK=G, especially on math? For these groups the critic receives no gradient, and with the smoothed prior (e.g., p~−i≈0.06\tilde p_{-i}\approx0.06 when K=0K=0, G=8G=8), all trajectories get small, uniformly negative advantages. GRPO assigns zero advantage in this case. How does this behavior affect training, and what happens if such groups are masked?

  2. How accurate is the critic beyond the single ALFWorld step-50 checkpoint? Can the authors report value-estimation error against Monte Carlo references at later policy checkpoints and on math (e.g., segment-level prefixes), where the evidence term Δπ\Delta^\pi is non-stationary and critic learning is hardest?

4 comments
AuthorsOpenReview ↗

Response to Reviewer bddg (Part 1/3)

W1: Credit assignment and bootstrapped advantage estimation.

Response: We agree that with γ=λ=1\gamma=\lambda=1 and terminal-only rewards, GAE reduces to A^t=R−V(ht)\hat A_t=R-V(h_t), without bootstrapping. However, this does not mean that all steps in a trajectory receive the same advantage. For two consecutive steps,

A^t−A^t+1=V(ht+1)−V(ht).\hat A_t - \hat A_{t+1} = V(h_{t+1}) - V(h_t).

While GRPO assigns the same outcome-based advantage to every step in a trajectory, GAVE uses history-dependent value estimates, allowing its advantages to vary across states. These state-dependent advantages are available even at λ=1\lambda=1, although GAE does not use bootstrapping in this setting.

To further evaluate GAVE with bootstrapping, we also trained PPO and GAVE with λ=0.9\lambda=0.9 and 0.950.95, keeping all other settings unchanged.

Table R1. ALFWorld success rates with Qwen3-4B under different GAE parameters.

Methodλ=1.0\lambda=1.0λ=0.95\lambda=0.95λ=0.9\lambda=0.9
PPO63.9 ± 1.968.42 ± 1.0367.25 ± 1.06
GAVE88.6 ± 1.487.83 ± 0.9289.71 ± 1.03

GAVE continues to substantially outperform PPO at both λ=0.9\lambda=0.9 and 0.950.95, showing that its advantage persists when bootstrapped value estimates are used in GAE.

We have clarified the distinction between state-dependent advantages and bootstrapping and added these results to Appendix E.4 (Table 13). These additions are highlighted in blue in the revised paper.


W2: Isolating the contribution of the group-conditional critic objective.

Response: We agree that the frozen-policy comparisons in Table 3 do not isolate the contribution of the critic objective during policy optimization. We have therefore added online comparisons on ALFWorld with Qwen3-4B, covering the controls you suggested:

  • (i) MSE/BCE Critic + Group Prior: Train conventional critics with their original MSE or BCE objectives and terminal-return targets, and apply the correction in Eq. (18) for PPO advantage estimation using the same smoothed leave-one-out prior estimator as GAVE.
  • (ii) Prior Only: Use the smoothed leave-one-out prior as the value estimate at every nonterminal state, without training a critic.
  • (iii) RLOO and RLOO + Critic: RLOO uses the mean reward of sibling rollouts as its baseline. For RLOO + Critic, we additionally subtract a standard MSE critic's predicted change in value from the initial state:

A^i,tRLOO+C=Ri−Rˉ−i−(V^(si,t)−V^(si,0)).\widehat A_{i,t}^{\mathrm{RLOO+C}} = R_i - \bar R_{-i} - \left(\widehat V(s_{i,t}) - \widehat V(s_{i,0})\right).

Here, Rˉ−i\bar R_{-i} is the mean reward of the other rollouts for the same prompt, and V^\widehat V is trained with standard MSE.

All methods use the same base model, rollout budget, and evaluation protocol. Policy-update settings are shared within the PPO variants and within the two RLOO variants.

Table R2. ALFWorld success rates with Qwen3-4B under different critic formulations.

MethodALFWorld success (%)
PPO (MSE)63.9 ± 1.9
MSE Critic + Group Prior70.4 ± 1.5
BCE Critic + Group Prior69.9 ± 4.3
Prior Only64.7 ± 1.9
RLOO67.6 ± 2.0
RLOO + Critic71.5 ± 2.4
GAVE88.6 ± 1.4

Prior Only performs similarly to standard PPO. GAVE achieves 88.6% success, compared with 70.4% for the strongest prior-corrected critic and 71.5% for RLOO + Critic. These results support the benefit of the group-conditional critic objective beyond incorporating the leave-one-out prior.

We have added these results to Section 4.4, Table 4, and the baseline definitions to Appendix E.1. These additions are highlighted in blue in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer bddg (Part 2/3)

W3: Additional baselines and training cost.

Response: Thank you for highlighting the missing baseline comparisons and the need to report training cost. We have added RLOO, DAPO, VinePPO, VC-PPO, and VAPO to our mathematical reasoning experiments with Qwen3-4B-Base. All methods use the same base model, training dataset, and evaluation protocol.

To examine whether additional rollouts can close the accuracy gap to GAVE, we also train GRPO with G=16G=16, doubling its default group size of G=8G=8. Table R3 reports accuracy in percent as mean ± standard deviation over three seeds.

Table R3. Mathematical reasoning accuracy with Qwen3-4B-Base.

MethodAIME24AIME25AMC23MATH-500Olympiad-BenchAvg.
PPO12.6 ± 0.19.4 ± 0.846.7 ± 0.076.8 ± 0.138.5 ± 0.136.8 ± 0.1
GRPO (G=8G=8)21.6 ± 0.518.3 ± 1.454.5 ± 0.982.2 ± 0.245.3 ± 0.344.4 ± 0.3
RLOO21.7 ± 1.618.2 ± 0.853.1 ± 1.081.5 ± 0.343.6 ± 0.443.6 ± 0.2
DAPO23.3 ± 1.119.9 ± 1.353.3 ± 1.281.8 ± 0.145.5 ± 0.444.8 ± 0.4
VinePPO11.2 ± 0.810.3 ± 0.947.9 ± 1.177.4 ± 0.238.8 ± 0.237.1 ± 0.1
VC-PPO11.0 ± 1.111.1 ± 0.642.2 ± 0.375.1 ± 0.135.4 ± 0.234.9 ± 0.1
VAPO12.2 ± 1.512.8 ± 2.246.0 ± 1.876.6 ± 0.138.2 ± 0.237.2 ± 0.8
GRPO (G=16G=16)22.6 ± 0.718.1 ± 1.255.5 ± 1.183.0 ± 0.346.2 ± 0.545.1 ± 0.3
GAVE24.8 ± 0.419.2 ± 0.757.8 ± 0.083.1 ± 0.147.0 ± 0.246.4 ± 0.1

GAVE achieves the highest average accuracy at 46.4%, compared with 44.8% for DAPO, the strongest of the five additional baselines. Increasing GRPO's group size improves its average accuracy from 44.4% to 45.1%, while GAVE retains a higher average accuracy.

We also report end-to-end training time and peak GPU memory under the same hardware configuration. Training time includes rollout generation and model optimization.

Table R4. Training cost on mathematical reasoning with Qwen3-4B-Base.

MethodWall-clock time (h)Peak GPU memory (GB)
PPO8.10122.76
GRPO (G=8G=8)10.06133.10
GRPO (G=16G=16)14.30148.98
GAVE12.45120.46

GAVE takes longer to train than standard GRPO (G=8G=8): 12.45 vs. 10.06 hours. We therefore also compare against GRPO with a larger group size (G=16G=16). This variant requires more training time (14.30 vs. 12.45 hours) and peak GPU memory (148.98 vs. 120.46 GB) than GAVE, but achieves lower average accuracy (45.1% vs. 46.4%).

Although PPO trains an additional critic, its responses are much shorter on average than GRPO's (1,235 vs. 2,300 tokens), which helps explain its lower overall training cost.

We have added the five additional baselines to Section 4.2, Table 2, and the larger-group comparison and training costs to Appendix E.2 (Tables 9 and 10). These additions are highlighted in blue in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer bddg (Part 3/3)

Q1: Homogeneous rollout groups and policy updates.

Response: We agree that all-failure (K=0K=0) and all-success (K=GK=G) groups provide no gradient for GAVE's evidence critic. These groups still inform the smoothed prompt prior used for value reconstruction and remain in PPO actor updates.

With γ=λ=1\gamma=\lambda=1, the advantage is A^i,t=Ri−V^(si,t)\hat A_{i,t}=R_i-\hat V(s_{i,t}). For all-failure groups, these advantages are negative before normalization, but their magnitudes need not be identical: the reconstructed values also depend on the history through the learned evidence term. GAVE can therefore produce nonzero, state-dependent advantages for these groups, whereas GRPO assigns zero outcome-based advantages.

To examine how frequently homogeneous groups occur, we analyzed rollout group composition on ALFWorld and mathematical reasoning with the 4B models, dividing training into three equal stages.

Table R5. Rollout group composition during GAVE training with the 4B models.

TaskTraining stageAll-failure (%)Mixed (%)All-success (%)
ALFWorldEarly32.042.925.1
Middle12.038.849.3
Late6.324.569.3
Overall16.835.447.9
MathEarly43.452.54.1
Middle33.355.611.1
Late29.454.616.0
Overall35.454.310.4

On mathematical reasoning, mixed groups account for more than half of the groups in each training stage and 54.3% overall. All-failure groups decrease from 43.4% to 29.4%, while all-success groups increase from 4.1% to 16.0%.

To assess the effect of homogeneous groups on policy learning, we also trained GAVE on mathematical reasoning with Qwen3-4B-Base while masking groups with K=0K=0 or K=GK=G from PPO actor updates.

Table R6. Mathematical reasoning accuracy after masking homogeneous groups from actor updates.

MethodAIME24AIME25AMC23MATH-500Olympiad-BenchAvg.
GAVE24.8 ± 0.419.2 ± 0.757.8 ± 0.083.1 ± 0.147.0 ± 0.246.4 ± 0.1
GAVE (masking)21.0 ± 1.620.4 ± 0.756.2 ± 0.283.0 ± 0.246.0 ± 0.445.3 ± 0.3

Masking homogeneous groups reduces average accuracy from 46.4% to 45.3%. This suggests that retaining these groups in actor updates benefits policy learning in this setting, despite their providing no gradient for the evidence critic.

We have added these analyses and clarifications to Appendix E.3 (Tables 11 and 12). These additions are highlighted in blue in the revised paper.


Q2: Critic accuracy at later policy checkpoints and on mathematical reasoning.

Response: We have extended the critic evaluation in Table 3 to later ALFWorld checkpoints (steps 100 and 150) and to mathematical reasoning (steps 50, 100, 150, and 200). At each checkpoint, we freeze the policy and estimate Monte Carlo reference values from 64 independent continuations per prefix.

Table R7. Critic MAE across ALFWorld checkpoints with Qwen3-4B.

MethodStep 50Step 100Step 150
Empirical Prior Only0.2500.1710.201
MSE Critic + Group Prior0.1580.1660.175
BCE Critic + Group Prior0.1690.1810.179
GAVE0.1160.1200.147

For mathematical reasoning, we evaluate four prefixes per trajectory, at 0%, 25%, 50%, and 75% of its length.

Table R8. Critic MAE across mathematical reasoning checkpoints with Qwen3-4B-Base.

MethodStep 50Step 100Step 150Step 200
Empirical Prior Only0.3140.3710.3750.362
MSE Critic + Group Prior0.2770.3510.3470.338
BCE Critic + Group Prior0.2640.3340.3110.329
GAVE0.2110.2730.2650.252

GAVE achieves the lowest MAE at every evaluated checkpoint on both tasks. These results extend the value-estimation gains beyond the original ALFWorld step-50 evaluation to later policy checkpoints and mathematical reasoning.

We have added these results to Appendix C.4, Table 8. These additions are highlighted in blue in the revised paper.

Reviewer_bddgOpenReview ↗

Official Comments from Reviewer bddg

Thanks for providing a detailed analysis of my concerns, as well as the additional baselines. I am happy to raise my score to 3. Good luck.

Reviewer_fkfMOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes GAVE, a method for improving critic learning in PPO by separating value estimation into a prompt-level success prior and history-dependent evidence. The prior is estimated from grouped rollouts, while the critic learns how the trajectory history changes the likelihood of success. The reconstructed values are then used for advantage estimation in PPO. The method is evaluated on interactive agent tasks, search-based question answering, and mathematical reasoning.

Critical strengths
  1. The decomposition is simple and well motivated. Grouped rollouts already provide information about prompt difficulty, so using this information to let the critic focus on history-dependent evidence is a sensible approach.

  2. The experimental results are encouraging. GAVE improves policy performance across several tasks and two model sizes. The separate fixed-policy critic evaluation also shows improvements in both overall and within-prompt value estimation, providing evidence that the method improves the critic itself rather than only downstream policy performance.

Critical weaknesses
  1. The paper does not provide an end-to-end baseline that combines a standard MSE or BCE critic with the same smoothed leave-one-out prior used during GAVE training. Table 3 provides a related comparison, but it uses a shared prior estimated from independent rollouts and evaluates value prediction under a frozen policy. An online comparison would better isolate whether the policy gains come from GAVE’s critic objective rather than from incorporating group-level prior information.

  2. Groups in which all rollouts succeed, or all fail, provide no training signal for the GAVE critic. This may occur frequently for very easy or difficult prompts, especially early in sparse-reward training. The paper does not report how common these groups are or how they affect performance.

Questions for authors
  1. How does end-to-end PPO perform when an MSE or BCE critic is combined with the same smoothed leave-one-out prior used by GAVE?

  2. What fraction of rollout groups contain both successful and unsuccessful outcomes during training, and how does GAVE perform when most groups contain only failures?

1 comment
AuthorsOpenReview ↗

Response to Reviewer fkfM

W1 & Q1: Isolating the contribution of the group-conditional critic objective.

Response: Thank you for suggesting this comparison. We agree that the frozen-policy evaluation in Table 3 does not establish how much the critic objective contributes to downstream policy improvement. We have therefore added end-to-end PPO comparisons on ALFWorld with Qwen3-4B:

  • MSE/BCE Critic + Group Prior: Train conventional critics with their original MSE or BCE objectives and terminal-return targets, and apply the correction in Eq. (18) for PPO advantage estimation. The correction uses the same smoothed leave-one-out prior estimator as GAVE and changes only the values supplied to the actor.
  • Prior Only: Use the smoothed leave-one-out prior as the value estimate at every nonterminal state, without training a critic.

In these online comparisons, the prior is computed from sibling rollouts within each training group, with G=8G=8 and α=0.5\alpha=0.5. All comparisons use the same base model, rollout budget, policy-update hyperparameters, and evaluation protocol.

Table R1. ALFWorld success rates with Qwen3-4B under different critic formulations.

MethodALFWorld success (%)
PPO (MSE)63.9 ± 1.9
Prior Only64.7 ± 1.9
MSE Critic + Group Prior70.4 ± 1.5
BCE Critic + Group Prior69.9 ± 4.3
GAVE88.6 ± 1.4

Prior Only performs similarly to standard PPO. The MSE and BCE critics with prior correction achieve 70.4% and 69.9% success, respectively, while GAVE reaches 88.6%, exceeding them by 18.2 and 18.7 percentage points. These comparisons support the benefit of the group-conditional critic objective beyond incorporating the empirical prompt prior during policy optimization.

We have added these results to Section 4.4, Table 4, and the baseline definitions to Appendix E.1. These additions are highlighted in blue in the revised paper.


W2 & Q2: Homogeneous groups and evidence-learning signals.

Response: We agree that all-failure and all-success groups provide no gradient for GAVE's evidence critic. However, their outcomes still inform the smoothed prompt prior, and these groups remain in PPO actor updates. Since reconstructed values also depend on the history through the learned evidence term, the absence of a critic gradient does not imply identical advantages or zero policy updates.

To quantify how frequently these groups occur, we analyzed rollout groups on ALFWorld and mathematical reasoning with the 4B models, dividing training into three equal stages.

Table R2. Rollout group composition during GAVE training with the 4B models.

TaskTraining stageAll-failure (%)Mixed (%)All-success (%)
ALFWorldEarly32.042.925.1
Middle12.038.849.3
Late6.324.569.3
Overall16.835.447.9
MathEarly43.452.54.1
Middle33.355.611.1
Late29.454.616.0
Overall35.454.310.4

Mixed groups account for 35.4% of ALFWorld groups and 54.3% of Math groups overall. They remain present throughout training, providing direct supervision for the shared evidence critic at all three stages.

To examine how GAVE performs when most groups contain only failures, we analyzed the first 20 steps of Math training. During this period, all-failure groups accounted for 59.5% of rollout groups and formed a majority in 19 of the 20 steps. Despite this, the mean on-policy rollout reward increased from 0.128 in steps 1–10 to 0.171 in steps 11–20. Meanwhile, 40.3% of groups remained mixed, providing supervision for the evidence critic.

While mixed groups provide the training signal for the evidence critic, homogeneous groups can still contribute to PPO actor updates. We tested their contribution by masking groups with K=0K=0 or K=GK=G from actor updates on mathematical reasoning, leaving the critic objective unchanged. Average accuracy decreased from 46.4% to 45.3%, suggesting that retaining these groups is beneficial for policy learning.

We have added the stagewise group-composition statistics and masking experiment to Appendix E.3 (Tables 11 and 12). These additions are highlighted in blue in the revised paper.

Reviewer_A4hPOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Group-Anchored Value Estimation (GAVE) for critic learning in LLM reinforcement learning with grouped rollouts and binary terminal rewards. The method decomposes state-value log-odds into a prompt-level success prior and history-dependent likelihood-ratio evidence. Conditioning on the number of successful rollouts yields a critic objective that removes the unknown prompt prior. During policy optimization, the learned evidence is combined with a smoothed leave-one-out empirical prior to reconstruct state values for PPO.

The author(s) prove the group-conditional identity and an expected absolute-error bound for value reconstruction. Experiments cover ALFWorld, WebShop, search-augmented QA, and mathematical reasoning. A separate frozen-policy experiment evaluates critic predictions against Monte Carlo continuation estimates.

In my opinion, the proposed decomposition is coherent, and the empirical results look promising. The principal theoretical arguments appear sound under their stated fixed-policy and sampling assumptions. My concerns are the incomplete attribution of downstream policy gains, the treatment of homogeneous rollout groups, and uncertainty in the critic-evaluation results.

Critical strengths
  1. A principled and implementable training objective: The method uses group information to change what the critic learns, rather than only changing its baseline at evaluation. Proposition 1 provides a clear justification for the group-dependent logit offset. Including the focal rollout's outcome in the training count is justified by the conditional prediction problem; reconstruction appropriately uses leave-one-out outcomes.

  2. A useful reconstruction guarantee: Proposition 2 establishes an expected contraction property for prior error when the history evidence is exact, together with a simple bound for evidence approximation error. The proof is more informative than a pointwise derivative bound, which could be large for particular states. The finite-sample corollary explicitly accounts for the bias introduced by smoothing.

  3. Broad downstream evaluation: The reported improvements extend across interactive agent tasks and mathematical reasoning, with two model sizes. The mathematical-reasoning results use three independent training runs and a fixed reporting checkpoint.

Critical weaknesses
  1. The downstream experiments do not fully isolate the contribution of the group-conditional critic objective: GAVE changes both critic training and the values supplied to the actor: it learns history evidence and introduces an empirical leave-one-out prompt prior. The frozen-policy experiments distinguish these ingredients to some extent, but the policy-optimization tables do not include the corresponding information-matched controls.

In my opinion, the most informative additions would be: (a) PPO using only the smoothed leave-one-out prior; (b) PPO using the existing MSE/BCE Critic + Group Prior reconstruction; (c) a critic trained with the empirical prior explicitly provided as an input or offset.

These controls would determine whether the policy gains require the proposed evidence-learning objective or can largely be obtained through group-based calibration of a conventional critic.

  1. Homogeneous groups provide no evidence-learning signal, and this limitation is insufficiently characterized: Under the exact endpoint convention, groups with K=0K=0 or K=GK=G have constant conditional predictions and contribute zero loss and zero critic gradient. For prompt success probability pp, the probability of a mixed group is P(0<K<G)=1−pG−(1−p)G.P(0<K<G)=1-p^G-(1-p)^G. For example, with G=8G=8 and p=0.01p=0.01, only approximately 7.7% of groups are informative for this objective.

This creates a potentially important limitation during early training on difficult tasks and later training on nearly solved tasks. Although the empirical prior still provides information, the evidence critic receives no direct update from homogeneous groups.

Questions for authors
  1. How much of the policy improvement remains relative to PPO using only the leave-one-out prior and PPO using a conventional critic with the same prior correction?

  2. Does the critic-evaluation advantage persist when using the algorithm's smoothed prior, and how sensitive is performance to α\alpha?

3 comments
AuthorsOpenReview ↗

Response to Reviewer A4hP (Part 1/2)

W1 & Q1: Isolating the contribution of the group-conditional critic objective.

Response: Thank you for suggesting these controls. We agree that the frozen-policy comparisons in Table 3 do not establish how much the critic objective contributes to downstream policy improvement. We have therefore added online PPO comparisons on ALFWorld with Qwen3-4B, covering the three types of controls you suggested:

  • (a) Prior Only: Use the smoothed leave-one-out prior as the value estimate at every nonterminal state, without training a critic.
  • (b) MSE/BCE Critic + Group Prior: Train conventional critics with their original MSE or BCE objectives, and apply the prior correction in Eq. (18) for PPO advantage estimation, using the same smoothed leave-one-out prior estimator as GAVE.
  • (c) Smoothed-Prior Offset: We explicitly provide the critic with the empirical prompt-level success prior, using its smoothed leave-one-out estimate p~−i(x)\widetilde p_{-i}(x) as a fixed logit offset during training:

Loffset(ψ)=E[ℓBCE(uψ(si,t)+logit⁡p~−i(x),Ri)].\mathcal{L}_{\mathrm{offset}}(\psi) = \mathbb{E}\left[\ell_{\mathrm{BCE}}\left(u_\psi(s_{i,t}) + \operatorname{logit}\widetilde p_{-i}(x), R_i\right)\right].

Here, ℓBCE\ell_{\mathrm{BCE}} denotes BCE with logits. We use the same value reconstruction as GAVE for PPO advantage estimation, isolating the effect of the critic training objective.

All comparisons use the same base model, rollout budget, policy-update hyperparameters, and evaluation protocol.

Table R1. ALFWorld success rates with Qwen3-4B under different critic formulations.

MethodALFWorld success (%)
PPO (MSE)63.9 ± 1.9
(a) Prior Only64.7 ± 1.9
(b) MSE Critic + Group Prior70.4 ± 1.5
(b) BCE Critic + Group Prior69.9 ± 4.3
(c) Smoothed-Prior Offset78.1 ± 1.9
GAVE88.6 ± 1.4

GAVE exceeds Prior Only and the strongest prior-corrected conventional critic by 23.9 and 18.2 percentage points, respectively. It also outperforms Smoothed-Prior Offset by 10.5 points. Since the offset baseline uses the same prior estimator and value reconstruction, this comparison supports the benefit of the group-conditional training objective beyond incorporating prompt-level prior information.

We have added these results to Section 4.4, Table 4, and the baseline definitions to Appendix E.1. These additions are highlighted in blue in the revised paper.


W2: Homogeneous groups and evidence-learning signals.

Response: We agree that homogeneous groups provide no gradient for GAVE's evidence critic. When all rollouts succeed or fail, their outcomes provide information about the overall prompt difficulty, but no within-group success–failure contrast for learning the evidence term. This is consistent with GAVE's decomposition: the prompt-level prior captures task difficulty, while the evidence critic learns from within-prompt differences. Although these groups provide no training signal for the evidence critic, they can still contribute to policy learning through advantage estimation and standard PPO updates.

To examine how frequently these cases arise during training, we analyzed the composition of rollout groups on ALFWorld and mathematical reasoning with the 4B models, dividing training into three equal stages.

Table R2. Rollout group composition during GAVE training with the 4B models.

TaskTraining stageAll-failure (%)Mixed (%)All-success (%)
ALFWorldEarly32.042.925.1
Middle12.038.849.3
Late6.324.569.3
Overall16.835.447.9
MathEarly43.452.54.1
Middle33.355.611.1
Late29.454.616.0
Overall35.454.310.4

On ALFWorld, the proportion of all-failure groups decreases from 32.0% to 6.3%, while all-success groups increase from 25.1% to 69.3%. Mixed groups remain present in all three stages, accounting for 35.4% of ALFWorld groups and 54.3% of Math groups overall. These groups provide training signals for the shared evidence critic, while homogeneous groups still contribute to the smoothed prior used for value reconstruction and remain in PPO updates.

We also examined their contribution to policy learning by masking homogeneous groups from actor updates on mathematical reasoning. This reduces average accuracy from 46.4% to 45.3%, suggesting that these groups still benefit policy optimization despite providing no gradient for the evidence critic.

We have added this discussion and these analyses to Appendix E.3 (Tables 11 and 12). These additions are highlighted in blue in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer A4hP (Part 2/2)

Q2: Smoothed-prior evaluation and sensitivity to the smoothing coefficient.

Response: Yes. GAVE retains its advantage in critic evaluation when using the smoothed leave-one-out prior. To examine sensitivity to the smoothing coefficient, we evaluated GAVE and the MSE/BCE Critic + Group Prior baselines across different values of α\alpha.

Table R3. Critic MAE under different prior-smoothing coefficients on ALFWorld.

Methodα=0\alpha=0α=0.1\alpha=0.1α=0.25\alpha=0.25α=0.5\alpha=0.5α=1.0\alpha=1.0α=2.0\alpha=2.0
MSE Critic + Group Prior0.15460.15800.16190.16650.17230.1789
BCE Critic + Group Prior0.16640.16890.17210.17620.18210.1897
GAVE0.11960.12010.12100.12280.12650.1334

At the default α=0.5\alpha=0.5, GAVE achieves an MAE of 0.1228, compared with 0.1665 for MSE Critic + Group Prior and 0.1762 for BCE Critic + Group Prior. GAVE achieves the lowest MAE across all tested values, although stronger smoothing increases prediction error for all three methods.

We also examined sensitivity during online PPO training on ALFWorld with Qwen3-4B, comparing α=0.25\alpha=0.25 and α=1.0\alpha=1.0 with our default α=0.5\alpha=0.5.

Table R4. ALFWorld success rates for GAVE with Qwen3-4B under different prior-smoothing coefficients.

Smoothing coefficientALFWorld success (%)
α=0.25\alpha=0.2586.3 ± 1.6
α=0.5\alpha=0.5 (default)88.6 ± 1.4
α=1.0\alpha=1.088.5 ± 0.7

Across the three tested values of α\alpha, GAVE's ALFWorld success rate ranges from 86.3% to 88.6%, with similar performance at α=0.5\alpha=0.5 and α=1.0\alpha=1.0.

We have added the online sensitivity results to Appendix E.4 (Table 14). These additions are highlighted in blue in the revised paper.

Thank you for answering all my questions. I increased my rating to 4.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Summary of Revisions

We thank the reviewers and the Area Chair for their careful reading and constructive feedback. We have revised the manuscript and conducted additional experiments in response to their comments. The main changes are summarized below and highlighted in blue in the revised paper.

  1. Contribution of the group-conditional critic objective (A4hP, fkfM, bddg). We conducted additional online PPO experiments on ALFWorld, including Prior Only, MSE/BCE Critic + Group Prior, Smoothed-Prior Offset, and RLOO variants. GAVE achieves 88.6% success, compared with 78.1% for the strongest control, indicating that the empirical prompt prior alone does not account for GAVE's improvement (Section 4.4, Table 4; Appendix E.1).

  2. Additional baselines and training cost (bddg). We expanded the mathematical reasoning comparisons to include RLOO, DAPO, VinePPO, VC-PPO, and VAPO, with GAVE achieving the highest average accuracy. We also doubled GRPO's rollout group size from 8 to 16 and measured end-to-end training time and peak GPU memory. Even with more rollouts and longer training time, GRPO (G=16G=16) achieves 45.1% average accuracy, compared with 46.4% for GAVE (Section 4.2, Table 2; Appendix E.2).

  3. Homogeneous rollout groups (A4hP, fkfM, bddg). We report the proportions of all-failure, mixed, and all-success groups throughout training on ALFWorld and mathematical reasoning, including the early stage of Math training when all-failure groups dominate. We further tested the effect of excluding homogeneous groups from PPO actor updates. Average Math accuracy drops from 46.4% to 45.3%, suggesting that these groups remain useful for policy learning despite providing no direct gradient for the evidence critic (Appendix E.3).

  4. Advantage estimation and hyperparameter sensitivity (A4hP, bddg). We clarified that GAVE provides state-dependent advantages when γ=λ=1\gamma=\lambda=1, even though the critic is not used for bootstrapping in this setting. Additional ALFWorld experiments with λ=0.9\lambda=0.9 and 0.950.95 confirm that GAVE's advantage over PPO persists when λ<1\lambda<1. We also examined sensitivity to the prior-smoothing parameter α\alpha (Appendix E.4).

  5. Critic evaluation across policy checkpoints (bddg). We extended the critic evaluation to later ALFWorld policy checkpoints and to mathematical reasoning. GAVE consistently achieves the lowest value-estimation MAE across all evaluated checkpoints on both tasks (Appendix C.4, Table 8).

Detailed responses to each reviewer are provided in the corresponding discussion threads. We welcome any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.