- Rating
- 3(+1)
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes GAVE, a PPO critic that exploits grouped rollouts. For binary terminal rewards, it decomposes value log-odds into a prompt-level success prior, estimated from sibling rollouts, and a history-dependent evidence term, learned by the critic via a group-count-offset BCE objective. Experiments on agent tasks and math with Qwen3-4B/8B show gains over PPO, GRPO, and GiGPO, as well as more accurate value estimates than standard critics.
Critical strengths
- Principled, simple idea: the Bayesian decomposition removes the prompt prior from the critic's target via a group-count offset (Prop. 1). The method is a drop-in change to the critic loss and value reconstruction within the standard PPO pipeline.
- Direct critic-level evidence: on a frozen-policy ALFWorld evaluation against Monte Carlo reference values, GAVE achieves lower overall and within-prompt error than information-matched baselines, and the gains are broadly distributed across prefixes.
Critical weaknesses
-
Credit-assignment claim is not supported by the actual setup. The paper motivates GAVE as improving credit assignment over intermediate decisions. However, the agent experiments use γ=λ=1, so GAE reduces to with no bootstrapping. In this regime, the critic acts only as a state-dependent baseline for variance reduction; it does not propagate value information across steps. This matters because it changes what the experiments actually demonstrate: better baselining, not better credit assignment. The authors should report agent-task results with λ<1 (e.g., 0.9/0.95) for both PPO and GAVE, and show whether GAVE's advantage persists or grows when the critic is used for bootstrapping.
-
The source of the policy gains is not isolated. With λ=1, GAVE's advantage at the first step is essentially , i.e., an RLOO-style leave-one-out baseline. The information-matched baselines (MSE/BCE critic + group prior) appear only in the frozen-policy critic evaluation (Table 3), not in RL training (Tables 1–2). The reported gains over PPO, GRPO, and GiGPO may therefore come largely from the leave-one-out prior rather than from the group-conditional evidence critic, which is the paper's main contribution. This is the main reason for my recommendation. In the Table 1/2 settings, the authors should add RL-training ablations: (i) a standard PPO critic combined with the same leave-one-out prior during training (Eq. 18 applied online); (ii) prior only, i.e., GAVE with ; (iii) RLOO, and RLOO plus a standard critic baseline.
-
The baselines are incomplete and not compute-matched. Math compares only against PPO and GRPO. It omits VinePPO, the most direct competitor for value estimation in LLM RL, as well as VC-PPO/VAPO (value pretraining, decoupled GAE), RLOO, and DAPO. GAVE also trains a full Qwen3-4B critic, while GRPO is critic-free, yet no wall-clock time, GPU memory, or compute-matched comparison is reported. The authors should add these baselines and report training cost, ideally with a compute-matched GRPO variant (e.g., more rollouts per prompt).
Questions for authors
-
What fraction of training groups are uninformative, i.e., or , especially on math? For these groups the critic receives no gradient, and with the smoothed prior (e.g., when , ), all trajectories get small, uniformly negative advantages. GRPO assigns zero advantage in this case. How does this behavior affect training, and what happens if such groups are masked?
-
How accurate is the critic beyond the single ALFWorld step-50 checkpoint? Can the authors report value-estimation error against Monte Carlo references at later policy checkpoints and on math (e.g., segment-level prefixes), where the evidence term is non-stationary and critic learning is hardest?
4 comments
Response to Reviewer bddg (Part 2/3)
W3: Additional baselines and training cost.
Response: Thank you for highlighting the missing baseline comparisons and the need to report training cost. We have added RLOO, DAPO, VinePPO, VC-PPO, and VAPO to our mathematical reasoning experiments with Qwen3-4B-Base. All methods use the same base model, training dataset, and evaluation protocol.
To examine whether additional rollouts can close the accuracy gap to GAVE, we also train GRPO with , doubling its default group size of . Table R3 reports accuracy in percent as mean ± standard deviation over three seeds.
Table R3. Mathematical reasoning accuracy with Qwen3-4B-Base.
| Method | AIME24 | AIME25 | AMC23 | MATH-500 | Olympiad-Bench | Avg. |
|---|---|---|---|---|---|---|
| PPO | 12.6 ± 0.1 | 9.4 ± 0.8 | 46.7 ± 0.0 | 76.8 ± 0.1 | 38.5 ± 0.1 | 36.8 ± 0.1 |
| GRPO () | 21.6 ± 0.5 | 18.3 ± 1.4 | 54.5 ± 0.9 | 82.2 ± 0.2 | 45.3 ± 0.3 | 44.4 ± 0.3 |
| RLOO | 21.7 ± 1.6 | 18.2 ± 0.8 | 53.1 ± 1.0 | 81.5 ± 0.3 | 43.6 ± 0.4 | 43.6 ± 0.2 |
| DAPO | 23.3 ± 1.1 | 19.9 ± 1.3 | 53.3 ± 1.2 | 81.8 ± 0.1 | 45.5 ± 0.4 | 44.8 ± 0.4 |
| VinePPO | 11.2 ± 0.8 | 10.3 ± 0.9 | 47.9 ± 1.1 | 77.4 ± 0.2 | 38.8 ± 0.2 | 37.1 ± 0.1 |
| VC-PPO | 11.0 ± 1.1 | 11.1 ± 0.6 | 42.2 ± 0.3 | 75.1 ± 0.1 | 35.4 ± 0.2 | 34.9 ± 0.1 |
| VAPO | 12.2 ± 1.5 | 12.8 ± 2.2 | 46.0 ± 1.8 | 76.6 ± 0.1 | 38.2 ± 0.2 | 37.2 ± 0.8 |
| GRPO () | 22.6 ± 0.7 | 18.1 ± 1.2 | 55.5 ± 1.1 | 83.0 ± 0.3 | 46.2 ± 0.5 | 45.1 ± 0.3 |
| GAVE | 24.8 ± 0.4 | 19.2 ± 0.7 | 57.8 ± 0.0 | 83.1 ± 0.1 | 47.0 ± 0.2 | 46.4 ± 0.1 |
GAVE achieves the highest average accuracy at 46.4%, compared with 44.8% for DAPO, the strongest of the five additional baselines. Increasing GRPO's group size improves its average accuracy from 44.4% to 45.1%, while GAVE retains a higher average accuracy.
We also report end-to-end training time and peak GPU memory under the same hardware configuration. Training time includes rollout generation and model optimization.
Table R4. Training cost on mathematical reasoning with Qwen3-4B-Base.
| Method | Wall-clock time (h) | Peak GPU memory (GB) |
|---|---|---|
| PPO | 8.10 | 122.76 |
| GRPO () | 10.06 | 133.10 |
| GRPO () | 14.30 | 148.98 |
| GAVE | 12.45 | 120.46 |
GAVE takes longer to train than standard GRPO (): 12.45 vs. 10.06 hours. We therefore also compare against GRPO with a larger group size (). This variant requires more training time (14.30 vs. 12.45 hours) and peak GPU memory (148.98 vs. 120.46 GB) than GAVE, but achieves lower average accuracy (45.1% vs. 46.4%).
Although PPO trains an additional critic, its responses are much shorter on average than GRPO's (1,235 vs. 2,300 tokens), which helps explain its lower overall training cost.
We have added the five additional baselines to Section 4.2, Table 2, and the larger-group comparison and training costs to Appendix E.2 (Tables 9 and 10). These additions are highlighted in blue in the revised paper.
Response to Reviewer bddg (Part 3/3)
Q1: Homogeneous rollout groups and policy updates.
Response: We agree that all-failure () and all-success () groups provide no gradient for GAVE's evidence critic. These groups still inform the smoothed prompt prior used for value reconstruction and remain in PPO actor updates.
With , the advantage is . For all-failure groups, these advantages are negative before normalization, but their magnitudes need not be identical: the reconstructed values also depend on the history through the learned evidence term. GAVE can therefore produce nonzero, state-dependent advantages for these groups, whereas GRPO assigns zero outcome-based advantages.
To examine how frequently homogeneous groups occur, we analyzed rollout group composition on ALFWorld and mathematical reasoning with the 4B models, dividing training into three equal stages.
Table R5. Rollout group composition during GAVE training with the 4B models.
| Task | Training stage | All-failure (%) | Mixed (%) | All-success (%) |
|---|---|---|---|---|
| ALFWorld | Early | 32.0 | 42.9 | 25.1 |
| Middle | 12.0 | 38.8 | 49.3 | |
| Late | 6.3 | 24.5 | 69.3 | |
| Overall | 16.8 | 35.4 | 47.9 | |
| Math | Early | 43.4 | 52.5 | 4.1 |
| Middle | 33.3 | 55.6 | 11.1 | |
| Late | 29.4 | 54.6 | 16.0 | |
| Overall | 35.4 | 54.3 | 10.4 |
On mathematical reasoning, mixed groups account for more than half of the groups in each training stage and 54.3% overall. All-failure groups decrease from 43.4% to 29.4%, while all-success groups increase from 4.1% to 16.0%.
To assess the effect of homogeneous groups on policy learning, we also trained GAVE on mathematical reasoning with Qwen3-4B-Base while masking groups with or from PPO actor updates.
Table R6. Mathematical reasoning accuracy after masking homogeneous groups from actor updates.
| Method | AIME24 | AIME25 | AMC23 | MATH-500 | Olympiad-Bench | Avg. |
|---|---|---|---|---|---|---|
| GAVE | 24.8 ± 0.4 | 19.2 ± 0.7 | 57.8 ± 0.0 | 83.1 ± 0.1 | 47.0 ± 0.2 | 46.4 ± 0.1 |
| GAVE (masking) | 21.0 ± 1.6 | 20.4 ± 0.7 | 56.2 ± 0.2 | 83.0 ± 0.2 | 46.0 ± 0.4 | 45.3 ± 0.3 |
Masking homogeneous groups reduces average accuracy from 46.4% to 45.3%. This suggests that retaining these groups in actor updates benefits policy learning in this setting, despite their providing no gradient for the evidence critic.
We have added these analyses and clarifications to Appendix E.3 (Tables 11 and 12). These additions are highlighted in blue in the revised paper.
Q2: Critic accuracy at later policy checkpoints and on mathematical reasoning.
Response: We have extended the critic evaluation in Table 3 to later ALFWorld checkpoints (steps 100 and 150) and to mathematical reasoning (steps 50, 100, 150, and 200). At each checkpoint, we freeze the policy and estimate Monte Carlo reference values from 64 independent continuations per prefix.
Table R7. Critic MAE across ALFWorld checkpoints with Qwen3-4B.
| Method | Step 50 | Step 100 | Step 150 |
|---|---|---|---|
| Empirical Prior Only | 0.250 | 0.171 | 0.201 |
| MSE Critic + Group Prior | 0.158 | 0.166 | 0.175 |
| BCE Critic + Group Prior | 0.169 | 0.181 | 0.179 |
| GAVE | 0.116 | 0.120 | 0.147 |
For mathematical reasoning, we evaluate four prefixes per trajectory, at 0%, 25%, 50%, and 75% of its length.
Table R8. Critic MAE across mathematical reasoning checkpoints with Qwen3-4B-Base.
| Method | Step 50 | Step 100 | Step 150 | Step 200 |
|---|---|---|---|---|
| Empirical Prior Only | 0.314 | 0.371 | 0.375 | 0.362 |
| MSE Critic + Group Prior | 0.277 | 0.351 | 0.347 | 0.338 |
| BCE Critic + Group Prior | 0.264 | 0.334 | 0.311 | 0.329 |
| GAVE | 0.211 | 0.273 | 0.265 | 0.252 |
GAVE achieves the lowest MAE at every evaluated checkpoint on both tasks. These results extend the value-estimation gains beyond the original ALFWorld step-50 evaluation to later policy checkpoints and mathematical reasoning.
We have added these results to Appendix C.4, Table 8. These additions are highlighted in blue in the revised paper.
Official Comments from Reviewer bddg
Thanks for providing a detailed analysis of my concerns, as well as the additional baselines. I am happy to raise my score to 3. Good luck.
Response to Reviewer bddg (Part 1/3)
W1: Credit assignment and bootstrapped advantage estimation.
Response: We agree that with γ=λ=1 and terminal-only rewards, GAE reduces to A^t=R−V(ht), without bootstrapping. However, this does not mean that all steps in a trajectory receive the same advantage. For two consecutive steps,
A^t−A^t+1=V(ht+1)−V(ht).
While GRPO assigns the same outcome-based advantage to every step in a trajectory, GAVE uses history-dependent value estimates, allowing its advantages to vary across states. These state-dependent advantages are available even at λ=1, although GAE does not use bootstrapping in this setting.
To further evaluate GAVE with bootstrapping, we also trained PPO and GAVE with λ=0.9 and 0.95, keeping all other settings unchanged.
Table R1. ALFWorld success rates with Qwen3-4B under different GAE parameters.
GAVE continues to substantially outperform PPO at both λ=0.9 and 0.95, showing that its advantage persists when bootstrapped value estimates are used in GAE.
We have clarified the distinction between state-dependent advantages and bootstrapping and added these results to Appendix E.4 (Table 13). These additions are highlighted in blue in the revised paper.
W2: Isolating the contribution of the group-conditional critic objective.
Response: We agree that the frozen-policy comparisons in Table 3 do not isolate the contribution of the critic objective during policy optimization. We have therefore added online comparisons on ALFWorld with Qwen3-4B, covering the controls you suggested:
Ai,tRLOO+C=Ri−Rˉ−i−(V(si,t)−V(si,0)).
Here, Rˉ−i is the mean reward of the other rollouts for the same prompt, and V is trained with standard MSE.
All methods use the same base model, rollout budget, and evaluation protocol. Policy-update settings are shared within the PPO variants and within the two RLOO variants.
Table R2. ALFWorld success rates with Qwen3-4B under different critic formulations.
Prior Only performs similarly to standard PPO. GAVE achieves 88.6% success, compared with 70.4% for the strongest prior-corrected critic and 71.5% for RLOO + Critic. These results support the benefit of the group-conditional critic objective beyond incorporating the leave-one-out prior.
We have added these results to Section 4.4, Table 4, and the baseline definitions to Appendix E.1. These additions are highlighted in blue in the revised paper.