- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses credit assignment in agentic RL using a frozen LLM as the critic model in two training settings: LVRL and LVOPD. In the LVRL setting, consecutive state-value differences predicted by the critic serve as step-level advantages, while anchoring the initial value to group success rate and the terminal value to actual reward to ensure that these advantages sum to the trajectory’s group-relative outcome. In the LVOPD setting, the same advantages selectively gate teacher distillation, allowing useful student behavior to differ from the teacher. Across AppWorld, BabyAI, and SWE-bench Verified, LVRL outperforms GRPO and PPO. LVOPD improves AppWorld results compared to two distillation baselines. Further Monte-Carlo analysis show agreement with the critic's predicted directions.
Critical strengths
Eeffective learning under sparse-reward cold starts. By drawing on a frozen LLM’s semantic knowledge, LVRL produces nonzero step-level advantages even when all rollouts fail, allowing useful intermediate actions to be reinforced while preserving the trajectory-level advantage sum.
Critical weaknesses
- The experiments do not isolate the proposed innovations from stronger supervision by the critic model.
- Lack of a baseline using the same frozen model (e.g, opus-4.6) for simpler step scoring or progress estimation.
- The ablation in Table 3 removes peer context and anchoring together, leaving their individual contributions unresolved.
- To substantiate the claimed efficiency advantage, the paper should at least report total API costs (summarization costs included), GPU-hours, wall-clock training time, and the total cost required to reach the same performance level. Ideally, the baselines should also be allowed to spend an equivalent additional budget on more rollouts to ensure the same inference budgets between different methods.
- The paper should also include one or two representative baselines from the most closely related step-level credit assignment methods (e.g, [1-2]) that are suitable for these environments. Outperforming GRPO/PPO alone is insufficient to fully establish the method’s relative advantages.
- The experiments of LVOPD does not consider the changes to the training objective. Comparison with Pure OPD adds an RL objective, while comparison with trajectory-gated OPD changes both the RL advantages and the distillation gate. Thus, a fixed LVRL objective with uniform, trajectory-level, proposed step-level, and random gates shoud be compared.
References:
- Feng, Lang, et al. "Group-in-group policy optimization for llm agent training." Advances in Neural Information Processing Systems 38 (2026): 46375-46408.
- Liu, Xiaoqian, et al. "Agentic reinforcement learning with implicit step rewards." International Conference on Learning Representations. 2026.
Questions for authors
See weaknesses above.
7 comments
Rebuttal (Part 2)
Q2: Separate ablations of group-conditioned calibration and endpoint anchoring
The ablation in Table 3 removes peer context and anchoring together, leaving their individual contributions unresolved.
A2:
We ablated each component separately on BabyAI (Qwen3-8B). We chose BabyAI because the frozen critic's signal is essential here to overcome the cold-start bottleneck where GRPO completely fails (reaching only 2.7% ID / 0.0% OOD). We only run each ablation once due to limited time.
| Variant | ID | OOD |
|---|---|---|
| Full LVRL | 75.0 ± 2.0 | 65.5 ± 3.0 |
| w/o peer rollouts (keeps and outcome anchors; drops peer summaries) | 67.2 | 45.8 |
| w/o anchoring (keeps peer summaries; drops and outcome anchors) | 71.2 | 60.4 |
- Policy-conditioned peer context matters most for generalization: Removing peer rollouts causes a severe drop in out-of-distribution transfer (down by 7.8% ID and 19.7% OOD), showing that in-context calibration to the current policy's behavior is critical when navigating unseen layouts.
- Endpoint anchoring ensures stable credit assignment: Removing the and terminal outcome anchors also degrades performance (down by 3.8% ID and 5.1% OOD), as unanchored values introduce unconstrained return drift across episodes.
We will expand Table 3 in the revised manuscript to report these separate ablations across environments.
Rebuttal (Part 3)
Q3: Cost and efficency
To substantiate the claimed efficiency advantage, the paper should at least report total API costs (summarization costs included), GPU-hours, wall-clock training time, and the total cost required to reach the same performance level. Ideally, the baselines should also be allowed to spend an equivalent additional budget on more rollouts to ensure the same inference budgets between different methods.
A3:
- API cost (summarization included): about $0.12 per AppWorld trajectory (Table 6, Appendix B.1), i.e. ~$4.0k per 500-update run (32,000 trajectories), and about $0.21 per BabyAI trajectory, i.e. ~$2.1k per 150-update run (9,600 trajectories), at Claude Opus 4.6 list prices with no caching or batch discounts.
- GPU-hours and wall-clock (measured on our rebuttal reruns, B200):
| Setting | Hardware | Min / update | Updates | Wall-clock | GPU-hours |
|---|---|---|---|---|---|
| AppWorld, Qwen3-8B | 1 node × 8 GPUs | 12.5–14.7 | 500 | 104–122 h | 830–980 |
| BabyAI, Qwen3-8B | 2 nodes × 8 GPUs | ~31 | 150 | ~78 h | ~1,250 |
Rollout generation and policy updates dominate each iteration and are identical between GRPO and LVRL (PPO additionally trains a critic head). LVRL adds no GPU compute; its only overhead is the parallelized critic phase after each rollout: 2 min of a 30-min BabyAI update (≈7%) without rate-limiting, rising to 6 min (≈20%) only when a shared API key was throttled.
- Cost to reach the same performance (Fig. 9, training reward):
- AppWorld (Qwen3-4B): GRPO ends at 0.68 after 500 updates. LVRL passes that level after 100–120 updates—using 4× less GPU time (critic overhead included) at $0.8–1.0k in undiscounted API cost.
- BabyAI (Qwen3-4B): LVRL passes 0.9 by update ~45, while GRPO and PPO stay near 0 for ~140 of the 150 updates and never reach that level within the budget.
- Giving GRPO an equal budget of extra rollouts:
- Fig. 9 already shows the rollout-budget axis: GRPO spends 4–5× more rollouts to reach the training reward LVRL reaches in ~100 updates.
- On AppWorld, GRPO's training reward is nearly flat over its last ~100 updates (Fig. 9), so additional rollouts at the end yield diminishing returns.
- Finally, the cost is not tied to Claude: with open-weights GLM-5.2 as the critic, LVRL still reaches 22.3 ± 4.1% vs. GRPO's 10.7 ± 2.8% on Qwen3-4B (Tables 1 and 4).
We will add per-method GPU-hours, wall-clock time, and total API cost for all runs to the appendix.
Rebuttal (Part 1)
Q1: LLM-as-a-Judge baseline with the same frozen model
A1:
We added this baseline on AppWorld (Qwen3-8B, Test-Challenge), with everything except the credit mechanism identical to LVRL: same Claude Opus 4.6, same rollout and update budget, and same evaluation protocol.
†1 seed; ± is the std over 3 evaluation runs.
As shown above, the same frontier model, at the same budget, gives no gain over GRPO when used as a conventional step judge. Thus, stronger supervision alone does not explain LVRL's gains. Training logs point to two failure modes that align with the components:
Finally, we would like to emphasize that LVRL's gains are not tied to one proprietary critic: on AppWorld (Qwen3-4B, Table 4), replacing Opus 4.6 with GLM-5.2 still reaches 22.3% (vs. 10.7% GRPO in Table 1 and 24.3% Opus 4.6 LVRL).