- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper notes that in modern LLM training, the generalized advantage estimator is typically used with , so it effectively becomes baseline subtraction. Since baseline subtraction doesn't change the expected gradient, it doesn't need to be accurate. The authors therefore argue that small critics can maintain performance while potentially saving computation. In particular, they propose to use a small MDP on top of the actor feature. From the experimental results, it seems that a warm start is also not needed (and, in some cases, it seems to drop performance). Their experiments use two agentic benchmarks, WebShop (a product-search and shopping environment) and the Berkeley Function Calling Leaderboard (BFCL) multi-turn tool-use benchmark, plus a transfer evaluation on -bench retail.
Critical strengths
-
The proposed idea is very simple and makes total sense. I agree that the critic effectively acts as a baseline subtraction, and I am not surprised that using a small critic doesn't hurt performance.
-
The authors perform several different tests to investigate their algorithmic choice, looking at the explained variance of the critic, performance of full-size vs. probe cold/warm start, synchronous/asynchronous training, and a fixed critic.
-
The authors also test a fixed critic (if I understand it correctly, trained at the beginning and never trained again), showing that there is no significant drop in performance.
Critical weaknesses
While the authors correctly compare their method with GRPO (which is critic-free), that comparison is a bit like comparing apples and oranges if you want to understand the critic's impact alone. I think that the paper's claims raise the questions: how small can a critic be? And can we entirely remove it?
Thus, the following comparisons are missing:
-
Size vs. performance. (Table 1 reports explained variance, not performance).
-
Performance of PPO without a critic.
Questions for authors
- I am not sure I really understood how the model selection worked (i.e., section 2.4)
3 comments
Thanks for your reply.
Dear authors, I've read your reply carefully. I will update my review accordingly and consider your changes to the paper during the reviewers-AC discussion.
Thank you for reading our response carefully and for considering the changes in the discussion. We appreciate your engagement throughout the review process!
We thank the reviewer for the positive assessment and the clear framing of the work's weaknesses. We added both comparisons, made the manuscript revisions in blue, and discuss the weaknesses below.
Summary of updates for LB1N:
Comparison with GRPO. The GRPO comparison establishes that the probe critic reaches performance comparable to the standard critic-free alternative at a matched episode budget, as Hou et al. (2026) do for their critic. The constant-baseline run below isolates the critic's effect within PPO, holding everything else fixed.
In the paper: §3, "Critics" paragraph; Table 2.
PPO without a critic. At λ≈1 the critic enters the advantage as a baseline, which lowers the variance of the policy gradient without changing its expectation. Removing the critic should therefore slow learning while leaving final success unchanged. We tested this with PPO using a constant baseline V≡μ on Qwen3-4B WebShop, three seeds, in the same pipeline. Final test success is similar (76.9%, App. D), and the fixed probe, warm-start probe and policy-sized critic lead the constant baseline by 6.9 points of validation success over the first 50 updates (p=2.3×10−2, Figure 6). Each update costs a full batch of multi-turn rollouts, so fewer updates save the most expensive part of agentic RL. Even a fixed probe, never updated during training, matches the policy-sized critic's sample efficiency. Hou et al. find the same early lag for a running-mean baseline, which in their asynchronous, long-horizon setting also ends below the critic.
In the paper: new App. D (Figure 6, closing paragraph on prior work); new Figure 2 and its paragraph in §4.1.
How small a critic can be. In task success, size matters little at λ≈1. On Qwen3-4B WebShop, PPO with a constant baseline, the 166K probe and the 3–4B policy-sized critic reach similar test success (App. D, Table 2). Critics matter for sample efficiency, as shown above, and for value accuracy. For value accuracy, the deployed [64,32] probe, with two nonlinear layers, is the smallest probe on the plateau of held-out explained variance (App. C, Table 4). Both a linear readout and a probe with a single nonlinear layer and as many parameters score below the deployed architecture over WebShop policy layers and seeds (sign test, p=2.9×10−11), so adding parameters to one layer does not recover what the second layer provides. Larger two-layer probes add little held-out accuracy (Table 1).
In the paper: §2.4, after the choice of [64,32]; App. C, last paragraph and Table 4.
§2.4 model selection. §2.4 now opens by stating selection as three steps. First, we roll out the frozen initial policy on the training tasks. Second, we train the candidate probes on a subset of the rollouts and evaluate generalization to a validation set consisting of unseen rollouts on the same tasks, as in standard actor-critic setups. Third, we deploy the smallest architecture whose held-out explained variance has levelled off, which is [64,32] in all four settings (Table 1).