acceptodds
Under review as a conference paper at ICLR 2027

A Probe Critic Matches a Policy-Sized Critic in Agentic PPO

Abstract

Proximal policy optimization (PPO) for language models typically uses a separate value network comparable in size to the policy, and agentic reinforcement learning has kept that design for trajectories that run to thousands of tokens across many turns. These policy-sized critics add considerable memory and compute and are pretrained before use. We show that a small probe trained on a language model agent's own activations can serve as its critic, at four orders of magnitude fewer parameters. We demonstrate that training the policy with such a probe nearly matches the performance of a critic the size of the policy, and holds that performance when the trajectories it learns from are several updates off-policy. We further find probe critics robust to how they are fit. A cold-start probe learns value alongside the policy, and a warm-start probe that never co-learns can still train the policy.

Primary area:
reinforcement learning
Keywords:
PPO, agents, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LB1NOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper notes that in modern LLM training, the generalized advantage estimator is typically used with λ=1\lambda=1, so it effectively becomes baseline subtraction. Since baseline subtraction doesn't change the expected gradient, it doesn't need to be accurate. The authors therefore argue that small critics can maintain performance while potentially saving computation. In particular, they propose to use a small MDP on top of the actor feature. From the experimental results, it seems that a warm start is also not needed (and, in some cases, it seems to drop performance). Their experiments use two agentic benchmarks, WebShop (a product-search and shopping environment) and the Berkeley Function Calling Leaderboard (BFCL) multi-turn tool-use benchmark, plus a transfer evaluation on τ2\tau^2-bench retail.

Critical strengths
  • The proposed idea is very simple and makes total sense. I agree that the critic effectively acts as a baseline subtraction, and I am not surprised that using a small critic doesn't hurt performance.

  • The authors perform several different tests to investigate their algorithmic choice, looking at the explained variance of the critic, performance of full-size vs. probe cold/warm start, synchronous/asynchronous training, and a fixed critic.

  • The authors also test a fixed critic (if I understand it correctly, trained at the beginning and never trained again), showing that there is no significant drop in performance.

Critical weaknesses

While the authors correctly compare their method with GRPO (which is critic-free), that comparison is a bit like comparing apples and oranges if you want to understand the critic's impact alone. I think that the paper's claims raise the questions: how small can a critic be? And can we entirely remove it?

Thus, the following comparisons are missing:

  1. Size vs. performance. (Table 1 reports explained variance, not performance).

  2. Performance of PPO without a critic.

Questions for authors
  • I am not sure I really understood how the model selection worked (i.e., section 2.4)
3 comments

We thank the reviewer for the positive assessment and the clear framing of the work's weaknesses. We added both comparisons, made the manuscript revisions in blue, and discuss the weaknesses below.

Summary of updates for LB1N:

  • Comparison with GRPO. Clarified (§3). The GRPO comparison shows the probe performs comparably to the standard critic-free alternative, as in Hou et al. The new constant-baseline run isolates the critic's effect within PPO.
  • PPO without a critic. New experiment (App. D, Figure 6). On Qwen3-4B WebShop a constant baseline reaches similar final success but learns more slowly. Critics fit before RL lead it by 6.9 points over the first 50 updates (p=2.3×10−2p = 2.3\times10^{-2}).
  • How small a critic can be. New experiment (§2.4, App. C). In task success size matters little at λ≈1\lambda\approx 1. For value accuracy the deployed two-layer probe is the smallest on the plateau, and a parameter-matched single-layer probe scores lower at every layer (p=2.9×10−11p = 2.9\times10^{-11}).
  • §2.4 model selection. Clarified (§2.4, first paragraph). Selection is now stated as three steps.

Comparison with GRPO. The GRPO comparison establishes that the probe critic reaches performance comparable to the standard critic-free alternative at a matched episode budget, as Hou et al. (2026) do for their critic. The constant-baseline run below isolates the critic's effect within PPO, holding everything else fixed.

In the paper: §3, "Critics" paragraph; Table 2.


PPO without a critic. At λ≈1\lambda\approx 1 the critic enters the advantage as a baseline, which lowers the variance of the policy gradient without changing its expectation. Removing the critic should therefore slow learning while leaving final success unchanged. We tested this with PPO using a constant baseline V≡μV\equiv\mu on Qwen3-4B WebShop, three seeds, in the same pipeline. Final test success is similar (76.9%, App. D), and the fixed probe, warm-start probe and policy-sized critic lead the constant baseline by 6.9 points of validation success over the first 50 updates (p=2.3×10−2p = 2.3\times10^{-2}, Figure 6). Each update costs a full batch of multi-turn rollouts, so fewer updates save the most expensive part of agentic RL. Even a fixed probe, never updated during training, matches the policy-sized critic's sample efficiency. Hou et al. find the same early lag for a running-mean baseline, which in their asynchronous, long-horizon setting also ends below the critic.

In the paper: new App. D (Figure 6, closing paragraph on prior work); new Figure 2 and its paragraph in §4.1.


How small a critic can be. In task success, size matters little at λ≈1\lambda\approx 1. On Qwen3-4B WebShop, PPO with a constant baseline, the 166K probe and the 3–4B policy-sized critic reach similar test success (App. D, Table 2). Critics matter for sample efficiency, as shown above, and for value accuracy. For value accuracy, the deployed [64,32] probe, with two nonlinear layers, is the smallest probe on the plateau of held-out explained variance (App. C, Table 4). Both a linear readout and a probe with a single nonlinear layer and as many parameters score below the deployed architecture over WebShop policy layers and seeds (sign test, p=2.9×10−11p = 2.9\times10^{-11}), so adding parameters to one layer does not recover what the second layer provides. Larger two-layer probes add little held-out accuracy (Table 1).

In the paper: §2.4, after the choice of [64,32]; App. C, last paragraph and Table 4.


§2.4 model selection. §2.4 now opens by stating selection as three steps. First, we roll out the frozen initial policy on the training tasks. Second, we train the candidate probes on a subset of the rollouts and evaluate generalization to a validation set consisting of unseen rollouts on the same tasks, as in standard actor-critic setups. Third, we deploy the smallest architecture whose held-out explained variance has levelled off, which is [64,32] in all four settings (Table 1).

Reviewer_LB1NOpenReview ↗

Thanks for your reply.

Dear authors, I've read your reply carefully. I will update my review accordingly and consider your changes to the paper during the reviewers-AC discussion.

Thank you for reading our response carefully and for considering the changes in the discussion. We appreciate your engagement throughout the review process!

Reviewer_KBGeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a simple probe critic for agentic PPO to replace the large policy-sized critic. They find that using policy's hidden representations and a small MLP critic can achieve comparable performance to a policy-sized critic. For experiments, they evaluate the method on 2 benchmarks using 2 kinds of LLM models, where their method show comparable results with other baselines while being more computational efficient. They also show further analysis of probe critic, such as comparisons of different probes and how probe uses info from hidden states, in which show cold-start or fixed critic also work well.

Critical strengths
  1. It proposes a simple and useful critic. Experiments show small probe critic can have comparable performance to policy-sized critic.
  2. Its motivation is interesting, which argues why a small critic might be enough.
Critical weaknesses
  1. The results don't show why the probe works very clearly. For example, I have several possible explanations: a. It could be that hidden states already contain more useful information; b. It could be the previous policy-sized critic is just overparameterized; c. It could be the critic barely matters when lambda close to 1, so any reasonable baseline could work well.
  2. The use case limited to lambda that is almost 1, which more like a design choice to me rather than a commonly used set up.
Questions for authors
  1. About why the probe works: Could you also run PPO with V=μ (just the running return mean) in the same pipeline? If that works, it might suggest the critic barely matters in this setting.
  2. The motivation is not very clear to me. How widely is this lambda setup in agentic PPO besides the literatures in paper?
  3. The introduction says the critic became a policy-sized value network in LLM PPO. If policy-sized critics are commonly used, what are the main reasons or advantages for others to use this setup, and can your probe critic still keep these advantages?
  4. Can you explain why sometimes policy-sized critic perform worse that linear readout in Table 1?
  5. The paper gives a brief explanation, but no ablation about why the probe critic uses the last layer's hidden states. Some prior works suggest intermediate layers could carry more task-relevant information that last layer.
3 comments

We thank the reviewer for the precise questions. We have made manuscript revisions in blue and discuss the weaknesses below.

Summary of updates for KBGe:

  • Q1 + W1: PPO with V=μV=\mu, and which explanation holds. New experiment (App. D, Figure 6). (c) sets the final success on Qwen3-4B WebShop, (a) determines sample efficiency, (b) follows from Table 1.
  • W2 + Q2: how common λ≈1\lambda\approx 1 is. Clarified (§2.2). Standard in large-scale PPO for language models, with new citations.
  • Q3: why policy-sized critics, and what the probe keeps. Answered (§2.1–2.2, App. G). The probe matches the policy-sized critic in task success, learning curves and under lag, and in value accuracy in three of four settings.
  • Q4: policy-sized critic below a linear readout. Clarified (§2.4, App. E). Decodability differs by model, and BFCL's 240 tasks bound the corpus for both critics.
  • Q5: intermediate layers. New experiment (§2.3, App. C). The final layer reaches 83% and 94% of the best layer.

Q1 + W1. Which explanation holds. All three hold, each for a different part of the result.

(c) At λ≈1\lambda\approx 1 the critic is a baseline, which changes the variance of the policy gradient but not its expectation. PPO with V≡μV\equiv\mu accordingly reaches final test success similar to the probe critics' (76.9% over three seeds on Qwen3-4B WebShop, App. D). Prior work agrees (Ahmadian et al., 2024; REINFORCE++), and Hou et al. find the same early lag.

(a) The hidden states do carry the information, since the return is decodable from them at every layer (App. C). That accuracy matters for sample efficiency, since lower gradient variance lets the policy reach a given success in fewer updates. Critics that are accurate from the first update gain validation success faster than V≡μV\equiv\mu, by 6.9 points on average over the first 50 updates of Qwen3-4B WebShop (p=2.3×10−2p = 2.3\times10^{-2}, Figure 6).

(b) The probe matches or exceeds the policy-sized critic's value accuracy in three of four settings, at 10−410^{-4} of its parameters and from the same rollouts (Table 1).

Beyond (c). Under asynchronous training, stale rollouts are clipped far more often, so the critic's values also decide which updates are kept. The probe still matches the policy-sized critic at up to 16 updates of lag (§4.2, Figure 3).

In the paper: App. D and Figure 6; §4.1 and Figure 2.


W2 + Q2. How common λ≈1\lambda\approx 1 is. λ≈1\lambda\approx 1 is the standard choice in large-scale PPO for language models. VAPO, VC-PPO (Yuan et al., 2025), Open-Reasoner-Zero (Hu et al., 2025) and the single-rollout asynchronous recipe of Hou et al. regress the critic on λ=1\lambda=1 returns, and DeepSeek-R1 reports that raising λ\lambda from 0.950.95 to 1.01.0 substantially improves PPO. In the paper: §2.2.


Q3. Why policy-sized critics, and what the probe keeps. Policy-sized critics date from RLHF-era PPO at the conventional λ=0.95\lambda=0.95 (Guo et al., 2025), where the advantage uses the critic's own values, so value errors bias the gradient and a larger critic reduces that bias. The probe keeps each measurable advantage. It matches or exceeds the policy-sized critic's value accuracy in three of four settings (Table 1) and matches its task success (Table 2), sample efficiency (Figure 2), and robustness under asynchronous training (Figure 3). It does so without a second model, and the cold-start probe needs no pretraining corpus. We observe 64 GB less critic training state, and an end-to-end run that takes 15–19% fewer GPU-hours (3.9–5.6 GPU-hours per run) than with the policy-sized critic, including its pretraining (App. G).

In the paper: §2.1–2.2; Tables 1–2; Figures 2–3; App. G.


Q4. Policy-sized critic below a linear readout. Both critics are trained on the same rollouts, and a probe's explained variance depends on how decodable the return is from its own model's hidden states. BFCL's 240 training tasks bound the effective sample size for both critics, the policy-sized critic must adapt 3-4B parameters, and a linear readout fits 2.6K parameters to features the policy already computes. Hou et al. find that scaling a policy-sized critic's value-pretraining corpus is essential, and BFCL is where it is smallest.

In the paper: §2.4; App. E.


Q5. Intermediate layers. As the reviewer expected, intermediate layers also encode the value. On WebShop the final layer reaches 83% and 94% of the best layer, and the architecture ranking that selected [64,32] holds at every layer (rank test, p<10−5p < 10^{-5}), so reading the final layer costs little. We read the final hidden state since the language model head reads it at every step, keeping it a stable probe input as the policy trains. Even a probe fit once and never updated trains the policy in every setting (§5). Probes that predict LLM answer correctness read the same layer (Zhang et al., 2025).

In the paper: §2.3; §5; App. C, Table 4.

Thanks for answering all my questions. For my weakness 2, I agree that λ≈1\lambda \approx 1 might be a common choice in large-scale LLM PPO, and the problem you propose is valid.

For my weakness 1, about why the probe critic works, thanks for honestly reporting the result of using the return mean as the critic and adding more analysis about why it works. I agree with your finding that a smaller critic can be enough under the near-one lambda setting. But I'm still not very convinced about the proposed method. I recognize the interesting empirical findings and good performance, but it seems like other simple critics could also work well. I'm still not very clear on how the specific properties of the problem motivate this particular design, beyond showing that the proposed method works well empirically.

I appreciate the empirical findings, but I still have some concern about the method contribution. I would rate it around 2.5.

Thank you for the updated assessment and your engagement during the rebuttal process. Interestingly, reading the value from the policy's own representation is the classic actor-critic design, which we discuss in Section 2.3. A3C (Mnih et al., 2016), IMPALA (Espeholt et al., 2018), PPO on Atari (Schulman et al., 2017) and AlphaGo Zero (Silver et al., 2017) all attach a small value head to the policy's network. Our proposed method is thus one of the arrangements PPO was originally introduced with, adapted to LLM agents. These value heads were designed for CNNs and LSTMs a decade ago and set aside for policy-sized critics when PPO moved to language models. We show that, surprisingly, they still work in agentic PPO, and we will emphasize this point in the final version.

Reviewer_7y5XOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Authors propose using probe critics (small MLP's on top of the features of the last layer of the policy transformer) for PPO rather than an independent, full-size critic transformer, for memory and computational efficiency. They show that for some tasks this has very little impact on the obtained performance.

Critical strengths
  • The proposed probe critic has the strength of requiring significantly less memory for training with PPO than full-sized models regularly used, as well as modest speed improvements.
  • Authors show that for the considered tasks, not much performance is lost from using probe critics as opposed to policy-sized critics.
  • There is some interesting analysis about the importance of continued training of the critic (Fig 3, fixed critic actually performs nearly as well as trained critic), and the effect of increasing off-policiness on probe critics.
Critical weaknesses
  • The biggest weakness in my view of the paper is that there is very little analysis about the reasons why probe critics do not lose significant performance, why we see different patterns in the different considered tasks, and on which tasks this pattern will hold or break. E.g. Table 1 shows significantly different patterns of explained variance comparing probe critics and the policy-sized critics when we move between models in the same task (e.g. WebShop Qwen3-4B really benefits from policy-sized critic, whereas granite-4.1-3b doesn’t), or same model and different tasks (on BFCL, probe critics can actually outperform policy-sized critic with Qwen3-4B, which is unintuitive by itself and deserves an explanation).

There is also a lack of analysis of how the strength of the value function used relates to the downstream test success. E.g., Fig. 3 shows that keeping the probe critic fixed during training performs nearly as well as keeping training it. What are the authors’ explanations for why this is the case? Is it that any good baseline will already help stabilize training for these tasks, regardless of whether it’s a good fit? E.g., would a constant non-state-dependent baseline work just as well?

I think the paper would benefit significantly from more discussion and analysis about why probe critics don’t seem to affect performance too much, including in which circumstances this can be expected, and if it would ever break down. At the moment, it is hard to get more general takeaways other than “for these specific tasks, it turns out that either the accuracy of the critic does not matter that much for performance, or you can get a good enough critic with much fewer parameters”.

  • There is also not much discussion or experimentation about why it is reasonable to only use the last layer for the critic probe. Does this affect the accuracy of the critic?

  • Little emphasis on the benefits of probe critics. The concrete benefits are only mentioned in one line in the main text (367-368), and there is no comparison about how probe critics compare to the critic-free baseline (since it does not train a critic at all, and since performance is comparable, there needs to be an argument about the trade-offs).

  • A contemporaneous work, “Your Language Model is Its Own Critic” (https://holi-lab.github.io/POISE/), is very related, and it would be good to add a citation.

Overall, the paper has interesting insights, but currently lacks analysis in multiple areas that would help generalize takeaways for a reader. I would consider raising my score if new analysis is submitted.

Questions for authors
  • Table 1: what is the authors’ analysis for: 1. the different patterns of performance seen across the two models (e.g. for WebShop Qwen3-4B policy-sized critic significantly outperforms all the considered alternatives, whereas for granite-4.1-3b some alternatives in fact outperform policy-sized critic); 2. the different pattern of performance across the two benchmarks (in BFCL policy-sized critic consistently underperform alternatives, unlike WebShop).
1 comment

We thank the reviewer for a careful review. We have made manuscript revisions in blue and discuss the weaknesses below.

Summary of updates for 7y5X:

  • W2: critic strength vs. test success, and why a fixed probe works. New experiments (Figure 2, App. D). On Qwen3-4B WebShop, even a never-updated probe matches the policy-sized critic's sample efficiency, and a constant baseline is 6.9 points behind over the first 50 updates (p=2.3×10−2p = 2.3\times10^{-2}).
  • W1 + W3 + Q1–Q2: Table 1's patterns, and when it breaks. New analysis (§2.4, App. C). Table 1's values track how decodable the return is from each model's hidden states. A fixed probe falls short where the representation drifts (§5.2).
  • W4: last layer. New experiments (§2.3, App. C). The final layer reaches 83% and 94% of the best layer.
  • W5: benefits vs. critic-free methods. Clarified with measured costs (§4.1, App. G). One rollout per prompt, no second model, and 15–19% fewer GPU-hours end to end in our runs.
  • W6: POISE. Added (§6).

W2. How critic strength relates to test success. On Qwen3-4B WebShop, the critics fit before RL (fixed probe, warm-start probe, policy-sized critic) beat PPO with a constant baseline V≡μV\equiv\mu in validation success at matched updates (three seeds each). Over the first 50 updates these three critics are 6.9 points ahead (p=2.3×10−2p = 2.3\times10^{-2}). Cold-start probes are ahead but not significantly. Hou et al. report the same lag for a running-mean baseline. Final test success is similar.

This is the theoretically expected behavior at λ≈1\lambda\approx 1. The critic enters the advantage as a baseline, which lowers gradient variance without changing its expectation. With a more accurate baseline the policy needs fewer updates to reach a given success, while in our setting every baseline reaches a similar final success. On Qwen3-4B WebShop, even a fixed probe never updated during training matches the sample efficiency of the policy-sized critic. It works because it is accurate from the first update, and on Qwen3-4B the representation it reads barely changes (§5.2).

In the paper: §4.1 and Figure 2; §5.1; App. D and Figure 6.


W1 + W3 + Q1–Q2. Why the patterns in Table 1 differ. A probe's explained variance measures how much of the return is decodable from its own model's hidden states, and the new layer sweep shows that this amount differs by model. Both critics are trained on the same data, so Table 1's differences reflect the models' representations. BFCL's 240 tasks bound the effective corpus for both critics, and the probe fits far fewer parameters to it (App. E). On WebShop, with 3,072 training tasks, the policy-sized critic has more data and leads on Qwen3-4B. On BFCL the probe leads on both models.

In the paper: §2.4; App. C, Table 4; App. E.


When it breaks. A fixed probe falls short where its policy's representation changes substantially, while a co-learned probe adapts to the change. On granite-4.1-3b the co-learned probe's first layer changes by 43–73% in norm and the fixed probe trails by 3–8 points, while on Qwen3-4B the first layer barely changes (5–8%) and the fixed probe matches. The probe's top value direction likewise keeps a cosine above 0.98 on Qwen3-4B but falls to 0.81 and 0.50 on granite-4.1-3b, suggesting granite's representation drifts further from the input the fixed probe was originally fit on.

In the paper: §5.2 and Figure 5.


W4. Why the last layer. We read the final hidden state because the language model head reads it at every step. The policy's own output depends on it, so it stays a reasonably stable input for a probe as the policy trains. Even a probe fit once and never updated trains the policy in every setting (§5). Probes that predict reasoning-model answer correctness read the same layer (Zhang et al., 2025). The cost in accuracy is small. On WebShop the final layer reaches 83% and 94% of the best layer, and the architecture ranking holds at every layer (rank test, p<10−5p < 10^{-5}).

In the paper: §2.3; §5; App. C, Table 4.


W5. Benefits over critic-free methods. GRPO needs eight rollouts per prompt and the probe one, at matching task success (Table 2). A group baseline can only be computed once every rollout of a prompt finishes, while single-rollout asynchronous training updates on each trajectory as soon as it completes. The probe gives a state-dependent baseline at every token, without a second policy-sized model. In our runs, we observe 64 GB less critic training state, and an end-to-end run that takes 15–19% fewer GPU-hours (3.9–5.6 GPU-hours per run) than with the policy-sized critic, including its pretraining (App. G).

In the paper: §4.1; App. G.


W6. POISE. POISE estimates prompt- and trajectory-level value rather than token values from two rollouts per prompt on single-turn reasoning. In the paper: §6.

Comments on the paper

2 comments

We thank the reviewers again for their thoughtful feedback. We would like to remind everyone that the discussion period ends on November 18, and we hope the reviewers will have time to consider our responses by then. We have responded to each review and uploaded a revised manuscript, with all changes in blue. The revision adds new RL runs and hundreds of experiments on value decodability, presented in a new learning-curve figure across all four settings (Figure 2) and two new appendices: PPO with a constant baseline (App. D) and probe architectures at every layer (App. C). We would welcome any follow-up questions or comments.

AuthorsOpenReview ↗

Summary for the Area Chair

For the AC's convenience, this comment summarizes the reviewers' assessment, concerns raised, and how our rebuttal and revised manuscript respond to each.

TL;DR. A 166K-parameter probe on the policy's final hidden state, four orders of magnitude smaller than the policy, replaces a policy-sized critic in agentic PPO, matching its task success on two environments (WebShop and BFCL) and two model families, and on WebShop under up to 16 updates of generator lag. The revision addresses all reviewers' concerns with new experiments, new analysis or clarification, including new RL runs and hundreds of value decodability experiments. All changes are in blue.


Strengths noted by the reviewers

  • Simple and well motivated. "The proposed idea is very simple and makes total sense" (LB1N). "A simple and useful critic" (KBGe).
  • Efficient. "Requiring significantly less memory for training with PPO than full-sized models regularly used, as well as modest speed improvements" (7y5X).
  • Comparable performance. Task success comparable to a policy-sized critic (7y5X, KBGe, LB1N).
  • Thorough analysis. "Several different tests," including a fixed critic and asynchronous training (LB1N), and "some interesting analysis" (7y5X).

Concerns and what the revision adds

  1. Constant baseline and critic accuracy (7y5X, KBGe, LB1N).
    New experiment (App. D, Figures 2 and 6).
    For the rebuttal, we ran three seeds of PPO with a constant baseline on Qwen3-4B WebShop. The critics fit before RL (the fixed probe, warm-start probe and policy-sized critic) lead it by 6.9 points of validation success over the first 50 updates (p=2.3×10−2p = 2.3\times10^{-2}), with similar final success, as expected when the critic acts as a baseline at λ≈1\lambda\approx 1. Even the fixed probe, never updated during training, matches the policy-sized critic's sample efficiency, and learning curves on all four settings show the co-learned probes closely tracking the policy-sized critic.
  2. Choice of the last layer (7y5X, KBGe).
    New experiment (App. C, Table 4).
    On WebShop, across six layers of both models and three seeds, the final layer reaches 83% and 94% of the best layer's explained variance, and the architecture ordering that selected [64,32] is consistent across layers (rank test, p<10−5p < 10^{-5}). We read the final layer because the language model head reads it at every step, and probes that predict reasoning-model answer correctness read the same layer (Zhang et al., 2025).
  3. Critic size (LB1N).
    New experiment (App. C).
    In task success, critic size matters little (concern 1, Table 2). The probe's value accuracy depends on its second nonlinear layer. On WebShop, a linear readout and a single-layer probe with as many parameters both score below the deployed probe at every layer and seed (p=2.9×10−11p = 2.9\times10^{-11}).
  4. Why the two critics rank differently across models and tasks in Table 1, and when a probe falls short (7y5X, KBGe).
    New analysis (§2.4, §5.2, App. C).
    A probe's accuracy depends on how much value information its model's hidden states carry, which differs by model. BFCL's 240 tasks limit the data available to both critics, and the probe, with far fewer parameters, leads there. On WebShop's 3,072 tasks the policy-sized critic leads on Qwen3-4B. A probe fit once and never updated falls behind on granite-4.1-3b, whose representation drifts more during training.
  5. What the probe keeps from a policy-sized critic, and the use of λ≈1\lambda\approx 1 (KBGe).
    Clarified with new citations (§2.2, Tables 1–2, Figures 2–3).
    VAPO, VC-PPO, Open-Reasoner-Zero and the single-rollout asynchronous recipe of Hou et al. all use λ=1\lambda=1 for the critic. The probe matches the policy-sized critic's task success, including under generator lag, closely tracks its learning curves, and matches or exceeds its value accuracy in three of four settings.
  6. Benefits over critic-free methods and GRPO (7y5X, LB1N).
    Clarified with measured costs (§4.1, App. G).
    The probe needs one rollout per prompt instead of GRPO's eight at matching task success, and in our runs takes 15–19% fewer GPU-hours end to end than the policy-sized critic, including the critic's pretraining.
  7. Related work and model selection (7y5X, LB1N).
    Added and clarified (§6, §2.4).
    POISE is now discussed, and §2.4 states the three selection steps.

Reviewer engagement

  • 7y5X wrote that they would consider raising their score if new analysis were submitted. Each of their requests is addressed above (concerns 1, 2, 4, 6 and 7).
  • KBGe replied that λ≈1\lambda\approx 1 "might be a common choice in large-scale LLM PPO, and the problem you propose is valid," recognized "interesting empirical findings and good performance," and now rates the paper around 2.5. Our reply addresses their remaining question.
  • LB1N replied that they read our response and will consider the changes in the reviewer–AC discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.