- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argues that value-free LLM RL methods lose fine-grained credit assignment and that discriminative critics are expressively limited. It proposes GenAC, a generative critic that reasons in chain-of-thought before predicting a value and is conditioned on the current actor. On math reasoning with Qwen3-8B-Base, GenAC improves average accuracy to 51.90 versus 50.39 for VC-PPO, with larger gains on hard benchmarks and better sample efficiency. Controlled scaling shows generative critics have lower approximation error, better scaling, and lower variance than discriminative critics.
Critical strengths
-
Tackles an important problem: credit assignment in LLM RL, where value-free methods like GRPO/RLOO lose fine-grained token-level credit.
-
Simple, intuitive method: a generative critic with chain-of-thought that outputs a value, plus In-Context Conditioning to keep it calibrated to the evolving actor.
-
Useful theoretical framing: discriminative one-shot critics are roughly TC0, while generative critics with CoT can be more expressive, up to P.
-
Reopens value-based RL for LLMs as a practical and worthwhile direction.
Critical weaknesses
The paper addresses an important problem with a simple, well-motivated method, but the contribution is incremental. For a top-tier venue, the theoretical link to RL optimization, the empirical scope, the compute-fairness comparisons, and the statistical rigor would likely need strengthening.
1. Theoretical weaknesses
-
The expressivity argument (TC0 vs. P) establishes a separation, not sufficiency. It never defines how much expressivity is enough for the RL task, nor links value approximation error to policy improvement or regret.
-
The separation assumes unbounded or sufficiently long chain-of-thought. The implementation caps generation length, so it is unclear whether the theoretical advantage survives under the actual CoT budget.
-
No concrete task-relevant value function is exhibited that a discriminative critic cannot approximate but a generative critic can. Without this, the theory remains suggestive rather than explanatory.
2. Methodological weaknesses
-
Segment-level credit assignment is heuristic. The paper does not analyze sensitivity to segmentation, and all tokens within a segment receive the same advantage, which may lose the fine-grained credit assignment the method aims to restore.
-
Parsing the generative critic’s integer score (0–10) into a value is brittle. Invalid outputs, refusals, non-integer outputs, and prompt-format sensitivity are not analyzed.
-
In-context conditioning (ICC) relies on an actor description and smoothed reward mean. It is unclear whether these are available in realistic on-policy settings without leakage, how often they are updated, and how sensitive training is to stale actor information.
-
The SFT stage depends on GPT-5 synthetic reasoning traces. The paper does not ablate whether self-generated traces, smaller models, or no SFT could work, raising reproducibility and cost concerns.
3. Empirical weaknesses
-
Scope is narrow: one model (Qwen3-8B-Base) and mostly math reasoning. No multi-turn dialogue, agentic tasks, code, open-ended generation, or larger models.
-
Baselines are limited to GRPO, RLOO, and VC-PPO. No comparisons to other generative value estimators, larger scalar critics at matched compute, process reward models, or other value-based methods.
Questions for authors
-
How much expressivity is sufficient? If we only need expressivity, why not simply scale up the base model? In Section 4.1, the authors establish a separation: a discriminative critic limited to one-shot scalar prediction is less expressive, while a generative critic with chain-of-thought can be more expressive. But they do not establish sufficiency. How much expressivity is actually enough for the RL problem you study? Could we define a task-relevant target for value or advantage approximation error, or a bound linking that error to policy improvement or regret, and then show GenAC reaches that threshold rather than merely outperforming the discriminative baseline? Specifically, does Figure 2(a) show saturation at that level, how does bounded versus unbounded CoT length affect the answer, and how do we reconcile the expressivity argument with the zero-shot GPT-5 result, where a highly expressive model still fails as a value estimator without training and policy conditioning?
-
On segmentation and credit assignment: How exactly are segments defined, and how sensitive is GenAC to that choice? Would token-level or response-level value modeling change the conclusions? Within a segment, do all tokens receive the same advantage, and does that lose the fine-grained credit assignment you aim to restore? The authors could provide some empirical results on this.
-
On baselines: Why compare only to GRPO, RLOO, and VC-PPO? How does GenAC compare to other value-based methods, larger scalar critics, reward-model-based critics, or recent generative value estimators?
8 comments
Author Response to Reviewer djpV (Part 2/4)
Weakness 1 and Question 1: Theoretical Framing and Expressivity
1.1. Why not simply scale up the base model (Q1)
TC0 characterizes constant-depth parallel computation, whereas P allows polynomial-time sequential computation (L177–180, L804–809). Scaling up a one-shot discriminative critic only increases parallel width within fixed depth without introducing sequential computation, so it remains in TC0. Empirically, Figure 2(a) scales discriminative critics up to 14B (the largest released Qwen3 dense base model) and shows they still fail to yield better value approximation.
1.2. Separation vs. sufficiency, RL link, and Figure 2(a) saturation (W1.1, Q1)
Complexity theory characterizes the capacity of a hypothesis class, while how much approximation accuracy suffices for RL on a real task is an empirical question rather than something theory can determine in closed form (L484). We thus present the theory as motivational while providing extensive empirical evidence that our recipe is practically sufficient: in Figure 2(a), discriminative critics stay flat across all sizes, while our generative critics improve monotonically with scale without saturating; at 8B, our generative critic already achieves stronger candidate ranking (Table 3) and out-of-distribution generalization (Table 4) that translate into sustained downstream RL gains (Figure 2(b), Table 1).
1.3. Bounded vs. unbounded CoT budget (W1.2, Q1)
Unbounded CoT is only an asymptotic requirement as input length approaches infinity. For finite length inputs, the required CoT length is also finite. Under our setup, critic generations are capped at 1024 tokens but average ~300 tokens, indicating that this cap is non-binding and already empirically sufficient for our task (Figure 2(a), Table 4).
1.4. Concrete task-relevant value function and role of theory (W1.3)
Proving a formal separation on an actual LLM task is intractable since neither natural-language tasks nor learned LLM policies have formal mathematical definitions. We therefore use the theoretical analysis in Section 3 as motivation and evaluate real-task value approximation empirically in Section 3 (Figure 2(a)). We will make this clearer in the revision.
1.5. Reconciling with zero-shot GPT-5 (Q1)
There is no contradiction because architectural expressivity only ensures that some instance in the hypothesis class can represent the target value function without guaranteeing the accuracy of any particular instance. Zero-shot GPT-5 is simply one untrained instance without knowledge of the evaluated actor in its parameters, so its high error in Table 2 (L366) does not contradict the expressiveness of the generative architecture.
Author Response to Reviewer djpV (Part 3/4)
Weakness 2 and Question 2: Segmentation, Score Parsing, ICC, and SFT Ablations
2.1. Segmentation (W2.1, Q2)
Our segmentation rule is detailed in Appendix C (L890–894). To test empirical sensitivity to segmentation granularity and SFT removal in end-to-end RL, we evaluate three new variants: GenAC-Coarse (~5x fewer segments per response), GenAC-Light (no SFT, 40 VP steps, subsampled critic batches), and GenAC-Coarse-Light (combining both):
| Algorithm | SFT Data | VP Steps | Granularity | Total Cost | AIME24 | AMC23 | MATH500 | AIME25 | Olympiad | Minerva | Average |
|---|---|---|---|---|---|---|---|---|---|---|---|
| GRPO | 0 | Response | 0.48x | 26.46 | 69.69 | 87.20 | 22.92 | 52.92 | 42.12 | 50.22 | |
| VC-PPO | 200 | Token | 1.00x | 25.83 | 70.78 | 87.54 | 21.67 | 53.50 | 43.02 | 50.39 | |
| GenAC | ✓ | 100 | Fine | 2.15x | 30.00 | 74.53 | 87.48 | 21.88 | 54.16 | 43.34 | 51.90 |
| GenAC-Coarse | ✓ | 100 | Coarse | 1.41x | 27.50 | 73.28 | 88.58 | 22.50 | 55.12 | 44.14 | 51.85 |
| GenAC-Light | 40 | Fine | 1.26x | 28.54 | 72.03 | 88.19 | 23.12 | 54.01 | 43.52 | 51.57 | |
| GenAC-Coarse-Light | 40 | Coarse | 1.05x | 27.92 | 73.91 | 88.15 | 21.88 | 54.55 | 43.59 | 51.66 |
- Sensitivity: Splitting at delimiters outside math follows standard practice (VinePPO, Kazemnejad et al., 2025), and because our critic always conditions on the full prefix , it does not rely much on exact boundary positions. Empirically, increasing the merge threshold in our split-then-merge rule (GenAC-Coarse and GenAC-Coarse-Light, which alters segment boundaries and reduces segment count by ~5x) yields nearly identical accuracy under both full and no-SFT initialization.
- Token-level vs. response-level value modeling: We do not use token-level generative value modeling because individual tokens carry little semantic meaning and generating CoT at every token is computationally prohibitive. We are also unsure what "response-level value modeling" refers to since treating a full response as a single action reduces multi-step decision-making to a bandit, leaving no intermediate value function to model.
- Intra-segment advantage sharing: Under a segment-level MDP (L248), each segment is one atomic action spanning multiple tokens, so tokens within that action naturally share its value and advantage; credit assignment remains fine-grained across the multiple sequential segment actions in each trajectory.
2.2. Score parsing reliability (W2.2): Emitting a verbalized discrete score with zero reward on parse failure (, L287) follows standard practice in RL-trained generative reward models (L791). Even when initialized directly from Qwen3-8B-Base without SFT, value pretraining reaches ~99% valid parsing within 10 steps and maintains >99.9% valid parsing throughout RL training.
2.3. ICC availability, leakage, and staleness (W2.3): In ICC, the actor size is static and the reward hint is a dataset-wide running average across past batches rather than using the current online batch's instance outcomes (Algorithm 1, L819), so it is always available on-policy without leakage. Sensitivity to staleness is bounded by Table 2 (L378), where even removing ICC completely causes only a modest drop.
2.4. Removing GPT-5 SFT (W2.4): Appendix D (Figure 5, L950) already shows that skipping GPT-5 SFT in value pretraining recovers nearly the same approximation quality. Extending this to end-to-end RL, GenAC-Light and GenAC-Coarse-Light in the table above initialize directly from Qwen3-8B-Base with zero GPT-5 data and still clearly outperform all baselines while cutting cost down to 1.05x–1.26x of VC-PPO.
Author Response to Reviewer djpV (Part 1/4)
We thank the reviewer for recognizing our motivation, method, and theoretical framing. Regarding the concerns in the Weakness summary, we clarify what our submission establishes and how we will further strengthen it in the revision:
Furthermore, we note that several concerns in the review are already explicitly included in our submission:
Below, we address each remaining question and present our new end-to-end RL runs.