acceptodds
Under review as a conference paper at ICLR 2027

Bringing Value Models Back: Generative Critics for LLM Reinforcement Learning

Abstract

Credit assignment is a central challenge in reinforcement learning (RL). Classical actor-critic methods address this challenge through fine-grained advantage estimation based on a learned value function. However, learned value models are often avoided in modern large language model (LLM) RL because conventional discriminative critics are difficult to train reliably. We revisit value modeling and argue that this difficulty is partly due to limited expressiveness. In particular, representation complexity theory suggests that value functions can be hard to approximate under the one-shot prediction paradigm used by existing value models, and our scaling experiments show that such critics do not improve reliably with scale. Motivated by this observation, we propose Generative Actor-Critic (GenAC), which replaces one-shot scalar value prediction with a generative critic that performs chain-of-thought reasoning before producing a value estimate. We further introduce In-Context Conditioning, which helps the critic remain calibrated to the current actor throughout training. GenAC improves value approximation, ranking reliability, and out-of-distribution generalization, and these gains translate into stronger downstream RL performance than both value-based and value-free baselines. Overall, our results suggest that stronger value modeling is a promising direction for improving credit assignment in LLM reinforcement learning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM RL, credit assignment, value modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_djpVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that value-free LLM RL methods lose fine-grained credit assignment and that discriminative critics are expressively limited. It proposes GenAC, a generative critic that reasons in chain-of-thought before predicting a value and is conditioned on the current actor. On math reasoning with Qwen3-8B-Base, GenAC improves average accuracy to 51.90 versus 50.39 for VC-PPO, with larger gains on hard benchmarks and better sample efficiency. Controlled scaling shows generative critics have lower approximation error, better scaling, and lower variance than discriminative critics.

Critical strengths
  1. Tackles an important problem: credit assignment in LLM RL, where value-free methods like GRPO/RLOO lose fine-grained token-level credit.

  2. Simple, intuitive method: a generative critic with chain-of-thought that outputs a value, plus In-Context Conditioning to keep it calibrated to the evolving actor.

  3. Useful theoretical framing: discriminative one-shot critics are roughly TC0, while generative critics with CoT can be more expressive, up to P.

  4. Reopens value-based RL for LLMs as a practical and worthwhile direction.

Critical weaknesses

The paper addresses an important problem with a simple, well-motivated method, but the contribution is incremental. For a top-tier venue, the theoretical link to RL optimization, the empirical scope, the compute-fairness comparisons, and the statistical rigor would likely need strengthening.

1. Theoretical weaknesses

  • The expressivity argument (TC0 vs. P) establishes a separation, not sufficiency. It never defines how much expressivity is enough for the RL task, nor links value approximation error to policy improvement or regret.

  • The separation assumes unbounded or sufficiently long chain-of-thought. The implementation caps generation length, so it is unclear whether the theoretical advantage survives under the actual CoT budget.

  • No concrete task-relevant value function is exhibited that a discriminative critic cannot approximate but a generative critic can. Without this, the theory remains suggestive rather than explanatory.

2. Methodological weaknesses

  • Segment-level credit assignment is heuristic. The paper does not analyze sensitivity to segmentation, and all tokens within a segment receive the same advantage, which may lose the fine-grained credit assignment the method aims to restore.

  • Parsing the generative critic’s integer score (0–10) into a value is brittle. Invalid outputs, refusals, non-integer outputs, and prompt-format sensitivity are not analyzed.

  • In-context conditioning (ICC) relies on an actor description and smoothed reward mean. It is unclear whether these are available in realistic on-policy settings without leakage, how often they are updated, and how sensitive training is to stale actor information.

  • The SFT stage depends on GPT-5 synthetic reasoning traces. The paper does not ablate whether self-generated traces, smaller models, or no SFT could work, raising reproducibility and cost concerns.

3. Empirical weaknesses

  • Scope is narrow: one model (Qwen3-8B-Base) and mostly math reasoning. No multi-turn dialogue, agentic tasks, code, open-ended generation, or larger models.

  • Baselines are limited to GRPO, RLOO, and VC-PPO. No comparisons to other generative value estimators, larger scalar critics at matched compute, process reward models, or other value-based methods.

Questions for authors
  1. How much expressivity is sufficient? If we only need expressivity, why not simply scale up the base model? In Section 4.1, the authors establish a separation: a discriminative critic limited to one-shot scalar prediction is less expressive, while a generative critic with chain-of-thought can be more expressive. But they do not establish sufficiency. How much expressivity is actually enough for the RL problem you study? Could we define a task-relevant target for value or advantage approximation error, or a bound linking that error to policy improvement or regret, and then show GenAC reaches that threshold rather than merely outperforming the discriminative baseline? Specifically, does Figure 2(a) show saturation at that level, how does bounded versus unbounded CoT length affect the answer, and how do we reconcile the expressivity argument with the zero-shot GPT-5 result, where a highly expressive model still fails as a value estimator without training and policy conditioning?

  2. On segmentation and credit assignment: How exactly are segments defined, and how sensitive is GenAC to that choice? Would token-level or response-level value modeling change the conclusions? Within a segment, do all tokens receive the same advantage, and does that lose the fine-grained credit assignment you aim to restore? The authors could provide some empirical results on this.

  3. On baselines: Why compare only to GRPO, RLOO, and VC-PPO? How does GenAC compare to other value-based methods, larger scalar critics, reward-model-based critics, or recent generative value estimators?

8 comments
AuthorsOpenReview ↗

Author Response to Reviewer djpV (Part 1/4)

We thank the reviewer for recognizing our motivation, method, and theoretical framing. Regarding the concerns in the Weakness summary, we clarify what our submission establishes and how we will further strengthen it in the revision:

  • Contribution: Learned critics in LLM RL are widely viewed as unstable and ineffective. We isolate value approximation directly, uncover that discriminative critics fail to scale with model size and are highly sensitive to random seeds, diagnose this as an architectural deficit of one-shot scalar prediction, and propose a generative value modeling framework that makes value learning accurate, robust, and scalable (L074, L480).
  • Theoretical link to RL: Section 3 uses representation complexity to motivate why a generative architecture is preferable to one-shot prediction (L484). We clarify the exact role and scope of this theoretical discussion under Weakness 1 and Question 1 below, and will revise Section 3 to state this more explicitly.
  • Empirical scope: Beyond downstream RL gains (Figure 2(b)), our submission evaluates value scaling across five model sizes (0.6B to 14B, Figure 2(a)), relative ranking (Table 3) and out-of-domain scientific reasoning (Table 4), and we discuss broader RL tasks under Weakness 3 and Question 3 below.
  • Compute-fairness comparisons: Under Weakness 2 and Question 2 below, we add new end-to-end RL ablations that remove the GPT-5 SFT stage entirely and reduce total compute down to 1.05x of VC-PPO while maintaining clear gains, which we will include in the revision.
  • Statistical rigor: Our submission reports 10 independent runs per model scale in Figure 2(a) and 3-seed RL baseline replications in Appendix E (Table 7).

Furthermore, we note that several concerns in the review are already explicitly included in our submission:

  1. Scaling discriminative critics to larger base models (W3.2, Q1, Q3): Figure 2(a) (L121), Section 3 (L192–202), and Table 6 (L918) already evaluate discriminative scalar critics across all five released Qwen3 dense base models from 0.6B up to 14B (the largest dense base model in the Qwen3 family).
  2. Ablating the removal of GPT-5 SFT (W2.4): Section 5.3.1 (L376–377) and Appendix D (Figure 5, L950–961) already ablate skipping the GPT-5 SFT stage entirely in value pretraining, and we extend this with new end-to-end RL runs without SFT under Weakness 2 and Question 2 below.
  3. Statistical rigor (Weaknesses intro): Figure 2(a) (L121, L193) reports mean and standard deviation across 10 runs per model size, and Appendix E (Table 7, L965–986, referenced at L351) provides 3-seed RL baseline replications.
  4. Segment definition and intra-segment broadcasting (W2.1, Q2): Appendix C (L890–894, referenced at L296) and Algorithm 1 (Lines 12 and 19–20, L824–835; L885–888) explicitly define our delimiter segmentation rule and the broadcasting of segment values to token level, and we add new segmentation granularity RL ablations under Weakness 2 and Question 2 below.
  5. ICC update rule (W2.3): Section 4.1 (L258) and Algorithm 1 (Lines 7 and 14, L819–826) explicitly specify that the ICC reward hint is updated at every iteration via a running average over training batches.

Below, we address each remaining question and present our new end-to-end RL runs.

AuthorsOpenReview ↗

Author Response to Reviewer djpV (Part 2/4)

Weakness 1 and Question 1: Theoretical Framing and Expressivity

1.1. Why not simply scale up the base model (Q1)

TC0 characterizes constant-depth parallel computation, whereas P allows polynomial-time sequential computation (L177–180, L804–809). Scaling up a one-shot discriminative critic only increases parallel width within fixed depth without introducing sequential computation, so it remains in TC0. Empirically, Figure 2(a) scales discriminative critics up to 14B (the largest released Qwen3 dense base model) and shows they still fail to yield better value approximation.

1.2. Separation vs. sufficiency, RL link, and Figure 2(a) saturation (W1.1, Q1)

Complexity theory characterizes the capacity of a hypothesis class, while how much approximation accuracy suffices for RL on a real task is an empirical question rather than something theory can determine in closed form (L484). We thus present the theory as motivational while providing extensive empirical evidence that our recipe is practically sufficient: in Figure 2(a), discriminative critics stay flat across all sizes, while our generative critics improve monotonically with scale without saturating; at 8B, our generative critic already achieves stronger candidate ranking (Table 3) and out-of-distribution generalization (Table 4) that translate into sustained downstream RL gains (Figure 2(b), Table 1).

1.3. Bounded vs. unbounded CoT budget (W1.2, Q1)

Unbounded CoT is only an asymptotic requirement as input length approaches infinity. For finite length inputs, the required CoT length is also finite. Under our setup, critic generations are capped at 1024 tokens but average ~300 tokens, indicating that this cap is non-binding and already empirically sufficient for our task (Figure 2(a), Table 4).

1.4. Concrete task-relevant value function and role of theory (W1.3)

Proving a formal separation on an actual LLM task is intractable since neither natural-language tasks nor learned LLM policies have formal mathematical definitions. We therefore use the theoretical analysis in Section 3 as motivation and evaluate real-task value approximation empirically in Section 3 (Figure 2(a)). We will make this clearer in the revision.

1.5. Reconciling with zero-shot GPT-5 (Q1)

There is no contradiction because architectural expressivity only ensures that some instance in the hypothesis class can represent the target value function without guaranteeing the accuracy of any particular instance. Zero-shot GPT-5 is simply one untrained instance without knowledge of the evaluated actor in its parameters, so its high error in Table 2 (L366) does not contradict the expressiveness of the generative architecture.

AuthorsOpenReview ↗

Author Response to Reviewer djpV (Part 3/4)

Weakness 2 and Question 2: Segmentation, Score Parsing, ICC, and SFT Ablations

2.1. Segmentation (W2.1, Q2)

Our segmentation rule is detailed in Appendix C (L890–894). To test empirical sensitivity to segmentation granularity and SFT removal in end-to-end RL, we evaluate three new variants: GenAC-Coarse (~5x fewer segments per response), GenAC-Light (no SFT, 40 VP steps, subsampled critic batches), and GenAC-Coarse-Light (combining both):

AlgorithmSFT DataVP StepsGranularityTotal CostAIME24AMC23MATH500AIME25OlympiadMinervaAverage
GRPO0Response0.48x26.4669.6987.2022.9252.9242.1250.22
VC-PPO200Token1.00x25.8370.7887.5421.6753.5043.0250.39
GenAC✓100Fine2.15x30.0074.5387.4821.8854.1643.3451.90
GenAC-Coarse✓100Coarse1.41x27.5073.2888.5822.5055.1244.1451.85
GenAC-Light40Fine1.26x28.5472.0388.1923.1254.0143.5251.57
GenAC-Coarse-Light40Coarse1.05x27.9273.9188.1521.8854.5543.5951.66
  • Sensitivity: Splitting at delimiters outside math follows standard practice (VinePPO, Kazemnejad et al., 2025), and because our critic always conditions on the full prefix (x,y≤k)(x, y_{\le k}), it does not rely much on exact boundary positions. Empirically, increasing the merge threshold in our split-then-merge rule (GenAC-Coarse and GenAC-Coarse-Light, which alters segment boundaries and reduces segment count by ~5x) yields nearly identical accuracy under both full and no-SFT initialization.
  • Token-level vs. response-level value modeling: We do not use token-level generative value modeling because individual tokens carry little semantic meaning and generating CoT at every token is computationally prohibitive. We are also unsure what "response-level value modeling" refers to since treating a full response as a single action reduces multi-step decision-making to a bandit, leaving no intermediate value function to model.
  • Intra-segment advantage sharing: Under a segment-level MDP (L248), each segment is one atomic action spanning multiple tokens, so tokens within that action naturally share its value and advantage; credit assignment remains fine-grained across the multiple sequential segment actions in each trajectory.

2.2. Score parsing reliability (W2.2): Emitting a verbalized discrete score with zero reward on parse failure (Rv=0R_v = 0, L287) follows standard practice in RL-trained generative reward models (L791). Even when initialized directly from Qwen3-8B-Base without SFT, value pretraining reaches ~99% valid parsing within 10 steps and maintains >99.9% valid parsing throughout RL training.

2.3. ICC availability, leakage, and staleness (W2.3): In ICC, the actor size is static and the reward hint rˉ\bar{r} is a dataset-wide running average across past batches rather than using the current online batch's instance outcomes (Algorithm 1, L819), so it is always available on-policy without leakage. Sensitivity to staleness is bounded by Table 2 (L378), where even removing ICC completely causes only a modest drop.

2.4. Removing GPT-5 SFT (W2.4): Appendix D (Figure 5, L950) already shows that skipping GPT-5 SFT in value pretraining recovers nearly the same approximation quality. Extending this to end-to-end RL, GenAC-Light and GenAC-Coarse-Light in the table above initialize directly from Qwen3-8B-Base with zero GPT-5 data and still clearly outperform all baselines while cutting cost down to 1.05x–1.26x of VC-PPO.

Reviewer_fHLaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the credit assignment problem in reinforcement learning for LLMs and argues for revisiting segment-level generative value models. The authors argue that conventional score-based critics, i.e., a scalar regression head on top of an LM, are difficult to use directly and suffer from high bias, partly because of limited expressiveness. They support this claim from two angles:

A theoretical analysis, grounded in the principles of chain-of-thought (CoT), of the limitations of producing only a scalar prediction. An empirical measurement of how the MSE varies across different methods and model sizes.

Based on this, the authors propose GenAC. The method first generates a CoT analysis and then outputs a success-probability score on a 0–10 scale. Unlike a standard scalar critic, GenAC splits the response into segments using heuristic rules and estimates a value once per segment. Each segment’s value is broadcast to all tokens within that segment and then used as the baseline for computing the PPO advantage.

The method is trained with Qwen3-8B-Base on mathematical reasoning datasets and evaluated on the relevant metrics. It achieves roughly 1 percentage point improvement over GRPO.

Critical strengths
  1. The authors bring the focus back to PPO-style value-based methods and revisit the value model. They identify the limitations of current critic models’ predictions and propose using CoT to enhance the critic’s capability.
  2. The critic is used only as a baseline. This makes it less susceptible to reward hacking than using it as a process reward model.
Critical weaknesses
  1. High cost, small gain. According to the FLOPs estimate in the authors’ appendix, the per-iteration compute is roughly 2.1× that of PPO and 3.8× that of GRPO, yet the actual wall-clock time is not reported. Since the decoding stage is memory-bandwidth-bound, the actual time cost is usually larger than the FLOPs ratio suggests. There does not seem to be a reasonable practical use case for the method: such a large overhead yields only about a 1 percentage point improvement.
  2. Too many confounding variables. The method introduces too many additional variables, which makes it impossible to determine the source of the gains. For example, GPT-5 is involved in generating the data used to train the critic via SFT.
  3. Insufficient experimental comparison. Since GenAC introduces finer-grained values, its advantages are also finer-grained. The amount of information obtained per update therefore differs from that of the baselines, and the convergence speed may differ as well. However, all comparisons use the checkpoint at 200 steps, which does not demonstrate that the baselines are sufficiently trained under this setting.
Questions for authors
  1. Please explain in detail, or provide an experimental comparison showing, the advantages and disadvantages of this method compared with directly using a generative process reward model. If we have a more accurate judge model, a more direct approach would be to use it for process reward prediction.
  2. Please explain in detail the future prospects and application scenarios of this method. This is a central point of the paper. If the current drawbacks (high time cost and low gain) can be addressed in the future, this would change the contribution of the paper itself.
8 comments
AuthorsOpenReview ↗

Author Response to Reviewer fHLa (Part 1/4)

We thank the reviewer for the constructive and insightful review, and provide detailed responses below.


Weakness 1: Cost, Wall-Clock Time, and Performance Gain

In the main paper, we deliberately chose an untuned, near-maximal feedback density (up to 35 segments per response) to establish whether stronger value modeling improves credit assignment (L298). We fully acknowledged cost as an important limitation (L481), and analyzed the full training cost and immediate ways to reduce it within our framework in Appendix F (L991).

To show that our framework remains effective under these cost reductions, we run three new variants: GenAC-Coarse (using ~5x coarser segmentation), GenAC-Light (skipping SFT stage, early stopping value pretraining stage, and subsampling critic training batches), and GenAC-Coarse-Light (combining both). We report their cost and performance below:

AlgorithmSFT DataVP StepsGranularityTotal CostAIME24AMC23MATH500AIME25OlympiadMinervaAverage
GRPO0Response0.48x26.4669.6987.2022.9252.9242.1250.22
VC-PPO200Token1.00x25.8370.7887.5421.6753.5043.0250.39
GenAC✓100Fine2.15x30.0074.5387.4821.8854.1643.3451.90
GenAC-Coarse✓100Coarse1.41x27.5073.2888.5822.5055.1244.1451.85
GenAC-Light40Fine1.26x28.5472.0388.1923.1254.0143.5251.57
GenAC-Coarse-Light40Coarse1.05x27.9273.9188.1521.8854.5543.5951.66

These variants reduce the total compute cost down to 1.05x of VC-PPO (less than half of original GenAC in the main paper) and remove any reliance on SFT data, while remaining comparable to original GenAC and clearly stronger than baselines.

We also agree that FLOPs and wall-clock time can diverge in autoregressive decoding. However, unlike actor generation that decodes a single long sequence (up to 8k tokens), critic generation spawns many independent requests per actor sequence (one per segment, 5-35 per sequence) with a much shorter generation length (at most 1k tokens, ~300 on average). This keeps critic generation compute-bound. As shown below, the measured per-step GPU wall-clock ratio (relative to GRPO) closely tracks our analytical FLOPs estimate:

VariantAnalytical FLOPs RatioMeasured Wall-Clock RatioDifference
GenAC-Coarse-Light2.10x2.27x+8.1%
GenAC-Light2.48x2.78x+12.1%

We agree that the absolute gain is modest in single-turn math where GRPO is already effective. However, as shown in Figure 2(b), a conventional discriminative critic (VC-PPO) is eventually caught up by GRPO and finishes with nearly identical performance, whereas replacing the critic with ours sustains a clear improvement throughout training. This difference shows that a generative critic can make value-based credit assignment meaningfully effective where a discriminative critic barely helps. We expect the benefits of generative value modeling to grow in longer-horizon or multi-turn settings where uniform credit assignment is much less effective, as we discuss in our response to Question 2 below.

AuthorsOpenReview ↗

Author Response to Reviewer fHLa (Part 2/4)

Weakness 2: Confounding Variables

We agree that the main paper could be clearer in isolating the effect of individual design choices, and we will revise the paper to reflect this.

We design the SFT stage following the standard practice in generative reward model literature to initialize the proper reasoning behavior that otherwise an 8B scale model does not reliably possess: following instructions, assessing problem difficulty, detecting errors, progress or confusion, answering with proper scoring format, and keeping the chain-of-thought within a controlled token budget. These abilities are complementary but beneficial for the value estimation that subsequent RL will optimize. GPT-5 is strong at these abilities, thus its reasoning traces are helpful, even though the value estimate they land on might be inaccurate. We already showed in Appendix D (L950) that skipping SFT during value pretraining recovers nearly the same MSE, with SFT mainly stabilizing the critic's chain-of-thought length. Extending this ablation to end-to-end RL in the table above (GenAC-Light and GenAC-Coarse-Light) confirms that skipping the SFT stage entirely still sustains downstream RL gains, showing that the SFT stage is beneficial but not necessary.

For other components, Table 2 (L378) in the main paper isolates the effect of each training stage and in-context conditioning, and the table above (GenAC-Coarse and GenAC-Coarse-Light) shows that varying the segmentation granularity (~5x fewer segments) also yields nearly identical performance. We will include these ablations and state the role of each component more explicitly in the revision. If there are other specific variables the reviewer is concerned about, we are happy to provide further clarification during the discussion period.


Weakness 3: 200-Step Comparison and Convergence Speed

We agree that fine-grained advantages provide more information per update and thus accelerate convergence. In fact, achieving faster policy improvement from a fixed amount of samples (increasing sample efficiency) is a primary benefit of credit assignment, which is why we evaluate all methods under the exact same RL rollout budget (L339).

In addition, VC-PPO can serve as the controlled baseline for fine-grained advantages. Both VC-PPO and GenAC converge faster in early training than uniform advantage baselines; however, constrained by the limitations of its discriminative critic, VC-PPO's advantage diminishes over time, whereas GenAC sustains its lead.

Finally, we argue that our setup of 200 total steps, 4 mini batch updates per step, 256 sequences per mini batch, corresponding to 800 actor gradient steps with batch size 256, is already a substantial training budget at 8B scale.

AuthorsOpenReview ↗

Author Response to Reviewer fHLa (Part 3/4)

Question 1: Comparison with Generative Process Reward Models

We touched on these differences in Section 1 (L096) and Appendix A (L779), and clarify them in more detail here. Conceptually, a process reward model (PRM) supplies an external dense reward that modifies the RL objective, while a value model is an internal RL component that estimates expected return given a defined outcome reward. They are orthogonal, non-conflicting techniques, and their practical trade-offs are consequences of this conceptual distinction, as detailed below.

Where generative PRMs have advantages:

  1. Policy-agnostic: Since reward functions are defined over states and actions, technically an offline-trained PRM can be used across all policies in its use case, e.g., Best-of-N selection or iterative refinement (as shown in GenPRM and ThinkPRM). Conversely, as value functions are defined over the evaluated policies, a generative critic cannot transfer to policies it is not trained on without adaptation.

  2. Intentional reward shaping: When the outcome reward is underspecified, an external PRM can reshape the objective to enforce preferences (e.g., training for step correctness in addition to answer correctness), while a value model strictly preserves the outcome reward without this flexibility.

Where generative value models have advantages:

  1. Not enlarging reward misspecification and reward hacking: The downside of PRMs modifying the reward function is that they substantially enlarge reward misspecification (e.g., it is difficult to specify intermediate rewards or balance process and outcome rewards), which RL optimization then exploits, leading to severe reward hacking. As a result, DeepSeek-R1 cites reward hacking and step ambiguity as key reasons for avoiding PRMs in large-scale RL, and GenPRM and ThinkPRM report no online RL training results. Optimizing with value models implements credit assignment cleanly without introducing this additional misspecification and hacking risk.

  2. Quality tolerance: When used as a baseline in advantage estimation, value prediction errors do not bias the expected policy gradient and affect only variance. In contrast, errors in a PRM directly corrupt the RL objective. This is practically important as both value models and PRMs are hard to train.

  3. Data scalability and online learning: As PRMs define external feedback signals, they fundamentally require external supervision sources that are less scalable and prevent online adaptation. Existing works use datasets with dense labels annotated by humans (e.g., PRM800K in ThinkPRM), LLM judges, or costly multi-rollout completions per step (e.g., Math-Shepherd, and Relative Progress Estimation in GenPRM) which actually compute offline value estimates. Value models instead can rely on the observed outcome reward as an unbiased value target. This is practically important, as scalable offline training is critical for accuracy while online adaptation keeps the critic calibrated and mitigates collapse.

For the reasons above, while PRMs are well-suited for test-time scaling of frozen policies, value models are more principled and practical for credit assignment in online RL, which is why our experiments focus on value-based and value-free advantage estimators rather than training against PRMs.

Reviewer_84roOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a novel generative reward modeling framework that leverages generative modeling and contrastive critic learning to address the challenging credit assignment problem in reinforcement learning. The proposed approach is conceptually sound and well-motivated, with the potential to improve learning stability and provide more informative reward signals.

Critical strengths
  1. The idea is reasonable, following the traditional actor-critical framework.
  2. The paper is easy for understanding.
Critical weaknesses

1. Insufficient Experimental Comparisons.

The experimental evaluation is insufficient to convincingly demonstrate the effectiveness of the proposed method, as it lacks comparisons with state-of-the-art generative and discriminative reward models. Including these baselines would provide a more comprehensive assessment of the proposed approach and help clarify its advantages over existing reward modeling methods.

2. Limited Methodological Novelty.

The methodological novelty of the proposed approach is somewhat unclear. Step-by-step reasoning has been widely explored in large language models, and combining such reasoning capabilities with generative reward modeling appears conceptually similar to the core idea presented in this paper. Therefore, the authors should further clarify the fundamental differences between their approach and existing reasoning-based reward modeling methods, highlighting the specific technical contributions beyond established frameworks.

Questions for authors
  1. How the model's effective compared with the SOTA reward model.
  2. The complexity of this generative reward model.
2 comments
AuthorsOpenReview ↗

Author Response to Reviewer 84ro (Part 1/2)

We thank the reviewer for the assessment of our idea and clarity as critical strengths. We would like to clarify a key distinction that the reviewer may have overlooked based on their summary of the paper as a generative reward modeling framework: what we propose is a generative value modeling framework, including both the value model and an online actor-critic algorithm built on top of it, instead of a reward model. The differences between reward functions and value functions are central to our motivation and design:

  1. We tackle the problem of credit assignment (decomposing the sparse outcome reward into fine-grained intermediate advantages for each action). Standard outcome reward functions define the sparse outcome without touching credit assignment.

  2. While dense reward functions (process reward models) also touch credit assignment, learning value models faces a unique challenge of conditioning on the actor. Value functions are defined over an evolving policy throughout RL training, whereas reward functions are not. Motivated by this, we learn this conditioning through both parameters and contexts, proposing joint RL training for the former, and in-context conditioning for the latter.

We also argue that value modeling is a fundamentally better solution than process reward modeling for credit assignment. In classical policy gradient, advantage is practically defined as return minus baseline. A value prediction enters as a baseline that doesn't bias the gradient in expectation, while a reward adds to return, changing the RL objective itself. This fact has important implications that favor value modeling as a better solution to credit assignment:

  • Adversarial robustness: Because reward is part of the objective and reward models are imperfect proxies for ground truth reward function, the policy can learn to utilize flaws of RMs to increase the RL objective while not optimizing what we want, known as the reward hacking issue. The value models cannot be optimized or utilized this way as they don't change the RL objective.

  • Quality tolerance: Reward models have to be well-trained since their quality directly impacts the training objective. A value model is less required in quality as it affects variance, not bias, of the RL objective. This relaxed requirement is practically important as both process reward models and value models are difficult to train.

  • Data scalability: Practically, process reward models require step-level annotation data that is hard to acquire at scale, while value models, even though access to ground truth value function is also costly, can resort to observed return as a noisy yet unbiased value estimate.

We have discussed the above distinctions explicitly in the title, abstract (L015), main paper (L096, L252, L366) and Appendix (L788). We believe that given the above clarification, the main concerns are addressed, so we briefly answer each specific point below.

AuthorsOpenReview ↗

Author Response to Reviewer 84ro (Part 2/2)

Weakness 1 and Question 1: Experimental Comparisons with SOTA Reward Models

As argued above, our proposed method is not a reward model. A proper experimental comparison is against other value-based methods and value-free methods in advantage estimation, which we covered in detail. We showed that our generative value model, compared to discriminative value models, is more accurate, more scalable across model sizes, and more robust across random seeds (Figure 2(a)). It achieves stronger relative ranking (Table 3), generalizes better out-of-distribution (Table 4), and as a result directly translates to better downstream RL performance (Figure 2(b)).


Weakness 2: Methodological Novelty

We agree that the generative architecture is a shared insight with the generative reward model literature (L097, L788). Our contribution is instead the diagnosis of the expressivity bottleneck of conventional discriminative value models (Figure 2(a)), and the adaptation of this architectural idea to a policy-conditioned value modeling task through value pretraining, online joint RL training, and in-context conditioning (Section 4).


Question 2: Complexity

We agree that the full setup in the main paper is a bit complex. We chose to keep the full pipeline and not to tune cost effectiveness, as the question this paper asks is whether better value modeling improves credit assignment, and answering it requires the best value model we can build (L298). We provided a full cost analysis and discussed immediate choices to reduce it in Appendix F (L991).

To show that our framework is general and remains effective under a much simpler and lighter setup, we run three new variants: GenAC-Coarse (using ~5x coarser segmentation), GenAC-Light (skipping the SFT stage, early stopping value pretraining, and subsampling critic training batches in joint RL), and GenAC-Coarse-Light (combining both):

AlgorithmSFT DataVP StepsGranularityTotal CostAIME24AMC23MATH500AIME25OlympiadMinervaAverage
GRPO0Response0.48x26.4669.6987.2022.9252.9242.1250.22
VC-PPO200Token1.00x25.8370.7887.5421.6753.5043.0250.39
GenAC✓100Fine2.15x30.0074.5387.4821.8854.1643.3451.90
GenAC-Coarse✓100Coarse1.41x27.5073.2888.5822.5055.1244.1451.85
GenAC-Light40Fine1.26x28.5472.0388.1923.1254.0143.5251.57
GenAC-Coarse-Light40Coarse1.05x27.9273.9188.1521.8854.5543.5951.66

These variants cut the total cost by more than half (down to 1.05x of VC-PPO) and remove any reliance on SFT data, while remaining clearly stronger than all baselines, showing that GenAC does not rely on a heavy pipeline to be effective.


Given that the main concerns stemmed from viewing GenAC as a reward model rather than a value model, we kindly ask the reviewer to reassess the quality of our work in light of these clarifications and the additional results.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Revision

We thank all reviewers for their time and constructive feedback. We have uploaded a revised manuscript incorporating the clarifications and new experiments from our responses, with all modifications highlighted in blue:

  • In Section 3, we clarify the motivational role of the asymptotic complexity separation and frame practical sufficiency under bounded reasoning length as an empirical question.
  • In Section 4.2, we explicitly mark the cold-start SFT stage as optional and outline how cost can be reduced within our framework.
  • In Section 5.4 (Tables 5 and 6, with details in Appendices C and F), we add SFT-free and coarser-segmentation RL variants that cut total compute down to 1.05x VC-PPO while preserving clear gains, report measured GPU wall-clock runtime, and compare against compute- and time-matched extended GRPO.
  • In Section 6, we discuss further cost optimization and extending GenAC to long-horizon, multi-turn agentic RL.

We hope our responses and the revised manuscript address the reviewers' concerns, and we are happy to clarify any remaining questions before the discussion period ends.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.