acceptodds
Under review as a conference paper at ICLR 2027

Disentangled Value Estimation: Rethinking Scalar Critics in Rubric-Based RL

Abstract

Value estimation is crucial for many reinforcement learning algorithms used to optimize large language models (LLMs). In critic-based methods such as Proximal Policy Optimization (PPO), an additional critic model predicts expected future outcomes for each generated token based on intermediate generation states, and this value is used to compute the advantages that guide policy optimization. Such value estimation is relatively well defined when the reward specification is fixed and consistent, but becomes more challenging in open-ended tasks where evaluation criteria vary across queries. We study this problem in RL with rubric rewards, where each query is associated with multiple query-specific criteria whose scores are collapsed into a scalar reward. A standard critic observes only the query and response prefix and is supervised by this aggregated reward, requiring it to implicitly recover the underlying reward specification before estimating values. To address this, we propose Disentangled Value Estimation (DVE), which uses a shared critic conditioned on one rubric at a time to estimate rubric-specific values, computes token-level advantages for each rubric, and aggregates them using the original rubric weights. Across five benchmarks spanning instruction following, chat, and scientific question answering, DVE outperforms the vanilla PPO and GRPO in 13 of 15 model-benchmark combinations. Further analysis shows that DVE achieves lower value prediction error, higher explained variance, stronger response-level ranking consistency, and more targeted token-level credit assignment. Code is available at Anonymous Repo.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning, Rubrics-as-reward, Value Estimation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hdH5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Disentangled Value Estimation (DVE) for rubric-based RL, which conditions a shared critic on one rubric at a time, trains it with rubric-level rewards, and aggregates rubric-specific advantages using the original rubric weights. The method is evaluated against PPO and GRPO on instruction following, chat, and scientific QA benchmarks with Qwen3 models of three sizes.

Critical strengths
  1. Clear motivation and simple design. The paper identifies a reasonable information mismatch in rubric-based RL, where the reward model sees the rubrics but the critic does not, and addresses it with a lightweight modification to PPO that could be easily integrated into existing frameworks.
Critical weaknesses
  1. Unclear ablation on disentanglement. DVE disentangles the critic input (one rubric at a time), the critic supervision (rubric-level rewards), and the advantage computation (per-rubric GAE), but the "w/o disentangle" variant in Table 2 collapses all three at once, so it is unclear which component drives the gains. Since disentanglement is the central claim of the paper, this matters for assessing the contribution. Notably, the per-rubric advantage aggregation appears equivalent to standard GAE with the weighted value, so a variant that takes all rubrics as input but predicts rubric-level values (e.g., with multiple heads) would help isolate the effect of rubric-level supervision.
  2. Uneven gains and incomplete baselines. While DVE shows large gains on MulDimIF, its improvements on several other benchmarks are marginal (e.g., 16.26 vs. 16.22 on Arena-Hard at 1.7B) and come from single training runs without reported variance. In addition, the comparison is limited to vanilla PPO and GRPO. It would be informative to also include multi-reward methods that decouple reward signals, such as GDPO [1], which normalizes each reward separately before aggregating advantages, and PPO variants with improved critics, such as VAPO [2], which the paper itself cites. The computational overhead of running the critic once per rubric is also not reported. [1] Liu, Shih-Yang, et al. "Gdpo: Group reward-decoupled normalization policy optimization for multi-reward rl optimization." arXiv preprint arXiv:2601.05242 (2026). [2] Yue, Yu, et al. "Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks." arXiv preprint arXiv:2504.05118 (2025).
Questions for authors
  1. Could the authors report results over multiple training seeds, at least for the settings with small margins?
  2. Since GAE is linear, aggregating per-rubric advantages (Eqs. 3-4) appears equivalent to standard GAE with Σ_k ŵ_k V^k as the value. Could the authors clarify what advantage-level disentanglement contributes beyond the rubric-conditioned critic?
  3. In Table 2, the "w/o disentangle" variant performs notably below vanilla PPO on MulDimIF at 1.7B (36.17 vs. 47.25). What might explain this?
  4. What is the training time and memory overhead of DVE compared with PPO, particularly in Chat and Science with many rubrics per query? How does vanilla PPO perform when its critic receives a matched amount of computation?
  5. Which checkpoint does each entry of Tables 1 and 2 report, for every method including GRPO and the ablation?
Reviewer_azwkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

DVE trains a shared critic to estimate values separately for each rubric, then combines the corresponding advantages using the original reward weights. Experiments compare it with PPO and GRPO across five benchmarks and three model sizes, alongside analyses of critic prediction quality.

Critical strengths

The method is clearly described and shows useful gains, particularly on instruction following. The all-rubrics scalar-critic ablation and evaluation on fixed response sets are helpful controls.

Critical weaknesses
  • The novelty seems limited. The main change is rubric-conditioned critic supervision; because GAE is linear, computing advantages separately and then combining them does not introduce a new estimator. The paper needs a clearer account of what this adds beyond existing conditioned and decomposed value methods.

  • The computational trade-off is unclear. The critic processes responses separately for each rubric, with a median of 27 rubrics per query in Chat and Science, yet no runtime, memory, or equal-compute comparison is reported. Without these measurements, it is difficult to judge whether the gains justify the additional work.

Questions for authors
  • What distinguishes DVE from existing conditioned or decomposed critics, and how would it compare with a shared-backbone critic that predicts all rubric values in one pass?
  • Can you report training time and memory usage, and compare DVE with the all-rubrics scalar critic under the same total compute budget?
Reviewer_rrYNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DVE trains a shared critic conditioned on one query-specific rubric at a time, computes rubric-level GAE estimates, and aggregates them with the original reward weights for PPO updates. The paper evaluates three model sizes across instruction-following, chat, and science tasks and reports gains in 13 of 15 model–benchmark combinations.

Critical strengths

Explicit rubric conditioning and per-rubric supervision make useful reward information available to the critic. The evaluation spans several model sizes and includes an all-rubrics scalar-critic ablation, with particularly substantial instruction-following gains.

Critical weaknesses
  1. GAE is linear, so aggregating rubric-level advantages is equivalent to ordinary GAE using the weighted aggregate critic. With gamma=lambda=1, the result is the scalar return minus the aggregate value. The distinct contribution is critic representation and supervision; the advantage computation itself does not retain a separate multi-aspect signal for the actor (pp. 3–4, 25).
  2. The existing ablation does not separate per-rubric supervision from additional critic computation and input factorization. Moreover, fixed-response value diagnostics use training-pool prompts and do not directly establish causal token-level credit assignment. A compute-matched critic comparison and held-out-prompt diagnostics would strengthen the explanation for the gains (pp. 6, 23–24).
Questions for authors
  1. Can you make the aggregate-GAE equivalence explicit and clarify which claimed benefits depend on the critic training objective rather than the advantage formula?
  2. What are the critic token/FLOP or wall-clock costs relative to the all-rubrics scalar ablation, and do the value-estimation improvements persist on held-out prompts/rubrics?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.