- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Disentangled Value Estimation (DVE) for rubric-based RL, which conditions a shared critic on one rubric at a time, trains it with rubric-level rewards, and aggregates rubric-specific advantages using the original rubric weights. The method is evaluated against PPO and GRPO on instruction following, chat, and scientific QA benchmarks with Qwen3 models of three sizes.
Critical strengths
- Clear motivation and simple design. The paper identifies a reasonable information mismatch in rubric-based RL, where the reward model sees the rubrics but the critic does not, and addresses it with a lightweight modification to PPO that could be easily integrated into existing frameworks.
Critical weaknesses
- Unclear ablation on disentanglement. DVE disentangles the critic input (one rubric at a time), the critic supervision (rubric-level rewards), and the advantage computation (per-rubric GAE), but the "w/o disentangle" variant in Table 2 collapses all three at once, so it is unclear which component drives the gains. Since disentanglement is the central claim of the paper, this matters for assessing the contribution. Notably, the per-rubric advantage aggregation appears equivalent to standard GAE with the weighted value, so a variant that takes all rubrics as input but predicts rubric-level values (e.g., with multiple heads) would help isolate the effect of rubric-level supervision.
- Uneven gains and incomplete baselines. While DVE shows large gains on MulDimIF, its improvements on several other benchmarks are marginal (e.g., 16.26 vs. 16.22 on Arena-Hard at 1.7B) and come from single training runs without reported variance. In addition, the comparison is limited to vanilla PPO and GRPO. It would be informative to also include multi-reward methods that decouple reward signals, such as GDPO [1], which normalizes each reward separately before aggregating advantages, and PPO variants with improved critics, such as VAPO [2], which the paper itself cites. The computational overhead of running the critic once per rubric is also not reported. [1] Liu, Shih-Yang, et al. "Gdpo: Group reward-decoupled normalization policy optimization for multi-reward rl optimization." arXiv preprint arXiv:2601.05242 (2026). [2] Yue, Yu, et al. "Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks." arXiv preprint arXiv:2504.05118 (2025).
Questions for authors
- Could the authors report results over multiple training seeds, at least for the settings with small margins?
- Since GAE is linear, aggregating per-rubric advantages (Eqs. 3-4) appears equivalent to standard GAE with Σ_k ŵ_k V^k as the value. Could the authors clarify what advantage-level disentanglement contributes beyond the rubric-conditioned critic?
- In Table 2, the "w/o disentangle" variant performs notably below vanilla PPO on MulDimIF at 1.7B (36.17 vs. 47.25). What might explain this?
- What is the training time and memory overhead of DVE compared with PPO, particularly in Chat and Science with many rubrics per query? How does vanilla PPO perform when its critic receives a matched amount of computation?
- Which checkpoint does each entry of Tables 1 and 2 report, for every method including GRPO and the ablation?