acceptodds
Under review as a conference paper at ICLR 2027

Riding the Waves of Uncertainty: Distributional Value Learning for LLMs with Verifiable Rewards

Abstract

Reinforcement learning with verifiable rewards (RLVR) trains large language models from feedback on completed responses, yet learning a useful value baseline from sparse terminal outcomes remains difficult. A conventional PPO critic regresses a scalar expected return; a multi-output value objective offers an alternative way to learn from the same feedback. We propose DistRLVR, a distributional actor–critic framework that predicts prefix-conditioned returns and uses their expectation as the PPO baseline. Dual Sample Replacement (dSR) constructs multi-step critic targets from existing trajectories and target-critic predictions, supplying additional supervision without generating additional responses. The predicted quantiles also support optional tail-aware advantage reweighting. Experiments on mathematical reasoning and executable code generation show improvements over PPO and GRPO, including gains over VAPO in mean code reward; instruction-following results further examine transfer to graded programmatic feedback. Scalar-target and capacity controls support an optimization-based interpretation of the gains: multi-output losses can improve value learning without requiring accurate reconstruction of the complete return law. DistRLVR with dSR provides an effective RLVR training recipe, while tail reweighting yields a task-dependent trade-off between average quality and repeated-sampling success. Code is available in the anonymous repository (https://anonymous.4open.science/r/DistRLVR_Anonymous-CD2E/).

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement, RLVR, Distributional RL, Post-Training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UdtHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes DistRLVR, which replaces PPO’s scalar critic with a distributional value predictor while retaining its mean as the default baseline. Dual Sample Replacement (dSR) constructs multi-step targets from existing trajectories and frozen-critic samples, with optional outcome-dependent Tail reweighting. Experiments cover mathematical reasoning, instruction following, code generation, and conversational QA.

Critical strengths

The paper carefully distinguishes optimization benefits from recovering additional distributional information, particularly for binary returns. The rollout-matched comparisons, scalar-objective and capacity controls, longer-response experiments, and cross-model evaluations are valuable.

Critical weaknesses
  1. The additional methodological contribution is limited and not fully isolated. dSR closely follows GMAC’s existing Sample Replacement mechanism, applied across multiple completions. Its gain over distributional TD(0) is only 0.279 percentage points in aggregate accuracy, although the Pass@128 improvement is larger. Moreover, the distributional TD(λ\lambda) control is insufficiently defined. It is unclear how its target differs from the mixture sampled by dSR.

  2. The optimization interpretation lacks direct critic-level evidence. Appendix D.12 samples independent continuations from fixed prefixes, but reports outcome variability rather than comparative prediction errors. This does not establish that distributional training produces a more useful mean baseline or more accurate advantages. Such diagnostics would strengthen the paper without requiring accurate reconstruction of the full return law.

  3. Statistical reporting is incomplete for the decisive comparisons. The detailed three-seed tables primarily cover the original unmatched-rollout PPO comparison. Comparable uncertainty estimates are needed for matched-KK baselines and the strongest code results. The code evaluation also needs clearer dataset-version, training-budget, and baseline-configuration details.

  4. The VAPO code Pass@k values require clarification. Table 29 reports 11.8%, 12.4%, and 15.2% at k=2,4,8k=2,4,8. Under the standard combinatorial estimator using a shared sample pool, concavity requires P8−P4≤2(P4−P2)P_8-P_4\le 2(P_4-P_2), which these values violate. Separate sampling pools could explain a deviation, but the evaluation protocol and uncertainty should then be specified.

Questions for authors
  1. What precisely distinguishes dSR from existing SR(λ\lambda), and how is the distributional TD(λ\lambda) control implemented?
  2. On identical prefixes under a frozen policy, does dSR improve mean-value or advantage-estimation accuracy against independent continuations?
  3. Can the authors clarify the code Pass@k computation and provide multi-seed results with fully specified matched baseline configurations?
Reviewer_zFcvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DistRLVR, a distributional actor–critic method for RLVR. It learns return distributions at each token prefix and uses their means as PPO value baselines. The proposed dual Sample Replacement (dSR) constructs multiple critic training targets from existing trajectories and critic predictions. Experiments on mathematical reasoning and other verifiable tasks show improvements over PPO and GRPO, with additional comparisons to VAPO.

Critical strengths
  1. The paper addresses a central difficulty in RLVR: learning values for response prefixes from sparse final rewards. Its main idea is to use a multi-output critic objective to improve value learning, while keeping the predicted mean as the standard PPO baseline.

  2. To train this critic, the paper adapts Sample Replacement to construct multiple multi-step targets from completed responses and target critic predictions. This gives the critic training targets at different backup depths, rather than relying only on one-step predictions when intermediate rewards are zero.

Critical weaknesses

The main concern is the limited novelty of dSR beyond existing SR. This weakens the paper’s main method contribution.

Questions for authors

1. Limited Novelty of dSR. GMAC has already introduced Sample Replacement (SR) and combined it with distributional critics and PPO. The proposed dSR follows the same target construction process, mainly extending SR to KK responses per prompt and MM targets per prefix. What are the main algorithmic differences between dSR and the original SR? How does dSR differ from existing distributional LLM PPO methods, such as DisPPO: Quantile-Based Distributional Reinforcement Learning for Large Language Models (DisPPO) ?

2. Unclear Performance Gains from dSR. Under the same rollout setting (K=8K=8), dSR improves aggregate accuracy by only 1.318 percentage points over GRPO and 0.279 points over distributional TD(0). These small gains may result from randomness in policy rollouts rather than the proposed method itself. How can the authors show that these improvements are due to dSR rather than differences in sampled responses?

3. Sensitivity to Rollout Count. Appendix D.11 shows that when K=2K=2, SR performs worse than distributional TD(0) on all six mathematical benchmarks. However, when K=8K=8, SR improves performance on five of them. Why does the effectiveness of dSR depend so strongly on the number of sampled responses? Does this suggest that dSR requires sufficient rollout coverage to work well, limiting its use when the rollout budget is small?

Reviewer_UVv2OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a return-distribution critic into LLM PPO training with terminal verification rewards to improve the existing algorithm. The authors propose dual Sample Replacement, which constructs multiple bootstrapped critic targets from existing responses, and provide an optional quantile-based variant. Experiments show performance improvements on mathematical reasoning and tasks with graded rewards.

Critical strengths
  • The paper identifies learning prefix values from terminal feedback as a key RLVR challenge. The proposed modification is straightforward to implement and compatible with the existing training pipeline.

  • The authors provide fairly extensive experimental results, and the reported findings are generally encouraging.

Critical weaknesses
  • The paper's novelty is limited, as its incremental contribution over direct precedents such as GMAC and DisPPO is neither clearly delineated nor adequately compared.
  • The paper lacks direct evaluation of critic-learning quality, making it unclear which training effects account for dSR's gains.
Questions for authors
  • Despite emphasizing uncertainty modeling, the main evidence concerns end-to-end performance. The authors should provide other experimental perspectives to demonstrate the role of each component.
  • It would be preferable for the authors to report uncertainty estimates, as some results appear similar to those of the baselines.
  • The illustration of Appendix F is currently unclear, and the authors should provide more details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.