- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes DistRLVR, which replaces PPO’s scalar critic with a distributional value predictor while retaining its mean as the default baseline. Dual Sample Replacement (dSR) constructs multi-step targets from existing trajectories and frozen-critic samples, with optional outcome-dependent Tail reweighting. Experiments cover mathematical reasoning, instruction following, code generation, and conversational QA.
Critical strengths
The paper carefully distinguishes optimization benefits from recovering additional distributional information, particularly for binary returns. The rollout-matched comparisons, scalar-objective and capacity controls, longer-response experiments, and cross-model evaluations are valuable.
Critical weaknesses
-
The additional methodological contribution is limited and not fully isolated. dSR closely follows GMAC’s existing Sample Replacement mechanism, applied across multiple completions. Its gain over distributional TD(0) is only 0.279 percentage points in aggregate accuracy, although the Pass@128 improvement is larger. Moreover, the distributional TD() control is insufficiently defined. It is unclear how its target differs from the mixture sampled by dSR.
-
The optimization interpretation lacks direct critic-level evidence. Appendix D.12 samples independent continuations from fixed prefixes, but reports outcome variability rather than comparative prediction errors. This does not establish that distributional training produces a more useful mean baseline or more accurate advantages. Such diagnostics would strengthen the paper without requiring accurate reconstruction of the full return law.
-
Statistical reporting is incomplete for the decisive comparisons. The detailed three-seed tables primarily cover the original unmatched-rollout PPO comparison. Comparable uncertainty estimates are needed for matched- baselines and the strongest code results. The code evaluation also needs clearer dataset-version, training-budget, and baseline-configuration details.
-
The VAPO code Pass@k values require clarification. Table 29 reports 11.8%, 12.4%, and 15.2% at . Under the standard combinatorial estimator using a shared sample pool, concavity requires , which these values violate. Separate sampling pools could explain a deviation, but the evaluation protocol and uncertainty should then be specified.
Questions for authors
- What precisely distinguishes dSR from existing SR(), and how is the distributional TD() control implemented?
- On identical prefixes under a frozen policy, does dSR improve mean-value or advantage-estimation accuracy against independent continuations?
- Can the authors clarify the code Pass@k computation and provide multi-seed results with fully specified matched baseline configurations?