- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This work presents RVPO. Authors suggest that aggregating multi-objective rewards via an arithmetic mean is not always useful because it can numerically offset critical failures in others (e.g., safety or formatting). Based on this motivation, the authors propose RVPO, which uses soft-min aggregation to emphasize lower reward scores, implicitly penalizing variance across reward objectives.
Critical strengths
- The proposed method is simple and can be easily integrated into existing critic-less RL frameworks without introducing additional architectural complexity.
- The authors provide a mathematical interpretation of SoftMin aggregation as an implicit variance penalty, which helps justify the proposed approach.
Critical weaknesses
My main concern is that the experimental results do not sufficiently support the motivation behind the proposed method. In abstract and Introduction the authors argue that arithmetic mean is not always useful because it can numerically offset critical failures in others (e.g., safety or formatting). But the paper doesn't address how safety or formatting improves by using RVPO. Although the authors report improvements in training dynamics, it remains unclear whether these improvements translate into better model behavior on unseen examples. Another concern is the limited and inconsistent empirical improvements. The choice of hyperparameters, particularly the scheduling of (k), appears largely heuristic, without a strong theoretical justification. Furthermore, the authors evaluate three RVPO variants (RVPO-min, RVPO-explicit, and RVPO), but none consistently outperforms the baselines across all settings in Section 6.1. In Section 6.2, GDPO achieves comparable or slightly better final performance than RVPO, despite the reported improvements in training dynamics. These results make it difficult to identify the practical advantages of RVPO over existing methods. For Minor Issues, the paper could benefit from clearer presentation and more detailed explanations. Some technical terms and experimental settings are introduced without sufficient explanation, making certain sections difficult to follow. Additional illustrations, more informative tables, and clearer definitions of key concepts would help readers better understand the proposed method and interpret the experimental results.
Questions for authors
See the Weakness
2 comments
Thank you for the response. My biggest concern was the claim of the safety/formatting and given the explanation my concerns are partially resolved. Therefore I increase the score. Thank you
Held-out behavior, baseline comparisons, and scope of claims
Thank you for the review and the positive feedback on the method’s mathematical interpretation and simple implementation.
Held-out behavior. HealthBench, GPQA-Diamond and BFCL-v3 are disjoint from training; HealthBench uses its own physician-written rubrics. These evaluations address generalization, but we agree that they do not establish a safety benefit. Safety was an illustrative example of a critical constraint, not an evaluated outcome. We will make that distinction explicit.
For formatting, held-out BFCL error rates are already low across GRPO, GDPO and RVPO: 0.44–0.48% at 1.5B and 0.13–0.22% at 3B. Figure 3 shows faster format learning during training, without an established held-out format gain.
Comparisons and k. Table 1 reports the highest final mean for RVPO at 3B and 14B, while GRPO-implicit and RVPO-explicit are competitive at 7B. We will make these comparisons clearer without implying that RVPO leads in every setting. We will also identify RVPO-min and RVPO-explicit explicitly as the hard-min and mean–variance ablations.
We selected the Medicine schedule using in-domain 7B evaluation and transferred it unchanged to 3B and 14B. The additional static-k probe does not establish an advantage for annealing. Appendix A.5 provides a training-statistics heuristic, k∗, for estimating the scale of k, but it is not a validated selection procedure for new tasks. We will clarify its scope, add the normalization illustration, and explain the definitions in section 3 more clearly.