acceptodds
Under review as a conference paper at ICLR 2027

RVPO: Risk-Sensitive Alignment via Variance Regularization

Abstract

Current critic-less RLHF methods aggregate multi-objective rewards via an arithmetic mean, leaving them vulnerable to constraint neglect: high-magnitude success in one objective can numerically offset critical failures in others (e.g., safety or formatting), masking low-performing "bottleneck" rewards vital for reliable multi-objective alignment. We propose Reward-Variance Policy Optimization (RVPO), a risk-sensitive framework that penalizes inter-reward variance during advantage aggregation, shifting the objective from "maximize sum" to "maximize consistency." Its LogSumExp (SoftMin) objective can be locally approximated by the mean reward minus an inter-objective variance penalty. We evaluate RVPO on rubric-based medical and scientific reasoning with up to 17 concurrent LLM-judged reward signals (Qwen2.5-3B/7B/14B) and on tool-calling with rule-based constraints (Qwen2.5-1.5B/3B). By discouraging the model from neglecting difficult constraints to exploit easier objectives, RVPO improves overall scores on HealthBench (0.254 vs. 0.207 for GDPO at 14B, averaged across five training seeds) and maintains competitive accuracy on GPQA-Diamond while reducing the late-stage degradation observed in other multi-reward methods, demonstrating that variance regularization mitigates constraint neglect across model scales without sacrificing general capabilities.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, RLHF, large language models, multi-objective optimization, reward aggregation, risk-sensitive optimization, variance regularization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_w8EUOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work presents RVPO. Authors suggest that aggregating multi-objective rewards via an arithmetic mean is not always useful because it can numerically offset critical failures in others (e.g., safety or formatting). Based on this motivation, the authors propose RVPO, which uses soft-min aggregation to emphasize lower reward scores, implicitly penalizing variance across reward objectives.

Critical strengths
  1. The proposed method is simple and can be easily integrated into existing critic-less RL frameworks without introducing additional architectural complexity.
  2. The authors provide a mathematical interpretation of SoftMin aggregation as an implicit variance penalty, which helps justify the proposed approach.
Critical weaknesses

My main concern is that the experimental results do not sufficiently support the motivation behind the proposed method. In abstract and Introduction the authors argue that arithmetic mean is not always useful because it can numerically offset critical failures in others (e.g., safety or formatting). But the paper doesn't address how safety or formatting improves by using RVPO. Although the authors report improvements in training dynamics, it remains unclear whether these improvements translate into better model behavior on unseen examples. Another concern is the limited and inconsistent empirical improvements. The choice of hyperparameters, particularly the scheduling of (k), appears largely heuristic, without a strong theoretical justification. Furthermore, the authors evaluate three RVPO variants (RVPO-min, RVPO-explicit, and RVPO), but none consistently outperforms the baselines across all settings in Section 6.1. In Section 6.2, GDPO achieves comparable or slightly better final performance than RVPO, despite the reported improvements in training dynamics. These results make it difficult to identify the practical advantages of RVPO over existing methods. For Minor Issues, the paper could benefit from clearer presentation and more detailed explanations. Some technical terms and experimental settings are introduced without sufficient explanation, making certain sections difficult to follow. Additional illustrations, more informative tables, and clearer definitions of key concepts would help readers better understand the proposed method and interpret the experimental results.

Questions for authors

See the Weakness

2 comments
AuthorsOpenReview ↗

Held-out behavior, baseline comparisons, and scope of claims

Thank you for the review and the positive feedback on the method’s mathematical interpretation and simple implementation.

Held-out behavior. HealthBench, GPQA-Diamond and BFCL-v3 are disjoint from training; HealthBench uses its own physician-written rubrics. These evaluations address generalization, but we agree that they do not establish a safety benefit. Safety was an illustrative example of a critical constraint, not an evaluated outcome. We will make that distinction explicit.

For formatting, held-out BFCL error rates are already low across GRPO, GDPO and RVPO: 0.44–0.48% at 1.5B and 0.13–0.22% at 3B. Figure 3 shows faster format learning during training, without an established held-out format gain.

Comparisons and k. Table 1 reports the highest final mean for RVPO at 3B and 14B, while GRPO-implicit and RVPO-explicit are competitive at 7B. We will make these comparisons clearer without implying that RVPO leads in every setting. We will also identify RVPO-min and RVPO-explicit explicitly as the hard-min and mean–variance ablations.

We selected the Medicine schedule using in-domain 7B evaluation and transferred it unchanged to 3B and 14B. The additional static-k probe does not establish an advantage for annealing. Appendix A.5 provides a training-statistics heuristic, k∗k^{*}, for estimating the scale of k, but it is not a validated selection procedure for new tasks. We will clarify its scope, add the normalization illustration, and explain the definitions in section 3 more clearly.

Thank you for the response. My biggest concern was the claim of the safety/formatting and given the explanation my concerns are partially resolved. Therefore I increase the score. Thank you

Reviewer_t4TLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents RVPO, a straightforward alternative to GDPO. The main change is that, instead of averaging the normalised reward from multiple objective channels, the paper uses softmin.

Critical strengths

(1) The paper is well-written; the method is clearly demonstrated, and it can be adapted with minimal changes. (2) The experimental results stand, and the limitations are frankly written.

Critical weaknesses
  1. (In section 5.2, first paragraph) The author mentioned that hyperparameters are selected based on final checkpoint performance, without a held-out dev set. Yet, in Table 4, the results are very noisy and sensitive to hyperparameter changes. k=1 best checkpoint score is 0.194, k=2 best checkpoint score is 0.225, but for annealed curricula the best score for k=1 to 2 is 0.143. These results suggest the improvement may be random and due to noise.
  2. The authors claim RVPO targets the bottleneck by penalizing variance across objectives. If you only look at the softmin of the multiple objective scores, it makes sense. But with the consideration that these scores are normalised within the group first, there could be another explanation: RVPO is mostly penalizing failures. An example: Z is normalized within the group, for a binary rubric a failure on an easy item (e.g. 15/16 pass) gets a very negative Z and dominates the softmin, while a rare success on a hard item (1/16 pass) gets a large positive Z that softmin normally just ignores. So RVPO seems to push the model to not fail on things it usually gets right, rather than improve on the hard items. This explanation fits the fact that the gain in the final checkpoint is more stable (in the main exp table). If possible, please help to provide per-objective scores during the training of RVPO and GDPO. If this is addressed, I will consider improve my score.
Questions for authors

See the weaknesses in my main problems. Also, in the abstract, the paper claims a large improvement by comparing RVPO with the worst baseline, GDPO. Could it be possible to change it to a better baseline?

1 comment
AuthorsOpenReview ↗

Normalization, per-objective trajectories, and choice of k

Thank you for the review and the helpful normalization example.

Normalization. We checked it on held-out samples, and with G=16 a failure when 15/16 responses pass gets Z ≈ −3.75 while a success when 1/16 pass gets Z ≈ +3.75. Holding other scores fixed, SoftMin is more sensitive to the former. This can help explain why it can preserve frequently satisfied criteria, rather than simply target the criteria with the lowest absolute success rates.

On 256 held-out RaR-Medicine prompts with G=16 responses per prompt from the base 7B model, we recomputed aggregate advantages offline:

criterion passed byGDPO advantage gapRVPO gap, k=2
1/16 responses+2.14+0.89
15/16 responses+2.45+5.07

The gap is E[A | pass] − E[A | fail].

Per-objective trajectories. Logged training pass rates for Essential, Important, Optional and Pitfall criteria do not show consistent separation between methods. We also tracked held-out criteria grouped by base-model pass rate: hard <25%, mid 25–75%, easy >75%. At the final 14B checkpoint, across 10 matched seeds:

criteriaGDPORVPOpaired difference [95% CI]
hard0.0880.143+0.056 [−0.055, +0.166]
mid0.2310.379+0.148 [−0.132, +0.428]
easy0.3010.486+0.186 [−0.158, +0.529]

RVPO has higher mean scores, but the intervals include zero. These don't show preferred treatment of hard criteria. We found at temperature 1 that the final checkpoints also hit the token limit more often than training examples (64% for GDPO, 39% for RVPO at 14B), making the criterion hard to separate from changes in generation behavior. We will add pass-rate curves for training and held-out prompts and explain this limitation in the revision

Choice of k. Table 4 is an exploratory single-run sweep. The 0.143 entry is the 1.0->2.0 schedule; the reported Medicine configuration is 0.5->2.0. The nearby poor results still show sensitivity. Selection used in-domain 7B evaluation without a separate development set, followed by transfer to 3B and 14B. A.5's k* heuristic estimates a scale, and is not a validated selection procedure for new tasks.

Abstract comparison. We will broaden the comparison beyond GDPO. At 14B, Table 1 reports best/final means of 0.254/0.217 for RVPO, 0.243/0.157 for RVPO-min (its hard-min ablation), and 0.242/0.143 for GRPO-implicit. At 7B, GRPO-implicit and RVPO-explicit have slightly higher final means than RVPO (0.160 versus 0.152). GDPO provides the direct mean-aggregation comparison, while GRPO-implicit uses a holistic judge score rather than separate rewards. We will clarify and report the strongest comparisons without implying that RVPO leads in every setting.

Reviewer_T3mAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies an important multi-reward reinforcement learning and proposes a simple method called RVPO based on GRPO and GDPO. The main idea is to introduce a variance penalty to prevent a high score on one reward from compensating for poor performance on other rewards. The method is straightforward: it replaces mean reward aggregation with the SoftMin. RVPO generally improves over GDPO on HealthBench, especially in terms of training stability, while the improvements on other benchmarks are relatively limited.

Critical strengths

The proposed RVPO is a simple and effective extension of GDPO. Its main strength is that the method is easy to understand and implement. The experiments also show some improvements in multi-reward learning, especially in reducing late-stage performance degradation on HealthBench.

Critical weaknesses

The performance of RVPO appears to be highly sensitive to the choice of k, but the paper does not provide a reliable way to select this hyperparameter for a new task. Moreover, k is selected based on in-domain evaluation without an independent validation set. Most importantly, some improvements are also not very large compared with the strongest baselines. I am concerned about how robust these improvements would be on new tasks.

RVPO penalizes reward variance by giving more attention to reward with lower Z-scores. However, a lower Z-score does not necessarily indicate a more important constraint. For example, a response may satisfy critical safety constraints but receive a low score on a noisy or less important reward. In this case, RVPO may penalize this response too heavily, potentially favoring balanced rewards over satisfying critical constraints. So this raises concerns about whether RVPO can truly address constraint neglect.

Questions for authors

Could the authors examine the behavior under different levels of reward sparsity? For example, by varying the overall success rate of a binary safety or formatting reward from 5% to 95%, and checking whether RVPO ever ranks constraint-satisfying responses below constraint-violating ones.

The paper considers noise in a single reward channel. What happens when multiple rewards are conflicting or have correlated noise?

1 comment
AuthorsOpenReview ↗

Reward sparsity, priority, and correlated noise

Thank you for the review and for recognizing the method’s simplicity and effectiveness.

Sparsity and priority. We ran the experiment you requested: one binary constraint with success rate from 5% to 95%, along with nine independent binary rewards, using the trainer's normalization.

  • At 5% success, RVPO (k=2) has about a tenth of GDPO's advantage gap, E[A | pass] − E[A | fail], and ranks a satisfying response below a failing response in 46% of these pairs, versus 21% for GDPO.
  • At 95% success, its gap is about twice GDPO's, with 4% ranking inversions, versus 21% for GDPO.

When other rewards conflict with the 5%-success constraint, its advantage gap is −0.79 under RVPO (k=2), versus −0.29 under GDPO. These simulations show that RVPO can underweight rarely satisfied constraints, especially when rewards conflict.

With independent rewards and k=0.5, the rare constraint retains about half of GDPO's gap, but this does not prove that annealing prevents the problem. In the additional five-seed static-k comparison, the annealed-minus-static-k=5 hard-criterion difference was +0.040 [95% CI −0.206, +0.286], with no established advantage for either setting.

We agree that low Z does not imply importance. Post-normalization weights can encode declared priorities, but does not guarantee constraint satisfaction. We will add the sparsity and conflict results and make this distinction explicit. The k-selection procedure also remains a limitation.

Multiple and correlated noise. At 3B, we corrupted two channels using independent or shared attempted flip masks with probability 0.25. Normalized aggregation weight on the corrupted pair (1 = equal share), mean ± sample SD over five seeds:

conditionRVPOhard-min
clean0.996 ± 0.0020.899 ± 0.013
independent flips1.101 ± 0.0041.530 ± 0.022
shared flips1.082 ± 0.0041.446 ± 0.016

RVPO concentrates less weight on the noisy channels than hard-min in these tests. Shared flips produced lower shares than independent flips; neither method identifies the channels as unreliable.

HealthBench corrupted−clean differences, mean ± sample SD over five complete seed pairs:

methodΔbest independent / sharedΔfinal independent / shared
GDPO+0.003 ± 0.010 / −0.006 ± 0.012−0.016 ± 0.069 / +0.019 ± 0.076
RVPO+0.002 ± 0.025 / +0.007 ± 0.017+0.001 ± 0.064 / +0.038 ± 0.058
hard-min+0.001 ± 0.022 / −0.006 ± 0.013+0.033 ± 0.076 / +0.011 ± 0.043

None are statistically significant (p ≥ 0.22), so these small-sample results do not establish general robustness to correlated judge errors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.