RVPO: Risk-Sensitive Alignment via Variance Regularization

Current critic-less RLHF methods aggregate multi-objective rewards via an arithmetic mean, leaving them vulnerable to constraint neglect: high-magnitude success in one objective can numerically offset critical failures in others (e.g., safety or formatting), masking low-performing "bottleneck" rewards vital for reliable multi-objective alignment. We propose Reward-Variance Policy Optimization (RVPO), a risk-sensitive framework that penalizes inter-reward variance during advantage aggregation, shifting the objective from "maximize sum" to "maximize consistency." Its LogSumExp (SoftMin) objective can be locally approximated by the mean reward minus an inter-objective variance penalty. We evaluate RVPO on rubric-based medical and scientific reasoning with up to 17 concurrent LLM-judged reward signals (Qwen2.5-3B/7B/14B) and on tool-calling with rule-based constraints (Qwen2.5-1.5B/3B). By discouraging the model from neglecting difficult constraints to exploit easier objectives, RVPO improves overall scores on HealthBench (0.254 vs. 0.207 for GDPO at 14B, averaged across five training seeds) and maintains competitive accuracy on GPQA-Diamond while reducing the late-stage degradation observed in other multi-reward methods, demonstrating that variance regularization mitigates constraint neglect across model scales without sacrificing general capabilities.