- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This work studies how to mitigate reward hacking in inference-time alignment methods, such as Best-of-n (BoN). To this end, the authors leverage an ensemble of reward models and give each candidate response a score based on a value-at-risk rule. Such a rule encourages responses that are favored by a large fraction of the reward models. The authors compare this method to BoN / Soft BoN / Best-of-Poisson with a single reward and a tuned -parameter using the HedgeTune algorithm of Khalaf et al. (2025)., as well as using an average over reward models and other risk notions. Their experimental results show modest gains, with the highest being 10% on MATH and the lowest being negative 5% on a coding task, compared to BoN with tuned , at the computational expense of using 14 reward models instead of one.
Critical strengths
The paper evaluate their method on a suite of benchmarks with a wide range of reward models and ablations, comparing different inference-time methods and risk notions. The idea is intuitive and straightforward to follow, and the topic is timely and relevant.
Critical weaknesses
Using reward ensembles is a standard way to mitigate reward hacking and has been thoroughly studied (Coste et al., 2024; Eisenstein et al., 2024). I encourage the authors to engage more with the wide literature on risk-sensitive / pessimistic regularization of reward scores in the context of inference-time alignment. Additionally, the insights in this work are discussed thoroughly in the references above. I believe the paper would benefit from an empirical comparison with those work. Hence, I find the contributions in the current version limited.
It also seems that UWO (uncertainty-penalized) for reward aggregation (Coste et al., 2024) performs just as well as the proposed method, as shown in the appendix. I would encourage these ablations being moved to the main text since they are extremely relevant.
Useful references include: Zou, M., et al. (2026). DARC: Disagreement-Aware Alignment via Risk-Constrained Decoding. ICML 2026. Hsu, H. et al. (2026). Best-of-Tails: Bridging Optimism and Pessimism in Inference-Time Alignment
Questions for authors
- Can you give more details on how and when your method beats UWO?
- Have you tried experimenting with choosing subsets of reward models? Perhaps using the full pool of 14 reward models is not the right way to go. You can do a global optimization, this might improve your performance since you discard obviously bad rewards. This is for future work; I do not expect this to be done during rebuttals.
2 comments
Thanks for your response, I appreciate it! Because the two methods perform similarly, I am not yet convinced of the utility of your framework or its novelty. I do appreciate the insights you have on the limits of test-time scaling and the need for better reward ensembles, but I do not think your paper, in its current form, illustrates these points properly. As such, I will maintain my score.
We thank the reviewer for the constructive feedback. We address each concern below.
W2 / Q1: Comparison with UWO and Conditions for VaR's Gains. We agree that UWO is a strong baseline and that its results deserve greater prominence. We will move the UWO comparison from the appendix into the main text (Table 2). At the pooled BoN level, VaR achieves 58.36% versus 58.29% for UWO; the difference is not statistically significant (95% CI [−0.86,+0.98]; Table 3). We also discuss this limitation in our response to Reviewer F8xQ (W2/Q2).
The methods differ in how they treat disagreement: UWO penalizes score dispersion through zˉ−λsd(z), whereas VaR directly optimizes a lower quantile. This makes VaR sensitive to the lower-tail distribution rather than only its mean and variance. Empirically, VaR exceeds UWO on GPQA (+0.97 pp), MATH (+0.29 pp), and IFEval (+0.75 pp), but underperforms on MMLU-Pro (−0.89 pp) and MBPP-Plus (−0.76 pp). These differences are descriptive rather than statistically established.
MATH has the lowest residual correlation (0.407) and highest top-1 disagreement (0.888; Table 2), consistent with greater error diversity. However, MBPP-Plus demonstrates that diversity alone is insufficient: a dominant individual RM can be suppressed by pessimistic aggregation. We will clarify this condition in Section 5 rather than claim that VaR consistently outperforms UWO.
W1: Novelty Relative to Ensemble and Risk-Sensitive Alignment Literature. We appreciate the references to Coste et al. (ICLR 2024), Eisenstein et al. (COLM 2024), DARC (Zou et al., 2026), and Best-of-Tails (Hsu et al., 2026). We acknowledge that reward-model ensembling, uncertainty penalties, and risk-sensitive inference-time selection have been studied previously.
Our contribution is not ensembling or pessimism in isolation, but: (i) a unified comparison of optimizer hedging and reward aggregation under the KL-regularized objective (Eq. 4), evaluated across BoN, SBoN, and BoP; (ii) a systematic evaluation of standardized low-quantile aggregation against Uniform, UWO, and worst-case aggregation; and (iii) a quantitative analysis of error diversity and evaluator competence, explaining both improvements and the MBPP-Plus failure (Table 2).
DARC studies disagreement-aware risk-constrained decoding, while Best-of-Tails examines optimism–pessimism tradeoffs in inference-time alignment. Our focus is the interaction between reward aggregation, optimizer calibration, and cross-model error structure across multiple selection mechanisms. We will expand Section 6 to clarify these distinctions and avoid overstating methodological novelty relative to prior risk-sensitive approaches.
Q2: Reward Model Subset Selection. Figure 7 (Appendix G) shows that subsets of approximately 5–6 reward models recover most of the full-ensemble gain, indicating that all 14 models are unnecessary. However, these results do not establish that optimizing model subsets improves performance. Selecting subsets using calibration-set accuracy and complementary error patterns is a promising extension, which we will identify as future work.