acceptodds
Under review as a conference paper at ICLR 2027

Curbing the Winner’s Curse: Reward Aggregation Mitigates Inference-Time Reward Hacking

Abstract

Inference-time alignment methods such as Best-of- steer a language model with a reward model, but because every reward model is an imperfect proxy, optimizing it too hard backfires: true quality rises, peaks, and then collapses, a failure known as inference-time reward hacking. A recent line of work hedges the optimizer, calibrating how much to sample so as to stop at the peak. We ask whether this is the most effective lever available, and show that it is not. Separating how far to optimize from what to optimize exposes two label-free levers. The optimizer lever is nearly exhausted: the ideal amount of optimization is strongly per-prompt, so an oracle with labels would gain roughly points, but this signal is label-gated and our best label-free per-prompt selector recovers under half a point. The decisive lever is to fix the reward: given a suite of reward models, we aggregate their per-prompt-standardized scores by a low quantile across models rather than their mean, a value-at-risk (VaR) rule that better aligns the proxy and flattens the hacking curve. Across five verifiable domains, three inference-time mechanisms (BoN, SBoN, and BoP), and reward models, VaR aggregation raises expected accuracy by points over hedging a typical single model, an order of magnitude more than the optimizer lever yields. A mechanistic, falsifiable account explains when it helps: aggregation rejects a response favored by one model's blind spot precisely when reward-model errors are diverse, which predicts both its gains on reasoning and QA and its failure on code, where errors are correlated. It is also immediately deployable: training-free, with no added parameters, a two-line reduction over reward-model scores, and a small, diverse set of models recovers the full gain, keeping it compute- and memory-light. These results recast inference-time reward hacking as a reward-model deficiency to be repaired, not merely an optimization budget to be tuned.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
reward hacking, best-of-n sampling, reward models, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7FnzOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This work studies how to mitigate reward hacking in inference-time alignment methods, such as Best-of-n (BoN). To this end, the authors leverage an ensemble of reward models and give each candidate response a score based on a value-at-risk rule. Such a rule encourages responses that are favored by a large fraction of the reward models. The authors compare this method to BoN / Soft BoN / Best-of-Poisson with a single reward and a tuned nn-parameter using the HedgeTune algorithm of Khalaf et al. (2025)., as well as using an average over reward models and other risk notions. Their experimental results show modest gains, with the highest being 10% on MATH and the lowest being negative 5% on a coding task, compared to BoN with tuned nn, at the computational expense of using 14 reward models instead of one.

Critical strengths

The paper evaluate their method on a suite of benchmarks with a wide range of reward models and ablations, comparing different inference-time methods and risk notions. The idea is intuitive and straightforward to follow, and the topic is timely and relevant.

Critical weaknesses

Using reward ensembles is a standard way to mitigate reward hacking and has been thoroughly studied (Coste et al., 2024; Eisenstein et al., 2024). I encourage the authors to engage more with the wide literature on risk-sensitive / pessimistic regularization of reward scores in the context of inference-time alignment. Additionally, the insights in this work are discussed thoroughly in the references above. I believe the paper would benefit from an empirical comparison with those work. Hence, I find the contributions in the current version limited.

It also seems that UWO (uncertainty-penalized) for reward aggregation (Coste et al., 2024) performs just as well as the proposed method, as shown in the appendix. I would encourage these ablations being moved to the main text since they are extremely relevant.

Useful references include: Zou, M., et al. (2026). DARC: Disagreement-Aware Alignment via Risk-Constrained Decoding. ICML 2026. Hsu, H. et al. (2026). Best-of-Tails: Bridging Optimism and Pessimism in Inference-Time Alignment

Questions for authors
  • Can you give more details on how and when your method beats UWO?
  • Have you tried experimenting with choosing subsets of reward models? Perhaps using the full pool of 14 reward models is not the right way to go. You can do a global optimization, this might improve your performance since you discard obviously bad rewards. This is for future work; I do not expect this to be done during rebuttals.
2 comments

We thank the reviewer for the constructive feedback. We address each concern below.

W2 / Q1: Comparison with UWO and Conditions for VaR's Gains. We agree that UWO is a strong baseline and that its results deserve greater prominence. We will move the UWO comparison from the appendix into the main text (Table 2). At the pooled BoN level, VaR achieves 58.36% versus 58.29% for UWO; the difference is not statistically significant (95% CI [−0.86,+0.98][-0.86,+0.98]; Table 3). We also discuss this limitation in our response to Reviewer F8xQ (W2/Q2).

The methods differ in how they treat disagreement: UWO penalizes score dispersion through zˉ−λ sd(z)\bar{z}-\lambda\,\mathrm{sd}(\mathbf{z}), whereas VaR directly optimizes a lower quantile. This makes VaR sensitive to the lower-tail distribution rather than only its mean and variance. Empirically, VaR exceeds UWO on GPQA (+0.97 pp), MATH (+0.29 pp), and IFEval (+0.75 pp), but underperforms on MMLU-Pro (−0.89-0.89 pp) and MBPP-Plus (−0.76-0.76 pp). These differences are descriptive rather than statistically established.

MATH has the lowest residual correlation (0.407) and highest top-1 disagreement (0.888; Table 2), consistent with greater error diversity. However, MBPP-Plus demonstrates that diversity alone is insufficient: a dominant individual RM can be suppressed by pessimistic aggregation. We will clarify this condition in Section 5 rather than claim that VaR consistently outperforms UWO.

W1: Novelty Relative to Ensemble and Risk-Sensitive Alignment Literature. We appreciate the references to Coste et al. (ICLR 2024), Eisenstein et al. (COLM 2024), DARC (Zou et al., 2026), and Best-of-Tails (Hsu et al., 2026). We acknowledge that reward-model ensembling, uncertainty penalties, and risk-sensitive inference-time selection have been studied previously.

Our contribution is not ensembling or pessimism in isolation, but: (i) a unified comparison of optimizer hedging and reward aggregation under the KL-regularized objective (Eq. 4), evaluated across BoN, SBoN, and BoP; (ii) a systematic evaluation of standardized low-quantile aggregation against Uniform, UWO, and worst-case aggregation; and (iii) a quantitative analysis of error diversity and evaluator competence, explaining both improvements and the MBPP-Plus failure (Table 2).

DARC studies disagreement-aware risk-constrained decoding, while Best-of-Tails examines optimism–pessimism tradeoffs in inference-time alignment. Our focus is the interaction between reward aggregation, optimizer calibration, and cross-model error structure across multiple selection mechanisms. We will expand Section 6 to clarify these distinctions and avoid overstating methodological novelty relative to prior risk-sensitive approaches.

Q2: Reward Model Subset Selection. Figure 7 (Appendix G) shows that subsets of approximately 5–6 reward models recover most of the full-ensemble gain, indicating that all 14 models are unnecessary. However, these results do not establish that optimizing model subsets improves performance. Selecting subsets using calibration-set accuracy and complementary error patterns is a promising extension, which we will identify as future work.

Thanks for your response, I appreciate it! Because the two methods perform similarly, I am not yet convinced of the utility of your framework or its novelty. I do appreciate the insights you have on the limits of test-time scaling and the need for better reward ensembles, but I do not think your paper, in its current form, illustrates these points properly. As such, I will maintain my score.

Reviewer_EtmXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies inference-time reward hacking: when a proxy reward model is optimized harder with Best-of-nn and related samplers, true quality first rises and then falls. It asks whether calibrating how far to optimize (HedgeTune, Khalaf et al., 2025) is the best mitigation, and compares two levers in an aggregated-proxy version of the KL-regularized objective (Eqs. 3–4): the optimization strength θ\theta and the proxy itself, built from several reward models. For the first lever, Adaptive HedgeTune predicts a per-prompt hedge from label-free features of the candidate pool and shrinks it toward the global hedge (Alg. 1, App. C). For the second, VaR aggregation z-scores each reward model's scores within a prompt and takes a low quantile across models (Eqs. 5–6), and is compared against the uniform mean, mean-minus-std (UWO), and the minimum.

All experiments are offline on the PPE Best-of-K benchmark: five verifiable domains (GPQA, MMLU-Pro, MATH, MBPP-Plus, IFEval) with about 500 prompts each and K=32K=32 candidates, 14 public reward models, and three samplers (BoN, SBoN, BoP). A per-prompt oracle hedge would beat global HedgeTune by 16.8 pp (BoN), but Adaptive HedgeTune recovers only +0.40 pp (Table 1), and per-prompt optima are barely predictable from label-free features (Rc2≤0.26R^2_c \le 0.26, Table 4). VaR over all 14 models reaches 58.36%, against 54.71% for HedgeTune averaged over single models (+3.65 pp, Table 2), but it loses on code (MBPP, −5.06 pp) and stays below the best single reward model (66.10%). The paper attributes this pattern to the diversity of reward-model errors (§5) and reports that 5–6 random models recover most of the gain (Fig. 7).

Critical strengths
  • The offline protocol is careful and reproducible, and the paper is candid about where the method fails. Accuracy is computed exactly from order statistics (Eq. 10, App. D), hedges and predictors are cross-fitted on held-out prompts, and code is linked. The paper reports the MBPP loss, the gap to the best single reward model (66.10 vs. 58.36), and the slightly negative Adaptive HedgeTune cases (GPQA −0.03, Table 5). This makes the numbers trustworthy as measurements, even where I disagree with their interpretation.
  • The negative result on per-prompt hedging is clear and well supported. Table 4 shows near-zero cross-validated R2R^2 for the per-prompt optimal hedge θt†\theta^\dagger_t from score statistics, per-feature regularizers (Elastic-Net, ARD) do not help, and the decision-focused variant gains little (Table 5). To me, this is the most convincing part of the paper.
Critical weaknesses
  • The claim that the reward lever is worth "an order of magnitude more" than the optimizer lever does not hold in a like-for-like comparison. The uniform mean of the same 14 models already reaches 57.94 (Table 2), so VaR adds only +0.42 pp over plain averaging, and it is worse than uniform on MMLU-Pro (−1.67). VaR is also indistinguishable from UWO (58.36 vs. 58.29 for BoN, Table 6, with similar gaps in Tables 7–8). VaR's own contribution is therefore about the size of Adaptive HedgeTune's +0.40 pp, which the paper dismisses as "not a source of real leverage" (§3), and none of these sub-point differences is shown to exceed prompt-level noise. What remains, that averaging reward models reduces Best-of-nn overoptimization, is established by Coste et al. (2024) and Eisenstein et al. (2024). I am fairly confident the latter already evaluated ensembles under Best-of-nn, and the authors should state what is new relative to it.
  • The paper leaves out the obvious deployable baseline. HedgeTune calibrates against rtr_t (p. 4), Adaptive HedgeTune regresses on label-derived targets (App. C, Eq. 7), and VaR's α\alpha is selected "via HedgeTune" on held-out prompts (footnote 3, p. 7). Every arm thus already uses true-reward labels on a calibration split, so choosing the best single reward model (or a weighted or subset ensemble) on the training folds is just as deployable, yet p. 5 dismisses "HedgeTune (best RM)" as unreachable. The oracle best reward model scores 66.10 against 58.36 for VaR (81.13 vs. 56.55 on MBPP), and with about 400 calibration prompts per domain I would expect a cross-fitted selection to recover much of that gap. If it does, the paper's practical recommendation reverses.

Other, less critical points:

  • The paper is badly written, and the problem and motivation are not clear from the introduction. The first paragraph presents Best-of-nn as a cheaper alternative to RLHF and DPO, which it "often matches" (p. 1). These are not alternatives: RLHF and DPO spend compute during training and Best-of-nn spends compute at inference, so they are independent levers that can be, and usually are, combined. Framing them as competing choices makes it unclear what problem the paper is solving and why inference-time reward hacking matters in practice.
  • The models and datasets are all old by current standards, which limits how much the conclusions say about inference-time scaling today. The 14 reward models (App. A) are a length heuristic and 13 scalar reward models released in 2024 or earlier (e.g., ArmoRM-Llama3-8B, Skywork-Reward-Llama-3.1-8B, Nemotron-4-340B-Reward), and the candidates come from the PPE benchmark (Frick et al., 2025). The benchmarks are MATH (2021), MBPP-Plus, IFEval, GPQA, and MMLU-Pro (2023–2024). The paper describes the candidates as coming from "frontier models" (p. 4), yet single-sample accuracy is 35% on MATH and 41% on GPQA (Table 2), far below current models. Today, Best-of-nn is typically run with much stronger generators and with generative or LLM-as-a-judge verifiers, where both the amount of over-optimization and the diversity of errors across reward models may be very different. Since the paper's main argument rests on error diversity across reward models, I would like to see at least one domain with current generators and verifiers.
  • The "falsifiable mechanism" in §5 is never tested. No measure of cross-model error correlation or diversity is computed, per domain or for the Fig. 7 subsets, so calling the MBPP loss a "confirmation" is post hoc. A simpler explanation fits Table 2: on MBPP, un-hedged Best-of-32 with a typical reward model is barely better than a single random sample (56.54 vs. 56.15), while the best reward model is far better (81.13), so any symmetric aggregate dilutes the one model that works. The claim in App. G that which models are included matters more than how many has the same problem.
  • Every "±" is a standard deviation over 4 fold-split seeds (footnote 2, p. 4), which captures only cross-fitting noise, and some cells are ±0.00 (Table 2). With about 500 binary-outcome prompts per domain, prompt-level standard errors are around 1–2 pp, larger than VaR vs. uniform (+0.42), VaR vs. UWO (+0.07), or Adaptive vs. global HedgeTune (+0.40).
  • nit: the opening sentence, "A dominant paradigm for improving large language models is to optimize a reward model" (p. 1), is either wrong or badly phrased. A language model is improved by optimizing it against a reward model, while improving the reward model itself is a separate line of work. It is a small point, but it is the first sentence of a paper whose thesis is about fixing the reward rather than the optimizer, and it reflects a general lack of attention to detail in the writing.
Questions for authors
  1. (Clarification) Exactly which true-reward labels does each arm use, and does selecting α\alpha "via HedgeTune" (footnote 3) use rtr_t on the calibration folds? If so, please add a cross-fitted baseline that selects the best single reward model on the training folds and then applies HedgeTune (ideally also a learned weighting or best-subset aggregate) to Tables 2 and 6–8. If this baseline beats VaR, I would read the paper's central recommendation very differently.
  2. (Suggestion) Please report paired, prompt-level bootstrap confidence intervals for VaR − Uniform, VaR − UWO, and Adaptive − Global HedgeTune, pooled and per domain. Unless VaR − Uniform clearly excludes zero, the contribution reduces to ensembling reward models, which is prior work.
  3. (Suggestion) To make §5 falsifiable, please compute a label-based diversity statistic per domain, e.g., the mean pairwise correlation of the residuals zt,k(m)−(z-scored) rt,kz^{(m)}_{t,k} - \text{(z-scored)}\, r_{t,k} or how often models agree on incorrect top picks, and show that it predicts VaR's gain across the five domains and the random subsets of Fig. 7. Does MBPP actually have the lowest error diversity?
  4. (Clarification) The base-4 and universal features in Table 3 include ensemble disagreement over all MM models' scores. Does Adaptive HedgeTune then need all 14 reward models at inference? If so, Table 9's "1 RM pass / candidate" is incorrect and the optimizer lever is not a single-model method. Have you also tried combining the levers (Adaptive HedgeTune on the VaR proxy)?
  5. (Clarification) With binary rtr_t and a fixed pool of 32, θt†\theta^\dagger_t seems essentially determined by whether the highest-scored candidate is correct. Is the bimodality in Fig. 2 and the 16.8 pp oracle gap largely this artifact? If so, the ceiling says little about whether the optimizer lever is inherently weaker. Please also report un-hedged Best-of-K (n=32n=32) accuracy for each aggregator, so the "flattening" claim can be quantified separately for Uniform and VaR.

My score could increase if the authors (i) show that VaR beats a cross-fitted best-reward-model baseline (Q1), (ii) show that VaR's margin over uniform averaging exceeds prompt-level noise (Q2), and (iii) measure the error-diversity mechanism claimed in §5 (Q3). Showing that the results hold with current generators and verifiers would also strengthen the paper considerably. The introduction would also need a substantial rewrite that states the problem and setting clearly.

3 comments

We thank the reviewer for the detailed feedback. We address the concerns below, including the requested additional experiments.

W1: "Order-of-Magnitude" Claim and Comparison Between Levers. The reported gains compare VaR aggregation against HedgeTune applied to a typical single RM (+3.65 pp), and Adaptive HedgeTune against global HedgeTune (+0.40 pp), yielding approximately 9×9\times. However, this comparison includes the benefit of ensembling; VaR's marginal improvement over Uniform is only +0.42 pp.

We acknowledge that VaR–Uniform and VaR–UWO differences are not statistically significant (Table 3). As discussed further in our response to Reviewer F8xQ (W2/Q2), the evidence primarily supports the benefit of reward aggregation, rather than a large advantage of the VaR rule itself. We will revise the abstract and Section 4 to clarify the comparison and avoid attributing the full ensemble gain to VaR.

W2 / Q1: Cross-Fitted Best-Single-RM Baseline. We implemented the requested baseline: for each training fold, we select the RM achieving the highest training-fold HedgeTune accuracy and evaluate it with its calibrated hedge on held-out prompts.

As shown in Table 1, the cross-fitted best RM achieves 65.44%, exceeding VaR (58.36%) and approaching the oracle best RM (66.10%). The largest gap occurs on MBPP-Plus (81.13% vs. 56.55%), followed by MMLU-Pro (75.88% vs. 67.35%).

We agree that this is a deployable and stronger baseline when representative labeled calibration data are available. Both HedgeTune and cross-fitted RM selection use calibration labels; therefore, our label-free claim concerns test-time inference and VaR's fixed aggregation rule, not the entire calibration procedure. VaR avoids supervised model-specific selection but does not outperform it in this setting. We will explicitly acknowledge this limitation and revise the practical recommendation (see also Reviewer F8xQ, W3/Q1).

W3: Introduction Framing and Motivation. We agree that BoN and training-time methods such as RLHF/DPO are complementary rather than competing alternatives. We will revise the introduction to clarify that inference-time reward hacking can occur regardless of how the reference model was trained, and that our focus is mitigating proxy overoptimization during inference-time selection.

W4: Evaluation with Older Models and Datasets. We acknowledge that PPE uses reward models and candidate generations that may not reflect current-generation systems. PPE provides a controlled setting with shared candidate pools, multiple RMs, and verifiable rewards, enabling consistent comparisons across BoN, SBoN, and BoP. However, our findings do not establish generalization to stronger generators or generative verifiers. We will explicitly state this limitation and identify evaluation on current-generation systems as future work.

W5 / Q3: Quantitative Validation of Error Diversity. We computed pairwise Pearson correlations of RM residuals and top-1 selection disagreement across all 91 RM pairs (Table 2; also reported in our response to Reviewer F8xQ, W4/Q3).

MATH exhibits the lowest residual correlation (0.407) and largest VaR gain (+10.30 pp), whereas GPQA has the highest correlation (0.491) and smallest positive gain (+2.91 pp). However, MBPP-Plus has relatively low correlation (0.428) despite a −5.06-5.06 pp loss.

This establishes that error diversity alone is insufficient: pessimistic aggregation also depends on the relative competence of the contributing RMs. In particular, a substantially stronger individual model can be diluted by weaker models, as illustrated by the MBPP-Plus oracle (81.13%) versus VaR (56.55%). We will refine the mechanism in Section 5 and avoid claiming that residual correlation alone predicts aggregation performance.

W6 / Q2: Prompt-Level Statistical Significance. We computed paired prompt-level bootstrap confidence intervals (B=10,000B=10{,}000). VaR significantly improves over HedgeTune (avg RM) by +3.61 pp (95% CI [+2.48,+4.77][+2.48,+4.77]). However, VaR–Uniform (+0.42 pp, CI [−0.51,+1.33][-0.51,+1.33]) and VaR–UWO (+0.07 pp, CI [−0.86,+0.98][-0.86,+0.98]) are not significant.

Full results appear in Table 3 and our response to Reviewer F8xQ (Q2). We will include these intervals in the revision and temper claims of VaR outperforming alternative aggregation rules.

Q4: Adaptive HedgeTune Inference Cost and Combining Levers. Adaptive HedgeTune trains its predictor offline using calibration data, while Table 9 (Appendix G) reports one RM forward pass per candidate at inference, compared with 14 for VaR. However, the ensemble-disagreement features in Table 3 require multiple RM scores if computed on a new prompt. Therefore, the one-pass cost applies only to configurations using features available from the selected RM; configurations requiring live ensemble disagreement incur additional RM evaluations. We will clarify this distinction and report inference costs consistently.

Adaptive HedgeTune and VaR act on different components of Eq. 4 and can be combined by applying the adaptive hedge to the VaR proxy. We have not established an empirical improvement for this combination and will distinguish this extension from the evaluated results.

Q5: Binary Rewards, Bimodality, and Oracle Headroom. We agree that binary correctness rewards and the fixed K=32K=32 candidate pool can contribute to the bimodal distribution of per-prompt optimal hedges. Consequently, the 16.8 pp oracle gap should be interpreted as a ceiling for this evaluation setting, not as evidence of a general limitation of optimizer adaptation.

Nevertheless, Adaptive HedgeTune achieves only +0.40 pp over global HedgeTune, while the evaluated label-free features have near-zero cross-validated predictability (Rc2≈0R_c^2\approx0; Table 4). These results support the limited effectiveness of our tested adaptive selector, without establishing an information-theoretic impossibility or generalization to continuous rewards.

We will clarify this distinction in Section 3. We also report un-hedged Best-of-32 accuracy for Uniform (57.69%) and VaR (57.73%) in Table 3, separating aggregation effects from hedge calibration.

Going through my review again, the other reviews, and the rebuttal, I would say that I am still left with doubts. In addition after looking at the model selection again, I find the outdated models (and datasets) to actually be a critical weakness. However, the authors have not had a lot of time to address this, so it is understandable that it is missing. Therefore, I will keep my score rather than dropping it. It is understandable if you prefer using open-weight models that you can run on local hardware, but the models should be updated to Qwen3.5 9B and Qwen3.8 27B amongst others.

Reviewer_F8xQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates inference-time reward hacking in Best-of-n and related methods. It compares two label-free approaches: adaptively controlling optimization strength for each prompt and aggregating multiple reward models using a low-quantile strategy (VaR). Experiments on the PPE benchmark show that adaptive hedging provides only limited improvements, while VaR aggregation achieves more substantial gains and reduces reward hacking in most domains, except code.

Critical strengths
  • The experimental protocol is careful, with cross-fitting over prompts and seeds, three inference-time mechanisms, and five domains.
  • The failure case in code is reported.
Critical weaknesses
  • Reward model ensembles for mitigating over-optimization have already been extensively studied, including conservative aggregation for Best-of-n (Coste et al., ICLR 2024) and inference-time reranking (Eisenstein et al., COLM 2024).
  • Most of the reported improvement over a typical single reward model comes from ensembling itself. VaR provides only a small improvement over the uniform mean and performs similarly to UWO (Tables 2 and 6). It also underperforms the uniform mean on MMLU-Pro. The significance of these additional gains is unclear.
  • The paper does not compare against a strong reward model selected using independent benchmark results. Furthermore, the proposed error-diversity explanation is plausible but is not directly validated through quantitative analysis of reward-model errors.
Questions for authors
  • How does VaR compare with a strong single reward model selected using an independent benchmark such as RewardBench?
  • Are the improvements of VaR over the uniform mean and UWO statistically significant? Could the authors report paired confidence intervals?
  • Can the proposed error-diversity mechanism be validated directly by measuring reward-model error correlations and relating them to aggregation performance?
  • How does the actual inference cost of VaR compare with using a single strong reward model, including reward-model forward passes?
4 comments
AuthorsOpenReview ↗

Response to Reviewer F8xQ

We thank the reviewer for their constructive feedback. We address each concern below.

W1: Novelty Relative to Reward Model Ensemble Literature. We agree that reward-model ensembling is established. Coste et al. (ICLR 2024) study ensembles and uncertainty-aware aggregation for mitigating reward overoptimization, while Eisenstein et al. (COLM 2024) investigate inference-time reranking ensembles.

Our contributions are more specific: (i) comparing optimizer hedging and reward aggregation under a shared KL-regularized objective (Eq. 4) across BoN, SBoN, and BoP; (ii) evaluating per-prompt-standardized low-quantile aggregation against Uniform, UWO, and worst-case aggregation; and (iii) quantifying how reward-model error structure relates to aggregation gains and failures.

Unlike Uniform, VaR directly scores the lower quantile of model evaluations, providing a distinct risk-sensitive criterion. We will clarify these distinctions without claiming novelty for ensembling itself.

W2: VaR Versus Uniform Mean and UWO. We agree that much of the improvement over a typical single RM comes from ensembling. VaR achieves 58.36% versus 57.94% for Uniform and 58.29% for UWO, and these pooled differences are not statistically significant (Q2 below).

VaR's distinction is its lower-tail criterion: Uniform averages standardized scores, UWO penalizes their standard deviation, whereas VaR directly evaluates a low quantile. This can suppress candidates receiving high scores from only a subset of models, without assuming that mean and variance sufficiently characterize the score distribution.

The distinction matters across domains. Relative to Uniform, VaR improves GPQA by +2.12 pp and MATH by +1.52 pp, but loses 1.67 pp on MMLU-Pro. Relative to UWO, it gains +0.97 pp on GPQA and +0.75 pp on IFEval, but loses 0.89 pp on MMLU-Pro. These differences are descriptive, not statistically established improvements.

Thus, our claim is not that VaR uniformly dominates existing aggregators, but that its explicit lower-tail criterion offers a different treatment of model disagreement, whose benefits depend on the underlying error structure. We will temper the accuracy claims accordingly.

W3 / Q1: Comparison Against a Strong Single Reward Model. We evaluated the cross-fitted best-single-RM baseline also requested by Reviewer EtmX. The RM is selected using labeled training folds and evaluated on disjoint held-out prompts. Table 1 includes this baseline, Nemotron-4-340B-Reward, and the oracle best RM.

Table 1: Extended Table 2 (BoN, accuracy ↑\uparrow %). Bold indicates the highest non-oracle accuracy. Best RM (xfit) uses labeled training folds; Oracle uses hindsight selection.

DatasetHedgeTune (avg RM)Uniform (14 RM)UWO (14 RM)VaR (ours)Best RM (xfit)Nemotron (HT)Oracle (best RM)
GPQA43.4844.2745.4246.3946.1147.0447.36
MMLU-Pro60.9169.0268.2467.3575.8869.4175.88
MATH48.4857.2658.4958.7860.6550.3160.65
MBPP-Plus61.6156.8657.3156.5581.1356.1581.13
IFEval59.0762.2962.0062.7563.4360.0265.49
Mean54.7157.9458.2958.3665.4456.5966.10

The cross-fitted best RM achieves 65.44%, exceeding VaR (58.36%) and approaching the oracle (66.10%). We acknowledge that supervised domain-specific RM selection is preferable when representative calibration labels are available. VaR instead provides a fixed aggregation rule without learning model-specific weights or selection from labels, although hedge calibration itself uses labels. Its advantage over a typical individually hedged RM (54.71%) does not establish superiority over supervised selection. We will clarify this distinction and limitation in the revision (see also Reviewer EtmX).

W4 / Q3: Quantitative Validation of Error Diversity. We measured pairwise Pearson correlations of residuals et,k(m)=zt,k(m)−rt,ke^{(m)}_{t,k}=z^{(m)}_{t,k}-r_{t,k} across all (142)=91\binom{14}{2}=91 RM pairs, together with top-1 selection disagreement. Results are shown below.

Table 2: Error diversity (BoN). VaR Δ\Delta is relative to HedgeTune (avg RM).

DomainResidual corrTop-1 disagreeVaR Δ\Delta
GPQA0.4910.868$+$2.91
MMLU-Pro0.4300.871$+$6.44
MATH0.4070.888$+$10.30
MBPP-Plus0.4280.892$-$5.06
IFEval0.4460.874$+$3.68

MATH has the lowest residual correlation (0.407) and largest VaR gain (+10.30 pp), whereas GPQA has the highest correlation (0.491) and smallest positive gain (+2.91 pp). However, MBPP-Plus contradicts a diversity-only explanation: despite relatively low correlation (0.428), VaR loses 5.06 pp.

This suggests that low correlation alone is insufficient. Pessimistic aggregation also requires informative disagreement among sufficiently competent models; otherwise, it can suppress a stronger individual RM. The MBPP-Plus oracle (81.13%) versus VaR (56.55%) illustrates this limitation. We will refine the mechanism in Section 5 accordingly, treating these measurements as correlational rather than causal evidence.

Q2: Statistical Significance and Paired Confidence Intervals. We computed paired prompt-level bootstrap confidence intervals (B=10,000B=10{,}000 resamples) across the five domains.

Table 3: Paired 95% bootstrap CIs (BoN, pooled).

ContrastDifference (pp)95% CIExcl. 0?
VaR −- HedgeTune avg RM+3.61+3.61[+2.48,+4.77][+2.48,+4.77]Yes
VaR −- Uniform+0.42+0.42[−0.51,+1.33][-0.51,+1.33]No
VaR −- UWO+0.07+0.07[−0.86,+0.98][-0.86,+0.98]No
Un-hedged Best-of-32 accuracy
Reference (n=1n=1)46.45%
Best-of-32 Uniform57.69%
Best-of-32 VaR57.73%

VaR significantly outperforms the typical individually hedged RM (+3.61 pp, 95% CI [+2.48,+4.77][+2.48,+4.77]), but its differences from Uniform and UWO do not exclude zero. We therefore attribute the statistically supported pooled gain primarily to reward aggregation, not VaR's specific quantile rule.

Un-hedged Best-of-32 accuracy is also similar for Uniform (57.69%) and VaR (57.73%), indicating that both mitigate overoptimization without establishing stronger flattening by VaR. We will report these intervals and qualify the corresponding claims.

Q4: Inference Cost Relative to a Single Strong RM. VaR with 14 models requires 14 RM forward passes per candidate versus one for single-RM HedgeTune (Appendix G, Table 9). The reported 0.02 ms/prompt is only the CPU quantile-reduction cost, excluding RM inference.

Figure 7 shows that removing the 340B Nemotron model reduces the summed RM parameter count by approximately 2.6×2.6\times with only a 0.07 pp accuracy reduction; subsets of 5–6 models recover most of the ensemble gain. Nevertheless, a strong single RM can be both more accurate and computationally cheaper when reliable labeled model selection is possible. We will explicitly acknowledge this accuracy–compute tradeoff.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.