acceptodds
Under review as a conference paper at ICLR 2027

Whether or Where? Tail Calibration Signals Finite-Budget Reward Hacking in Best-of-N Selection

Abstract

Best-of- (BoN) inference returns the candidate with the largest proxy reward, so increasing increasingly tests the proxy's extreme tail rather than its average ranking quality. We separate two questions: whether that tail admits a finite optimum, and where the optimum becomes visible. The exact BoN value is a Beta-weighted integral of conditional true reward over within-prompt score quantiles. Under a bounded, homoskedastic additive model, gaussian, laplace, and regularly varying errors respectively imply continued improvement, a tilted plateau, and return to the prior mean. A 162-cell phase map supports this scoped taxonomy; 12 additional heteroskedastic cells show that even conditional Gaussian noise can reverse the limit when variance is largest near low true reward. We then study fixed real pools at and . The latter spans five verifiable domains, four base policies, and 13 learned reward models (65 cells). Six learned-model cells have negative top-12.5% tail excess, all on MBPP+; one has a simultaneous interior downturn, and a cross-fitted tail-triggered cap yields multiplicity-controlled gains in four cells. Although only 9/130 diagnostic folds are positive, tail risk achieves ROC AUC [95% cluster-bootstrap CI ] and average precision [], versus prevalence . On the six negative-tail cells, the cap and a same-pool cross-fitted rank-Soft-BoN baseline gain and correctness, respectively. Tail calibration is thus not a universal replacement for ranking metrics or a new optimizer; it is a score-transform-invariant, mechanism-localizing audit that produces a competitive held-out budget rule.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
reward hacking, best-of-N sampling, reward model evaluation, tail calibration, inference-time alignment, test-time scaling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CwDvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzes how the conditional true utility at each proxy-score quantile determines Best-of-N (BoN) performance. It develops shape-transfer results, an additive-noise tail classification, and a heteroskedastic extension, then proposes a tail-excess diagnostic that gates calibration-selected budget caps. Validation combines synthetic models, controlled reward-model input perturbations, and held-out analysis of public PPE candidate pools.

Critical strengths
  1. The marginal-value identity and shape-transfer theorem explain precisely how calibration-curve increments constrain BoN behavior, with useful continuous and finite-pool formulations.
  2. The prior-tilt and heteroskedastic analyses provide concrete mechanisms for different limiting behaviors, including why conditionally Gaussian errors need not protect against harmful selection.
  3. Exact without-replacement evaluation, tie handling, cross-fitting, and simultaneous or clustered uncertainty estimates improve the rigor of the empirical analysis.
  4. The released score/label artifacts support numerical verification. The paper also reports negative controls, rare-event uncertainty, and nonsignificant comparator differences rather than presenting uniformly positive results.
Critical weaknesses
  1. The theoretical narrative does not establish the claimed diagnostic contribution

W1 — Incomplete positioning of the contribution. PPE already evaluates full BoN curves, and HedgeTune calibrates operating points; these precedents narrow the claimed gap beyond the introduction's characterization of existing evaluation as bulk-focused. Directly relevant regularized BoN, Soft-BoN, and smoothing/regret analyses [1–3] (see below) are also omitted, leaving the incremental methodological contribution insufficiently distinguished.

W2 — The diagnostic does not measure the failure characterized by the theory. Equation (2) compares top-band utility with random selection, whereas a downturn concerns negative marginal BoN utility. Tail excess can remain positive despite a genuine peak and subsequent decline, so the implemented negative-excess gate can miss the budget failures motivating the paper. The shape-transfer theorem does not resolve this gap.

W3 — Asymptotic and finite-budget claims are conflated. Theorem 3.3 determines limiting utility, not the existence or location of finite-budget peaks; non-log-concave light-tailed mixtures can also peak. Likewise, Lemma 3.5 supplies a sensitivity bound, not the turning-point identification used in Appendix D.1. The claimed separation of whether failure occurs from where it occurs therefore exceeds the established results.

  1. The validation does not establish sufficient added value or reliability

W4 — Limited evidence for the proposed diagnostic's added value. Only one learned PPE cell has a statistically supported interior downturn, and neither tail risk nor the cap significantly outperforms rank accuracy or rank-Soft-BoN, respectively (Tables 3 and 6). The clearest controlled failure is template-confounded, while the essential baseline of calibration-argmax budgeting without the tail gate is missing.

W5 — Reproducible implementation defects affect supporting evidence. Fixed-bin midpoint integration in tail_sensitivity.py can manufacture a finite optimum for a monotone calibration curve. The mathematical-answer checker also produces unambiguous false positives and false negatives. These issues require corrected sensitivity analyses and a label audit; they should not be conflated with the separately implemented exact finite-pool estimator or the external PPE labels.

Specific verification findings and corrections:

  • Counterexample relevant to W2. Consider the valid, smooth, single-peaked calibration curve

    \mu=\frac{3}{10},\qquad T_{7/8}=\frac{7}{64}>0.$$ Direct substitution into Equation (1) gives $$V_{N+1}-V_N=\frac{9-N}{5(N+1)(N+2)(N+3)}.$$ BoN therefore peaks at budgets 9 and 10 and subsequently declines, while the population negative-tail trigger does not activate. This challenges the diagnostic's coverage, not the BoN identities.
  • Code checks relevant to W5. On 256 equally spaced rank values from the identity calibration curve, the submitted histogram integrator reports an optimum of 14 with 16 bins and 23 with 32 bins, although the population BoN curve is strictly increasing. Separately, check_correct.py accepts the output 10000 against gold 1000, but rejects The answer is 0.5 against gold 0.5. The prevalence of affected labels in the actual experiment remains unknown.

  • Appendix discrepancies. Proposition A.7(iv) assumes only two-sided comparability of the endpoint density, but its proof uses an asymptotic equivalent. Corollary A.10 is stated for arbitrary calibration curves, while its proof establishes a special-case contamination result. The pointwise identity in Proposition A.8 also does not imply the averaged-sign equivalence asserted in Appendix B.

  • Minor reference issues. Coste et al. (2023a/b) and Lambert et al. (2024b/c) duplicate the same respective papers; Section 2 also repeats author names in several citations.

Additional references relevant to W1:

[1] Yuu Jinnai et al. Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignment. NAACL 2025, pp. 9321–9347. arXiv:2404.01054.

[2] Claudio Mayrink Verdun et al. Soft Best-of-n Sampling for Model Alignment. 2025. arXiv:2505.03156.

[3] Gholamali Aminian et al. Best-of-N through the Smoothing Lens: KL Divergence and Regret Analysis. ICLR 2026. arXiv:2507.05913.

Questions for authors
  1. Regarding W1: What is the principal methodological advance beyond Khalaf et al./HedgeTune and PPE's BoN evaluation, after accounting for the directly relevant work in [1–3]?
  2. Regarding W2: Which class of budget-induced declines is the negative-tail trigger intended to detect? How does the proposed audit handle the positive-tail counterexample above?
  3. Regarding W3: Which finite-budget conclusions follow rigorously from Theorem 3.3 and Lemma 3.5, rather than additional assumptions or heuristics? Can the overstatements and listed appendix discrepancies be corrected without changing the claimed contribution?
  4. Regarding W4: Under identical folds and tuning budgets, what does the tail gate add over direct calibration-argmax budgeting? What evidence supports its usefulness beyond the template-confounded control and the few code-task failures?
  5. Regarding W5: After correcting the histogram integration and answer checker, do the affected sensitivity findings and controlled downturn remain? Please quantify changed labels and report the corrected comparisons.
Reviewer_cv9NOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies when sampling more answers and selecting the one with the highest reward-model score can actually reduce correctness. The key idea is that a larger candidate pool increases the chance of finding a wrong answer that receives an unusually high score, so a reward model’s reliability among its highest-scoring answers matters for deciding how many candidates to sample. The authors develop a theoretical account of this behavior and propose a practical check: use answers with known correctness to examine whether the highest-scoring candidates remain reliable, then reduce the sampling budget when they do not. Through simulations and experiments on math, coding, and other verifiable tasks, they show that this check can identify harmful selection behavior and guide budget choices that improve correctness on separate test data. The main contribution is connecting reward-model reliability to sampling budgets; their method performs similarly to existing selection approaches rather than establishing a clear performance advantage.

Critical strengths

None. I could not identify substantive strength.

Critical weaknesses
  1. The proposed method's advantage over existing metrics is not fully established. Given the Table 3 and reported confidence interval the ordinary rank (existing method) accuracy predicts problematic cases almost as well as the proposed tail statistic. The numerical value of proposed method is higher but the confidence is wider than difference ([−0.003, 0.042] for AUC and [−0.015, 0.215] for average precision). The empirical evidence is not enough to claim that the proposed method has an advantage.
  2. Based on the data provided in Table 5 it is hard to conclude that the rise-then-fall pattern as N increases is a general phenomenon in real data. Eurus-7B peaks at N=2 and then declines. It is the only case marked “D,” meaning a statistically supported rise-then-fall pattern. The Skywork-Llama-8B case the decline does not meet statistical criteria. Other models start to fall as N goes beyond 1. The theory and the synthetic data works well but the claim's generalization to real data seems limited.
  3. I don't see much value in this method as to calculate this tail statistics, it essentially needs to do similar to what BoN does: generating many candidates, obtaining correctness and RM label and calculating correctness in different percentile bins. At this point simply doing BoN will tell you whether going up in N makes the performance worse. I don't see how the proposed method could predict this with much less compute or effort.
Questions for authors
  1. See weakness 1: could the authors offer and explanation or more comparison experiments to establish advantage?
  2. See weakness 2: could the authors provide more real data evidence supporting this claim?
  3. See weakness 3
Reviewer_v24rOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies why generating more candidate answers and selecting the one with the highest reward-model score can sometimes reduce actual answer quality. It connects this behavior to the reliability of the reward model among its highest-scoring answers. The paper provides theoretical results under several error models and proposes a diagnostic for deciding when to limit the number of candidates. Experiments include synthetic data, controlled language-model evaluations, and public benchmarks.

Critical strengths
  1. Relevant problem and useful theoretical insights. The paper addresses an important limitation of best-of-(N) selection: additional computation does not necessarily improve performance. The analysis helps explain how errors among the highest-scoring answers can cause performance to deteriorate.

  2. Careful experimental evaluation. The experiments cover several settings and use held-out data to evaluate the selected budgets. The authors also acknowledge cases where their diagnostic performs similarly to existing metrics, which supports a balanced interpretation of the results.

Critical weaknesses

1.The paper is not clearly written and is difficult to follow. The presentation introduces many definitions and theoretical results without sufficiently explaining their intuition or how they connect to the proposed practical procedure. The main contribution and the assumptions needed for the central claims are consequently difficult to identify.

2.The practical benefit remains unclear. The proposed diagnostic performs similarly to ordinary rank accuracy, and the budget cap offers comparable performance to the soft-selection baseline. The paper also lacks a direct comparison with simply choosing the best-performing budget on validation data. Since the proposed procedure already requires labeled calibration data, this comparison is important for establishing its added value.

Questions for authors
  1. What is the main new contribution relative to existing analyses of best-of-(N) selection and methods for choosing its sampling budget?

  2. How does the proposed procedure compare with directly selecting the best budget on the same calibration data, without using the tail diagnostic?

  3. Which assumptions are necessary for the claim that the error-tail behavior determines whether performance will decline? Some statements appear broader than the formal results establish.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.