acceptodds
Under review as a conference paper at ICLR 2027

Efficient Best-of-N Policy Evaluation for Inference-Time Alignment

Abstract

Best-of- (BoN) is a common inference-time alignment method that selects the highest-scoring response among samples from a reference model. Evaluating BoN policies from logged data is challenging under sample-only access because standard off-policy estimators require density ratios that depend on unavailable response likelihoods. In this paper, we propose a sample-only framework for evaluating and selecting BoN policies without access to these likelihoods. We show that the order-statistic structure of BoN allows the required density ratios to be expressed through score-rank probabilities that are estimable from samples alone. We then develop a doubly robust estimator of the BoN policy value (BoN-DR) that efficiently reuses a shared auxiliary sample pool across candidate budgets. We establish valid asymptotic inference even under reward estimator misspecification and prove the efficiency of our BoN-DR estimator. Since larger budgets can amplify errors in the score function and lead to reward overoptimization, we derive two selection rules: (i) maximizing the estimated policy value and (ii) maximizing a lower confidence bound on the improvement over the reference policy, which accounts for estimation uncertainty and provides a no-harm guarantee. Across synthetic experiments and GSM8K with multiple reference and reward models, our framework accurately estimates BoN policy values and selects effective sampling budgets.

Primary area:
causal reasoning
Keywords:
off-policy evaluation, causal inference, doubly robust estimation, uncertainty quantification, policy selection, Best-of-N sampling, inference-time alignment, large language models

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_ndPfOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies off-policy evaluation of BoN policies, which is particular important in the LLM/generative-AI setting. Here we have access to a base model \pi_b and there is as unknown reward function \mu, and we want to estimate the value (expected reward) of the BoN policy induced by \pi_b, with sampling budget N. Traditionally this is solved via importance weighting, but in the gen-AI setting we don't have access to the importance weights. Thus, the first key insight of the paper is that, by exploiting the structure of the BoN policy, one can use U-statistics to get an unbiased estimator for the importance weights. This leads to consistent value estimator, with a "standard" doubly-robust guarantee. The paper also considers the model- or estimator-selection problem, where specifically you want to choose the sampling budget N in the BoN; they give two estimators with asymptotic guarantees. Finally the paper conducts experiments, both synthetic and in an real LLM setting, as validation.

Critical strengths

The paper studies a real and interesting problem, has somewhat non-trivial technical insights, is comprehensive (with detailed theoretical statements and empirical validation), and is well-presented.

Critical weaknesses

I wouldn't call this critical, but I think there is one main point for the authors to think about:

  1. Although asymptotic guarantees are useful, finite sample guarantees are the gold standard for off-policy evaluation, as they provide a much more fine-grained measure of estimator quality (see e.g., https://arxiv.org/pdf/1511.03722 or https://arxiv.org/abs/1612.01205, or many other related papers). In this particular setting, there are two main concerns. 1a. We do not see the role of M or N in the basic bound (Theorem 4.2). Although we get a \sqrt{n} rate, it's plausible that the pre-multiplier scales exponentially with N; indeed the g_N term has exponential-in-N quantities, and this could mean that the estimator is actually still quite poor in finite sample. This would not be super surprising, because intuitively the importance weights for BoN-N explode as N increases, as the BoN policy moves further an further away from \pi_b, but it would be a good thing to point out explicitly. 1b. The asymptotic guarantees hide a lot of interesting behavior in the estimator selection problem. To me (and as pointed out in several prior works: https://arxiv.org/abs/2002.07729v2, https://jonathannlee.com/files/tucker2021improved.pdf) the estimator selection problem is much more interesting in finite sample. Particularly, here it's natural to expect that both BoN policy value and finite-sample variance are increasing with N, which means that asymptotically we would select the largest N, but in finite sample we probably would not (too high variance). The point is that doing everything in finite sample would capture more of the nuance of the problem and also lead to some non-trivial conceptual points. At the same time, I think it is extremely doable. There are concentration inequalities for U-statistics, which seem directly applicable. I think including this, possibly in the appendix would be great!

(Also I think it would be worthwhile to cite some of the papers on estimator selection. There are a bunch more: https://arxiv.org/abs/1604.00923, https://arxiv.org/abs/2502.08021v3, the two mentioned above, and possibly others!)

Questions for authors

Can you look into finite sample guarantees?

3 comments

Thank you very much for your thoughtful review and your helpful comments! Below, we address all your comments and suggestions. We have incorporated all new results into the revised version of our paper.

> W + Q: Finite-sample guarantees

Thank you for this suggestion! We followed your suggestion and derived finite-sample guarantees to provide additional insights into the dependence on the BoN budget and the behavior of budget selection, which are especially relevant for high Nmax⁡N_{\max}. We have addressed your comments as follows:

  • 1(a) Role of NN and MM: We have added a finite-sample analysis in a new Appendix F. Our main result establishes simultaneous high-probability concentration bounds for the BoN-DR estimator, with an error rate of

O(Nlog⁡(J/δ)n+Nlog⁡(J/δ)n),O\left(\sqrt{\frac{N\log(J/\delta)}{n}}+\frac{N\log(J/\delta)}{n}\right),

and explicit dependence on the auxiliary sample size MM. Importantly, the dependence on NN is polynomial rather than exponential. Intuitively, despite the powers appearing in gNg_N, the BoN importance weights (and their U-statistic estimates) are bounded by NN.

  • 1(b) Budget selection in finite samples: Furthermore, we derive an additional finite-sample budget selection rule based on empirical Bernstein confidence intervals. This rule provides high-probability no-harm guarantees relative to the behavior policy and oracle-regret bounds. In particular, it captures your observation that smaller budgets may be preferable in finite samples when larger budgets have excessive estimation uncertainty. As a small addition, the largest NN is not necessarily optimal even asymptotically in our setting, as reward overoptimization can cause the true policy value to decrease with increasing NN.

Empirical evaluation: We have also added new experiments to evaluate both results on GSM8K (new Appendix F.1, Tables 9 and 10). The finite-sample intervals cover the oracle value in every run, and the finite-sample rule never selects a harmful budget. As expected, both are conservative: the rule selects much smaller budgets (mean 1.0 to 4.1, compared with 24.0 to 106.7 for the asymptotic rule) and thus gives up part of the gain.

Action: We have added the finite-sample guarantees and their empirical evaluation to our new Appendix F, and we cross-reference them in the main text.

> Additional references

Thank you also for pointing us to the relevant literature on finite-sample OPE (Jiang & Li, 2016; Wang et al., 2017) and estimator selection (Thomas & Brunskill, 2016; Su et al., 2020; Tucker & Lee, 2021; Liu et al., 2025). We have incorporated these references and discussed them in our extended related work.

Action: We have included the suggested references in our extended related work.

Reviewer_ndPfOpenReview ↗

Awesome!

This is great! I appreciate you taking the time to look into these things and the updated results look very nice. I'll increase my score.

Thank you very much for your kind reply and for increasing your score! We are really grateful for your suggestion of finite-sample guarantees, which nicely complement our asymptotic results and clearly strengthen our paper.

Reviewer_dNBgOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

Best-of-N (BoN) is a common inference-time alignment method that selects the highest-scoring response among N samples from a reference model. However, Evaluating BoN policies from logged data is challenging under sample-only access. To this end, this paper proposes BoN-DR, a doubly robust estimator that efficiently reuses a shared auxiliary sample pool across candidate budgets. The authors also propose two methods for selecting the budget: (i) maximizing the estimated policy value and (ii) maximizing a lower confidence bound on the improvement over the reference policy.

Critical strengths
  • The presentation is really great and the visualizations are of good quality.
  • The studied problem is important to the real world.
  • Strong theoretical results and useful empiritical validations.
Critical weaknesses
  • In my understanding, the paper mainly uses GSM8K for the real LLM experiments. However, inference-stage alignment in practice always incorporates more dimensions, such as safety, helpfulness, and instruction following. Therefore, Relying on GSM8K is kind of limited experimental setup. The authors can also try some safety alignment datasets, such as BeaverTails.
Questions for authors

See the weakness.

3 comments

Thank you very much for your positive review! We were happy to read that you found the studied problem important to the real world and that you appreciated our presentation and our theoretical results. Below, we address your suggestion in detail. We mark the resulting paper changes as Action, and we highlight all changes in blue in the revised PDF.

> W: Experiments beyond GSM8K (e.g., safety alignment on BeaverTails)

Thank you for this helpful suggestion. A strength of our framework is that it is agnostic to the choice of the downstream reward, so it applies directly to other alignment dimensions such as safety, helpfulness, or instruction following, given a corresponding reward signal. To demonstrate this, we followed your suggestion and added new experiments on BeaverTails (Ji et al., 2023) with safety as the downstream reward (see our new Appendix E.5), using the same three reference models and two score functions as on GSM8K. The findings from GSM8K carry over:

(1) BoN-DR accurately recovers the policy value. At N=64N=64, BoN-DR is essentially unbiased in all six configurations (absolute bias at most 0.2 percentage points) with coverage between 0.90 and 0.95, whereas the Plug-in estimator overestimates the safety rate by up to 5 percentage points and never covers the oracle value. BoN-DR is also much more precise than BoN-IPW, whose intervals are 2.2 to 3.3$\times$ wider.

(2) Improvement-LCB guards against harmful budgets. Safety alignment is a particularly relevant application for our no-harm guarantee, since a harmful budget choice here means deploying a less safe model. We therefore zoom in on Qwen with OASST, which is the configuration where reward overoptimization occurs: the score keeps increasing with N, while the safety rate at large budgets falls below that of the reference model, so that Naïve BoN is harmful in every repetition. Here, Improvement-LCB with BoN-DR lowers the harm rate from 22% (Value-max) to 1% while retaining the same gain. In the remaining five configurations, larger budgets are beneficial and no selection rule incurs harm.

→\rightarrow Our framework thus extends from mathematical reasoning to safety alignment, which shows its broad applicability. In particular, our Improvement-LCB rule is especially valuable for such safety-critical tasks, where practitioners want to be on the safe side: it avoids the budgets that would make the model less safe while retaining the gain.

Action: We added the BeaverTails experiments to our new Appendix E.5, including policy value estimation and budget selection. We also reference the new experiments in Section 6.

We hope that these additions further strengthen your positive assessment, and we are happy to answer any further questions.

References

Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. BeaverTails: Towards improved safety alignment of LLM via a human-preference dataset. In Advances in Neural Information Processing Systems (NeurIPS), 2023.

Thanks for the authors' rebuttal. My concerns are addressed. I will keep my positive score.

Thank you for your reply! We are happy to hear that your concerns are addressed, and thank you for maintaining your positive score.

Reviewer_eEmROpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper concerns policy evaluation with access only to sampling, with applications to hyperparameter selection for best-of-N sampling. The paper is quite interesting and exploits structure in BoN sampling to construct a likelihood-free representation, a statistically-interesting estimator, and an algorithm for selecting N. The theoretical results are interesting, and the method shows significant empirical improvements over the plug-in baseline.

Critical strengths
  1. Likelihood-Free DR Estimation Without Propensity Bias: Replacing intractable sequence likelihood ratios πNs(a∣x)πb(a∣x)\frac{\pi_N^s(a \mid x)}{\pi_b(a \mid x)} with finite-pool score-rank U-statistics neatly solves the black-box OPE problem for BoN. Crucially, because the weight estimator w^i,Ns,M\hat{w}_{i,N}^{s,M} is exactly conditionally unbiased for any finite M≥N−1M \ge N-1, finite auxiliary sampling only introduces O(1/M)O(1/M) variance rather than nuisance bias—yielding valid n\sqrt{n}-inference under arbitrary reward-model misspecification without the product-rate conditions required by standard DR.
  2. Practical Shared-Pool Reuse Across Budgets: Reusing a single auxiliary pool per prompt across both DR components and all candidate budgets N∈NN \in \mathcal{N}, coupled with a joint multiplier bootstrap to handle cross-budget correlation, makes uncertainty-aware grid search over NN practical and yields large variance reductions over BoN-IPW.
Critical weaknesses
  1. Finite-Sample Undercoverage and Violation of the No-Harm Bound at Large NN : Because w^i,Ns,M\hat{w}_{i,N}^{s,M} is zero whenever the logged action's rank in the auxiliary pool is below N−1N-1, only an O(1/N)O(1/N) fraction of prompts receive non-zero debiasing weights at large budgets (effective sample size drops from 3,9733,973 to ∼25\sim 25 at N=256N=256 and ∼10\sim 10 at N=512N=512; Appendix D.6). Relying on fixed-NN Gaussian asymptotics in this heavy-tailed regime causes BoN-DR's nominal 95%95\% CIs to undercover at N=256N=256 (75%–88%75\%\text{–}88\% coverage on GSM8K; Table 2). Consequently, under severe overoptimization where N⋆=1N^\star = 1 (Llama and Qwen with OASST; Table 5 & Appendix D.6), BoN-DR + Improvement-LCB incurs a 12%–14%12\%\text{–}14\% harm rate at Nmax⁡=256N_{\max}=256 (and 24%–33%24\%\text{–}33\% at Nmax⁡=512N_{\max}=512), exceeding the nominal α=5%\alpha = 5\% level.
  2. High Online Generation Cost (M≥Nmax⁡−1M \ge N_{\max}-1 per Prompt) and Stationarity Requirement: Unbiasedness requires generating at least Nmax⁡−1N_{\max}-1 fresh auxiliary responses from πb(⋅∣xi)\pi_b(\cdot \mid x_i) for every logged prompt xix_i (over 2M2\text{M} LLM generations for the GSM8K validation set at M=511M=511). This is computationally expensive at large Nmax⁡N_{\max} and assumes zero distribution shift between the historical logging policy and the current black-box API.
Questions for authors
  1. Mitigating large-NN undercoverage in BoN-DR: At large NN, BoN-DR's debiasing term is non-zero on very few prompts, causing Gaussian/Wald intervals to undercover and inflating the harm rate of Improvement-LCB beyond α=5%\alpha = 5\% (Tables 2 & 5). Could finite-sample heavy-tailed adjustments (e.g., weight clipping, self-normalization, or studentized empirical Bernstein bounds) restore nominal coverage and safety at large NN without sacrificing BoN-DR's variance advantage over BoN-IPW?
  2. Behavior when M<N−1M < N - 1: Generating M≥Nmax⁡−1M \ge N_{\max}-1 auxiliary samples per logged prompt can be prohibitive when Nmax⁡N_{\max} is large. How does a biased plug-in rank estimator gN(F^x(a),F^x−(a))g_N(\hat{F}_x(a), \hat{F}_x^-(a)) (as in Appendix F.3) perform when M≪Nmax⁡M \ll N_{\max} (e.g., M=32M = 32 for Nmax⁡=256N_{\max} = 256)?
  3. Score-ties assumption in Proposition 4.1: Proposition 4.1 (line 212) assumes distinct actions have distinct scores, which fails for discrete LLM-judge rubrics, whereas the proof in Appendix B.2 (line 816) holds for general score ties under uniform tie-breaking. Can this restriction be removed from the main text?
4 comments

Thank you for your positive review and your helpful comments. Below, we have drafted careful responses to your suggestions. We have incorporated all points marked with Action into the revised version of our paper.

> W1 + Q1: Finite-sample undercoverage and no-harm guarantee at large N

Thank you for this advice, which helped us to substantially strengthen the paper. We acknowledge that the asymptotic approximation can undercover at large N, where only few prompts receive non-zero weight. Following your suggestion of empirical Bernstein bounds, we have added finite-sample guarantees in our new Appendix F: (i) a simultaneous concentration bound for BoN-DR (Theorem F.1), and (ii) a finite-sample Improvement-LCB rule based on empirical-Bernstein radius (Maurer & Pontil, 2009) (Theorem F.2). Importantly, both hold without relying on asymptotic normality. We have also evaluated them empirically on GSM8K (Appendix F.1, Tables 9 and 10):

  • Coverage is ensured. At N=256, the finite-sample intervals cover the oracle value in 100% of the repetitions in all six configurations, compared with 75% to 90% for the asymptotic Wald intervals.

  • No-harm is ensured. The finite-sample Improvement-LCB rule has a harm rate of 0.00 in all six configurations, including the two overoptimized OASST configurations where the asymptotic rule has harm rates of 0.12 and 0.14.

We would like to be transparent that these guarantees come at a cost. At N=256, the finite-sample interval has a width of 2.30, which exceeds the range of the policy value and is therefore not informative at the largest budget. Likewise, the finite-sample rule selects much smaller budgets (mean selected budget 1.0 to 4.1, compared with 24.0 to 106.7 for the asymptotic rule) and thus gives up part of the gain when larger budgets are beneficial (e.g., 11.74 vs. 20.57 accuracy points for Gemma with Skywork).

Regarding the question on BoN-DR's variance advantage: the leading term of the empirical-Bernstein radius is variance-adaptive and thus benefits from the lower variance of BoN-DR, but its range term grows linearly in N and dominates at large budgets. Hence, the adjustment restores coverage and safety, but currently not without sacrificing gain. We therefore see the two rules as complementary: the asymptotic rule as the default, and the finite-sample rule whenever a strict no-harm guarantee is required. Weight clipping and self-normalization are interesting alternatives that would trade the exact finite-pool unbiasedness of our weights for lower variance, which we consider a promising direction for future work.

Action: We have added the finite-sample guarantees (new Theorems F.1 and F.2) and their empirical evaluation to our new Appendix F.

> W2: Generation cost and stationarity requirement

Thank you for raising these two points.

(a) Generation cost. We would like to emphasize that the shared auxiliary pool is a strength of our method, as it makes our evaluation efficient in terms of generations and labels:

(i) Efficient wrt. generations. A direct Monte Carlo implementation of the BoN-DR score would draw fresh responses for every expectation, every rank probability, and every candidate budget NN. Our pooled estimator draws a single pool per prompt and reuses it for both terms of BoN-DR and for all candidate budgets. Further, a pool of M=Nmax⁡−1M = N_{\max}-1 auxiliary responses is already sufficient for unbiasedness and valid inference (Theorem 4.2(a)). In our GSM8K experiments, we used all 511 remaining responses only because we had already stored them to compute the oracle policy values, and M=255M=255 would have sufficed for Nmax⁡=256N_{\max}=256. Hence, every generated response is used for every candidate budget, which keeps the number of generations relatively small.

(ii) Efficient wrt. labels. Our method needs only one observed reward per prompt (that of the logged response), and the auxiliary responses require no true reward labels at all. In contrast, the closest existing approach (Khalaf et al., 2025) needs the true rewards of all sampled responses. Since reward labels such as human feedback are typically the expensive resource in practice, our method is substantially more data-efficient.

Finally, we generate the pool once and offline, so that no additional cost arises at deployment. In total, evaluating all candidate budgets takes n⋅Nmax⁡n \cdot N_{\max} generations including the logged responses, which equals the cost of serving the nn logged prompts once with BoN at budget Nmax⁡N_{\max}.

⇒\Rightarrow A practitioner pays the generation cost once and can then evaluate and compare all budgets up to Nmax⁡N_{\max} without bias. We discuss smaller pools with M<Nmax⁡−1M < N_{\max}-1, which give up this unbiasedness, in our response to Q2.

(b) Stationarity. Our guarantees assume that the logged and the auxiliary responses come from the same reference model (Assumption B.1(i)). This corresponds to the standard assumption in off-policy evaluation that the logging policy is known and fixed (e.g., Dudík et al., 2011), and it is what allows us to evaluate BoN policies exactly without response likelihoods. We would like to emphasize that this is the typical use case of our framework: a practitioner collects feedback on the responses of a deployed model and wants to know whether BoN improves this same model. Using the running example of our paper, a car manufacturer collects user feedback on its in-car assistant and then evaluates BoN for this assistant, so that the logged and the auxiliary responses come from the same model by design. Moreover, since the auxiliary responses require no labels, they can be generated together with the logged responses, in which case no shift can occur. We agree, however, that extending our framework to a distribution shift between logging and evaluation is an interesting direction for future work, and we now state this requirement as a limitation.

Action: We added the requirement of auxiliary responses from the same reference model as a limitation to our conclusion (Section 6).

> Q2: Behavior when M<N−1M < N-1

Thank you for this interesting question, which is relevant for reducing the generation cost further. We added a new experiment for exactly the setting you suggest (M=32M=32, Nmax⁡=256N_{\max}=256) in our new Appendix E.6. There, we replace our U-statistics by the empirical-CDF construction gN(F^x,F^x−)g_N(\hat F_x, \hat F_x^-) from only M=32M=32 auxiliary draws (as in the appendix you refer to, now Appendix G.3), which remains computable for N>MN > M. We then compare the resulting biased versions of BoN-DR, BoN-IPW, and Plug-in with our estimators on GSM8K (Figure 15). We have three main findings:

(1) The empirical-CDF construction is feasible. It allows us to evaluate budgets N>MN > M at a fraction of the generation cost, and it stays close to the oracle value as long as the budget is small relative to the pool.

(2) Its bias grows with the budget. For N≫MN \gg M, the biased BoN-IPW and BoN-DR estimators deviate increasingly from the oracle value. The reason is that a pool of MM draws cannot resolve score ranks beyond the top 1/M1/M fraction, so that the empirical-CDF weights are no longer centered at one. The biased Plug-in estimator does not benefit from the small pool either and remains biased in all six configurations (absolute bias between 1.0 and 9.6 percentage points at N=256N=256).

(3) Our U-statistic estimators remain unbiased at every budget. They closely track the oracle value over the whole grid, in line with their finite-pool unbiasedness (Lemma B.2). This shows a main strength of our construction: it gives exact unbiasedness from a finite pool, exactly where the more straightforward approach fails.

⇒\Rightarrow We therefore recommend our U-statistic estimators with a candidate grid of Nmax⁡≤M+1N_{\max} \leq M+1. If only a smaller pool is affordable, corrections such as self-normalized weights (in the spirit of your suggestions in Q1) could reduce the bias of the empirical-CDF construction, although without the finite-pool unbiasedness of our U-statistics. We see this as an interesting direction for future work.

Action: We added the experiment with M=32M=32 to our new Appendix E.6 (Figure 15).

> Q3: Score-ties assumption in Proposition 4.1

Thank you for your careful reading of our proofs. We stated the distinct-score assumption in the main text only for ease of exposition. Indeed, it is not needed, since our proof in Appendix B.2 already covers general score ties under uniform tie-breaking, and we have therefore removed it. Our estimators carry over directly: the weight estimator in Eq. (9) remains unchanged, and the plug-in estimator in Eq. (7) averages μ^\hat{\mu} over the draws that share a score, which reduces to our original estimator when distinct actions have distinct scores. →\rightarrow Our framework is thus directly applicable to discrete score functions such as LLM-judge rubrics.

Action: We removed the distinct-score assumption from Proposition 4.1 and added the treatment of tied scores to Sections 3 and 4.2.

References

Miroslav Dudík, John Langford, and Lihong Li. Doubly robust policy evaluation and learning. In International Conference on Machine Learning (ICML), 2011.

Hadi Khalaf, Claudio Mayrink Verdun, Alex Oesterling, Himabindu Lakkaraju, and Flavio Calmon. Inference-time reward hacking in large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2025.

Andreas Maurer and Massimiliano Pontil. Empirical Bernstein bounds and sample variance penalization. In Conference on Learning Theory (COLT), 2009.

Thank you for the clarification and finite sample guarantees, these are great to see. I have raised my score.

Thank you very much for your positive response and for raising your score! We are very glad that you appreciate the clarification and the finite-sample guarantees. Thanks again also for your detailed review, which substantially helped us improve the paper.

Comments on the paper

1 comment

Dear reviewers and AC,

Thank you for handling our submission and for the constructive reviews. We were very happy to see that the reviewers found the studied problem important for the real world (dNBg, ndPf), the likelihood-free BoN-DR estimator and the shared-pool reuse interesting and practical (eEmR), the theoretical results strong (dNBg, eEmR), and the paper clear and well-presented (eEmR, dNBg, ndPf).

We have carefully addressed every concern in our individual responses and uploaded a revised manuscript, in which we highlight all changes in blue. In our responses, we cite works by author and year, and the reference list of the revised paper contains all cited works. The main additions are:

  • Finite-sample guarantees (new Appendix F; eEmR, ndPf). We have derived a simultaneous finite-sample concentration bound for BoN-DR, which shows that the error depends only polynomially on the budget NN and makes the role of the pool size MM explicit. We have further added a finite-sample Improvement-LCB rule with a no-harm guarantee. On GSM8K, this rule incurs no harm in any configuration, at the price of more conservative budget choices.

  • BeaverTails experiments (new Appendix E.5; dNBg). We have extended our evaluation to a new dataset with safety as the downstream reward. BoN-DR remains essentially unbiased, whereas the Plug-in estimator overestimates the safety rate by up to 5 percentage points, and BoN-IPW intervals are 2.2 to 3.3$\times$ wider. Where large budgets reduce safety, our Improvement-LCB lowers the harm rate from 22% to 1%.

  • Small auxiliary pool (new Appendix E.6; eEmR). We have performed new experiments to evaluate biased versions of our estimators that use only M=32M = 32 auxiliary draws for budgets up to Nmax⁡=256N_{\max} = 256. Their bias grows with the budget, whereas our U-statistic estimators remain unbiased at every budget, which supports choosing the candidate grid such that Nmax⁡≤M+1N_{\max} \leq M + 1.

  • Score ties (Sections 3 and 4; eEmR). We have generalized Proposition 4.1 and our estimators to arbitrary score ties under uniform tie-breaking, which makes our framework directly applicable to discrete LLM-judge scores.

  • Related work and clarifications (eEmR, ndPf). We have added the suggested literature on finite-sample OPE and estimator selection to Appendix A. In the main text, we now spell out more clearly the role of NN and MM after Theorem 4.2, and we state the requirement of auxiliary responses from the same reference model as a limitation.

Taken together, these revisions provide new theoretical guarantees, new experimental evidence, and a clearer picture of our framework's capabilities and limitations. We believe they significantly strengthen the paper, and we truly appreciate the reviewers' help in improving our work.

Best regards,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.