acceptodds
Under review as a conference paper at ICLR 2027

Speculative Evaluation of Stochastic LLMs

Abstract

Evaluating a stochastic large language model is costly: benchmark scores estimate expected performance from randomized rollouts, yet uniform repetition ignores sharp differences in task-level rollout variance. We ask how to minimize the variance of a fixed-benchmark mean under an exact rollout budget. We develop Speculative Evaluation with a Hierarchical Bayesian Neyman (HBN) policy with pilot size and stage weight jointly chosen ex ante. It runs a short uniform pilot, pools per-task success counts with a hierarchical Bayesian model, and uses posterior expectations of task-level sampling variances for exact positive-integer Neyman allocation. To mitigate the pilot synchronization barrier, HBN-async speculatively executes continuations from partial pilot feedback and retains those selected by the final allocation. Across six checkpoints and 18 benchmark groups, we evaluate 107 nondegenerate benchmark-checkpoint profiles. For rollout budgets of 8-64 per task, Speculative Evaluation reduces variance relative to Uniform by 12.8%-33.6% on average across profiles, outperforming hindsight-tuned empirical and independent Bayesian baselines. Real-generation experiments that account for the pilot synchronization barrier show that HBN-async mitigates its overhead, helping translate statistical efficiency into practical evaluation benefits.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
LLM evaluation, stochastic evaluation, adaptive sampling, Neyman allocation, hierarchical Bayesian modeling, compute-efficient evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4u6kOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to allocate a fixed rollout budget to estimate a stochastic LLM’s average benchmark performance more precisely. It retains all tasks but adjusts their evaluation counts. Hierarchical Bayesian Neyman (HBN) allocation combines initial samples across tasks to allocate more evaluations to those with greater estimated variability, while preserving unbiasedness through a two-stage estimator with predetermined weights. HBN-async starts likely follow-up evaluations before all initial results arrive to reduce waiting. Experiments across 107 profiles from six Qwen checkpoints and 18 benchmark groups report lower average estimation variance than uniform allocation. Timing experiments use real model generation with simulated correctness feedback. HBN-async remains slower than uniform allocation for the same number of evaluations, but needs fewer evaluations to reach similar estimation accuracy. At larger budgets in the tested setting, this translates into overall time savings.

Critical strengths
  1. The paper offers a useful approach to estimating benchmark performance more precisely under a limited evaluation budget. HBN shares information across tasks to allocate repeated evaluations while preserving an unbiased estimate under the stated assumptions. HBN-async complements this design by starting likely follow-up evaluations early without changing which results enter the final estimate. The contribution lies in combining these statistical and execution choices for low-budget stochastic LLM evaluation.

  2. The experiments support the effectiveness of the proposed allocation method compared with existing approaches. Across 107 empirical profiles, HBN reduces mean estimation variance by 12.8%–33.6% relative to uniform allocation over the tested budgets. It also achieves lower average error than retrospectively tuned empirical, independent Bayesian, and sequential allocation baselines, although the gains over the latter two are smaller. These results support the value of the specific design beyond the established benefit of adaptive sampling.

Critical weaknesses
  1. The gains over IBN and TS-Neyman are small, and their stability is uncertain. The statistical experiments treat success probabilities estimated from 1,024 generations per task as fixed values, but these estimates can vary with a different set of generations. The reported Monte Carlo uncertainty does not account for this source of variation. This limits confidence that the method provides a consistent improvement over these stronger baselines.

  2. The reported time savings need further validation with real scoring feedback and its associated costs. The paper demonstrates time savings at approximately matched estimation accuracy for larger budgets, but the timing experiments use simulated correctness feedback and exclude scoring time. In practice, running tests on generated code delays feedback, and answer correctness may also be related to generation time. These factors can affect which follow-up requests HBN-async starts early and how long it must wait. This leaves the practical time advantage of the complete method uncertain beyond the controlled setting.

Questions for authors

Please explain how you would address the two weaknesses above, as they are the main reasons for my rating.

7 comments
AuthorsOpenReview ↗

Rebuttal (1/3)

Thank you for identifying profile-estimation uncertainty and real scoring feedback as the two main concerns. Your single question (Q1) asks us to address both weaknesses, which we label W1 and W2 in their original order. We have added experiments addressing each, with the following scope and limitations.

Response to W1 / Q1 — Bootstrap stability of the estimated profiles. We agree that the original MC intervals condition on the estimated profiles. To assess sensitivity to the observed response sample, we generated 32 bootstrap sets of all 107 profiles. Within each task, we resampled 1,024 binary outcomes with replacement from its 1,024 observed outcomes and recomputed the success rate. For a task with k observed successes, this is exactly equivalent to drawing k∗∼Binomial⁡(1024,k/1024)k^* \sim \operatorname{Binomial}(1024,k/1024) and setting p∗=k∗/1024p^*=k^*/1024.

All original policy parameters remain fixed; no hindsight search is repeated after resampling. HBN retains its reference prior and ex-ante m,w; IBN retains its originally selected alpha and m,w; TS retains its originally selected alpha/m0. Each method still updates from its new pilot and adapts its allocation. Oracle recomputes the optimal positive-integer allocation using each bootstrap profile's probabilities; it remains an infeasible full-information reference.

We report results in the Table 1 format (Uniform=100%; lower is better). Within each bootstrap replicate, we average normalized risk across the 107 profiles, using each profile's own Uniform variance as the normalizer. Entries are then mean ± sample SD across the 32 replicates. Oracle/IBN/HBN report variance; TS reports actual MSE, including adaptive-estimation bias.

BudgetOracleTuned IBNTS-NeymanHBN
8(40.403 ± 0.057)%(89.520 ± 0.038)%(88.906 ± 0.653)%(87.223 ± 0.036)%
16(39.507 ± 0.058)%(81.885 ± 0.053)%(82.562 ± 0.816)%(80.090 ± 0.054)%
32(39.088 ± 0.058)%(74.087 ± 0.076)%(74.984 ± 0.730)%(72.984 ± 0.070)%
64(38.885 ± 0.059)%(66.839 ± 0.075)%(67.110 ± 0.673)%(66.347 ± 0.062)%

“Tuned IBN” refers to tuning on the original profiles, with parameters frozen during bootstrap evaluation. We use 256 pilot draws per profile/replicate for IBN/HBN and 256 complete trajectories for TS. The SDs include finite replay noise; Oracle uses analytic variance, so its SD reflects bootstrap variation alone.

HBN retains a positive mean advantage over both baselines at every budget. Against IBN, the profile-averaged gain is positive in all 32 bootstrap replicates at every budget. Against TS, it is positive in all 32 replicates at b=8/16/32 and in 28 of 32 at b=64. All 95% t intervals for the mean paired gains over the bootstrap replicates exclude zero (31 degrees of freedom). These intervals concern the bootstrap-average gains; the table's individual-method SDs are not intervals for the differences.

These results support robustness to resampling the observed responses without retuning the baselines. As an empirical bootstrap, this procedure preserves tasks with all-success or all-failure observations at probabilities 1 or 0; it assesses stability within the observed empirical response distributions.

AuthorsOpenReview ↗

Rebuttal (2/3)

Response to W2 / Q1 — End-to-end evaluation with actual scoring. Thank you for suggesting this additional evaluation. Our original experiment used synthetic rewards to align final allocations across synchronous and asynchronous execution, so that differences in the work allocated to individual tasks would not confound the comparison of execution overhead. We agree that evaluating the full pipeline with actual correctness feedback provides complementary evidence.

We apologize that time and resource constraints during the rebuttal limit the additional evaluation we can provide to two representative workloads. We report these results below and will expand the evaluation as much as possible in the next revision.

We evaluated MATH-100 and LiveCodeBench-medium (100 tasks each) with Qwen2.5-7B-Instruct on 8 H20 GPUs. Each profile/budget has 24 independent paired blocks comparing U(b), HBN-async, and rounded-equivalent Uniform (UeqU_{\mathrm{eq}}): 192 blocks and 576 policy executions. All policies use the same 64 scoring workers and a 30-second code-execution timeout.

We use the time-normalization convention of Table 4, now with actual correctness feedback and scoring overhead, and report Math and Code separately. Time entries are point estimates calculated from the reported mean runtimes.

Math (MATH-100)

BudgetRounded-equivalent budgetRounded-equivalent Uniform time change vs. U(b)HBN-async time change vs. U(b)Δ\Delta (pp)
88-3.30%+5.48%-8.78
1619+4.97%+2.54%+2.43
3242+12.25%+10.00%+2.25
6491+21.56%+12.09%+9.47

Code (LiveCodeBench-medium)

BudgetRounded-equivalent budgetRounded-equivalent Uniform time change vs. U(b)HBN-async time change vs. U(b)Δ\Delta (pp)
89+11.56%+6.87%+4.69
1620+22.66%+3.71%+18.94
3246+40.87%-8.10%+48.97
64104+61.26%-10.56%+71.82

The observed timing gains are larger on Code. Math shows modest timing gains relative to rounded-equivalent Uniform at b=16/32/64 and is slower at b=8. On Code, HBN-async is also faster than same-budget Uniform at b=32/64. Further analysis supports reduced scoring workload as the main explanation.

Code workload metricb=32b=64
Answers generated and scored by each policy3,2006,400
Scoring CPU time change, HBN vs Uniform−10.01%−11.40%
Output-token count change, HBN vs Uniform−1.41%−1.57%

HBN reduced sampling of several costly, frequently timing-out tasks with low observed success rates, while increasing sampling of tasks with intermediate success rates. Examples at b=64 show both directions; success rates, timeout rates, and per-answer scoring times below are measured under Uniform, and sample counts are means over 24 runs.

TaskObserved successTimeout rateScoring seconds / answerUniform samplesHBN samples
abc353_d0.00%96.74%29.146439.08
abc353_c0.00%88.15%26.686439.25
abc363_c51.04%4.04%13.7464169.96

these findings support a workload reduction from adaptive allocation, rather than an isolated scheduling speedup.

AuthorsOpenReview ↗

Rebuttal (3/3)

Planned revisions (W1/W2; Q1). We will incorporate the bootstrap stability results, real-scoring experiments, and workload analysis into the manuscript, including the resampling protocol, uncertainty measures, and both runtime and precision outcomes.

We will also clarify principled ex-ante parameter selection as an important contribution: the pilot size m and stage weight w are jointly determined by minimizing expected estimator variance under a specified reference prior. This principle provides the design used before evaluation; the stability experiments then assess the resulting policies without retuning their parameters.

We thank you for your time and constructive suggestions, which have helped us clarify the contribution and strengthen the empirical evidence.

Reviewer_E3VVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to spend a fixed budget (i.e., number of rollout) when estimating a stochastic LLM's mean accuracy on a specific benchmark. It is designed primarily for binary outcome tasks. The paper proposes a new method, speculative evaluation with hierarchical Bayesian Neyman (HBN) policy. HBN runs a uniform pilot, pools pilot counts in a hierarchical binomial model, and allocates the remaining rollouts by exact integer Neyman allocation on posterior expected task variances. To reduce the machine waiting time, it proposes a async version of HBN, which starts continuations early from partial pilot results and uses fixed request IDs to avoid latency-dependent selections. The authors perform experiments on 6 Qwen model checkpoints x 18 benchmarks. The results show that it can reduce variance by 12.8-33.6% comparing with uniform baseline. Real-generation timing shows a wall-clock benefit for budgets of 16 and above.

Critical strengths
  1. The paper proposes a simple but practical way to allocate rollout budget in stochastic LLM evaluation, which applies to any binary tasks. HBN spends a short pilot to learn which tasks/prompts are noisy then shift the remaining rollout towards them. This estimate stays unbiased and no tuning is needed. The experiments on 6 checkpoints and 18 benchmarks show a 13-34% reduction on variance compared with baseline. It improves on almost every profile which is quite nice.

  2. The paper proposes a async variants of the method, and it reduces most of the waiting time for the pilot. The results show that HBN async variant only adds 3-7% time compare to uniform baseline. Each rollout is expensive in settings like long reasoning traces, so a drop-in way to get tighter error bars from the same budget is a useful contribution.

  3. The evaluation is well designed and broad. It goes beyond just statistical simulation, which measures also actual generation time. The author perform experiments on real generation and show that HBN-async only adds 3-7% time compare to uniform baseline. I would consider this as a contribution itself, as it shows efficiency of the method and make it practical as a replacement for the baseline.

Critical weaknesses

The benefit of hierarchical pooling is the core idea of the method, however, it is not well supported by the experiments. The paper presents pooling pilot counts across tasks as what makes HBN work. A simpler baseline (IBN) that drops pooling and uses an independent Beta prior seems does similarly. And the advantage of HBN diminishes as the budget grows. I think the author need to prove (or justify) what benefit the proposed HBN brings us. Otherwise, I would think the main benefits come from classical two stage Neyman allocation with a smoothed pilot. This will make the novelty of the paper weaker.

Questions for authors

I am curious how much hierarchical pooling contributes to the performance gain. Do you have the numbers for IBN with a fixed default prior (e.g., alpha = 0.5 and alpha = 1) under the same ex-ante pilot-and-weight design as HBN? It would also help to know, at each budget, the fraction of profiles on which HBN beats tuned IBN, with Monte Carlo standard errors.

2 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

Thank you for asking us to clarify the contribution of hierarchical pooling. We report the requested fixed-default IBN comparisons (Q1(a)) and profile-level win fractions with MC errors (Q1(b)), and clarify both the original ablation and the role of our ex-ante Bayesian design (W1).

Response to W1 — Pooling ablation and principled ex-ante design. The original IBN baseline was designed to ablate hierarchical pooling: it retains the two-stage estimator and exact integer Neyman allocator, but replaces the hierarchical model with independent Beta priors. We agree that classical two-stage Neyman allocation contributes to the improvement over Uniform; the full improvement should not be attributed to pooling alone.

Our contribution also includes a Bayesian principle for jointly determining the pilot size m and stage weight w before evaluation: minimizing expected estimator variance under a specified reference prior. This addresses design choices that classical allocation alone leaves open. For example, Blackwell et al. provide empirical simulation guidance for pilot size and use sample-size-based stage weights, without an ex-ante joint optimization principle for m,w. We apply our principle to IBN as well, optimizing each candidate's m,w under its matched Beta prior. Thus, IBN's competitive performance already benefits from this shared design contribution. Once the prior is specified, these designs require no tuning on evaluation-profile performance.

Response to W1 / Q1(a) — Fixed-default IBN with comparable ex-ante design. For IBN with alpha=0.5 or 1, we retain its existing m,w jointly optimized before evaluation under the matched Beta(alpha,alpha) prior. This uses the same ex-ante design principle as HBN; it does not force the same numerical m,w across different priors. Neither default is tuned on the evaluation profiles. Both were already included in the original 23-point IBN search grid, so the following results expose candidates within the reported hindsight envelope.

Across 107 profiles with 8,192 replay draws per profile and budget, mean Uniform-normalized variance is shown below. Entries are mean ± one MC standard error, in percentage points of Uniform variance (Uniform=100%).

BudgetHBNIBN alpha=0.5IBN alpha=1Tuned IBN
887.2476 ± 0.004994.3768 ± 0.002298.3914 ± 0.001489.5572 ± 0.0054
1680.1233 ± 0.006087.2921 ± 0.003091.3723 ± 0.001981.9287 ± 0.0061
3273.0299 ± 0.006080.3851 ± 0.003485.4118 ± 0.002274.1607 ± 0.0065
6466.4368 ± 0.005573.7076 ± 0.003179.1440 ± 0.002366.9634 ± 0.0058

HBN has lower mean variance than both fixed defaults at every budget, with a substantially larger advantage than against hindsight-tuned IBN. The much smaller gap against tuned IBN must be interpreted with its tuning advantage: for each budget, IBN selects one global alpha retrospectively by mean risk on the same 107 evaluation profiles, after each candidate receives its own matched-prior m,w. It does not select a different alpha for each profile. HBN's design uses no evaluation-profile performance. The original comparison therefore deliberately gives IBN extra information; it is not an equally informed ex-ante comparison.

Response to W1 / Q1(b) — Profile-level wins and the hindsight advantage. Below, a win means lower estimated variance on a profile. Entries are win percentages ± approximate MC standard errors in percentage points; all denominators are 107. Win-fraction MC errors use a Gaussian perturbation approximation based on each profile's difference and its simulation SE. They are conditional on the fixed profiles, designs, and selected alpha, not binomial sampling errors over a population of profiles.

BudgetHBN vs IBN alpha=0.5HBN vs IBN alpha=1HBN vs Tuned IBN
889.72 ± 0.0399.07 ± 0.3770.09 ± 1.02
1690.65 ± 0.1597.20 (SE <0.01)62.62 ± 0.86
3292.52 ± 0.5496.26 (SE <0.01)43.93 ± 1.06
6492.52 ± 0.4196.26 (SE <0.01)41.12 ± 0.57

HBN does not win on a majority of profiles against hindsight-tuned IBN at b=32/64, although its mean variance remains lower because gains on winning profiles outweigh losses elsewhere. The hindsight envelope is a deliberately advantaged benchmark, not a fair comparison under equal ex-ante information: IBN's prior is selected after observing performance on the evaluation profiles, while HBN receives no such tuning information. We therefore encourage interpreting these win fractions alongside the fixed-prior results, which more directly compare policies without hindsight tuning. Against both specified default priors, HBN wins on a large majority of profiles at every budget, with a substantially larger mean variance improvement.

AuthorsOpenReview ↗

Rebuttal (2/2)

Planned revisions (W1; Q1(a)/Q1(b)). We will add the fixed-prior IBN results and profile-level win fractions with MC errors to the manuscript, and clearly distinguish the hindsight-tuned benchmark from comparisons using priors specified before evaluation.

We will strengthen the contribution statement and method discussion to emphasize principled ex-ante parameter selection as an important contribution: jointly optimizing m,w under a specified prior to minimize expected estimator variance. We will explain that IBN already benefits from this shared design principle, alongside clarifying the separate contribution of hierarchical pooling.

We thank you for your time and constructive suggestions, which have helped us clarify the contribution and strengthen the empirical evidence.

Reviewer_LXSGOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+2)
Read review
Paper summary

The paper studies how to allocate repeated LLM evaluations across a fixed benchmark to estimate mean accuracy more precisely. Its HBN method uses a uniform pilot, a hierarchical Beta–Binomial model, and exact integer Neyman allocation, with the pilot size and estimator weight chosen before evaluation. HBN-async executes speculative continuations while awaiting pilot feedback. Across 107 empirical benchmark–checkpoint profiles, HBN reports 12.8%–33.6% lower variance than Uniform. Separate real-generation timing experiments, using synthetic correctness feedback, assess the execution overhead.

Critical strengths
  1. Careful treatment of adaptive estimation. The method preserves unbiasedness by combining the pilot and fresh continuation estimates with a precommitted weight. The asynchronous implementation retains designated request IDs rather than whichever responses finish first. These choices address real statistical pitfalls.

  2. Substantial statistical evaluation. The profiles are estimated from 1,024 real responses per problem, and the comparisons include hindsight-tuned controls and an adapted TS-Neyman baseline [1]. The conditional-profile results provide credible evidence of variance reduction over Uniform.

Reference

[1] Morikawa, Kosuke, Mst Moushumi Pervin, and Jae Kwang Kim. "TS-Neyman: Posterior Sampling for Adaptive Stratified Estimation." arXiv preprint arXiv:2606.08418 (2026).

Critical weaknesses
  1. The additional methodological contribution is not sufficiently established. Two-stage Neyman designs with unbiased combination of stages and shrinkage for binary outcomes already exist [1], as does exact integer allocation through marginal variance reductions [2]. The headline gains against Uniform therefore do not establish what the hierarchical treatment adds. In Table 1, HBN improves over tuned IBN by only 0.6–2.4 percentage points of Uniform-normalized variance; the margin over the adapted TS-Neyman baseline is similarly small (Table 8). Moreover, IBN uses a globally tuned symmetric Beta prior, whereas HBN adapts the mean and dispersion within each profile. A simpler pooled empirical-Bayes Beta(a,b) allocation is not tested. Without that comparison, the paper does not establish that its additional modeling and design machinery provides a substantial advance.

  2. The practical contribution lacks an end-to-end evaluation with actual correctness feedback. Section 4.3 uses real generation but assigns synthetic Bernoulli rewards independently of the generated answers, and excludes judging and code-execution latency. This controls the final allocation, but removes the within-task relationship between correctness and latency that affects partial-pilot speculation. Each setting is also timed only once at a single concurrency configuration. Table 4 consequently combines conditional-profile variance estimates with a separate timing experiment, rather than demonstrating the precision–runtime tradeoff in an actual evaluation pipeline. The limitations are acknowledged, but they affect the central evidence for HBN-async.

Reference

[1] Blackwell, Matthew, Nicole E. Pashley, and Dominic Valentino. "Batch adaptive designs to improve efficiency in social science experiments." 2022,

[2] Wright, Tommy. "Two optimal exact sample allocation algorithms: Sampling variance decomposition is key." Statistics (2016): 03.

Questions for authors
  1. How does HBN compare with a pooled empirical-Bayes Beta(a,b)Beta(a,b) allocation using the same two-stage estimator and integer allocator? With comparable pilot/weight design, how much improvement remains from integrating over the hierarchical posterior?

  2. Does HBN-async retain its precision–runtime advantage when it receives the generated answers’ actual correctness labels, including judging or code-execution latency? Repeated measurements on representative math and code workloads would materially affect my assessment.

9 comments
AuthorsOpenReview ↗

Rebuttal (1/3)

Thank you for identifying the missing pooled-EB comparison and the need for actual scoring feedback. We have added both experiments and distinguish the statistical contribution from what the new systems evidence establishes. W1/W2 and Q1/Q2 below refer to the weaknesses and questions in their original order.

Response to W1 / Q1 — Hierarchical integration beyond pooled empirical Bayes. We fitted a shared Beta(a,b) prior by Beta-binomial marginal maximum likelihood on all pilot counts, then used each task's posterior expected Bernoulli variance with the same two-stage estimator, exact integer allocator, and rollout budget as HBN. EB's pilot size and estimator weight (m,w) are jointly optimized ex ante to minimize its expected estimator variance under HBN's reference prior: independent ξ,δ∼Beta(1,1)\xi, \delta \sim Beta(1,1). The design uses the same search space and simulation budget as HBN, without tuning on evaluation-profile performance.

We extend Table 1 with pooled EB and the appendix's Bernoulli TS-Neyman baseline, retaining the original Uniform-normalized scale (Uniform = 100%; lower is better). TS reports Uniform-normalized MSE, including adaptive-estimation bias; for the other methods, MSE equals variance. † Hindsight tuning selects one global configuration per budget on the evaluation profiles. EB and HBN use ex-ante m,w designs under the same reference prior. EB and HBN entries show mean ± one MC standard error on the Uniform=100% scale, conditional on the fixed profiles and designs (8,192 replay draws per profile).

BudgetOracleTuned EN†Tuned IBN†TS-Neyman†EBHBN
840.734%94.324%89.557%88.602%(87.638 ± 0.005)%(87.248 ± 0.005)%
1639.880%92.646%81.929%82.094%(80.481 ± 0.006)%(80.123 ± 0.006)%
3239.479%91.936%74.161%74.718%(73.355 ± 0.006)%(73.030 ± 0.006)%
6439.286%91.469%66.963%67.222%(66.714 ± 0.006)%(66.437 ± 0.005)%

HBN outperforms EB at every budget, with an advantage consistently resolved beyond Monte Carlo error. All paired 95% MC intervals for the gain over EB exclude zero, using 8,192 replay draws per profile and budget and conditioning on the fixed profiles and designs.

Response to W1 — Two distinct contributions.

  1. A principled ex-ante design for pooled two-stage evaluation. Blackwell et al. (§§4–5) do not provide a principle for jointly selecting the pilot size m and stage weight w before evaluation: their pilot-size recommendations rely on empirical simulations, and their stage weights follow relative batch sizes. We provide this design principle by jointly minimizing expected estimator variance under a specified reference prior. Once that prior is specified, m,w are determined without tuning on evaluation-profile performance. This is an important contribution of our framework, applicable to both hierarchical Bayes and pooled EB, building on established Bayesian decision theory.

  2. An additional gain from hierarchical integration under the same design principle. We equip pooled EB with our reference-prior m,w optimization procedure, using the same reference prior, estimator, and integer allocator as HBN. To our knowledge, this complete EB specification has not appeared in prior work. Its close performance therefore already benefits from our first contribution. Within this shared framework, HBN integrates over the hierarchical posterior instead of plugging in fitted hyperparameters, aligning allocation with the same Bayesian risk criterion. It achieves a small but consistent improvement over EB at every tested budget, with all paired 95% MC intervals above zero. Both methods use reusable, low-dimensional offline design computations whose cost is expected to be marginal relative to LLM evaluation.

The original IBN and TS-Neyman comparisons also deliberately favor the baselines through hindsight tuning. IBN receives a matched-prior ex-ante m,w for every candidate alpha, then one alpha is selected per budget on the evaluation profiles. The TS appendix similarly selects one global alpha/m0 pair per budget. These are demanding hindsight benchmarks, rather than equally informed ex-ante policies; we will make this distinction more prominent.

For fixed-prior comparisons, we use Beta(0.5,0.5) and Beta(1,1). IBN retains its matched-prior ex-ante m,w for each prior. Bernoulli TS uses the same fixed warm-start m0=2 at every budget, the smallest warm-start in the appendix grid; we report these configurations directly, without minimizing over warm-starts. These are fixed configurations of our Bernoulli adaptation, not prescribed defaults of the original TS paper. Both priors are already included in the respective hindsight grids.

AuthorsOpenReview ↗

Rebuttal (2/3)

BudgetIBN Beta(0.5,0.5)IBN Beta(1,1)TS Beta(0.5,0.5), m0=2TS Beta(1,1), m0=2HBN
894.377 ± 0.00298.391 ± 0.00194.144 ± 0.14296.081 ± 0.14687.248 ± 0.005
1687.292 ± 0.00391.372 ± 0.00287.531 ± 0.13290.538 ± 0.13680.123 ± 0.006
3280.385 ± 0.00385.412 ± 0.00280.110 ± 0.12182.969 ± 0.12573.030 ± 0.006
6473.708 ± 0.00379.144 ± 0.00272.439 ± 0.11175.246 ± 0.11466.437 ± 0.005

Mean Uniform-normalized MSE ± one MC standard error, in percentage points (Uniform=100%; lower is better), across 107 profiles with 8,192 replays per profile. For HBN and IBN, MSE equals variance; TS includes adaptive-estimation bias. MC errors condition on the fixed profiles and configurations.

HBN outperforms all four fixed configurations at every budget, with substantially larger gains than against the hindsight-tuned baselines. This makes the tuning advantage of the original comparisons explicit.

Response to W2 / Q2 — Actual correctness feedback and repeated end-to-end measurement. Thank you for suggesting this additional evaluation. Our original experiment used synthetic rewards to align final allocations across synchronous and asynchronous execution, so that differences in the work allocated to individual tasks would not confound the comparison of execution overhead. We agree that evaluating the full pipeline with actual correctness feedback provides complementary evidence.

We apologize that time and resource constraints during the rebuttal limit the additional evaluation we can provide to two representative workloads. We report these results below and will expand the evaluation in the next revision.

We evaluated MATH-100 and LiveCodeBench-medium (100 tasks each) with Qwen2.5-7B-Instruct on 8 H20 GPUs. Each profile/budget has 24 independent paired blocks comparing U(b), HBN-async, and rounded-equivalent Uniform (UeqU_{\mathrm{eq}}): 192 blocks and 576 policy executions. All policies use the same 64 scoring workers and a 30-second code-execution timeout.

We use the time-normalization convention of Table 4, now with actual correctness feedback and scoring overhead, and report Math and Code separately. Time entries are point estimates calculated from the reported mean runtimes.

Math (MATH-100)

BudgetRounded-equivalent budgetRounded-equivalent Uniform time change vs. U(b)HBN-async time change vs. U(b)Δ\Delta (pp)
88-3.30%+5.48%-8.78
1619+4.97%+2.54%+2.43
3242+12.25%+10.00%+2.25
6491+21.56%+12.09%+9.47

Code (LiveCodeBench-medium)

BudgetRounded-equivalent budgetRounded-equivalent Uniform time change vs. U(b)HBN-async time change vs. U(b)Δ\Delta (pp)
89+11.56%+6.87%+4.69
1620+22.66%+3.71%+18.94
3246+40.87%-8.10%+48.97
64104+61.26%-10.56%+71.82

The observed timing gains are larger on Code. Math shows modest timing gains relative to rounded-equivalent Uniform at b=16/32/64 and is slower at b=8. On Code, HBN-async is also faster than same-budget Uniform at b=32/64. Further analysis supports reduced scoring workload as the main explanation.

Code workload metricb=32b=64
Answers generated and scored by each policy3,2006,400
Scoring CPU time change, HBN vs Uniform−10.01%−11.40%
Output-token count change, HBN vs Uniform−1.41%−1.57%

HBN reduced sampling of several costly, frequently timing-out tasks with low observed success rates, while increasing sampling of tasks with intermediate success rates. Examples at b=64 show both directions; success rates, timeout rates, and per-answer scoring times below are measured under Uniform, and sample counts are means over 24 runs.

TaskObserved successTimeout rateScoring seconds / answerUniform samplesHBN samples
abc353_d0.00%96.74%29.146439.08
abc353_c0.00%88.15%26.686439.25
abc363_c51.04%4.04%13.7464169.96

these findings support a workload reduction from adaptive allocation, rather than an isolated scheduling speedup.

AuthorsOpenReview ↗

Rebuttal (3/3)

Planned revisions. We will incorporate the pooled-EB and fixed-prior comparisons, the expanded Table 1, and the real-scoring results and workload analysis into the manuscript. We will also make the hindsight tuning of the original baselines more explicit.

We will revise the contribution statement and related-work discussion to emphasize principled ex-ante parameter selection as an important contribution: jointly determining the pilot size m and stage weight w by minimizing expected estimator variance under a specified reference prior. We will clarify that the EB comparator also receives this design procedure, so its close performance reflects the additional benefit of hierarchical integration within a shared design framework.

We thank you for your time and constructive suggestions, which have helped us clarify the contribution and strengthen the empirical evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.