- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to allocate a fixed rollout budget to estimate a stochastic LLM’s average benchmark performance more precisely. It retains all tasks but adjusts their evaluation counts. Hierarchical Bayesian Neyman (HBN) allocation combines initial samples across tasks to allocate more evaluations to those with greater estimated variability, while preserving unbiasedness through a two-stage estimator with predetermined weights. HBN-async starts likely follow-up evaluations before all initial results arrive to reduce waiting. Experiments across 107 profiles from six Qwen checkpoints and 18 benchmark groups report lower average estimation variance than uniform allocation. Timing experiments use real model generation with simulated correctness feedback. HBN-async remains slower than uniform allocation for the same number of evaluations, but needs fewer evaluations to reach similar estimation accuracy. At larger budgets in the tested setting, this translates into overall time savings.
Critical strengths
-
The paper offers a useful approach to estimating benchmark performance more precisely under a limited evaluation budget. HBN shares information across tasks to allocate repeated evaluations while preserving an unbiased estimate under the stated assumptions. HBN-async complements this design by starting likely follow-up evaluations early without changing which results enter the final estimate. The contribution lies in combining these statistical and execution choices for low-budget stochastic LLM evaluation.
-
The experiments support the effectiveness of the proposed allocation method compared with existing approaches. Across 107 empirical profiles, HBN reduces mean estimation variance by 12.8%–33.6% relative to uniform allocation over the tested budgets. It also achieves lower average error than retrospectively tuned empirical, independent Bayesian, and sequential allocation baselines, although the gains over the latter two are smaller. These results support the value of the specific design beyond the established benefit of adaptive sampling.
Critical weaknesses
-
The gains over IBN and TS-Neyman are small, and their stability is uncertain. The statistical experiments treat success probabilities estimated from 1,024 generations per task as fixed values, but these estimates can vary with a different set of generations. The reported Monte Carlo uncertainty does not account for this source of variation. This limits confidence that the method provides a consistent improvement over these stronger baselines.
-
The reported time savings need further validation with real scoring feedback and its associated costs. The paper demonstrates time savings at approximately matched estimation accuracy for larger budgets, but the timing experiments use simulated correctness feedback and exclude scoring time. In practice, running tests on generated code delays feedback, and answer correctness may also be related to generation time. These factors can affect which follow-up requests HBN-async starts early and how long it must wait. This leaves the practical time advantage of the complete method uncertain beyond the controlled setting.
Questions for authors
Please explain how you would address the two weaknesses above, as they are the main reasons for my rating.
7 comments
Rebuttal (2/3)
Response to W2 / Q1 — End-to-end evaluation with actual scoring. Thank you for suggesting this additional evaluation. Our original experiment used synthetic rewards to align final allocations across synchronous and asynchronous execution, so that differences in the work allocated to individual tasks would not confound the comparison of execution overhead. We agree that evaluating the full pipeline with actual correctness feedback provides complementary evidence.
We apologize that time and resource constraints during the rebuttal limit the additional evaluation we can provide to two representative workloads. We report these results below and will expand the evaluation as much as possible in the next revision.
We evaluated MATH-100 and LiveCodeBench-medium (100 tasks each) with Qwen2.5-7B-Instruct on 8 H20 GPUs. Each profile/budget has 24 independent paired blocks comparing U(b), HBN-async, and rounded-equivalent Uniform (): 192 blocks and 576 policy executions. All policies use the same 64 scoring workers and a 30-second code-execution timeout.
We use the time-normalization convention of Table 4, now with actual correctness feedback and scoring overhead, and report Math and Code separately. Time entries are point estimates calculated from the reported mean runtimes.
Math (MATH-100)
| Budget | Rounded-equivalent budget | Rounded-equivalent Uniform time change vs. U(b) | HBN-async time change vs. U(b) | (pp) |
|---|---|---|---|---|
| 8 | 8 | -3.30% | +5.48% | -8.78 |
| 16 | 19 | +4.97% | +2.54% | +2.43 |
| 32 | 42 | +12.25% | +10.00% | +2.25 |
| 64 | 91 | +21.56% | +12.09% | +9.47 |
Code (LiveCodeBench-medium)
| Budget | Rounded-equivalent budget | Rounded-equivalent Uniform time change vs. U(b) | HBN-async time change vs. U(b) | (pp) |
|---|---|---|---|---|
| 8 | 9 | +11.56% | +6.87% | +4.69 |
| 16 | 20 | +22.66% | +3.71% | +18.94 |
| 32 | 46 | +40.87% | -8.10% | +48.97 |
| 64 | 104 | +61.26% | -10.56% | +71.82 |
The observed timing gains are larger on Code. Math shows modest timing gains relative to rounded-equivalent Uniform at b=16/32/64 and is slower at b=8. On Code, HBN-async is also faster than same-budget Uniform at b=32/64. Further analysis supports reduced scoring workload as the main explanation.
| Code workload metric | b=32 | b=64 |
|---|---|---|
| Answers generated and scored by each policy | 3,200 | 6,400 |
| Scoring CPU time change, HBN vs Uniform | −10.01% | −11.40% |
| Output-token count change, HBN vs Uniform | −1.41% | −1.57% |
HBN reduced sampling of several costly, frequently timing-out tasks with low observed success rates, while increasing sampling of tasks with intermediate success rates. Examples at b=64 show both directions; success rates, timeout rates, and per-answer scoring times below are measured under Uniform, and sample counts are means over 24 runs.
| Task | Observed success | Timeout rate | Scoring seconds / answer | Uniform samples | HBN samples |
|---|---|---|---|---|---|
| abc353_d | 0.00% | 96.74% | 29.14 | 64 | 39.08 |
| abc353_c | 0.00% | 88.15% | 26.68 | 64 | 39.25 |
| abc363_c | 51.04% | 4.04% | 13.74 | 64 | 169.96 |
these findings support a workload reduction from adaptive allocation, rather than an isolated scheduling speedup.
Rebuttal (3/3)
Planned revisions (W1/W2; Q1). We will incorporate the bootstrap stability results, real-scoring experiments, and workload analysis into the manuscript, including the resampling protocol, uncertainty measures, and both runtime and precision outcomes.
We will also clarify principled ex-ante parameter selection as an important contribution: the pilot size m and stage weight w are jointly determined by minimizing expected estimator variance under a specified reference prior. This principle provides the design used before evaluation; the stability experiments then assess the resulting policies without retuning their parameters.
We thank you for your time and constructive suggestions, which have helped us clarify the contribution and strengthen the empirical evidence.
Rebuttal (1/3)
Thank you for identifying profile-estimation uncertainty and real scoring feedback as the two main concerns. Your single question (Q1) asks us to address both weaknesses, which we label W1 and W2 in their original order. We have added experiments addressing each, with the following scope and limitations.
Response to W1 / Q1 — Bootstrap stability of the estimated profiles. We agree that the original MC intervals condition on the estimated profiles. To assess sensitivity to the observed response sample, we generated 32 bootstrap sets of all 107 profiles. Within each task, we resampled 1,024 binary outcomes with replacement from its 1,024 observed outcomes and recomputed the success rate. For a task with k observed successes, this is exactly equivalent to drawing k∗∼Binomial(1024,k/1024) and setting p∗=k∗/1024.
All original policy parameters remain fixed; no hindsight search is repeated after resampling. HBN retains its reference prior and ex-ante m,w; IBN retains its originally selected alpha and m,w; TS retains its originally selected alpha/m0. Each method still updates from its new pilot and adapts its allocation. Oracle recomputes the optimal positive-integer allocation using each bootstrap profile's probabilities; it remains an infeasible full-information reference.
We report results in the Table 1 format (Uniform=100%; lower is better). Within each bootstrap replicate, we average normalized risk across the 107 profiles, using each profile's own Uniform variance as the normalizer. Entries are then mean ± sample SD across the 32 replicates. Oracle/IBN/HBN report variance; TS reports actual MSE, including adaptive-estimation bias.
“Tuned IBN” refers to tuning on the original profiles, with parameters frozen during bootstrap evaluation. We use 256 pilot draws per profile/replicate for IBN/HBN and 256 complete trajectories for TS. The SDs include finite replay noise; Oracle uses analytic variance, so its SD reflects bootstrap variation alone.
HBN retains a positive mean advantage over both baselines at every budget. Against IBN, the profile-averaged gain is positive in all 32 bootstrap replicates at every budget. Against TS, it is positive in all 32 replicates at b=8/16/32 and in 28 of 32 at b=64. All 95% t intervals for the mean paired gains over the bootstrap replicates exclude zero (31 degrees of freedom). These intervals concern the bootstrap-average gains; the table's individual-method SDs are not intervals for the differences.
These results support robustness to resampling the observed responses without retuning the baselines. As an empirical bootstrap, this procedure preserves tasks with all-success or all-failure observations at probabilities 1 or 0; it assesses stability within the observed empirical response distributions.