- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how to allocate a fixed number of response generations across benchmark items to estimate a model's mean accuracy. It proposes sequential policies that pool target responses within historical difficulty strata or regularize item accuracies using a historical difficulty order. The policies prioritize estimated response variability while bounding differences in item evaluation counts; Itemwise + Order also uses fitted and interpolated probabilities in the final estimate. Experiments on AIME 2025, HMMT February 2025, arXivMath, and GPQA Diamond simulate responses from empirical item accuracies. The paper reports lower estimation errors across several budgets and reference-information settings, including a reported 17% generation reduction at a ranking-recovery threshold in an AIME experiment.
Critical strengths
-
The paper addresses how to distribute repeated generations when estimating a fixed benchmark score. This is relevant to small reasoning benchmarks, where stochastic responses make stable evaluation expensive.
-
Transferring difficulty structure while estimating accuracies from the target model's own responses is a reasonable design for few-reference or weak-reference settings. The policies are simple, and their pilot evaluations are explicitly included in the budget.
Critical weaknesses
-
Proposition 2 analyzes allocation proportional to estimated standard deviations, whereas §3 repeatedly selects the admissible item or stratum with the largest standard deviation. Count constraints do not make these rules equivalent, even with accurate priorities. The analysis therefore does not establish how closely the proposed greedy rule approximates the oracle allocation.
-
The experiments treat accuracies estimated from 8–128 stored responses per item as ground truth, then simulate Bernoulli outcomes. The reported Monte Carlo errors omit uncertainty in those probabilities and the reference pools. Representative validation with fresh model responses is needed to substantiate the practical evaluation gains.
Questions for authors
-
How does repeatedly selecting the largest admissible standard deviation track the proportional allocation analyzed in Proposition 2? Can you compare it with a policy that explicitly tracks those proportions, using the same estimates and budget constraints?
-
What bias and variance does the actual estimator exhibit when observations guide allocation and are reused in the final score? Please include the low-budget regime with unobserved items, or clearly delimit which policies and regimes the theory covers.
-
Can Uniform sampling be combined with the same final isotonic fitting and interpolation as Itemwise + Order? Please match pilot budgets and count constraints to isolate the benefit of adaptive selection.
2 comments
Thank you for the revisions and additional experiments. I appreciate the corrected allocation rule and the ablation separating sampling from final estimation. However, replacing the original selection rule with proportional-allocation tracking seems a substantive methodological change, rather than merely a clarification. I will maintain my rating.
We appreciate your constructive feedback and recognition of our approach's relevance with few or weak reference models. We address your concerns point by point below.
Thank you for identifying this discrepancy. The original rule, which selects the admissible item (or stratum) with the largest estimated standard deviation, does not necessarily track proportional allocation. We revised the allocation rule and now state it in Equation 6. The revised rule selects the admissible item (or stratum) with the largest difference between its target count, computed from the estimated proportional allocation, and its current count. We also updated the manuscript's experimental results using this rule.
To address the second part of your question, we compared the two rules using the same estimation formulas and budget constraints across ten target models on AIME 2025. Entries report MAE in percentage points.
For Itemwise + Order at l=1, the two rules select exactly the same items at every step. The two rules achieve comparable mean absolute error at both budgets, while the revised rule makes the connection to proportional allocation explicit.
We agree that reusing observations that guide allocation can introduce bias. We examine this effect through a two-stage evaluation on AIME 2025 and MATH-500 with m<n, so some items remain unobserved. First, we collect pilot responses sampling uniformly within each stratum and use them to determine how the remaining budget is allocated across strata. We then collect additional responses according to this fixed allocation. We compute the final estimate using either the additional responses alone or both the pilot and additional responses. We summarize the results below, where bias, RMSE, and MAE are reported in percentage points and variance in squared percentage points.
Including pilot responses increases the magnitude of the bias but reduces variance, leading to lower RMSE. This supports their reuse in this setting.
We will further clarify that our theory assumes the pilot responses are independent of those used for final estimation and does not directly cover their reuse.
Yes. We apply the same isotonic fitting (used by Itemwise + Order) to responses collected through Uniform sampling on AIME 2025 with an evaluation budget of m=20. Both final estimators use the same sampled responses, so the comparison isolates the effect of isotonic estimation. It reduces mean absolute error from 6.13 to 5.33 percentage points. We added this comparison to Section 5.4.
Thank you for highlighting this concern. For benchmarks with sufficient responses per item (ex. 32 responses per item on AIME 2025), we updated the evaluation protocol to use available model responses directly instead of generating outcomes from estimated item-level accuracies. In each repetition, we randomly split each item's responses into two subsets: one is used for adaptive sampling and accuracy estimation (ex. 4 responses), while the other defines the held-out benchmark score (ex. the remaining 28 responses), used as a proxy for the true accuracy when calculating estimation error. Both the estimate and the held-out score therefore vary across repetitions. The main improvements of our methods persist under this revised protocol, described in Section 5.