acceptodds
Under review as a conference paper at ICLR 2027

Efficient Evaluation of Reasoning Benchmarks with Limited Historical Information

Abstract

Accuracy estimates on reasoning benchmarks can be noisy, with uncertainty arising from the small number of benchmark items and the stochastic generation of responses by language models. Repeated evaluation can reduce this uncertainty but requires additional generations, raising the question of how to allocate a generation budget across items to minimize error in estimating benchmark accuracy. We formulate this problem as sequential item selection and develop adaptive sampling policies that leverage intermediate evaluation results from the target model to guide subsequent sampling. Our sampling policies also incorporate structural information about item difficulty from historical evaluations of other models on the benchmark, such as difficulty strata or a relative difficulty order. We provide theoretical analysis showing that informative difficulty strata can reduce estimation error, while a reliable relative difficulty order can tighten an estimation error bound. Experiments show that the resulting policies reduce error in estimating benchmark accuracy across a range of budgets and levels of available historical information. We summarize practical guidelines for choosing an evaluation strategy according to the historical information available, enabling efficient and reliable evaluation.

Primary area:
datasets and benchmarks
Keywords:
benchmark, evaluation, language model, LLM, sampling, stratified, adaptive, reasoning, budget

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_URgDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to allocate a fixed number of response generations across benchmark items to estimate a model's mean accuracy. It proposes sequential policies that pool target responses within historical difficulty strata or regularize item accuracies using a historical difficulty order. The policies prioritize estimated response variability while bounding differences in item evaluation counts; Itemwise + Order also uses fitted and interpolated probabilities in the final estimate. Experiments on AIME 2025, HMMT February 2025, arXivMath, and GPQA Diamond simulate responses from empirical item accuracies. The paper reports lower estimation errors across several budgets and reference-information settings, including a reported 17% generation reduction at a ranking-recovery threshold in an AIME experiment.

Critical strengths
  1. The paper addresses how to distribute repeated generations when estimating a fixed benchmark score. This is relevant to small reasoning benchmarks, where stochastic responses make stable evaluation expensive.

  2. Transferring difficulty structure while estimating accuracies from the target model's own responses is a reasonable design for few-reference or weak-reference settings. The policies are simple, and their pilot evaluations are explicitly included in the budget.

Critical weaknesses
  1. Proposition 2 analyzes allocation proportional to estimated standard deviations, whereas §3 repeatedly selects the admissible item or stratum with the largest standard deviation. Count constraints do not make these rules equivalent, even with accurate priorities. The analysis therefore does not establish how closely the proposed greedy rule approximates the oracle allocation.

  2. The experiments treat accuracies estimated from 8–128 stored responses per item as ground truth, then simulate Bernoulli outcomes. The reported Monte Carlo errors omit uncertainty in those probabilities and the reference pools. Representative validation with fresh model responses is needed to substantiate the practical evaluation gains.

Questions for authors
  1. How does repeatedly selecting the largest admissible standard deviation track the proportional allocation analyzed in Proposition 2? Can you compare it with a policy that explicitly tracks those proportions, using the same estimates and budget constraints?

  2. What bias and variance does the actual estimator exhibit when observations guide allocation and are reused in the final score? Please include the low-budget regime with unobserved items, or clearly delimit which policies and regimes the theory covers.

  3. Can Uniform sampling be combined with the same final isotonic fitting and interpolation as Itemwise + Order? Please match pilot budgets and count constraints to isolate the benefit of adaptive selection.

2 comments

We appreciate your constructive feedback and recognition of our approach's relevance with few or weak reference models. We address your concerns point by point below.

Q1 (& W1). How does repeatedly selecting the largest admissible standard deviation track proportional allocation? Can you compare it with a policy that explicitly tracks those proportions?

Thank you for identifying this discrepancy. The original rule, which selects the admissible item (or stratum) with the largest estimated standard deviation, does not necessarily track proportional allocation. We revised the allocation rule and now state it in Equation 6. The revised rule selects the admissible item (or stratum) with the largest difference between its target count, computed from the estimated proportional allocation, and its current count. We also updated the manuscript's experimental results using this rule.

To address the second part of your question, we compared the two rules using the same estimation formulas and budget constraints across ten target models on AIME 2025. Entries report MAE in percentage points.

Sampling policymmllOriginal ruleRevised rule
Itemwise + Order1515.7325.732
Stratified + Order1515.9176.106
Itemwise + Order5023.1343.133
Stratified + Order5023.0673.084

For Itemwise + Order at l=1l=1, the two rules select exactly the same items at every step. The two rules achieve comparable mean absolute error at both budgets, while the revised rule makes the connection to proportional allocation explicit.

Q2. What bias and variance does the actual estimator exhibit when observations guide allocation and are reused in the final score, including the low-budget regime with unobserved items?

We agree that reusing observations that guide allocation can introduce bias. We examine this effect through a two-stage evaluation on AIME 2025 and MATH-500 with m<nm<n, so some items remain unobserved. First, we collect pilot responses sampling uniformly within each stratum and use them to determine how the remaining budget is allocated across strata. We then collect additional responses according to this fixed allocation. We compute the final estimate using either the additional responses alone or both the pilot and additional responses. We summarize the results below, where bias, RMSE, and MAE are reported in percentage points and variance in squared percentage points.

BenchmarkmmPilot responses in final estimationBiasVarianceRMSEMAE
AIME 202520Excluded0.01893.8369.6887.652
AIME 202520Included-0.04944.4436.6695.301
MATH-500250Excluded0.0162.1871.4791.116
MATH-500250Included0.0351.8631.3661.029

Including pilot responses increases the magnitude of the bias but reduces variance, leading to lower RMSE. This supports their reuse in this setting.

We will further clarify that our theory assumes the pilot responses are independent of those used for final estimation and does not directly cover their reuse.

Q3. Can Uniform sampling be combined with the same final isotonic fitting and interpolation as Itemwise + Order?

Yes. We apply the same isotonic fitting (used by Itemwise + Order) to responses collected through Uniform sampling on AIME 2025 with an evaluation budget of m=20m=20. Both final estimators use the same sampled responses, so the comparison isolates the effect of isotonic estimation. It reduces mean absolute error from 6.13 to 5.33 percentage points. We added this comparison to Section 5.4.

W2. The reported Monte Carlo errors omit uncertainty in the estimated probabilities and reference pools. Representative validation with fresh model responses is needed.

Thank you for highlighting this concern. For benchmarks with sufficient responses per item (ex. 32 responses per item on AIME 2025), we updated the evaluation protocol to use available model responses directly instead of generating outcomes from estimated item-level accuracies. In each repetition, we randomly split each item's responses into two subsets: one is used for adaptive sampling and accuracy estimation (ex. 4 responses), while the other defines the held-out benchmark score (ex. the remaining 28 responses), used as a proxy for the true accuracy when calculating estimation error. Both the estimate and the held-out score therefore vary across repetitions. The main improvements of our methods persist under this revised protocol, described in Section 5.

Thank you for the revisions and additional experiments. I appreciate the corrected allocation rule and the ablation separating sampling from final estimation. However, replacing the original selection rule with proportional-allocation tracking seems a substantive methodological change, rather than merely a clarification. I will maintain my rating.

Reviewer_LB6TOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors present adaptive sampling policies for better accuracy estimation in reasoning benchmarks using historical evaluation to determine how many generations each item of the benchmark needs according to its uncertainty. Three adaptive approaches are defined (Stratified + Estimate, Stratified + Order, and Itemwise + Order) each differing by the way they compute a priority according to which the items will be sequentially evaluated. Moreover, the authors include an analysis of how the difficulty grouping and a reliable ordering improve the error bound of the accuracy estimation. Evaluation is performed across 4 different reasoning benchmarks, reporting a MAE improvement against stratified only, uniform sampling and other existing efficient-evaluation methods.

Critical strengths

The paper is well organised, presenting information where it helps understanding, and allowing the reading to flow smoothly. The evaluation spans 4 different reasoning benchmarks including math (AIME 2025, HMMT Feb. 2025, arXivMath) and science (GPQA Diamond).

Critical weaknesses

My main concern regards the fact that the presented sampling policies are based on the concept of uncertainty, that from my understanding is something highly tied to the capabilities of the model: the less capable the model the more different are the results in the different roll outs of an item. Moreover, in the main paper is not clear how the models are used in combination: which models have been used as the reference models and which is the target model, along with the fact that is not clear how the results are aggregated.

Questions for authors
  1. Could you expand more on the experimental setting? Specifically, when you refer to target model, which model is that? Same for the reference models from which you extract the historical data. [clarification]
  2. Do you have experiments investigation how same/cross-family architectures affect the error? What I mean by this is that if having reference models be the same as the target model doesn't bring any advantage or viceversa. [suggestion/future work]
2 comments

Thank you for your positive assessment of the paper's organization and questions about reference models. Our responses to your questions follow.

Q1. When you refer to target model, which model is that? Same for the reference models from which you extract the historical data.

In the main experiments, each model serves as the target in turn, using all other models on the same benchmark as references. For example, when Qwen3-4B is the target on AIME 2025, we average each item's accuracy across the other nine models and order the items by these averages, from highest to lowest. We average evaluation errors over all ten targets. Table 2 additionally considers restricted sets of one or three reference models. We will clarify this setting more explicitly in Section 5.

Q2. How do same/cross-family architectures affect the error?

Thank you for suggesting this direction for future work. As a preliminary investigation, we compare Olmo and non-Olmo reference models using Olmo-3-7B-Think, the highest-accuracy Olmo model in our AIME 2025 data, as the target at m=20m=20. We use three Olmo references (Olmo-3-7B-RL-Zero-Code, Olmo-3-7B-RL-Zero-Math, and Olmo-3.1-7B-RL-Zero-Math) and three non-Olmo references (Qwen3-4B, DeepSeek-R1-0528-Qwen3-8B, and DeepSeek-R1-Distill-Llama-8B). We report mean absolute error (MAE) in percentage points.

Sampling policyReference modelsMAE
UniformNone6.779
Itemwise + OrderOther Olmo models5.842
Itemwise + OrderNon-Olmo models5.497

Both reference sets improve over Uniform. Using non-OLMo references yields slightly lower MAE than using Olmo references, suggesting that references from the same model family do not necessarily provide greater benefits. Identifying when same-family references offer larger gains remains an interesting direction for future work.

Thank you for your responses to Q1 and Q2. My main concern, that the uncertainty-based ordering may depend on the target model's capability, remains open, so I will keep my current score.

Reviewer_NEFmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to estimate LLM reasoning benchmark accuracy with fewer generations. The authors propose adaptive sampling strategies that use historical difficulty information from other models (difficulty groups or rankings), to decide which questions to evaluate next. They also provide theoretical analysis and experiments on several reasoning benchmarks, showing reduced estimation error compared with uniform sampling and existing methods, even when historical information is limited.

Critical strengths

The paper aims to address the practical problem of accurately estimating LLM reasoning benchmark performance with a limited number of stochastic generations. Using historical difficulty strata or relative difficulty rankings instead of relying on detailed historical accuracy estimates, is a simple and useful idea. The paper provides theoretical results showing when informative strata and reliable difficulty orderings can reduce estimation error, as well as how the sampling imbalance constraint relates to the oracle allocation. The experiments cover four reasoning benchmarks and show promising improvements over uniform sampling and existing methods, even with few or weaker reference models.

Critical weaknesses

The method relies heavily on historical difficulty orderings transferring across models, but does not provide a practical way to assess their reliability for a new target model. The choice of the number of strata h also seems critical, as it affects difficulty grouping, estimation quality, and the pilot budget, but is not ablated.

The experiments also do not clearly separate the benefits of adaptive sampling from those of using historical difficulty information to improve accuracy estimation. A more targeted ablation would help clarify the contribution of each.

The experiments mainly use simulated outcomes on relatively small reasoning benchmarks. It would be useful to evaluate larger benchmarks and more expensive end-to-end tasks, such as Terminal-Bench, where reducing evaluation costs could have greater practical value. These tasks may also have more diverse difficulty patterns, making historical difficulty ordering harder to transfer. The generalizability of the method to these settings remains unclear.

Questions for authors
  1. How can the reliability of historical difficulty orderings be assessed for a new target model with a limited evaluation budget?
  2. How sensitive are the results to the choice of h? Is there a practical guideline for selecting it?
  3. Can the authors provide an ablation that separates the improvement from adaptive sampling and difficulty-informed accuracy estimation?
  4. How well would the method generalize to larger or more expensive end-to-end benchmarks, where question difficulty may be more heterogeneous and evaluation costs are higher?
1 comment

We thank the reviewer for recognizing the simplicity and usefulness of our approach and suggesting further analyses to better assess its effectiveness. We respond to each question below.

Q3. Separating the improvement from adaptive sampling and difficulty-informed accuracy estimation.

Our method Itemwise + Order uses historical difficulty ordering for (1) adaptive sampling, where order-constrained accuracy estimates determine item-level allocation proportions, and (2) final estimation through isotonic fitting. To assess the contribution of each component, we compare the four combinations of sampling strategies and final estimators on AIME 2025 with an evaluation budget of m=20m=20. Rows compare uniform and adaptive sampling policies, while columns compare the empirical mean and isotonic estimate. Empirical estimation averages observed item accuracies without isotonic fitting. Entries are mean absolute error in percentage points.

Sampling policyMAE of empirical meanMAE of averaged isotonic estimates
Uniform6.135.33
Adaptive5.764.97

Both order-informed allocation and isotonic final estimation reduce the error.

Q1. How can the reliability of historical difficulty orderings be assessed for a new target model?

The collected target responses allow us to assess ordering reliability using Kendall's tau between the historical ordering and observed accuracy ordering over evaluated items. When agreement is weak, final estimation can use empirical item means without isotonic fitting. In the comparison below, both final estimators use the same observations collected by Itemwise + Order under random historical orderings. The three error columns report MAE in percentage points.

BenchmarkmmKendall's tauMAE of UniformMAE of Itemwise + Order (empirical mean)MAE of Itemwise + Order (averaged isotonic estimates)
AIME 2025150.0137.987.959.12
HMMT Feb. 2025150.0238.108.099.37
arXivMath200.0397.327.269.75
Terminal-Bench 2.0440.0174.734.726.61
GPQA Diamond990.0083.263.244.71
MATH-500250−0.0061.131.122.02

The observed correlations are near zero. Although allocation uses random historical orders, empirical final estimation achieves MAE comparable to Uniform across all six benchmarks. We have added this analysis to Section 5.4.

Q2. How sensitive are the results to the choice of h? Is there a practical guideline for selecting it?

We choose the number of strata hh based on the number of items nn in the benchmark: h=5h=5 for n≤50n\leq50, h=10h=10 for 50<n≤10050<n\leq100, and h=20h=20 otherwise. Increasing hh allows finer grouping by difficulty but requires more pilot responses (2h2h in total), leaving less budget for subsequent adaptive sampling. Figures 5 and 6 in the appendix show that the proposed choices yield MAE close to the minimum observed in each setting, although the value of hh that minimizes MAE varies across benchmarks and evaluation budgets.

Q4. How well would the method generalize to larger or more expensive end-to-end benchmarks?

We added results evaluating our methods on MATH-500 (a larger benchmark with n=500n=500 problems) and Terminal-Bench 2.0 (an end-to-end benchmark with 89 terminal tasks). Figure 3 shows improvements over Uniform, consistent with the other benchmarks. Dataset descriptions and evaluation protocols appear in Appendices C.2 and D.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.