acceptodds
Under review as a conference paper at ICLR 2027

BayesAME: Bayesian Active Model Evaluation

Abstract

Evaluating large generative models across benchmarks is time-consuming and costly, motivating methods that can estimate full benchmark performance by evaluating models on only a subset of items (coreset). While current literature mostly requires the practitioner to input a coreset size, we introduce BayesAME, a sequential active Bayesian framework that uses historical data and an information-gain criterion to dynamically select items to include in the coreset thus allowing automatic determination of its size. Additionally, we show how a multi-target extension further reduces coreset size by leveraging model correlations. Extensive experiments demonstrate that BayesAME consistently outperforms existing methods. Crucially, our analysis resolves recent skepticism in the literature by showing that a) non-random coreset selection is advantageous over random selection and b) one can reliably improve upon the random sample mean baseline.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Efficient model evaluation, active learning, Gaussian processes, LLM benchmarking

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cWBUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BayesAME, a sequential Bayesian framework for efficient evaluation of large generative models. Rather than requiring a predefined coreset size, the method sequentially selects benchmark items and automatically terminates evaluation when both the estimated performance fluctuation and posterior uncertainty fall below user-specified thresholds. Historical evaluations from reference models are used to construct a Gaussian prior over latent target-model abilities, while an information-gain criterion determines which benchmark item should be evaluated next.

The paper further introduces a multi-target extension based on a linear model of coregionalization to exploit correlations among multiple target models. The framework is evaluated across six benchmarks using binary and continuous scores, under both interpolation and extrapolation regimes and with varying numbers of reference models. The experiments compare against several sequentially adapted model-evaluation methods and show that BayesAME generally achieves favorable estimation accuracy versus evaluation cost.

Critical strengths
  1. Most existing methods assume that the evaluation budget is known in advance, whereas in many practical settings the more relevant question is how many benchmark items are required to obtain a sufficiently reliable estimate. The proposed sequential Bayesian formulation provides a coherent way to combine performance estimation, posterior uncertainty, active item acquisition, and stopping within a single framework.
  2. The experiments cover multiple benchmarks, varying numbers of reference models, binary and continuous scores, interpolation and extrapolation regimes, and both single-target and multi-target settings. The use of a large number of random initialization trajectories and the comparison against random sampling, regression, IRT, ProEval, and other reference-model-based estimators provide substantial empirical evidence that exploiting historical model behavior can improve efficient evaluation.
Critical weaknesses
  1. The reliability of the Bayesian stopping criterion is not sufficiently established, despite being central to the paper's main contribution. The method terminates when $ 2z_{0.975}\sqrt{\operatorname{Var}(R\mid s_C)} \leq \epsilon_2 $ together with a stability condition on the point estimate. However, this stopping decision relies directly on the posterior uncertainty being well calibrated under the assumed probabilistic model. The paper primarily evaluates downstream RMSE and cost--accuracy trade-offs, but does not appear to report the empirical coverage of the nominal 95%95\% credible intervals or the probability that the algorithm terminates while the true benchmark performance lies outside the claimed uncertainty interval. This issue is particularly important because the likelihood model is Gaussian even for binary and bounded scores, which the authors themselves acknowledge is misspecified from a support perspective. A Bayesian credible interval is only as reliable as the underlying probabilistic model; therefore, good average RMSE alone does not establish that the stopping rule provides reliable uncertainty control.
  2. The claimed benefit of active, non-random selection depends substantially on heuristic safeguards and on favorable reference-score structure.
Questions for authors
  1. What is the empirical coverage probability of the nominal 95% posterior credible interval I0.95(R)\mathcal{I}_{0.95}(R) across targets, seeds, benchmarks, and interpolation/extrapolation regimes?
  2. How sensitive is the method to the heuristic parameters governing the switch between active and random selection, particularly N0N_0, the ℓ∞\ell_\infty tolerance used for continuous scores, and the choice of σ2\sigma^2? Please also report, for each experimental setting, what fraction of the acquisition trajectory actually uses the information-gain criterion rather than random selection. This would clarify how much of the observed improvement can truly be attributed to active selection.
  3. In the multi-target experiments, the ``high-correlation'' group is formed from the ten most correlated target models. How are these correlations computed? If full benchmark evaluations of the target models are used to identify this group, then this construction uses information that would not be available in the intended evaluation setting. Can the authors demonstrate similar benefits when target correlation must be inferred solely from the initial 10%10\% evaluation subset?
  4. The multi-target variant initially evaluates all target models on 10%10\% of the benchmark before active acquisition begins. How does this fixed initialization cost affect the claimed efficiency advantage when the number of target models is large? A comparison of total model--item evaluations, including this initialization cost, against independent single-target evaluation would help establish when the multi-target formulation is actually beneficial.
4 comments

We thank the reviewer for their thoughtful review. Below, we address their comments and questions.

What is the empirical coverage probability of the nominal 95% posterior credible interval?

We added a calibration analysis in Appendix C.6 of the paper (text in red) and we report the findings in the general comment above.

What fraction of the acquisition trajectory actually uses the information-gain criterion rather than random selection?

Whether a (target model, seed)-run uses the information-gain criterion or random selection is determined globally at the start of the trajectory based on the information provided by the reference models. In the table of the general comment, we report the percentage of the trajectories (over which we aggregate for the plots in the paper) that use the information-gain criterion. In particular:

  • For 50% and 90% of the reference models, 100% of trajectories use the information-gain criterion.
  • For 10% of the reference models, for GPQA with binary scores 0% of the trajectories use the information-gain criterion, whereas for MMLU-Pro with continuous scores 63% (interpolation) and 65% (extrapolation)of trajectories use the information-gain criterion.

Hyperparameter sensitivity

Sensitivity to N0N_0. Please see our response to Reviewer VWA5 and Appendix C.7 in the updated paper.

Sensitivity to σ2\sigma^2. We vary σ2\sigma^2 on GPQA with binary scores:

  1. Nearly-unique reference regime (50% and 90% reference models): We consider σ2∈10−6,10−4,5×10−3,10−1\sigma^2\in\\{10^{-6},10^{-4},5\times10^{-3},10^{-1}\\}. Figures 16 and 17 in the updated paper and Table 1 below (Table 5 in Appendix C.7) show that σ2∈10−6,10−4,5×10−3\sigma^2\in\\{10^{-6},10^{-4},5\times10^{-3}\\} yield similar RMSE, stopping coreset sizes and coverage. Increasing σ2\sigma^2 to 10−110^{-1} worsens BayesAME's RMSE, particularly under extrapolation (Figure 17, top two rows), while BayesAME-RS's RMSE remains largely insensitive (Figure 17, bottom two rows).

  2. Non-unique reference regime (10% reference models): We consider σ2∈10−3,10−2,10−1,1\sigma^2\in\\{10^{-3},10^{-2},10^{-1},1\\}. Figure 18 shows that RMSE is comparable across values, with the default σ2=10−1\sigma^2=10^{-1} yielding slightly lower values. However, sensitivity is more pronounced in interval coverage and stopping (Table 2 below, Table 6 in Appendix C.7): σ2=10−3\sigma^2=10^{-3} produces substantial undercoverage and tolerance-exceedance rates of 9.5-17.2%. The default σ2=10−1\sigma^2=10^{-1} yields conservative intervals and exceedance rates of at most 0.1%, but median coreset size at stopping is at least 99%. The intermediate choice σ2=10−2\sigma^2=10^{-2} offers a less conservative trade-off: at stopping threshold ϵ2=0.03\epsilon_2=0.03, median coreset size at stopping falls to 90-93%, with exceedance rates of 2.8-3.3%.

Table 1. Nearly-unique reference regime (50% and 90% reference models). Cfixed(c)C_{\mathrm{fixed}}(c) reports coverage with c%c\% of evaluated items and Cstop(ϵ2)C_{\mathrm{stop}}(\epsilon_2) reports coverage with stopping threshold ϵ2\epsilon_2. Estop(ϵ2)E_{\mathrm{stop}}(\epsilon_2) estimates Pr⁡(∣R^τ−R⋆∣>ϵ2/2)\Pr(\lvert\hat R_\tau-R^\star\rvert>\epsilon_2/2). The final two columns report the median percentage evaluated at stopping.

SplitRefsσ2\sigma^2MethodCfixed(50C_{\mathrm{fixed}}(50\\%)Cfixed(90C_{\mathrm{fixed}}(90\\%)Cstop(0.01)C_{\mathrm{stop}}(0.01)Cstop(0.03)C_{\mathrm{stop}}(0.03)Estop(0.01)E_{\mathrm{stop}}(0.01)Estop(0.03)E_{\mathrm{stop}}(0.03)Evaluated$(\epsilon_2=0.01)$Evaluated(\epsilon_2=0.03)\
Interpolation50%1e-6BayesAME0.9290.9290.9340.9260.0390.04897%81%
1e-40.9200.9310.9290.9260.0400.04597%81%
5e-30.9280.9360.9310.9280.0370.04297%81%
1e-10.9400.9360.9610.9470.0270.03399%86%
1e-6BayesAME-RS0.9400.9400.9630.9590.0250.02699%86%
1e-40.9400.9420.9620.9590.0250.02799%86%
5e-30.9410.9420.9650.9560.0240.02899%86%
1e-10.9580.9570.9750.9630.0190.01899%87%
--RS-Mean0.9440.9540.9860.9700.0090.013100%93%
90%1e-6BayesAME0.9160.9480.9580.9390.0310.04397%81%
1e-40.9270.9490.9680.9390.0210.04397%81%
5e-30.9180.9550.9680.9430.0240.04097%81%
1e-10.9200.9440.9700.9560.0190.02698%86%
1e-6BayesAME-RS0.9410.9460.9620.9520.0280.02898%86%
1e-40.9410.9460.9620.9520.0280.02898%86%
5e-30.9420.9460.9640.9520.0260.02998%86%
1e-10.9600.9650.9770.9670.0180.01998%86%
--RS-Mean0.9440.9540.9860.9700.0090.013100%93%
Extrapolation50%1e-6BayesAME0.9200.9160.9390.9090.0320.04798%84%
1e-40.9130.9150.9220.9180.0400.04698%84%
5e-30.9090.9240.9240.9170.0420.04599%85%
1e-10.9370.9390.9700.9390.0210.035100%88%
1e-6BayesAME-RS0.9500.9410.9760.9470.0170.025100%90%
1e-40.9510.9410.9770.9460.0170.025100%90%
5e-30.9500.9420.9770.9460.0170.025100%89%
1e-10.9540.9490.9790.9520.0140.021100%89%
--RS-Mean0.9480.9480.9870.9690.0110.009100%96%
90%1e-6BayesAME0.9220.9370.9720.9370.0210.03898%83%
1e-40.9320.9320.9740.9350.0200.03498%83%
5e-30.9190.9270.9600.9200.0330.04398%84%
1e-10.9150.9530.9710.9590.0190.017100%87%
1e-6BayesAME-RS0.9430.9430.9700.9420.0230.02999%88%
1e-40.9430.9430.9700.9420.0240.02999%88%
5e-30.9420.9430.9690.9430.0240.02999%88%
1e-10.9510.9530.9730.9490.0220.02699%88%
--RS-Mean0.9480.9480.9870.9690.0110.009100%96%

Table 2. Non-unique reference regime (10% reference models).

SplitRefsσ2\sigma^2MethodCfixed(50C_{\mathrm{fixed}}(50\\%)Cfixed(90C_{\mathrm{fixed}}(90\\%)Cstop(0.01)C_{\mathrm{stop}}(0.01)Cstop(0.03)C_{\mathrm{stop}}(0.03)Estop(0.01)E_{\mathrm{stop}}(0.01)Estop(0.03)E_{\mathrm{stop}}(0.03)Evaluated$(\epsilon_2=0.01)$Evaluated(\epsilon_2=0.03)\
Interpolation10%1e-3BayesAME-RS0.7300.6260.7230.6820.1720.11497%82%
1e-20.9250.9020.9420.9130.0310.033100%90%
1e-11.0001.0000.9971.0000.0010.000100%99%
11.0001.0001.0001.0000.0000.000100%100%
--RS-Mean0.9440.9540.9860.9700.0090.013100%93%
Extrapolation10%1e-3BayesAME-RS0.7200.5940.7710.6830.1420.09599%88%
1e-20.9100.8910.9480.9080.0280.028100%93%
1e-11.0001.0001.0001.0000.0010.000100%100%
11.0001.0001.0001.0000.0000.000100%100%
--RS-Mean0.9480.9480.9870.9690.0110.009100%96%

In the multi-target experiments, the ``high-correlation'' group is formed from the ten most correlated target models. How are these correlations computed?

We appreciate this thoughtful question, as it allows us to clarify both the experimental design and the assumed operating scenario, and also opens the doors for alternative scenarios.

The algorithm never accesses target correlation information. Cross-model correlations are inferred online via the latent coregionalization weights, which are learned only from the sequentially evaluated data (starting from the initial 10% random subset). Selecting the 10 most correlated target models was done offline purely as a controlled experimental setup to evaluate the method's behavior under the two low and high-correlation regimes. In practice, practitioners typically know a priori when target models are in a high-correlation regime (e.g. when evaluating checkpoints from the same training run). This is the operating scenario we assume. In scenarios where target correlation is completely unknown beforehand, an exciting future extension of our framework would be to use the initial 10% subset not only to initialize the coregionalization weights, but also to assess target models' correlation to decide whether to deploy the single or multi-target model.

To verify if high correlation can be inferred from the initial subset, we performed the ten-model selection procedure on 100 random subsets formed by only 10% of items (55 items) for GPQA. For binary scores, the selected groups have an average within-group correlation (measured using 100% of items) of 0.485, compared with 0.487 for the group selected using 100% of items. For continuous scores, the corresponding values are 0.497 and 0.524.

The mean overlap with the models selected using 100% of items is 5.14 out of ten models for binary scores and 7.81 for continuous scores. For binary scores, this variation partly reflects switching between two distinct clusters of highly correlated models. These results therefore confirm the feasibility of identifying strongly correlated target models from the 10% initial evaluation subset.

The multi-target variant initially evaluates all target models on of the benchmark before active acquisition begins. How does this fixed initialization cost affect the claimed efficiency advantage when the number of target models is large? A comparison of total model--item evaluations, including this initialization cost, against independent single-target evaluation would help establish when the multi-target formulation is actually beneficial.

The initialization cost is already fully accounted for in all reported multi-target results. In Figures 2, 12 and 13, the x-axis represents the per-target coreset size including the 10% initialization items, which is why the multi-target curves begin at n=10%Nn = 10\%N. Since both single and multi-target evaluate the same KtK_t target models, the total number of model–item evaluations is Kt,nˉK_t \\,\bar{n} in both cases, so the cost-accuracy plots directly compare total evaluations. We make this explicit in Table 3 below, which reports the proportion of the benchmark that multi-target and single-target BayesAME must evaluate per target model to reach the same RMSE. We report this for the three RMSE values reached by multi-target BayesAME at stopping for ϵ2∈0.01,0.02,0.03\epsilon_2 \in\\{0.01, 0.02, 0.03\\}.

Table 3. Multi-target vs. independent single-target BayesAME at equal RMSE. Coreset sizes are % of NN per target and the multi-target sizes include the 10% initialization. Values are given at the stopping RMSE obtained by multi-target BayesAME with ϵ2=0.01/0.02/0.03\epsilon_2 = 0.01/0.02/0.03

TargetsBenchmark (score)Multi-target nˉ\bar nSingle-target nˉ\bar n at equal RMSEEvaluations saved by multi-target
High corr.IFEval (binary)92.6 / 81.3 / 66.694.4 / 85.6 / 72.11.9% / 5.0% / 7.6%
High corr.GPQA (binary)96.8 / 89.2 / 77.597.9 / 93.4 / 86.41.1% / 4.5% / 10.3%
High corr.GPQA (cont.)91.2 / 72.2 / 52.693.3 / 79.3 / 61.42.3% / 9.0% / 14.4%
Low corr.IFEval (binary)94.9 / 87.4 / 76.894.4 / 87.4 / 75.0−0.5% / 0.0% / −2.4%
Low corr.GPQA (binary)97.4 / 91.3 / 82.096.9 / 90.5 / 80.1−0.5% / −0.9% / −2.4%
Low corr.GPQA (cont.)94.0 / 81.0 / 64.890.1 / 73.4 / 56.9−4.4% / −10.4% / −14.0%

Multi-target BayesAME saves around 7.6%–14.4% of all model–item evaluations when targets are correlated. When targets are uncorrelated it costs more, in particular, up to 2.4% for binary scores and up to 14% for continuous scores, consistent with the negative transfer discussed in Sec. 5.2.

AuthorsOpenReview ↗

Follow-up Before Discussion Ends

Dear Reviewer, thank you again for your time and thoughtful review. As the discussion window closes soon, we would be very grateful to hear whether our response and additional experiments addressed your concerns, or if anything remains unclear.

Reviewer_Ch95OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BayesAME, a sequential Bayesian framework for efficient model evaluation that automatically determines the number of benchmark items to evaluate. It uses historical model scores to construct a Gaussian prior, updates performance estimates and uncertainty as new observations arrive, and selects items based on information gain. A multi target extension further exploits correlations among target models. Experiments on six benchmarks, covering interpolation and extrapolation settings with binary and continuous scores, demonstrate improved estimation accuracy and cost–accuracy tradeoffs over existing baselines, with additional efficiency gains when target models are highly correlated.

Critical strengths

1.The paper presents a well structured Bayesian framework with clear mathematical formulations and detailed derivations of posterior inference, performance estimation, and information gain based selection. This provides a solid methodological foundation for the proposed approach. The paper addresses a practical limitation of efficient benchmarking by proposing a well-motivated approach to determining coreset size automatically, rather than requiring a fixed budget.

2.The experiments cover six benchmarks, multiple baselines, and both interpolation and extrapolation settings. These evaluations provide useful empirical support for the proposed method.

Critical weaknesses

1.The proposed stopping criterion relies on posterior uncertainty estimates, but the paper does not evaluate their calibration or empirical coverage. Without such analysis, it remains unclear whether the stopping criterion reliably reflects the actual estimation error, particularly in extrapolation settings.

2.The paper reports substantial benefits from continuous scores, but the experiments change both the reference scores and the target evaluation metric. Besides, the paper lacks qualitative analysis, particularly an examination of failure cases that could clarify when and why BAYESAME underperforms.

Questions for authors

Could the authors clarify whether the benefit of continuous reference information has been evaluated while keeping the target metric fixed? For example, comparing binary and continuous reference scores when predicting binary benchmark accuracy would help isolate the contribution of richer reference information.

3 comments

Calibration and coverage

We added a calibration analysis in App C.6 of the paper as well as in the general comment above.

Benefits of continuous reference information

This is a very interesting point. In our experiments, we use the same score type for the reference and target models.

To understand whether using continuous reference scores improves estimation of the target model's performance computed using binary scores, we compared using both binary and continuous reference models' scores on GPQA. With continuous reference models' scores, BayesAME always uses the information-gain criterion to select new items. In contrast, binary reference models' scores trigger the fallback to random selection with 10% of the reference pool.

Table 1 below (Table 7 in App C.8 of the revised paper) and Figure 19 in App C.8 show moderate RMSE gains from continuous references for BayesAME with 10% of the reference pool (that is, a setting where binary reference scores are not very informative) and under interpolation with 50%, whereas in the other settings there is no clear winner.

While continuous reference information can lead to earlier stopping, calibration slightly degrades. Table 2 below (Table 8 in App C.8) shows that, for BayesAME with 10% references and ϵ2=0.03\epsilon_2=0.03, the median percentage evaluated falls from 99% to 81% under interpolation and from 100% to 85% under extrapolation, while tolerance-exceedance rates rise from 0% to 9.6% and 7.6%, respectively.

Table 1. Each entry reports RMSE(×100) at 10/25/50% of items evaluated. Bold and italic values denote the best and second-best distinct results, respectively, for each split, reference-pool size, and coreset size. †: BayesAME falls back to random selection and coincides with BayesAME-RS.

SplitRefsBayesAME(binary)BayesAME(continuous)BayesAME-RS(binary)BayesAME-RS(continuous)RS-Mean
Interpolation10%5.45/3.20/1.84†5.18/3.09/1.825.45/3.20/1.845.45/3.20/1.915.77/3.40/1.99
50%4.74/2.75/1.584.50/2.67/1.584.99/2.92/1.725.06/2.96/1.735.77/3.40/1.99
90%4.46/2.57/1.544.46/2.59/1.534.91/2.85/1.685.00/2.88/1.685.77/3.40/1.99
Extrapolation10%5.66/3.38/1.97†5.59/3.30/1.895.66/3.38/1.975.90/3.47/1.986.08/3.57/2.09
50%5.10/2.99/1.755.18/2.99/1.735.30/3.15/1.835.46/3.12/1.826.08/3.57/2.09
90%4.84/2.85/1.644.91/2.83/1.675.19/3.06/1.795.30/3.05/1.786.08/3.57/2.09

Table 2. Coverage is reported at fixed evaluation percentages c=50c=50\\%/90\\%, Cfixed(c)C_{\mathrm{fixed}}(c), and at stopping, Cstop(ϵ2)C_{\mathrm{stop}}(\epsilon_2). Estop(ϵ2)E_{\mathrm{stop}}(\epsilon_2) estimates Pr⁡(∣R^τ−R⋆∣>ϵ2/2)\Pr(\lvert\hat R_\tau-R^\star\rvert>\epsilon_2/2). The final two columns report the median percentage of items evaluated at stopping.

SplitRefsMethodScoresCfixed(50C_{\mathrm{fixed}}(50\\%)Cfixed(90C_{\mathrm{fixed}}(90\\%)Cstop(0.01)C_{\mathrm{stop}}(0.01)Cstop(0.03)C_{\mathrm{stop}}(0.03)Estop(0.01)E_{\mathrm{stop}}(0.01)Estop(0.03)E_{\mathrm{stop}}(0.03)Evaluated(ε₂=0.01)Evaluated(ε₂=0.03)
Interpolation10%BayesAMEBinary1.0001.0000.9971.0000.0010.000100%99%
BayesAMEContinuous0.8690.8560.8990.8560.0700.09697%81%
BayesAME-RSBinary1.0001.0000.9971.0000.0010.000100%99%
BayesAME-RSContinuous0.9420.9460.9740.9520.0180.024100%90%
50%BayesAMEBinary0.9200.9310.9290.9260.0400.04597%81%
BayesAMEContinuous0.9120.9190.9480.9070.0350.06697%80%
BayesAME-RSBinary0.9400.9420.9620.9590.0250.02799%86%
BayesAME-RSContinuous0.9370.9470.9620.9480.0260.03398%86%
90%BayesAMEBinary0.9270.9490.9680.9390.0210.04397%81%
BayesAMEContinuous0.9190.9200.9500.9170.0380.05597%79%
BayesAME-RSBinary0.9410.9460.9620.9520.0280.02898%86%
BayesAME-RSContinuous0.9420.9410.9560.9460.0320.03498%85%
AllRS-Mean--0.9440.9540.9860.9700.0090.013100%93%
Extrapolation10%BayesAMEBinary1.0001.0001.0001.0000.0010.000100%100%
BayesAMEContinuous0.8760.8420.9000.8380.0570.07699%85%
BayesAME-RSBinary1.0001.0001.0001.0000.0010.000100%100%
BayesAME-RSContinuous0.9480.9470.9780.9640.0150.017100%93%
50%BayesAMEBinary0.9130.9150.9220.9180.0400.04698%84%
BayesAMEContinuous0.9040.8850.9570.8980.0280.06498%83%
BayesAME-RSBinary0.9510.9410.9770.9460.0170.025100%90%
BayesAME-RSContinuous0.9490.9450.9700.9470.0240.025100%89%
90%BayesAMEBinary0.9320.9320.9740.9350.0200.03498%83%
BayesAMEContinuous0.9120.8970.9580.9140.0300.05698%82%
BayesAME-RSBinary0.9430.9430.9700.9420.0240.02999%88%
BayesAME-RSContinuous0.9440.9450.9710.9460.0200.03099%88%
AllRS-Mean--0.9480.9480.9870.9690.0110.009100%96%

Thank you for the detailed response and additional experiments. The calibration analysis and the comparison with a fixed binary target metric are helpful, although calibration under extrapolation and the inconsistent gains from continuous reference scores remain concerns. I retain my positive assessment of 3.

Thank you for your reply, we are pleased you found our response and additional experiments helpful, and that you retain a positive assessment of our work.

Reviewer_VWA5OpenReview ↗
Rating
3
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BayesAME, a Bayesian active learning framework for efficient large language model evaluation. It aims to accurately estimate a model's performance on a full benchmark by evaluating only a small subset of benchmark items, while automatically determining the required evaluation sample size.

The authors conduct experiments on six benchmarks, covering different numbers of reference models, interpolation and extrapolation settings, as well as binary and continuous scores. Experimental results demonstrate that BayesAME outperforms existing methods in performance estimation across most settings.

Critical strengths
  1. The research problem is practically important. As the scale of large language models and the number of evaluation tasks continue to grow, conducting evaluations on full benchmarks becomes increasingly expensive. This paper investigates how to automatically determine the required number of evaluation samples while maintaining reliable performance estimation, which has clear practical value.

  2. Framework is comprehensive and well-motivated. This paper integrates Bayesian performance estimation, information-gain-based active sample selection, and a dynamic stopping mechanism into a unified framework. Compared with methods that require a predefined coreset size, this design offers greater flexibility and practicality.

  3. Experimental evaluation is comprehensive, and the results are competitive. The experiments cover multiple benchmarks and evaluation settings, providing convincing evidence of the effectiveness of the proposed method.

Critical weaknesses
  1. The reliability of the automatic stopping mechanism requires further validation. One of the key contributions of this paper is automatically determining the coreset size based on posterior uncertainty. However, the current experiments mainly evaluate the method using RMSE and cost-accuracy trade-offs, without directly examining the calibration of the Bayesian credible intervals. In particular, under extrapolation settings, target models may exhibit substantially different behaviors from reference models. Such distribution shifts could cause the posterior variance to underestimate the actual estimation error, potentially leading to premature stopping.

  2. The parameter N0 used to determine whether active selection is applied is mainly chosen empirically. I suggest that the authors provide a sensitivity analysis of N0 to further evaluate the robustness of the proposed selection strategy.

  3. Additional computational overhead deserves further investigation. Although BayesAME reduces the number of benchmark items that need to be evaluated, it also introduces additional computational costs associated with Bayesian posterior inference, information gain computation, and hyperparameter optimization, particularly in the multi-target setting.

Questions for authors

Please refer to Weaknesses.

3 comments

Thank you for the thoughtful review. We appreciate your positive comments and address your concerns below.

Further validation of the reliability of the automatic stopping mechanism.

We added a calibration analysis in Appendix C.6 of the paper (text in red), and we report the findings in the general comment above.

The parameter N0 used to determine whether active selection is applied is mainly chosen empirically. I suggest that the authors provide a sensitivity analysis of N0 to further evaluate the robustness of the proposed selection strategy.

Thanks for raising this point. We first recall that active selection is used when B≥N−N0B\ge N-N_0, otherwise, the method falls back to random selection recovering BayesAME-RS. Table 1 below (Table 4 in Appendix C.7 of the revised paper) reports the percentage of seed-target runs using active selection on GPQA with binary scores. For N0∈(5,200)N_0\in(5,200), the selection decisions almost exactly coincide with those under the default N0=10N_0=10, that is, random selection is used with 10% of the reference pool and active selection with 50% or 90%, in both interpolation and extrapolation. Smaller thresholds can unnecessarily disable beneficial active selection.

Conversely, setting N0=NN_0=N forces active selection even with 10% of the reference pool, where many items share the same reference-score vectors and estimation accuracy deteriorates, including at stopping (see Figure 15 in the revised paper). These results support the default choice while showing robustness over a broad range of thresholds.

Table 1. Percentage of seed–target runs using active selection for each threshold N0N_0. Non-unique item counts are summarized by their median and range across runs. The default is N0=10N_0=10.

SplitRefsNon-unique items: median(range)N0=0N_0=055101020205050200200400400
Interpolation10%335 (314–410)0%0%0%0%0%0%98%
50%3 (0–8)6%92%100%100%100%100%100%
90%0 (0–2)80%100%100%100%100%100%100%
Extrapolation10%349 (315–433)0%0%0%0%0%0%96%
50%1 (0–7)30%99%100%100%100%100%100%
90%0 (0–0)100%100%100%100%100%100%100%

In Appendix C.7, we also provide a sensitivity analysis for σ2\sigma^2.

Additional computational overhead

We thank the reviewer for raising this practical consideration. In LLM benchmarking, the primary bottleneck is GPU inference time and cost. Because GPU evaluation is very resource-intensive, reducing the number of evaluated items saves substantial GPU hours, easily outweighing the CPU operations required by the method. The overhead of BayesAME itself depends on the setting:

  1. Single-Target BayesAME (negligible overhead): Information-gain scores have closed-form expressions in terms of posterior quantities and can be computed in parallel across candidate items. Between hyperparameter updates, posterior inference uses Cholesky rank-one updates with a per-iteration cost of only O(min⁡(n,B)2)\mathcal{O}(\min(n,B)^2), where nn is the number of evaluated items and BB is the number of buckets as described in Section 3.1, lines 122–124. Furthermore, hyperparameters are scalar and refitted only periodically, initializing each optimization from previous estimates to facilitate convergence.
  2. Multi-target BayesAME (higher overhead): This variant incurs higher computational cost because it jointly learns the latent coregionalization weights and covariance parameters across all KtK_t models (Appendix A.4) periodically. To reduce this cost, we use an adaptive optimization budget, starting with 500 gradient steps per parameter update and reducing this to 100 once the coregionalization weights stabilize. Additionally, to limit the cost of active selection in this setting, we iterate deterministically through the target models (Section 3.2) rather than searching jointly over all (item, model) pairs. In practice, the multi-target variant represents a trade-off, it invests additional compute in order to save more LLM evaluations across multiple models. When minimizing computational overhead is the priorityy, the single-target variant provides optimal performance with negligible overhead.

Thank you for your detailed response. My concerns have been largely addressed. I remain the positive score.

Thank you for your reply, we are pleased we largely addressed your concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Calibration analysis

We reply to the point regarding the calibration analysis raised by the three reviewers.

We agree that the reliability of the stopping rule relies on the posterior distribution being well-calibrated and that RMSE alone does not assess calibration. To directly validate the calibration of our stopping mechanism, we have conducted an extensive calibration analysis that can be found in Appendix C.6.of the revised paper.

We present below the results on GPQA with binary scores and MMLU-Pro with continuous scores. Specifically, in Tables 1 and 2 below, we report:

a) Coverage at fixed percentages of evaluated items: Coverage of the 95% credible interval centered at R^\hat R when the percentage of evaluated items is 50% and 90%.

b) Coverage at stopping: Coverage of the 95% credible interval centered at R^\hat R at stopping for stopping threshold ϵ2∈0.01,0.03\epsilon_2\in\\{0.01, 0.03\\}.

c) Tolerance-exceedance rate: The probability at stopping that the estimate R^\hat R misses the ground-truth R⋆R^\star by more than the stopping threshold, that is, Pr⁡(∣R^−R⋆∣>ϵ2/2)\Pr(|\hat R - R^\star| > \epsilon_2/2). Under a well-calibrated posterior, this should be ≤0.05\le 0.05 when R^\hat R is the posterior mean.

d) IG used. For BayesAME, the percentage of runs that used the information-gain criterion rather than random selection.

Table 1: GPQA with binary scores.

SettingRefsMethoda) Coverage at 5050\\% / 90\\% of evaluated itemsb) Coverage at stopping for ϵ2=0.01/0.03\epsilon_2=0.01 / 0.03c) Pr⁡(∣R^−R⋆∣>ϵ2/2)\Pr(\lvert\hat R-R^\star\rvert>\epsilon_2/2) for ϵ2=0.01/0.03\epsilon_2=0.01 / 0.03d) IG used
GPQA, interp.10%BayesAME1.000 / 1.0000.997 / 1.0000.000 / 0.0000%
BayesAME-RS1.000 / 1.0000.997 / 1.0000.000 / 0.000–
RS-Mean0.944 / 0.9540.986 / 0.9700.009 / 0.013–
50%BayesAME0.920 / 0.9310.929 / 0.9260.040 / 0.045100%
BayesAME-RS0.940 / 0.9420.962 / 0.9590.025 / 0.027–
RS-Mean0.944 / 0.9540.986 / 0.9700.009 / 0.013–
90%BayesAME0.927 / 0.9490.968 / 0.9390.021 / 0.043100%
BayesAME-RS0.941 / 0.9460.962 / 0.9520.028 / 0.028–
RS-Mean0.944 / 0.9540.986 / 0.9700.009 / 0.013–
GPQA, extrap.10%BayesAME1.000 / 1.0001.000 / 1.0000.000 / 0.0000%
BayesAME-RS1.000 / 1.0001.000 / 1.0000.000 / 0.000–
RS-Mean0.948 / 0.9480.987 / 0.9690.011 / 0.009–
50%BayesAME0.913 / 0.9150.922 / 0.9180.040 / 0.046100%
BayesAME-RS0.951 / 0.9410.977 / 0.9460.017 / 0.025–
RS-Mean0.948 / 0.9480.987 / 0.9690.011 / 0.009–
90%BayesAME0.932 / 0.9320.974 / 0.9350.020 / 0.034100%
BayesAME-RS0.943 / 0.9430.970 / 0.9420.024 / 0.029–
RS-Mean0.948 / 0.9480.987 / 0.9690.011 / 0.009–

Table 2: MMLU-Pro with continuous scores.

SettingRefsMethoda) Coverage at 5050\\% / 90\\% of evaluated itemsb) Coverage at stopping for ϵ2=0.01/0.03\epsilon_2=0.01 / 0.03c) Pr⁡(∣R^−R⋆∣>ϵ2/2)\Pr(\lvert\hat R-R^\star\rvert>\epsilon_2/2) for ϵ2=0.01/0.03\epsilon_2 = 0.01 / 0.03d) IG used
MMLU-Pro, interp.10%BayesAME0.932 / 0.9580.929 / 0.9230.070 / 0.06763%
BayesAME-RS0.953 / 0.9540.957 / 0.9510.043 / 0.049–
RS-Mean0.940 / 0.9460.954 / 0.9370.046 / 0.059–
50%BayesAME0.929 / 0.9800.930 / 0.9140.069 / 0.079100%
BayesAME-RS0.918 / 0.9300.927 / 0.9280.072 / 0.067–
RS-Mean0.940 / 0.9460.954 / 0.9370.046 / 0.059–
90%BayesAME0.935 / 0.9890.934 / 0.9280.066 / 0.062100%
BayesAME-RS0.924 / 0.9390.926 / 0.9290.072 / 0.065–
RS-Mean0.940 / 0.9460.954 / 0.9370.046 / 0.059–
MMLU-Pro, extrap.10%BayesAME0.893 / 0.8600.842 / 0.8980.159 / 0.10265%
BayesAME-RS0.958 / 0.9540.957 / 0.9620.043 / 0.038–
RS-Mean0.941 / 0.9500.953 / 0.9430.046 / 0.058–
50%BayesAME0.908 / 0.8820.864 / 0.9000.137 / 0.100100%
BayesAME-RS0.953 / 0.9460.945 / 0.9550.055 / 0.045–
RS-Mean0.941 / 0.9500.953 / 0.9430.046 / 0.058–
90%BayesAME0.925 / 0.9130.894 / 0.9190.104 / 0.081100%
BayesAME-RS0.954 / 0.9460.941 / 0.9580.059 / 0.042–
RS-Mean0.941 / 0.9500.953 / 0.9430.046 / 0.058–

On GPQA, BayesAME exhibits some interval undercoverage, but tolerance-exceedance rates at stopping remain below 5% across all settings. On MMLU-Pro, its intervals at stopping are less well-calibrated, particularly under extrapolation, where exceedance rates range from 8.1% to 15.9%. Increasing the reference pool from 10% to 90% improves both coverage at stopping and tolerance-exceedance rates in this setting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.