- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes BayesAME, a sequential Bayesian framework for efficient evaluation of large generative models. Rather than requiring a predefined coreset size, the method sequentially selects benchmark items and automatically terminates evaluation when both the estimated performance fluctuation and posterior uncertainty fall below user-specified thresholds. Historical evaluations from reference models are used to construct a Gaussian prior over latent target-model abilities, while an information-gain criterion determines which benchmark item should be evaluated next.
The paper further introduces a multi-target extension based on a linear model of coregionalization to exploit correlations among multiple target models. The framework is evaluated across six benchmarks using binary and continuous scores, under both interpolation and extrapolation regimes and with varying numbers of reference models. The experiments compare against several sequentially adapted model-evaluation methods and show that BayesAME generally achieves favorable estimation accuracy versus evaluation cost.
Critical strengths
- Most existing methods assume that the evaluation budget is known in advance, whereas in many practical settings the more relevant question is how many benchmark items are required to obtain a sufficiently reliable estimate. The proposed sequential Bayesian formulation provides a coherent way to combine performance estimation, posterior uncertainty, active item acquisition, and stopping within a single framework.
- The experiments cover multiple benchmarks, varying numbers of reference models, binary and continuous scores, interpolation and extrapolation regimes, and both single-target and multi-target settings. The use of a large number of random initialization trajectories and the comparison against random sampling, regression, IRT, ProEval, and other reference-model-based estimators provide substantial empirical evidence that exploiting historical model behavior can improve efficient evaluation.
Critical weaknesses
- The reliability of the Bayesian stopping criterion is not sufficiently established, despite being central to the paper's main contribution. The method terminates when $ 2z_{0.975}\sqrt{\operatorname{Var}(R\mid s_C)} \leq \epsilon_2 $ together with a stability condition on the point estimate. However, this stopping decision relies directly on the posterior uncertainty being well calibrated under the assumed probabilistic model. The paper primarily evaluates downstream RMSE and cost--accuracy trade-offs, but does not appear to report the empirical coverage of the nominal credible intervals or the probability that the algorithm terminates while the true benchmark performance lies outside the claimed uncertainty interval. This issue is particularly important because the likelihood model is Gaussian even for binary and bounded scores, which the authors themselves acknowledge is misspecified from a support perspective. A Bayesian credible interval is only as reliable as the underlying probabilistic model; therefore, good average RMSE alone does not establish that the stopping rule provides reliable uncertainty control.
- The claimed benefit of active, non-random selection depends substantially on heuristic safeguards and on favorable reference-score structure.
Questions for authors
- What is the empirical coverage probability of the nominal 95% posterior credible interval across targets, seeds, benchmarks, and interpolation/extrapolation regimes?
- How sensitive is the method to the heuristic parameters governing the switch between active and random selection, particularly , the tolerance used for continuous scores, and the choice of ? Please also report, for each experimental setting, what fraction of the acquisition trajectory actually uses the information-gain criterion rather than random selection. This would clarify how much of the observed improvement can truly be attributed to active selection.
- In the multi-target experiments, the ``high-correlation'' group is formed from the ten most correlated target models. How are these correlations computed? If full benchmark evaluations of the target models are used to identify this group, then this construction uses information that would not be available in the intended evaluation setting. Can the authors demonstrate similar benefits when target correlation must be inferred solely from the initial evaluation subset?
- The multi-target variant initially evaluates all target models on of the benchmark before active acquisition begins. How does this fixed initialization cost affect the claimed efficiency advantage when the number of target models is large? A comparison of total model--item evaluations, including this initialization cost, against independent single-target evaluation would help establish when the multi-target formulation is actually beneficial.
4 comments
Table 1. Nearly-unique reference regime (50% and 90% reference models). reports coverage with of evaluated items and reports coverage with stopping threshold . estimates . The final two columns report the median percentage evaluated at stopping.
| Split | Refs | Method | Evaluated$(\epsilon_2=0.01)$ | Evaluated(\epsilon_2=0.03)\ | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Interpolation | 50% | 1e-6 | BayesAME | 0.929 | 0.929 | 0.934 | 0.926 | 0.039 | 0.048 | 97% | 81% |
| 1e-4 | 0.920 | 0.931 | 0.929 | 0.926 | 0.040 | 0.045 | 97% | 81% | |||
| 5e-3 | 0.928 | 0.936 | 0.931 | 0.928 | 0.037 | 0.042 | 97% | 81% | |||
| 1e-1 | 0.940 | 0.936 | 0.961 | 0.947 | 0.027 | 0.033 | 99% | 86% | |||
| 1e-6 | BayesAME-RS | 0.940 | 0.940 | 0.963 | 0.959 | 0.025 | 0.026 | 99% | 86% | ||
| 1e-4 | 0.940 | 0.942 | 0.962 | 0.959 | 0.025 | 0.027 | 99% | 86% | |||
| 5e-3 | 0.941 | 0.942 | 0.965 | 0.956 | 0.024 | 0.028 | 99% | 86% | |||
| 1e-1 | 0.958 | 0.957 | 0.975 | 0.963 | 0.019 | 0.018 | 99% | 87% | |||
| -- | RS-Mean | 0.944 | 0.954 | 0.986 | 0.970 | 0.009 | 0.013 | 100% | 93% | ||
| 90% | 1e-6 | BayesAME | 0.916 | 0.948 | 0.958 | 0.939 | 0.031 | 0.043 | 97% | 81% | |
| 1e-4 | 0.927 | 0.949 | 0.968 | 0.939 | 0.021 | 0.043 | 97% | 81% | |||
| 5e-3 | 0.918 | 0.955 | 0.968 | 0.943 | 0.024 | 0.040 | 97% | 81% | |||
| 1e-1 | 0.920 | 0.944 | 0.970 | 0.956 | 0.019 | 0.026 | 98% | 86% | |||
| 1e-6 | BayesAME-RS | 0.941 | 0.946 | 0.962 | 0.952 | 0.028 | 0.028 | 98% | 86% | ||
| 1e-4 | 0.941 | 0.946 | 0.962 | 0.952 | 0.028 | 0.028 | 98% | 86% | |||
| 5e-3 | 0.942 | 0.946 | 0.964 | 0.952 | 0.026 | 0.029 | 98% | 86% | |||
| 1e-1 | 0.960 | 0.965 | 0.977 | 0.967 | 0.018 | 0.019 | 98% | 86% | |||
| -- | RS-Mean | 0.944 | 0.954 | 0.986 | 0.970 | 0.009 | 0.013 | 100% | 93% | ||
| Extrapolation | 50% | 1e-6 | BayesAME | 0.920 | 0.916 | 0.939 | 0.909 | 0.032 | 0.047 | 98% | 84% |
| 1e-4 | 0.913 | 0.915 | 0.922 | 0.918 | 0.040 | 0.046 | 98% | 84% | |||
| 5e-3 | 0.909 | 0.924 | 0.924 | 0.917 | 0.042 | 0.045 | 99% | 85% | |||
| 1e-1 | 0.937 | 0.939 | 0.970 | 0.939 | 0.021 | 0.035 | 100% | 88% | |||
| 1e-6 | BayesAME-RS | 0.950 | 0.941 | 0.976 | 0.947 | 0.017 | 0.025 | 100% | 90% | ||
| 1e-4 | 0.951 | 0.941 | 0.977 | 0.946 | 0.017 | 0.025 | 100% | 90% | |||
| 5e-3 | 0.950 | 0.942 | 0.977 | 0.946 | 0.017 | 0.025 | 100% | 89% | |||
| 1e-1 | 0.954 | 0.949 | 0.979 | 0.952 | 0.014 | 0.021 | 100% | 89% | |||
| -- | RS-Mean | 0.948 | 0.948 | 0.987 | 0.969 | 0.011 | 0.009 | 100% | 96% | ||
| 90% | 1e-6 | BayesAME | 0.922 | 0.937 | 0.972 | 0.937 | 0.021 | 0.038 | 98% | 83% | |
| 1e-4 | 0.932 | 0.932 | 0.974 | 0.935 | 0.020 | 0.034 | 98% | 83% | |||
| 5e-3 | 0.919 | 0.927 | 0.960 | 0.920 | 0.033 | 0.043 | 98% | 84% | |||
| 1e-1 | 0.915 | 0.953 | 0.971 | 0.959 | 0.019 | 0.017 | 100% | 87% | |||
| 1e-6 | BayesAME-RS | 0.943 | 0.943 | 0.970 | 0.942 | 0.023 | 0.029 | 99% | 88% | ||
| 1e-4 | 0.943 | 0.943 | 0.970 | 0.942 | 0.024 | 0.029 | 99% | 88% | |||
| 5e-3 | 0.942 | 0.943 | 0.969 | 0.943 | 0.024 | 0.029 | 99% | 88% | |||
| 1e-1 | 0.951 | 0.953 | 0.973 | 0.949 | 0.022 | 0.026 | 99% | 88% | |||
| -- | RS-Mean | 0.948 | 0.948 | 0.987 | 0.969 | 0.011 | 0.009 | 100% | 96% |
Table 2. Non-unique reference regime (10% reference models).
| Split | Refs | Method | Evaluated$(\epsilon_2=0.01)$ | Evaluated(\epsilon_2=0.03)\ | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Interpolation | 10% | 1e-3 | BayesAME-RS | 0.730 | 0.626 | 0.723 | 0.682 | 0.172 | 0.114 | 97% | 82% |
| 1e-2 | 0.925 | 0.902 | 0.942 | 0.913 | 0.031 | 0.033 | 100% | 90% | |||
| 1e-1 | 1.000 | 1.000 | 0.997 | 1.000 | 0.001 | 0.000 | 100% | 99% | |||
| 1 | 1.000 | 1.000 | 1.000 | 1.000 | 0.000 | 0.000 | 100% | 100% | |||
| -- | RS-Mean | 0.944 | 0.954 | 0.986 | 0.970 | 0.009 | 0.013 | 100% | 93% | ||
| Extrapolation | 10% | 1e-3 | BayesAME-RS | 0.720 | 0.594 | 0.771 | 0.683 | 0.142 | 0.095 | 99% | 88% |
| 1e-2 | 0.910 | 0.891 | 0.948 | 0.908 | 0.028 | 0.028 | 100% | 93% | |||
| 1e-1 | 1.000 | 1.000 | 1.000 | 1.000 | 0.001 | 0.000 | 100% | 100% | |||
| 1 | 1.000 | 1.000 | 1.000 | 1.000 | 0.000 | 0.000 | 100% | 100% | |||
| -- | RS-Mean | 0.948 | 0.948 | 0.987 | 0.969 | 0.011 | 0.009 | 100% | 96% |
In the multi-target experiments, the ``high-correlation'' group is formed from the ten most correlated target models. How are these correlations computed?
We appreciate this thoughtful question, as it allows us to clarify both the experimental design and the assumed operating scenario, and also opens the doors for alternative scenarios.
The algorithm never accesses target correlation information. Cross-model correlations are inferred online via the latent coregionalization weights, which are learned only from the sequentially evaluated data (starting from the initial 10% random subset). Selecting the 10 most correlated target models was done offline purely as a controlled experimental setup to evaluate the method's behavior under the two low and high-correlation regimes. In practice, practitioners typically know a priori when target models are in a high-correlation regime (e.g. when evaluating checkpoints from the same training run). This is the operating scenario we assume. In scenarios where target correlation is completely unknown beforehand, an exciting future extension of our framework would be to use the initial 10% subset not only to initialize the coregionalization weights, but also to assess target models' correlation to decide whether to deploy the single or multi-target model.
To verify if high correlation can be inferred from the initial subset, we performed the ten-model selection procedure on 100 random subsets formed by only 10% of items (55 items) for GPQA. For binary scores, the selected groups have an average within-group correlation (measured using 100% of items) of 0.485, compared with 0.487 for the group selected using 100% of items. For continuous scores, the corresponding values are 0.497 and 0.524.
The mean overlap with the models selected using 100% of items is 5.14 out of ten models for binary scores and 7.81 for continuous scores. For binary scores, this variation partly reflects switching between two distinct clusters of highly correlated models. These results therefore confirm the feasibility of identifying strongly correlated target models from the 10% initial evaluation subset.
The multi-target variant initially evaluates all target models on of the benchmark before active acquisition begins. How does this fixed initialization cost affect the claimed efficiency advantage when the number of target models is large? A comparison of total model--item evaluations, including this initialization cost, against independent single-target evaluation would help establish when the multi-target formulation is actually beneficial.
The initialization cost is already fully accounted for in all reported multi-target results. In Figures 2, 12 and 13, the x-axis represents the per-target coreset size including the 10% initialization items, which is why the multi-target curves begin at . Since both single and multi-target evaluate the same target models, the total number of model–item evaluations is in both cases, so the cost-accuracy plots directly compare total evaluations. We make this explicit in Table 3 below, which reports the proportion of the benchmark that multi-target and single-target BayesAME must evaluate per target model to reach the same RMSE. We report this for the three RMSE values reached by multi-target BayesAME at stopping for .
Table 3. Multi-target vs. independent single-target BayesAME at equal RMSE. Coreset sizes are % of per target and the multi-target sizes include the 10% initialization. Values are given at the stopping RMSE obtained by multi-target BayesAME with
| Targets | Benchmark (score) | Multi-target | Single-target at equal RMSE | Evaluations saved by multi-target |
|---|---|---|---|---|
| High corr. | IFEval (binary) | 92.6 / 81.3 / 66.6 | 94.4 / 85.6 / 72.1 | 1.9% / 5.0% / 7.6% |
| High corr. | GPQA (binary) | 96.8 / 89.2 / 77.5 | 97.9 / 93.4 / 86.4 | 1.1% / 4.5% / 10.3% |
| High corr. | GPQA (cont.) | 91.2 / 72.2 / 52.6 | 93.3 / 79.3 / 61.4 | 2.3% / 9.0% / 14.4% |
| Low corr. | IFEval (binary) | 94.9 / 87.4 / 76.8 | 94.4 / 87.4 / 75.0 | −0.5% / 0.0% / −2.4% |
| Low corr. | GPQA (binary) | 97.4 / 91.3 / 82.0 | 96.9 / 90.5 / 80.1 | −0.5% / −0.9% / −2.4% |
| Low corr. | GPQA (cont.) | 94.0 / 81.0 / 64.8 | 90.1 / 73.4 / 56.9 | −4.4% / −10.4% / −14.0% |
Multi-target BayesAME saves around 7.6%–14.4% of all model–item evaluations when targets are correlated. When targets are uncorrelated it costs more, in particular, up to 2.4% for binary scores and up to 14% for continuous scores, consistent with the negative transfer discussed in Sec. 5.2.
Follow-up Before Discussion Ends
Dear Reviewer, thank you again for your time and thoughtful review. As the discussion window closes soon, we would be very grateful to hear whether our response and additional experiments addressed your concerns, or if anything remains unclear.
We thank the reviewer for their thoughtful review. Below, we address their comments and questions.
We added a calibration analysis in Appendix C.6 of the paper (text in red) and we report the findings in the general comment above.
Whether a (target model, seed)-run uses the information-gain criterion or random selection is determined globally at the start of the trajectory based on the information provided by the reference models. In the table of the general comment, we report the percentage of the trajectories (over which we aggregate for the plots in the paper) that use the information-gain criterion. In particular:
Sensitivity to N0. Please see our response to Reviewer VWA5 and Appendix C.7 in the updated paper.
Sensitivity to σ2. We vary σ2 on GPQA with binary scores:
Nearly-unique reference regime (50% and 90% reference models): We consider σ2∈10−6,10−4,5×10−3,10−1. Figures 16 and 17 in the updated paper and Table 1 below (Table 5 in Appendix C.7) show that σ2∈10−6,10−4,5×10−3 yield similar RMSE, stopping coreset sizes and coverage. Increasing σ2 to 10−1 worsens BayesAME's RMSE, particularly under extrapolation (Figure 17, top two rows), while BayesAME-RS's RMSE remains largely insensitive (Figure 17, bottom two rows).
Non-unique reference regime (10% reference models): We consider σ2∈10−3,10−2,10−1,1. Figure 18 shows that RMSE is comparable across values, with the default σ2=10−1 yielding slightly lower values. However, sensitivity is more pronounced in interval coverage and stopping (Table 2 below, Table 6 in Appendix C.7): σ2=10−3 produces substantial undercoverage and tolerance-exceedance rates of 9.5-17.2%. The default σ2=10−1 yields conservative intervals and exceedance rates of at most 0.1%, but median coreset size at stopping is at least 99%. The intermediate choice σ2=10−2 offers a less conservative trade-off: at stopping threshold ϵ2=0.03, median coreset size at stopping falls to 90-93%, with exceedance rates of 2.8-3.3%.