- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies benchmark compression through the behavior of uniformly sampled subsets and shows that Random is often accurate but occasionally produces large errors when a subset is systematically easier or harder than the full benchmark. Based on a theoretical analysis of this common-shift effect, the paper proposes BestRand, which uses a few probe models to select the most balanced candidate among multiple random subsets. The method is evaluated extensively across diverse text, multimodal, and agent benchmarks.
Critical strengths
- Simple and practically effective method supported by broad empirical evaluation.
BestRand is exceptionally simple, requires no learned estimator for new target models, and consistently improves average MAE over Random across diverse settings. Extensive assessments provide credible evidence that lightweight probe-guided selection is practically useful rather than being tailored to a small number of benchmarks. Additional experiments such as OOD, ablations, etc., further strengthen the empirical scope of the work.
- Interpretable diagnosis of an important failure mode of random benchmark subsampling.
The common-shift perspective provides a clear explanation for why a random subset may become systematically easier/harder compared to the full benchmark. Finite-population moment analysis and the decomposition of common-shift variance are conceptually useful, and they motivate a lightweight method for suppressing unfavorable random draws. More broadly, the finding that Random is already strong, but can be made more reliable through inexpensive balancing, is informative for future research.
Critical weaknesses
- The theoretical and statistical evidence does not fully support the paper’s claims.
The theory establishes a narrower result than the presentation sometimes suggests. Theorem 1 provides a common-shift-based lower bound on the MAE tail rather than a characterization of the full MAE tail, and Theorem 2’s mode being close to the limiting support endpoint does not by itself imply a high probability of meeting a small error tolerance. Moreover, Theorem 3 analyzes an idealized selector using independently sampled full-benchmark candidates and the exact full-benchmark probe mean, whereas the actual BestRand implementation draws candidates from a shared finite pool and uses a noisy pool mean. Thus, the theoretical claims do not directly guarantee contraction of the total MAE or its upper tail for the implemented algorithm. In addition, only three seeds are repeated for the competing stochastic baselines, in contrast to 10,000 trials for Random and BestRand, which does not correspond as well; the claim against competing methods is not yet adequately supported.
- The practical cost advantage and methodological advantage over closely related sampling strategies are not sufficiently established.
The reported construction cost counts historical model-item records, but it does not evaluate total end-to-end cost after combining probe inference and subsequent target-model evaluation. An equal-budget comparison against a larger Random subset and a break-even analysis as a function of the number of target models are needed to substantiate the low-cost claim. In addition, BestRand is closely related to re-randomization and covariate-balanced sampling, yet the experimental comparison focuses mainly on more complex benchmark-compression methods. Direct comparisons with simple probe-stratified sampling, balanced sampling, deterministic mean balancing, or a full-reference probe-matching variant would help establish whether BestRand provides an advantage beyond standard balancing strategies.
Questions for authors
- Stability and theoretical justification.
Could the authors repeat the high-error-risk comparison using sufficiently many trials for all methods? Please also clarify how much of the improvement comes from reducing the common shift and whether the result holds when using the pool reference rather than the exact full-benchmark mean.
- End-to-end cost.
How does BestRand compare with Random and simple balanced-sampling methods under the same total cost, including construction and target-model evaluation? A break-even analysis over the number of target models would clarify when BestRand is genuinely cost-effective.
- Ranking utility
BestRand improves score MAE but not Spearman ranking over Random. Therefore, if model selection or ranking is a primary use case, its practical benefit appears limited. What do the authors think about this limitation?
Satisfactory answers and supporting results for the questions above (especially Questions 1 and 2), would address my main concerns and could raise my recommendation.
4 comments
Response to Reviewer PueL Part 2
[PueL Q4] Given BestRand's close connection to re-randomization and covariate balancing, how does it compare with simple probe-based sampling and balancing methods?
[PueL A4] Thank you for suggesting these comparisons. We extend the main and OOD experiments with four baselines using the same five probes and candidate pool as BestRand:
- Stratified sampling groups items by mean probe score and samples proportionally from each group.
- Clustered sampling groups items by their probe-response vectors and samples proportionally from each cluster.
- Nearest-mean selection selects items whose probe-response vectors are closest to the pool mean.
- Threshold acceptance accepts a random subset once its probe mean is sufficiently close to the pool mean, with at most 128 draws.
| Method | Text | Multimodal | Agent | Overall | OOD |
|---|---|---|---|---|---|
| Stratified sampling | 3.149 | 3.085 | 5.659 | 3.493 | 1.779 |
| Clustered sampling | 3.113 | 3.019 | 5.681 | 3.458 | 1.777 |
| Nearest-mean selection | 10.847 | 9.998 | 14.425 | 11.164 | 13.692 |
| Threshold acceptance | 3.308 | 3.143 | 5.940 | 3.646 | 1.858 |
| BestRand | 2.923 | 2.869 | 5.429 | 3.268 | 1.692 |
BestRand achieves the lowest MAE in both the main and OOD experiments, with the best results in every reported column.
.
.
[PueL Q5] Under the same total cost, how does BestRand compare with larger Random subsets and simple balancing methods? When does its probe cost become worthwhile?
[PueL A5] Thank you for this practical suggestion. BestRand achieves lower MAE at the same total evaluation cost. Each method uses its maximum affordable subset, allowing Random to spend its saved probe cost on more items. We compare both newly evaluated probes and existing evaluation results, which are equally available to all probe-based methods.
| Setting | Random | Clustered sampling | Threshold acceptance | BestRand |
|---|---|---|---|---|
| 1 new probe, 30 targets | 2.342 | 2.339 | 2.459 | 2.308 |
| 1 previously evaluated model, 1 target | 2.342 | 2.147 | 2.264 | 2.128 |
| 2 previously evaluated models, 1 target | 2.342 | 2.153 | 2.250 | 2.077 |
With one new probe, BestRand becomes cheaper than Random after about 16 target evaluations. At the same MAE, its lower per-target cost offsets the initial probe evaluation. The costs below use one full-benchmark evaluation of one model as one unit.
| Cost component | Random | TinyBenchmarks | ZipBench | BestRand, 1 probe | BestRand, 3 probes |
|---|---|---|---|---|---|
| Probe evaluation | 0 | 5.0000 | 5.0000 | 0.6807 | 1.8753 |
| Evaluation per target | 0.3014 | 0.2627 | 0.2544 | 0.2582 | 0.2211 |
In practice, existing evaluation results make the savings immediate. These results naturally arise in two situations:
- Users may evaluate one or two models before finding full-benchmark evaluation too expensive or slow. Those completed evaluations can directly provide BestRand's probe scores.
- Without full-benchmark results, users cannot check a random subset's errors or decide how much to compress. Even Random therefore typically needs full evaluations of one or two models. These results can also supply BestRand's probe scores, improving accuracy and stability without extra probe cost.
Response to Reviewer PueL Part 3
[PueL Q6] BestRand improves MAE but not Spearman correlation over Random. What practical benefit does it offer when model selection or ranking is the primary goal?
[PueL A6] Thank you for raising this point. We focus on the more demanding task of accurate score estimation. As discussed in Section 3 and Appendix E.1, leading methods already achieve an average SP of 0.969 at 50% retention. Yauney et al. [1] (ICLR 26 Oral) likewise show high overall rank correlation with very small subsets in Figure 1. Section 5.1 further explains that shared score errors can preserve rankings while increasing MAE, making accurate scores a stricter evaluation target.
Our additional experiment shows that BestRand retains useful ranking accuracy for closely scoring models. The table below presents five representative benchmarks at 50% retention with five probes. The last two columns report correct pairwise ordering rates for full-benchmark score gaps within one or two percentage points.
| Benchmark | MAE | SP | Ranking accuracy, score gap ≤ 0.01 | Ranking accuracy, score gap ≤ 0.02 |
|---|---|---|---|---|
| MMLU-Pro | 0.285 | 0.9975 | 90.6% | 94.2% |
| HellaSwag | 0.205 | 0.9973 | 85.4% | 92.1% |
| BBEH | 0.331 | 0.9891 | 78.7% | 87.8% |
| PAWS | 0.263 | 0.9918 | 78.6% | 90.9% |
| Drafter-Bench | 0.364 | 0.9900 | 76.7% | 88.0% |
.
.
[PueL Q7] Theorem 1 gives a tail lower bound, while Theorem 2 places the mode near the limiting lower endpoint. How do these support claims about the full MAE tail and high-probability accuracy?
[PueL A7] Thank you for this helpful question.
-
Theorem 1 characterizes the full MAE tail under the joint Gaussian approximation. Equation 5 specifies the joint model-error distribution, which determines the full MAE distribution and its tail probabilities. Equation 6 separately extracts the common-shift contribution as a lower bound. Our theoretical predictions are highly consistent with experimental results across 70 text, multimodal, and agent benchmarks, as illustrated in Figures 1 and 9.
-
Theorem 2 provides an intuitive characterization of the observed low-error peak. It helps readers understand this phenomenon by quantitatively describing how the mode approaches the limiting lower endpoint as more models are evaluated. Figures 1 and 9 show that the peak near the lower end accompanies a large fraction of low-error draws. Figure 15 further demonstrates close agreement between theoretical and experimental shapes and modes with 5, 10, and 20 models. Thus, the empirical MAE distributions show that small errors occur frequently, while Theorem 2 quantitatively explains the observed low-error peak.
.
.
[1] How Reliable is Language Model Micro-Benchmarking? ICLR 2026.
Thanks for the detailed rebuttal. Most of my concerns are resolved and therefore I raise my score.
Response to Reviewer PueL Part 1
Thank you for recognizing BestRand's simplicity, consistent MAE improvements, broad evaluation, and interpretable diagnosis of random sampling's instability. We also appreciate your positive assessment of our OOD experiments and ablations. Your suggestions help us strengthen the theoretical presentation and practical evidence.
.
.
[PueL Q1]Can you repeat the high-error-risk comparison with enough trials for all methods, given that competing baselines use only three seeds versus 10,000 for Random and BestRand?[PueL A1]Thank you for this helpful suggestion. Actually Our existing ten-seed results support the same conclusion. Appendix Figure 7 reports a comparison with ten seeds for BestRand and competing baselines, while Random retains 10,000 trials. Its results closely agree with Figure 6.Our new 100-seed experiment yields consistent results. Following your suggestion, we repeat the comparison with 100 seeds for every method. BestRand's risk estimates remain close to those in Figures 6 and 7, retaining a 67.58% relative risk reduction against Random, ZipBench, and TinyBenchmarks.
.
.
[PueL Q2]How much of BestRand's improvement in MAE and high-error risk comes from reducing the common shift?[PueL A2]Thank you for suggesting this analysis. We add a controlled ablation across 70 main benchmarks and 18 OOD benchmarks using 100 paired seeds. The five probes, candidate pools, and candidate subsets are identical across settings. Disabling common-direction matching removes most of BestRand's gains.For each probe, let dj be its subset-minus-pool score difference and dˉ their average. The identity ∑jdj2=5dˉ2+∑j(dj−dˉ)2 splits the objective into a: common-direction matching and b: between-probe variation matching.
Random takes the first candidate without screening. The other settings select candidates using a, b, or both, retaining their original weights. High-error risk uses the same fixed Random P70 MAE threshold across settings. Parentheses report relative reductions from Random.
Removing the common term loses roughly two-thirds to three-quarters of the improvement. Comparing + b with + a + b, disabling a loses 74.5% of the main MAE improvement, 73.5% of the risk reduction, and 66.9% of the OOD MAE improvement, relative to the full method's gains over Random. Common-only matching also improves on Random, while the full objective performs best. These ratios quantify the effect of this design component rather than a unique attribution of target-model error sources.
.
.
[PueL Q3]Theorem 3 uses independent full-benchmark candidates and the exact probe mean. Does its analysis extend to shared-pool candidates and a noisy pool mean, and does it guarantee lower total MAE or high-error risk?[PueL A3]Thank you for highlighting this gap. Both our theoretical discussion and existing experiments address the finite-pool implementation.The analysis accounts for finite-pool reference error. Theorem 3 analyzes the idealized selector. The unnumbered triangle inequality in Section 2.2, lines 261–265, connects it to the practical implementation by bounding full-reference imbalance with the matching error within the pool plus the pool-reference error. Candidate search reduces the former, while increasing the pool size reduces the latter in mean square. The reference becomes exact at full-benchmark coverage.
Existing experiments directly support finite-pool BestRand. Figure 16 shows that matching the pool mean also improves matching to the full-benchmark mean as more candidates are considered. Figure 17 further shows that finite-pool BestRand reduces common-shift fluctuations nearly as much as full-reference selection across 24 benchmarks. Figures 2 and 3 and Table 1 also demonstrate lower total MAE and high-error risk for the practical implementation, providing direct empirical support for these improvements.