acceptodds
Under review as a conference paper at ICLR 2027

Random Is Almost Enough? Stable and Low-Cost Benchmark Compression via BestRand

Abstract

Benchmark compression reduces the often prohibitive time and cost of model evaluation by evaluating models on a small benchmark subset that closely matches full-benchmark results. To quantify estimation error, we derive an accurate analytical approximation to the error distribution when estimating full-benchmark scores from a subset, showing that most subsets yield small errors, while a few produce much larger errors, resulting in a long-tailed distribution. This means random sampling alone can achieve surprisingly strong performance with high probability, but remains unstable due to rare large errors. We further show that this instability is inherent to modern foundation models and persists even as more models are evaluated. Based on these findings, we introduce BestRand, a stable and low-cost benchmark compression method that preserves the efficiency of random sampling while substantially reducing its high-error tail. We validate BestRand on 70+ text, multimodal, and agent benchmarks, and further extend it to VLA evaluation. Compared with competitive methods, BestRand reduces high-error risk by 50.2% and estimation error by 16.4% on average, while requiring only 51.3% of the construction cost. It further reduces OOD estimation error by 21.8% on average across score-range, model-family, and temporal shifts. Together, our theory provides practical guidance for understanding and predicting the accuracy and reliability of benchmark compression, while BestRand enables stable and accurate compression at low cost. All code and data will be released.

Primary area:
datasets and benchmarks
Keywords:
Efficient, Low-cost, Benchmark, Theory-inspired

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PueLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies benchmark compression through the behavior of uniformly sampled subsets and shows that Random is often accurate but occasionally produces large errors when a subset is systematically easier or harder than the full benchmark. Based on a theoretical analysis of this common-shift effect, the paper proposes BestRand, which uses a few probe models to select the most balanced candidate among multiple random subsets. The method is evaluated extensively across diverse text, multimodal, and agent benchmarks.

Critical strengths
  1. Simple and practically effective method supported by broad empirical evaluation.

BestRand is exceptionally simple, requires no learned estimator for new target models, and consistently improves average MAE over Random across diverse settings. Extensive assessments provide credible evidence that lightweight probe-guided selection is practically useful rather than being tailored to a small number of benchmarks. Additional experiments such as OOD, ablations, etc., further strengthen the empirical scope of the work.

  1. Interpretable diagnosis of an important failure mode of random benchmark subsampling.

The common-shift perspective provides a clear explanation for why a random subset may become systematically easier/harder compared to the full benchmark. Finite-population moment analysis and the decomposition of common-shift variance are conceptually useful, and they motivate a lightweight method for suppressing unfavorable random draws. More broadly, the finding that Random is already strong, but can be made more reliable through inexpensive balancing, is informative for future research.

Critical weaknesses
  1. The theoretical and statistical evidence does not fully support the paper’s claims.

The theory establishes a narrower result than the presentation sometimes suggests. Theorem 1 provides a common-shift-based lower bound on the MAE tail rather than a characterization of the full MAE tail, and Theorem 2’s mode being close to the limiting support endpoint does not by itself imply a high probability of meeting a small error tolerance. Moreover, Theorem 3 analyzes an idealized selector using independently sampled full-benchmark candidates and the exact full-benchmark probe mean, whereas the actual BestRand implementation draws candidates from a shared finite pool and uses a noisy pool mean. Thus, the theoretical claims do not directly guarantee contraction of the total MAE or its upper tail for the implemented algorithm. In addition, only three seeds are repeated for the competing stochastic baselines, in contrast to 10,000 trials for Random and BestRand, which does not correspond as well; the claim against competing methods is not yet adequately supported.

  1. The practical cost advantage and methodological advantage over closely related sampling strategies are not sufficiently established.

The reported construction cost counts historical model-item records, but it does not evaluate total end-to-end cost after combining probe inference and subsequent target-model evaluation. An equal-budget comparison against a larger Random subset and a break-even analysis as a function of the number of target models are needed to substantiate the low-cost claim. In addition, BestRand is closely related to re-randomization and covariate-balanced sampling, yet the experimental comparison focuses mainly on more complex benchmark-compression methods. Direct comparisons with simple probe-stratified sampling, balanced sampling, deterministic mean balancing, or a full-reference probe-matching variant would help establish whether BestRand provides an advantage beyond standard balancing strategies.

Questions for authors
  1. Stability and theoretical justification.

Could the authors repeat the high-error-risk comparison using sufficiently many trials for all methods? Please also clarify how much of the improvement comes from reducing the common shift and whether the result holds when using the pool reference rather than the exact full-benchmark mean.

  1. End-to-end cost.

How does BestRand compare with Random and simple balanced-sampling methods under the same total cost, including construction and target-model evaluation? A break-even analysis over the number of target models would clarify when BestRand is genuinely cost-effective.

  1. Ranking utility

BestRand improves score MAE but not Spearman ranking over Random. Therefore, if model selection or ranking is a primary use case, its practical benefit appears limited. What do the authors think about this limitation?

Satisfactory answers and supporting results for the questions above (especially Questions 1 and 2), would address my main concerns and could raise my recommendation.

4 comments
AuthorsOpenReview ↗

Response to Reviewer PueL Part 1

Thank you for recognizing BestRand's simplicity, consistent MAE improvements, broad evaluation, and interpretable diagnosis of random sampling's instability. We also appreciate your positive assessment of our OOD experiments and ablations. Your suggestions help us strengthen the theoretical presentation and practical evidence.

.

.

[PueL Q1] Can you repeat the high-error-risk comparison with enough trials for all methods, given that competing baselines use only three seeds versus 10,000 for Random and BestRand?

[PueL A1] Thank you for this helpful suggestion. Actually Our existing ten-seed results support the same conclusion. Appendix Figure 7 reports a comparison with ten seeds for BestRand and competing baselines, while Random retains 10,000 trials. Its results closely agree with Figure 6.

Our new 100-seed experiment yields consistent results. Following your suggestion, we repeat the comparison with 100 seeds for every method. BestRand's risk estimates remain close to those in Figures 6 and 7, retaining a 67.58% relative risk reduction against Random, ZipBench, and TinyBenchmarks.

Quantity, %Original Fig. 6Original Fig. 7Matched 100 seeds
BestRand: above-mean risk14.1012.7813.92
BestRand: worst-30% risk7.946.117.44
BestRand: worst-5% risk0.180.000.14
Relative risk reduction versus three baselines68.0671.0667.58

.

.

[PueL Q2] How much of BestRand's improvement in MAE and high-error risk comes from reducing the common shift?

[PueL A2] Thank you for suggesting this analysis. We add a controlled ablation across 70 main benchmarks and 18 OOD benchmarks using 100 paired seeds. The five probes, candidate pools, and candidate subsets are identical across settings. Disabling common-direction matching removes most of BestRand's gains.

For each probe, let djd_j be its subset-minus-pool score difference and dˉ\bar d their average. The identity ∑jdj2=5dˉ2+∑j(dj−dˉ)2\sum_j d_j^2=5\bar d^2+\sum_j(d_j-\bar d)^2 splits the objective into a: common-direction matching and b: between-probe variation matching.

Random takes the first candidate without screening. The other settings select candidates using a, b, or both, retaining their original weights. High-error risk uses the same fixed Random P70 MAE threshold across settings. Parentheses report relative reductions from Random.

AblationMain MAE ↓Main risk, % ↓OOD MAE ↓
Random baseline3.77729.581.931
+ a: Common-direction matching3.350 (↓11.3%)15.87 (↓46.3%)1.720 (↓10.9%)
+ b: Between-probe variation matching3.644 (↓3.5%)25.37 (↓14.2%)1.852 (↓4.1%)
+ a + b: Full BestRand3.255 (↓13.8%)13.74 (↓53.5%)1.692 (↓12.4%)

Removing the common term loses roughly two-thirds to three-quarters of the improvement. Comparing + b with + a + b, disabling a loses 74.5% of the main MAE improvement, 73.5% of the risk reduction, and 66.9% of the OOD MAE improvement, relative to the full method's gains over Random. Common-only matching also improves on Random, while the full objective performs best. These ratios quantify the effect of this design component rather than a unique attribution of target-model error sources.

.

.

[PueL Q3] Theorem 3 uses independent full-benchmark candidates and the exact probe mean. Does its analysis extend to shared-pool candidates and a noisy pool mean, and does it guarantee lower total MAE or high-error risk?

[PueL A3] Thank you for highlighting this gap. Both our theoretical discussion and existing experiments address the finite-pool implementation.

  1. The analysis accounts for finite-pool reference error. Theorem 3 analyzes the idealized selector. The unnumbered triangle inequality in Section 2.2, lines 261–265, connects it to the practical implementation by bounding full-reference imbalance with the matching error within the pool plus the pool-reference error. Candidate search reduces the former, while increasing the pool size reduces the latter in mean square. The reference becomes exact at full-benchmark coverage.

  2. Existing experiments directly support finite-pool BestRand. Figure 16 shows that matching the pool mean also improves matching to the full-benchmark mean as more candidates are considered. Figure 17 further shows that finite-pool BestRand reduces common-shift fluctuations nearly as much as full-reference selection across 24 benchmarks. Figures 2 and 3 and Table 1 also demonstrate lower total MAE and high-error risk for the practical implementation, providing direct empirical support for these improvements.

AuthorsOpenReview ↗

Response to Reviewer PueL Part 2

[PueL Q4] Given BestRand's close connection to re-randomization and covariate balancing, how does it compare with simple probe-based sampling and balancing methods?

[PueL A4] Thank you for suggesting these comparisons. We extend the main and OOD experiments with four baselines using the same five probes and candidate pool as BestRand:

  • Stratified sampling groups items by mean probe score and samples proportionally from each group.
  • Clustered sampling groups items by their probe-response vectors and samples proportionally from each cluster.
  • Nearest-mean selection selects items whose probe-response vectors are closest to the pool mean.
  • Threshold acceptance accepts a random subset once its probe mean is sufficiently close to the pool mean, with at most 128 draws.
MethodTextMultimodalAgentOverallOOD
Stratified sampling3.1493.0855.6593.4931.779
Clustered sampling3.1133.0195.6813.4581.777
Nearest-mean selection10.8479.99814.42511.16413.692
Threshold acceptance3.3083.1435.9403.6461.858
BestRand2.9232.8695.4293.2681.692

BestRand achieves the lowest MAE in both the main and OOD experiments, with the best results in every reported column.

.

.

[PueL Q5] Under the same total cost, how does BestRand compare with larger Random subsets and simple balancing methods? When does its probe cost become worthwhile?

[PueL A5] Thank you for this practical suggestion. BestRand achieves lower MAE at the same total evaluation cost. Each method uses its maximum affordable subset, allowing Random to spend its saved probe cost on more items. We compare both newly evaluated probes and existing evaluation results, which are equally available to all probe-based methods.

SettingRandomClustered samplingThreshold acceptanceBestRand
1 new probe, 30 targets2.3422.3392.4592.308
1 previously evaluated model, 1 target2.3422.1472.2642.128
2 previously evaluated models, 1 target2.3422.1532.2502.077

With one new probe, BestRand becomes cheaper than Random after about 16 target evaluations. At the same MAE, its lower per-target cost offsets the initial probe evaluation. The costs below use one full-benchmark evaluation of one model as one unit.

Cost componentRandomTinyBenchmarksZipBenchBestRand, 1 probeBestRand, 3 probes
Probe evaluation05.00005.00000.68071.8753
Evaluation per target0.30140.26270.25440.25820.2211

In practice, existing evaluation results make the savings immediate. These results naturally arise in two situations:

  1. Users may evaluate one or two models before finding full-benchmark evaluation too expensive or slow. Those completed evaluations can directly provide BestRand's probe scores.
  2. Without full-benchmark results, users cannot check a random subset's errors or decide how much to compress. Even Random therefore typically needs full evaluations of one or two models. These results can also supply BestRand's probe scores, improving accuracy and stability without extra probe cost.
AuthorsOpenReview ↗

Response to Reviewer PueL Part 3

[PueL Q6] BestRand improves MAE but not Spearman correlation over Random. What practical benefit does it offer when model selection or ranking is the primary goal?

[PueL A6] Thank you for raising this point. We focus on the more demanding task of accurate score estimation. As discussed in Section 3 and Appendix E.1, leading methods already achieve an average SP of 0.969 at 50% retention. Yauney et al. [1] (ICLR 26 Oral) likewise show high overall rank correlation with very small subsets in Figure 1. Section 5.1 further explains that shared score errors can preserve rankings while increasing MAE, making accurate scores a stricter evaluation target.

Our additional experiment shows that BestRand retains useful ranking accuracy for closely scoring models. The table below presents five representative benchmarks at 50% retention with five probes. The last two columns report correct pairwise ordering rates for full-benchmark score gaps within one or two percentage points.

BenchmarkMAESPRanking accuracy, score gap ≤ 0.01Ranking accuracy, score gap ≤ 0.02
MMLU-Pro0.2850.997590.6%94.2%
HellaSwag0.2050.997385.4%92.1%
BBEH0.3310.989178.7%87.8%
PAWS0.2630.991878.6%90.9%
Drafter-Bench0.3640.990076.7%88.0%

.

.

[PueL Q7] Theorem 1 gives a tail lower bound, while Theorem 2 places the mode near the limiting lower endpoint. How do these support claims about the full MAE tail and high-probability accuracy?

[PueL A7] Thank you for this helpful question.

  1. Theorem 1 characterizes the full MAE tail under the joint Gaussian approximation. Equation 5 specifies the joint model-error distribution, which determines the full MAE distribution and its tail probabilities. Equation 6 separately extracts the common-shift contribution as a lower bound. Our theoretical predictions are highly consistent with experimental results across 70 text, multimodal, and agent benchmarks, as illustrated in Figures 1 and 9.

  2. Theorem 2 provides an intuitive characterization of the observed low-error peak. It helps readers understand this phenomenon by quantitatively describing how the mode approaches the limiting lower endpoint as more models are evaluated. Figures 1 and 9 show that the peak near the lower end accompanies a large fraction of low-error draws. Figure 15 further demonstrates close agreement between theoretical and experimental shapes and modes with 5, 10, and 20 models. Thus, the empirical MAE distributions show that small errors occur frequently, while Theorem 2 quantitatively explains the observed low-error peak.

.

.

[1] How Reliable is Language Model Micro-Benchmarking? ICLR 2026.

Thanks for the detailed rebuttal. Most of my concerns are resolved and therefore I raise my score.

Reviewer_hPdJOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to estimate a model’s score on an entire benchmark using a small subset. The authors derive the distribution of subset estimation error. Based on this finding, the authors propose BestRand: sample multiple random subsets, and select the subset whose average score from a small set of probe models over the candidate pool is closest to the reference value.

Critical strengths
  1. The theoretical framework is well-established. This work is one of the rare studies in the re-evaluation domain that includes complete theoretical analysis, which I believe is worthy of learning for researchers in this field.
  2. Substantial experimental efforts. The paper contains abundant experiments, and the scale of experiments convinces me that the conclusions are reliable.
Critical weaknesses
  1. The theory in this paper mainly analyzes the Common Shift and its variance, and demonstrates that probe-guided selection can reduce the fluctuation of Common Shift. However, Common Shift constitutes only one component of the overall MAE. Reducing the variance of Common Shift does not directly guarantee the concurrent reduction of the overall MAE or the tail probability of large errors. Furthermore, the theorem analyzes an idealized subset selection procedure, which is not fully consistent with the practical implementation of BestRand using a finite candidate pool.

  2. The paper highlights the low-cost advantage of BestRand, yet experiments primarily measure computational overhead only at the subset construction stage, without fully accounting for the full pipeline consisting of probe model inference, subset construction, and subsequent target model evaluation.

Since BestRand requires additional evaluation via the probe model, whereas naive random sampling incurs nearly no construction overhead, its practical efficiency gain hinges on the reuse rate of the compressed subset, the inference cost of the probe model, and the specific evaluation scenario.

Questions for authors

See Weakness.

2 comments
AuthorsOpenReview ↗

Response to Reviewer hPdJ Part 1

Thank you for recognizing our complete theoretical analysis and extensive, convincing experiments. We hope the following responses address your concerns.

.

.

[hPdJ Q1] Common shift is only one component of MAE, so reducing its variance alone does not guarantee lower overall MAE or large-error probability. What supports these improvements?

[hPdJ A1] Thank you for this insightful comment. Our theory explains the common-shift mechanism, and our experiments verify lower high-error risk and overall MAE.

  1. Probe matching reduces common-shift fluctuations. Equation 11 shows how probe matching can reduce the common-shift scale ss toward s1−R2s\sqrt{1-R^2}. Equation 6 gives

    \\;\geq\\; \underbrace{\Pr(|\bar E| > t)}_{\text{common-shift tail probability}} \\;\approx\\; 2\Phi\\!\left(-\frac{t}{s}\right).$$ For fixed $t>0$, as $s$ decreases, $2\Phi(-t/s)$ decreases, so the common-shift tail probability decreases within this Gaussian approximation. This lowers the bound on the overall high-error probability on the left; whether that probability itself decreases is verified experimentally below. This explains the shared source of large errors that BestRand seeks to reduce.
  2. Experiments show fewer large errors and lower overall MAE. Figure 1 shows that ss strongly predicts empirical MAE P95 with R2=0.993R^2=0.993, closely matching the theoretical relationship between common-shift scale and high-error risk. Figures 2 and 3 and Table 1 further show that BestRand suppresses high-error draws and lowers mean MAE across 70+ LLM, VLM, VLA, and agent benchmarks. Relative to Random, ZipBench, and TinyBenchmarks, high-error risk decreases by 50.2% and mean MAE by 16.4% on average.

.

.

[hPdJ Q2] The theorem analyzes idealized subset selection, while BestRand uses a finite candidate pool. How is this gap between theory and implementation addressed?

[hPdJ A2] Thank you for highlighting this gap. Both our theoretical discussion and existing experiments address the finite-pool implementation.

  1. The theory explicitly accounts for the finite-pool reference error. Section 2.2 bounds the full-reference imbalance by the matching error within the pool plus the error of the pool reference. Candidate search reduces the matching error, while increasing the pool size reduces the reference error in mean square. The reference becomes exact at full-benchmark coverage. This explains how the practical selection criterion approximates the idealized one.

  2. Existing experiments directly support the finite-pool implementation. Figure 16, page 61, shows that although BestRand matches only the pool mean, its selected subsets also better match the full-benchmark probe mean as more candidates are considered. Figure 17, page 62, further shows that finite-pool BestRand reduces common-shift fluctuations nearly as much as full-reference selection across 24 benchmarks. These experimental trends closely agree with the theoretical predictions.

AuthorsOpenReview ↗

Response to Reviewer hPdJ Part 2

[hPdJ Q3] Is BestRand still cost-effective when probe inference, subset construction, and target evaluation are all included, especially compared with Random?

[hPdJ A3] Thank you for your constructive suggestion. We compare lifetime costs, including probe inference and target evaluation.

Lower setup and testing costs than learned methods. BestRand uses only five probes on a limited pool, keeping setup costs low, as supported by Figures 4b and 5. The table below compares these costs, with one unit of inference cost defined as evaluating one model on the full benchmark. Subset construction is generally inexpensive, and its runtime is reported in seconds. At the same MAE, BestRand needs fewer test items than learned methods, so each model costs less to evaluate.

Cost componentTinyBenchmarksZipBenchBestRand (1 probe)BestRand (3 probes)
Probe inference cost5.00005.00000.6807 (↓86.39%)1.8753 (↓62.49%)
One-time subset construction time (s)56.15 s28.41 s0.0027 s (↓99.99%)0.0031 s (↓99.99%)
Evaluation cost per new model0.26270.25820.2544 (↓1.47%)0.2211 (↓14.37%)

Lower testing cost than Random. Across 70 text, multimodal, and agent benchmarks, Random occasionally selects subsets with very large errors, raising its average MAE. Since accurate score estimation is central to benchmark compression, we compare testing costs at the same MAE. BestRand reaches this accuracy with fewer items, reducing the cost of each target evaluation.

Cost componentRandomBestRand (1 probe)BestRand (3 probes)
Probe inference cost00.68071.8753
One-time subset construction time (s)0.00026 s0.0027 s0.0031 s
Evaluation cost per new model0.30140.2582 (↓14.33%)0.2211 (↓26.64%)

We agree that practical savings depend on subset reuse. Benchmark compression typically targets repeated evaluation of models or checkpoints, so reuse is generally high. Even when probe responses must be newly collected, cumulative inference cost falls below Random’s after 16–24 evaluations. Users may also adopt compression after evaluating one to three models and recognizing the high time or monetary cost. BestRand can reuse these evaluation records as probe responses, avoiding additional inference.

Reviewer_t1fROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents BestRand, a benchmark compression method that works by evaluating a set of probe models on multiple randomly sampled subsets of the benchmark items, and using the most representative of these subsets to evaluate future models. There are theoretical results with Gaussian approximations to justify this method. On experiments with 5 historical models used for fitting the subset, this performs baselines in terms of MAE, both if the historical models are chosen randomly, and from a different distribution than the target models.

Critical strengths
  • The approach outperforms the considered baselines in the considered low-resource setting
  • The inclusion of various OOD settings into the experiments is nice
Critical weaknesses
  • I am not sure I understand the motivation for theorem 2. The fact that random sampling is enough, with high probability, follows from bog-standard concentration results without any need for Gaussian approximations etc. More broadly, the theory section seems overly long and complicated, without yielding a very strong payoff.
  • It is relatively clear that unlike for (uncited) prior work like Wu et al 2026 https://arxiv.org/pdf/2601.20251 the proposed method can fail arbitrarily badly when the assumed correlation structure breaks.
  • Key baselines are omitted. For example, the cited Zhang et al. 2025a claims that their suggested AIPW method clearly outperforms random sampling in the OOD setting, so it is unclear whether BestRand is actually SOTA.
  • Unlike prior work, the paper focuses on a regime with very few available models. While this seems reasonable for some applications, it could be motivated better, and highlighted more clearly. I presume that the reason most baselines underperform random sampling is precisely due to that change in evaluation regime.
Questions for authors
  • What are the right-facing triangles in Figure 5 representing?
7 comments
AuthorsOpenReview ↗

Response to Reviewer t1fR Part 1

Thank you for recognizing BestRand's strong performance. We provide the following clarifications and additional experiments to address your concerns.

.

.

[t1fR Q1] Standard concentration bounds already explain why Random is accurate. What does Theorem 2 add, and is the theoretical treatment necessary?

[t1fR A1] Thank you for bringing up this point. Concentration bounds describe deviations around the mean, while Theorem 2 characterizes the mode. The mode marks the peak of the MAE distribution, identifying the most frequent error region across random subsets. In the right-skewed distributions considered here, occasional large errors raise the mean above this peak. Therefore, for the phenomenon described in this paper, the mode is more suitable than the mean. Our Theorem 2 provides a precise characterization of the mode, helping readers better understand this phenomenon.

Error statisticHellaSwagLiveCodeBenchMathVistaMMMU-ProBFCLOSWorld
Mean0.7602.5501.9531.7701.0203.580
Mode0.4301.7711.4601.3800.8003.030

Theorem 2 explains why Random is almost enough and motivates BestRand's design. As MM grows, the mode approaches the lower endpoint of the limiting MAE distribution. Together with the observed concentration at small errors, this suggests that random sampling already provides good candidates, motivating BestRand to select reliable subsets from them at low cost. The theoretical predictions are highly consistent with empirical results across 70+ LLM, VLM, VLA, and agent benchmarks (Figures 1, 9, and 15).

.

.

[t1fR Q2] BestRand may fail when the assumed correlation structure breaks. How does its reliability compare with Wu et al. 2026?

[t1fR A2] Thank you for bringing this work to our attention. We will cite FAQ and add a comparison in the revised paper. We make structural assumptions to keep the analysis tractable. Appendix F.2 already validates the assumed structure empirically, showing that the assumptions are closely satisfied across diverse LLM, VLM, and agent benchmarks.

Although these assumptions may not hold perfectly in real data, our existing experiments demonstrate strong performance across 70+ text, multimodal, and agent benchmarks, with further validation on VLA tasks. This supports BestRand's broad applicability even when the assumed structure is not perfectly matched.

In [t1fR A3], we add comparisons with FAQ across these benchmarks and under OOD shifts, showing that BestRand achieves lower MAE in both settings.

AuthorsOpenReview ↗

Response to Reviewer t1fR Part 2

[t1fR Q3] AIPW from Zhang et al. 2025a improves over Random under OOD shifts. Does BestRand retain its advantage against this missing baseline?

[t1fR A3] Thank you for highlighting AIPW as a missing baseline.

(1) Our existing evaluation already covers 15 baselines. According to Table 1, Section 4, and Appendix C.3, BestRand achieves the strongest overall performance among these methods. Our contribution is a theoretical framework and a practical method for accurate, stable, and low-cost benchmark compression, without claiming state-of-the-art performance.

(2) We add comparisons with AIPW and FAQ in both main and OOD experiments. Under the same evaluation settings, BestRand achieves lower MAE in the main experiments and lower mean OOD MAE than both methods.

Main experiments. Mean MAE in percentage points, averaged across the six retention ratios in Table 1.

Domain typeRandomtinyBenchmarksZipBenchFAQAIPWBestRand
Text3.473.653.233.223.302.92
Multimodal3.243.403.193.152.942.87
Agent6.146.276.015.735.875.43
Overall4.284.444.144.034.043.74

OOD experiments. Mean MAE in percentage points. Settings and aggregation follow Table 2.

MethodMid→TailsLow→HighHigh→LowFamily OODTemporal OODOOD mean
Random1.812.021.781.972.011.95
tinyBenchmarks2.342.463.032.202.272.36
FAQ2.132.202.212.032.012.07
AIPW1.641.851.661.811.721.75
BestRand1.601.741.611.701.731.69

.

.

[t1fR Q4] The few-model setting seems reasonable, but needs clearer motivation and emphasis. Does it explain why most baselines underperform Random?

[t1fR A4] Thank you for this helpful comment. As you note, this setting is reasonable for practical applications.

(1) Using only a few historical models is more practical. Collecting complete benchmark responses from tens or hundreds of models can be prohibitively costly, making methods that require such records difficult to use in practice. Our five-model setting reflects this constraint.

(2) Our paper already evaluates settings with abundant historical models. Figure 4(b) compares 2, 10, 100, and 1,000 construction models. BestRand performs best with few models and remains competitive with the strongest alternatives even with hundreds or thousands, making it practical across historical-data budgets. The table below compares MAE across construction-model budgets.

Method2 models10 models100 models1,000 models
AnchorPoints17.883.844.635.85
tinyBenchmarks2.101.761.431.41
BestRand1.701.581.481.41

.

.

[t1fR Q5] What do the right-facing triangles in Figure 5 represent?

[t1fR A5] Thank you for pointing this out. The triangles indicate that the MAE of the AnchorPoints remains above 2% across the tested construction budgets. We will make the figure clearer in the revision.

  • I do not find the discussion around theorem 2 convincing at all. Concentration directly implies that most random subsets will have an empirical mean close to their expectation (i.e. the score on the whole benchmark), and thus a small MAE. On the other hand, it is not clear at all why the position of the mode of a continuous distribution would say much about why "random subsampling is almost enough".

  • I acknowledge that the method performs well in the tested OOD settings, but remain worried about forgoing theoretical worst-case guarantees for relatively minor empirical improvements. I would also strongly suggest to add the most competitive additional baselines to the updated main text, rather than hiding them in the appendix.

  • "Our contribution is a theoretical framework [...] without claiming state-of-the-art performance." I am slightly confused, since you seem to be demonstrating state of the art performance (albeit by a small margin), even after accounting for the additional baselines (for the specific considered setting), while I view the theoretical framework as the weakest part of this paper.

  • "Using only a few historical models is more practical." I understand that this can be the case in some settings (while historical plenty historical data is available in many other settings). My point was that the paper does not do a good job of actually motivating these settings, and the intro and abstract do not particularly highlight this shift in setting compared to most prior work, which I would characterize as somewhat misleading, especially as the claimed large improvements compared to other methods mostly rely on the shifted setting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.