- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Balance of Benchmarks (BoB) for aggregating sparse, heterogeneous benchmark scores. BoB separates two questions: 1) which evidence to retain and 2) how much influence each benchmark should receive. It combines benchmark-specific score equating with inverse-density weighting in a semantic embedding space, so crowded benchmark neighborhoods receive less total influence.
Critical strengths
-
Good problem formulation: Separating evidence retention from benchmark influence is conceptually useful and clearer than treating benchmark selection and weighting as the same decision.
-
Careful experimental decomposition: The paper separately tests adding more evidence and robustness to benchmark re-listing, and explicitly studies sparse reporting effects.
Critical weaknesses
- Related work and positioning must be strengthened.
The related work section misses several closely related efforts on benchmark redundancy and aggregation bias. In particular, Uzunoglu et al. (https://arxiv.org/abs/2509.25671) studies how uneven subdomain representation can distort aggregate scores; Siska et al. (https://aclanthology.org/2024.acl-long.560/) analyze how benchmark distributional assumptions and semantic similarity affect rankings; Wu et al. (https://arxiv.org/abs/2509.23488) studies functional overlap across benchmarks.
These works should be discussed to sharpen BoB's novelty.
- The empirical case for semantic density weighting is limited.
The strongest held out prediction result is primarily an evidence retention result rather than a BoB specific weighting result. In the strict panel, BoB improves by +0.035 when moving from the curated to full pool, but equal weight equating improves by about +0.037. Therefore, the direct BoB vs. equal-weight differences are close to zero and statistically inconclusive.
Density weighting clearly improves robustness to complete duplicated columns, but this is a favorable intervention for a method explicitly designed to share weight among duplicates. More importantly, at 25% copy coverage the direction reverses and density weighting performs worse than equal weighting. The paper should sharpen its practical claim, where the evidence supports multiplicity robustness under substantial overlap more strongly than improved model ranking in general.
- Description similarity is a weakly validated proxy for benchmark redundancy.
BoB's core weighting rule treats proximity between benchmark description embeddings as a proxy for related benchmark content, while explicitly acknowledging that it does not guarantee shared questions, protocols, or capabilities. This is consequential because the inferred semantic geometry directly determines benchmark influence.
The paper provides encoder and bandwidth sensitivity analyses, but I would like stronger validation that description space proximity corresponds to functional capability overlap that should actually be downweighted. A benchmark pair can sound semantically similar while measuring different skills, or use different language while eliciting the same underlying capability.
Questions for authors
I am overall open to increase my score if the authors address the concerns I highlight in weaknesses. Below are additional questions:
-
Can the authors validate description-embedding similarity against an independent notion of benchmark capability overlap, rather than mainly testing embedding-model sensitivity?
-
What practical recommendation should readers take from density weighting given that it provides essentially no held-out predictive improvement over equal-weight equating and can reverse under low reporting overlap?
-
How sensitive are latent model rankings to selective or outcome-dependent reporting in WildScores?
5 comments
I appreciate the authors' rebuttal.
I find the responses to W1, Q2, and Q3 satisfactory. Below are my remaining and new questions regarding W2 and W3/Q1:
W2: I am not sure if I am convinced duplicate listing is a realistic problem. If a benchmark suffers from duplicate (or near-duplicate) listings, then it must be a terrible benchmark. I am not sure if today's benchmarks would suffer from this, given that many of them are quite costly to run and don't have a large body of problems (e.g. like MMLU does). Do the authors have a convincing argument why this is a real problem and we should care about their solution?
W3/Q1: I cannot contextualize the precision results. What would be a very strong baseline here, and what would be its precision? Does the precision of 0.146, which description similarity achieves, mean a strong performance? Just by the numerical scale of precision, from 0 to 1, 0.146 looks rather low, and hence weak. I understand it's stronger than the base rate, but it's (0.146 vs. 0.018) not an astonishing overperformance in my understanding.
Also, a humble and unrelated (to above) suggestion I have for the paper is to be thoughtful about the appendix length. There are many sub-subsections. I understand appendix is for sharing relatively unimportant details for those who are curious, but it gets hard for the reader to properly spend their attention when appendix is tens of pages.
I appreciate the time and effort the reviewers put into the rebuttals.
Thank you for the follow-up. As you point out, many benchmarks are expensive to run, so we want to make use of every eligible score. Our goal is to build a broad, balanced picture of model performance while controlling how much related benchmarks count together.
(W2) The same question can contribute to more than one benchmark score in a composite index, even when both benchmarks are carefully designed. OpenVLM’s default overall score averages eight benchmark scores, including MathVista testmini and MMStar [3]. MMStar selects questions from existing datasets to build an evaluation that requires visual input. Its released data identify MathVista as the source of 345 of its 1,500 questions [4]. For example, MathVista testmini item 404 and MMStar item 399 use the same timetable and ask the same question with the same four choices [4, 5]. Both benchmarks provide useful coverage, but that question contributes to two scores in the published average.
The authors of the Epoch Capabilities Index describe a related case. Splitting one benchmark into two entries of similar difficulty can roughly double its weight in their fit, even when the underlying question-level data stay the same [1, Section 4.4].
We tested what happens when a developer and a third party both measure the same benchmark. Both scores can tell us something useful, but giving each score an equally weighted entry increases the benchmark’s share of the index. BoB computes weights from the descriptions of the benchmarks each model reports. When two scores are known to measure the same benchmark, our extension assigns weight to that benchmark once per model and divides it among the available scores. Both scores enter the fit, which reduces the need to remove related entries by hand to limit their combined weight. Keeping the full eligible pool improved held-out prediction with both BoB and equal-weight equating.
We tested the extension on 824 Artificial Analysis scores for benchmarks that the same models already had developer scores for. With the same added scores, the remeasured benchmarks’ average share of model weight rose by 7.88 percentage points with equal weights per entry and 0.35 points with current BoB using a shared description. With the new rule, the benchmarks kept their original weight share. Held-out correlations were close to those of current BoB in both panels.
(W3/Q1) On the BIG-bench shared-input test, character n-gram TF–IDF reached 0.276 average precision, compared with 0.146 for description embeddings on the same pairs and descriptions. We therefore view 0.146 as a modest result for retrieving literal input reuse.
We also asked whether descriptions can identify related task designs when the questions differ. The BBEH authors list which BBH task each of their 23 new tasks was designed to replace [2, Table 1]. After we removed task titles and explicit references to the predecessor, Qwen3 embeddings reached 0.537 AP at 0.6B and 0.485 at 4B, compared with 0.415 for character TF–IDF on the same descriptions. This gives us independent evidence that descriptions can help find related task designs across suites. These tasks can still contribute distinct evidence about model performance at different levels of difficulty.
Thank you also for the suggestion about the appendix. We are shortening it by bringing related analyses together and moving detailed tables to the supplement so the main comparisons are easier to find.
References
[1] Ho et al. (2025). A Rosetta Stone for AI Benchmarks. arXiv:2512.00193, Section 4.4.
[2] Kazemi et al. (2025). BIG-Bench Extra Hard. ACL, Table 1.
[3] Duan et al. (2024). VLMEvalKit: An Open-Source ToolKit for Evaluating Large Multi-Modality Models. ACM Multimedia 2024, pp. 11198–11201. ACM paper · Section 3, OpenVLM scoring.
[4] Chen et al. (2024). Are We on the Right Way for Evaluating Large Vision-Language Models? MMStar release · Item 399.
[5] Lu et al. (2024). MathVista. Testmini item 404.
Thank you for the related-work pointers and the questions. The general response describes the one change we made to the weighting, and below we call the two versions the old and the new weighting.
(W1) The revised Section 2 now discusses the three papers you name, and Appendix A.6 compares BoB with the benchmark signatures of Wu et al. in a table. Uzunoglu et al. study uneven subdomain coverage inside one benchmark, Siska et al. show that correlated test prompts can change rankings, and Wu et al. measure which benchmarks overlap in what they test. These works study how benchmarks relate. BoB studies what to do with such relations when scores are combined and each model reports a different set of benchmarks. In that case one weight per benchmark loses most of its benefit. We show this for our old weighting and for clone-robust weights from [4], which the revision tests directly (Appendix I.8), and the revision computes the weight for each model from the benchmarks it reports. The overlap measure of Wu et al. could replace description similarity under our framework, which we have not tested yet (Appendix A.6).
(W2) The held-out gain is a retention result, as the review says, and equal weights share it. The revised abstract now states the claim as you suggest, that the value of density weighting is robustness to repeated listings and not better ranking in general. The new weighting also removes the reversal at 25% coverage in every copy test (general response). And since complete copies are an easy case for a method built to share weight, the revision adds real cases, which are third-party re-measurements and benchmark families that are listed several times (Appendix I.9).
(Q1, W3) We tested description similarity on BIG-bench with two independent checks. Among subtasks of the same parent task, it finds pairs that share inputs far more often than chance (average precision 0.146 against a base rate of 0.018). Its link to how similarly models score, once overall ability is removed, is at most weak, with an interval that includes zero, and BIG-bench has no shared inputs across parent tasks to test (Appendix I.3). So we cannot show that similar descriptions mean overlapping capability, and the revision no longer relies on it. It says only that the weights control how much entries with almost the same description count together, and it does not claim that low-weight benchmarks are redundant (Section 5). On your two cases, benchmarks that only sound similar share little weight, because the kernel is narrow. In WildScores only 8 pairs of benchmarks have a kernel similarity of at least one half, and the entries that share most weight are versions of one benchmark, such as the six RULER lengths (Appendix E). Benchmarks that use different wording for the same capability are not detected, which is a limit we state. We judge similarity from content on purpose. Guidance on composite indicators says overlap should be judged from what the indicators contain and not only from their correlation [6], and weights that ignore score values cannot be shaped by the scores.
(Q2) Our recommendation has three parts. First, use all eligible benchmarks, because this improves held-out prediction with equal weights and with both weightings. Second, use the new weighting when benchmarks are listed more than once. Its prediction is close to that of equal weights, and in every copy test the ranking moves less than with equal weights, also when few models report the copies. Third, when someone re-measures a benchmark, give the re-measurement the same description as the benchmark, so that the two share weight. In the real cases the direction is the same in most instances, with clear effects for RULER and AIME and small effects for the rest (Appendices I.8 and I.9).
(Q3) This cannot be measured inside WildScores, because the scores that developers did not report are unknown. We therefore applied the real WildScores reporting patterns to BIG-bench, where all scores are known. Agreement with held-out rankings drops for BoB and for equal weights alike (Spearman from 0.230 to 0.084 with the old weighting). When scores are hidden depending on their values, rankings move more than when the same number is hidden at random (5.0 to 6.6 places on average against 3.7, at 25% coverage, Table 54). So rankings are sensitive to selective reporting, and neither weighting corrects for it (Appendices I.6 and I.7).
We hope the revision clarifies the contribution. The claim is now limited to robustness to repeated listings, which the evidence supports.
References
[4] Berriaud, D., Wangsutthitham, S., and Wattenhofer, R. (2026). Breaking the Illusion of Artificial Consensus: Clone-Robust Weighting in Arbitrary Metric Spaces. arXiv 2602.24024.
[6] Nardo, M., Saisana, M., Saltelli, A., Tarantola, S., Hoffmann, A., and Giovannini, E. (2008). Handbook on Constructing Composite Indicators: Methodology and User Guide. OECD Publishing.