acceptodds
Under review as a conference paper at ICLR 2027

Balance of Benchmarks: Evidence Retention and Multiplicity-Aware Model Comparison

Abstract

Benchmark aggregation couples two quantities that should be set separately: how much evidence a comparison uses and how much influence each benchmark receives. Curated suites control influence by selecting benchmarks, potentially excluding informative observations, while equal weighting of a broader pool lets repeated or closely related benchmarks gain influence through sheer numbers. We introduce Balance of Benchmarks (BoB), which retains eligible observations, fits benchmark-specific response curves to place different difficulty ranges on a shared ability scale, and down-weights benchmarks in crowded semantic neighborhoods using description similarity. We evaluate evidence retention and density weighting through separate interventions on WildScores, a sparse collection of 148 developer-reported benchmarks, with 14 Artificial Analysis benchmarks as a controlled reference. Holding fitting models fixed and withholding source-lineage families, expanding one historical ten-benchmark suite to the full eligible pool raises BoB's held-out Spearman correlation by 0.035 over 50 targets (Holm p=0.013). Density weighting reduces sensitivity to complete re-listings: after four copies are added, mean absolute rank displacement falls from 1.95 under equal-weight equating to 0.63. Reporting masks reveal a further distinction: conserving a group's column weight leaves each model's observed weight dependent on its reporting coverage. Together, these results clarify how evidence retention, benchmark multiplicity, and reporting coverage jointly shape model comparison.

Primary area:
datasets and benchmarks
Keywords:
large language model evaluation, benchmark aggregation, score equating, semantic density weighting, sparse reporting

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4ywkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Balance of Benchmarks (BoB) for aggregating sparse, heterogeneous benchmark scores. BoB separates two questions: 1) which evidence to retain and 2) how much influence each benchmark should receive. It combines benchmark-specific score equating with inverse-density weighting in a semantic embedding space, so crowded benchmark neighborhoods receive less total influence.

Critical strengths
  • Good problem formulation: Separating evidence retention from benchmark influence is conceptually useful and clearer than treating benchmark selection and weighting as the same decision.

  • Careful experimental decomposition: The paper separately tests adding more evidence and robustness to benchmark re-listing, and explicitly studies sparse reporting effects.

Critical weaknesses
  1. Related work and positioning must be strengthened.

The related work section misses several closely related efforts on benchmark redundancy and aggregation bias. In particular, Uzunoglu et al. (https://arxiv.org/abs/2509.25671) studies how uneven subdomain representation can distort aggregate scores; Siska et al. (https://aclanthology.org/2024.acl-long.560/) analyze how benchmark distributional assumptions and semantic similarity affect rankings; Wu et al. (https://arxiv.org/abs/2509.23488) studies functional overlap across benchmarks.

These works should be discussed to sharpen BoB's novelty.

  1. The empirical case for semantic density weighting is limited.

The strongest held out prediction result is primarily an evidence retention result rather than a BoB specific weighting result. In the strict panel, BoB improves by +0.035 when moving from the curated to full pool, but equal weight equating improves by about +0.037. Therefore, the direct BoB vs. equal-weight differences are close to zero and statistically inconclusive.

Density weighting clearly improves robustness to complete duplicated columns, but this is a favorable intervention for a method explicitly designed to share weight among duplicates. More importantly, at 25% copy coverage the direction reverses and density weighting performs worse than equal weighting. The paper should sharpen its practical claim, where the evidence supports multiplicity robustness under substantial overlap more strongly than improved model ranking in general.

  1. Description similarity is a weakly validated proxy for benchmark redundancy.

BoB's core weighting rule treats proximity between benchmark description embeddings as a proxy for related benchmark content, while explicitly acknowledging that it does not guarantee shared questions, protocols, or capabilities. This is consequential because the inferred semantic geometry directly determines benchmark influence.

The paper provides encoder and bandwidth sensitivity analyses, but I would like stronger validation that description space proximity corresponds to functional capability overlap that should actually be downweighted. A benchmark pair can sound semantically similar while measuring different skills, or use different language while eliciting the same underlying capability.

Questions for authors

I am overall open to increase my score if the authors address the concerns I highlight in weaknesses. Below are additional questions:

  • Can the authors validate description-embedding similarity against an independent notion of benchmark capability overlap, rather than mainly testing embedding-model sensitivity?

  • What practical recommendation should readers take from density weighting given that it provides essentially no held-out predictive improvement over equal-weight equating and can reverse under low reporting overlap?

  • How sensitive are latent model rankings to selective or outcome-dependent reporting in WildScores?

5 comments

Thank you for the related-work pointers and the questions. The general response describes the one change we made to the weighting, and below we call the two versions the old and the new weighting.

(W1) The revised Section 2 now discusses the three papers you name, and Appendix A.6 compares BoB with the benchmark signatures of Wu et al. in a table. Uzunoglu et al. study uneven subdomain coverage inside one benchmark, Siska et al. show that correlated test prompts can change rankings, and Wu et al. measure which benchmarks overlap in what they test. These works study how benchmarks relate. BoB studies what to do with such relations when scores are combined and each model reports a different set of benchmarks. In that case one weight per benchmark loses most of its benefit. We show this for our old weighting and for clone-robust weights from [4], which the revision tests directly (Appendix I.8), and the revision computes the weight for each model from the benchmarks it reports. The overlap measure of Wu et al. could replace description similarity under our framework, which we have not tested yet (Appendix A.6).

(W2) The held-out gain is a retention result, as the review says, and equal weights share it. The revised abstract now states the claim as you suggest, that the value of density weighting is robustness to repeated listings and not better ranking in general. The new weighting also removes the reversal at 25% coverage in every copy test (general response). And since complete copies are an easy case for a method built to share weight, the revision adds real cases, which are third-party re-measurements and benchmark families that are listed several times (Appendix I.9).

(Q1, W3) We tested description similarity on BIG-bench with two independent checks. Among subtasks of the same parent task, it finds pairs that share inputs far more often than chance (average precision 0.146 against a base rate of 0.018). Its link to how similarly models score, once overall ability is removed, is at most weak, with an interval that includes zero, and BIG-bench has no shared inputs across parent tasks to test (Appendix I.3). So we cannot show that similar descriptions mean overlapping capability, and the revision no longer relies on it. It says only that the weights control how much entries with almost the same description count together, and it does not claim that low-weight benchmarks are redundant (Section 5). On your two cases, benchmarks that only sound similar share little weight, because the kernel is narrow. In WildScores only 8 pairs of benchmarks have a kernel similarity of at least one half, and the entries that share most weight are versions of one benchmark, such as the six RULER lengths (Appendix E). Benchmarks that use different wording for the same capability are not detected, which is a limit we state. We judge similarity from content on purpose. Guidance on composite indicators says overlap should be judged from what the indicators contain and not only from their correlation [6], and weights that ignore score values cannot be shaped by the scores.

(Q2) Our recommendation has three parts. First, use all eligible benchmarks, because this improves held-out prediction with equal weights and with both weightings. Second, use the new weighting when benchmarks are listed more than once. Its prediction is close to that of equal weights, and in every copy test the ranking moves less than with equal weights, also when few models report the copies. Third, when someone re-measures a benchmark, give the re-measurement the same description as the benchmark, so that the two share weight. In the real cases the direction is the same in most instances, with clear effects for RULER and AIME and small effects for the rest (Appendices I.8 and I.9).

(Q3) This cannot be measured inside WildScores, because the scores that developers did not report are unknown. We therefore applied the real WildScores reporting patterns to BIG-bench, where all scores are known. Agreement with held-out rankings drops for BoB and for equal weights alike (Spearman from 0.230 to 0.084 with the old weighting). When scores are hidden depending on their values, rankings move more than when the same number is hidden at random (5.0 to 6.6 places on average against 3.7, at 25% coverage, Table 54). So rankings are sensitive to selective reporting, and neither weighting corrects for it (Appendices I.6 and I.7).

We hope the revision clarifies the contribution. The claim is now limited to robustness to repeated listings, which the evidence supports.

References

[4] Berriaud, D., Wangsutthitham, S., and Wattenhofer, R. (2026). Breaking the Illusion of Artificial Consensus: Clone-Robust Weighting in Arbitrary Metric Spaces. arXiv 2602.24024.

[6] Nardo, M., Saisana, M., Saltelli, A., Tarantola, S., Hoffmann, A., and Giovannini, E. (2008). Handbook on Constructing Composite Indicators: Methodology and User Guide. OECD Publishing.

I appreciate the authors' rebuttal.

I find the responses to W1, Q2, and Q3 satisfactory. Below are my remaining and new questions regarding W2 and W3/Q1:

W2: I am not sure if I am convinced duplicate listing is a realistic problem. If a benchmark suffers from duplicate (or near-duplicate) listings, then it must be a terrible benchmark. I am not sure if today's benchmarks would suffer from this, given that many of them are quite costly to run and don't have a large body of problems (e.g. like MMLU does). Do the authors have a convincing argument why this is a real problem and we should care about their solution?

W3/Q1: I cannot contextualize the precision results. What would be a very strong baseline here, and what would be its precision? Does the precision of 0.146, which description similarity achieves, mean a strong performance? Just by the numerical scale of precision, from 0 to 1, 0.146 looks rather low, and hence weak. I understand it's stronger than the base rate, but it's (0.146 vs. 0.018) not an astonishing overperformance in my understanding.

Also, a humble and unrelated (to above) suggestion I have for the paper is to be thoughtful about the appendix length. There are many sub-subsections. I understand appendix is for sharing relatively unimportant details for those who are curious, but it gets hard for the reader to properly spend their attention when appendix is tens of pages.

I appreciate the time and effort the reviewers put into the rebuttals.

Thank you for the follow-up. As you point out, many benchmarks are expensive to run, so we want to make use of every eligible score. Our goal is to build a broad, balanced picture of model performance while controlling how much related benchmarks count together.

(W2) The same question can contribute to more than one benchmark score in a composite index, even when both benchmarks are carefully designed. OpenVLM’s default overall score averages eight benchmark scores, including MathVista testmini and MMStar [3]. MMStar selects questions from existing datasets to build an evaluation that requires visual input. Its released data identify MathVista as the source of 345 of its 1,500 questions [4]. For example, MathVista testmini item 404 and MMStar item 399 use the same timetable and ask the same question with the same four choices [4, 5]. Both benchmarks provide useful coverage, but that question contributes to two scores in the published average.

The authors of the Epoch Capabilities Index describe a related case. Splitting one benchmark into two entries of similar difficulty can roughly double its weight in their fit, even when the underlying question-level data stay the same [1, Section 4.4].

We tested what happens when a developer and a third party both measure the same benchmark. Both scores can tell us something useful, but giving each score an equally weighted entry increases the benchmark’s share of the index. BoB computes weights from the descriptions of the benchmarks each model reports. When two scores are known to measure the same benchmark, our extension assigns weight to that benchmark once per model and divides it among the available scores. Both scores enter the fit, which reduces the need to remove related entries by hand to limit their combined weight. Keeping the full eligible pool improved held-out prediction with both BoB and equal-weight equating.

We tested the extension on 824 Artificial Analysis scores for benchmarks that the same models already had developer scores for. With the same added scores, the remeasured benchmarks’ average share of model weight rose by 7.88 percentage points with equal weights per entry and 0.35 points with current BoB using a shared description. With the new rule, the benchmarks kept their original weight share. Held-out correlations were close to those of current BoB in both panels.

(W3/Q1) On the BIG-bench shared-input test, character n-gram TF–IDF reached 0.276 average precision, compared with 0.146 for description embeddings on the same pairs and descriptions. We therefore view 0.146 as a modest result for retrieving literal input reuse.

We also asked whether descriptions can identify related task designs when the questions differ. The BBEH authors list which BBH task each of their 23 new tasks was designed to replace [2, Table 1]. After we removed task titles and explicit references to the predecessor, Qwen3 embeddings reached 0.537 AP at 0.6B and 0.485 at 4B, compared with 0.415 for character TF–IDF on the same descriptions. This gives us independent evidence that descriptions can help find related task designs across suites. These tasks can still contribute distinct evidence about model performance at different levels of difficulty.

Thank you also for the suggestion about the appendix. We are shortening it by bringing related analyses together and moving detailed tables to the supplement so the main comparisons are easier to find.

References

[1] Ho et al. (2025). A Rosetta Stone for AI Benchmarks. arXiv:2512.00193, Section 4.4.

[2] Kazemi et al. (2025). BIG-Bench Extra Hard. ACL, Table 1.

[3] Duan et al. (2024). VLMEvalKit: An Open-Source ToolKit for Evaluating Large Multi-Modality Models. ACM Multimedia 2024, pp. 11198–11201. ACM paper · Section 3, OpenVLM scoring.

[4] Chen et al. (2024). Are We on the Right Way for Evaluating Large Vision-Language Models? MMStar release · Item 399.

[5] Lu et al. (2024). MathVista. Testmini item 404.

Reviewer_pnteOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary
  1. The paper studies two decisions in benchmark aggregation: which observations to retain and how much weight each benchmark receives. BoB combines benchmark-specific response curves with inverse-density weights computed from benchmark descriptions.
  2. Experiments use 148 WildScores benchmarks and a 14-benchmark Artificial Analysis reference. Expanding the evidence pool improves held-out ranking under both BoB and equal-weight equating. Density weighting reduces sensitivity to complete duplicate listings. Further analyses show why this benefit can weaken or reverse under partial reporting.
Critical strengths
  1. The experiments separate evidence retention from duplicate sensitivity. Curated and full-pool fits use the same model rows, while duplicate-listing tests add no new measurements. This makes the two effects easier to interpret. Table 3 also makes clear that retention gains are not unique to density weighting.
  2. The reporting-mask analysis provides a useful explanation of failure cases. Preserving a benchmark group’s total weight does not preserve its weight for every model. Section 4.2 derives this effect and tests it under partial reporting. The reversal also appears with an affine estimator, which strengthens the mechanism-level finding.
Critical weaknesses
  1. The retention result has limited validation under more comparable evaluation protocols. Removing 38 columns with unresolved protocol differences leaves a positive effect, but it is inconclusive after correction (Table 16). The strict panel retains only three targets. The study also tests one curated suite, and the main analysis followed inspection of earlier results. These limits weaken confidence that the gain transfers to other benchmark pools.
  2. The practical advantage of semantic weighting remains narrow. It provides no statistically resolved predictive gain over equal-weight equating (Table 11). Its clearest benefit comes from controlled re-listings. At 25% copy coverage, it increases mean rank displacement from 0.684 to 0.761 (Table 4). Since low reporting overlap also occurs among real semantic neighbors, the experiments do not yet establish when this weighting should be preferred in practice.
Questions for authors
  1. Was any corpus or benchmark suite kept untouched during method and analysis development? If so, does it reproduce the retention gain under a frozen protocol?
  2. Can the existing results identify observable reporting-overlap conditions under which density weighting reliably helps? The current tests show both benefits and reversals, but leave the practical choice unclear.
3 comments

Thank you for the questions on independent validation and on when density weighting helps. The general response describes the one change we made to the weighting, and below we call the two versions the old and the new weighting.

(Q1) We did not set aside a corpus during development. For the revision we used a BIG-bench score matrix that we had not used before, and we fixed the analysis before running the comparisons. All its scores follow one protocol (zero-shot, temperature zero). Using all tasks predicts held-out tasks better than a selected set, and the gain is clear after correction (0.0825 for the old weighting, 95% interval from 0.058 to 0.107, Holm-adjusted p=0.0003, Appendix I.2). Equal weights show the same gain (0.079), and the estimates are positive in all nine suite settings of Table 52, over 131 held-out targets. This speaks to three limits you list for the WildScores result, which are mixed protocols, few targets and a single suite. BIG-bench shares task origins with BBH and BBEH in WildScores, which the paper states (Section 4.3 and Appendix I.1).

(Q2) For the old weighting, the condition is how many of the same models report the related benchmarks. When at least half of them do, and each model has enough observed weight, the ranking moves less on average than with equal weights when copies are added (0.056 against 0.221 on BIG-bench, with some exceptions, Appendix I.5). WildScores does not meet this condition. The new weighting follows from the mechanism you point to in Section 4.2. Since keeping a group's total weight does not keep it for every model, we compute the weight for each model from the benchmarks it reports. In our copy tests this removes the need for the condition. The ranking moves less than with equal weights in all 88 tests, including the 25% coverage case cited in the review (0.151, against 0.684 for equal weights and 0.761 for the old weighting). Its prediction is close to that of equal weights, and on the WildScores strict panel the 95% interval of the difference includes zero. So the practical choice is the new weighting whenever benchmarks are listed more than once, and equal weights lose nothing in prediction but let repeated listings count more. Real cases point the same way in most instances (Appendices I.8 and I.9).

We hope this addresses both concerns. The retention result appears again on data we had not used, and the new weighting keeps its benefit when few models report the related benchmarks, in the tests we ran.

Thank you for the detailed response. The response has alleviated my main concerns.

However, I remain cautious about the broader applicability of these results. The evidence supports retaining more reported scores within the tested model populations and reducing ranking changes caused by repeated listings. It does not yet establish whether the method also improves fine-grained comparisons among frontier models. For these models, most benchmarks may be near saturation, and small score differences may be affected by measurement noise. I consider reliable differentiation among frontier models an important and pressing priority for evaluation. Still, I understand that this is not the paper’s direct objective, and I do not regard it as a requirement for acceptance.

Considering the authors’ response and the additional evidence, I am willing to raise my rating to Weak Accept.

Thank you for raising the question of whether aggregation helps distinguish frontier models when their scores are close.

Combining scores can reduce dependence on any one measurement. When many scores are near the ceiling, reliable comparisons also depend on having benchmarks that can distinguish the leading models.

In Artificial Analysis, retaining more scores modestly improved agreement with held-out rankings for closely ranked leading models. We used the same 209 pair–benchmark comparisons, selecting pairs within three rank positions in the top half of 19 recent checkpoints using the other benchmarks. Expanding the fitting pool from four benchmarks to eight raised agreement from 68.6% to 71.1% with BoB and to 70.6% with equal-weight equating. The median held-out gap was 6.5 percentage points.

Across six splits of the saved MathArena responses, the second half preserved the order in 24 of 32 comparisons and reversed it in eight for pairs whose HMMT, AIME, and BRUMO averages were both at least 95% and 0.56 percentage points apart in the first half, so gaps this small still need uncertainty estimates.

We also tested how weighting affects the estimate of a single benchmark score. For 21 model configurations matched by checkpoint and reasoning mode, we estimated HMMT using one saved response per question, the nearby scores from AIME and BRUMO, and the more distant scores from IFBench and AA-LCR. We calibrated the auxiliary scores using other model groups and compared each estimate with the mean accuracy of the three remaining HMMT responses, rotating which response served as input.

Weighting ruleHMMTAIME + BRUMOIFBench + AA-LCRMean squared error (pp²)
Equal weights20.0%40.0%40.0%78.20
Fixed HMMT weight, inverse density for other scores47.3%23.9%28.8%50.65
BoB47.3%46.7%6.0%32.93

Equal weighting gave more stable estimates across input choices (mean within-model variance, 6.83 versus 12.09 pp²). BoB gave lower error against the held-out HMMT scores. With HMMT fixed at 47.3% and the same inverse-density factors, applying relevance weights to the other four scores lowered the error from 50.65 to 32.93, with a paired 95% interval for the reduction that includes zero (−4.02 to 70.54 pp²).

We hope these results clarify what aggregation can contribute to comparisons among closely ranked frontier models.

Reviewer_s42uOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Balance of Benchmarks (BoB), a method for aggregating sparse benchmark scores into a shared model-ability estimate. It combines benchmark-specific response curves with semantic inverse-density weighting to retain available evidence while reducing the influence of closely related benchmarks. Validation uses WildScores, containing 354 model configurations and 148 benchmarks, with Artificial Analysis as a denser reference. Controlled experiments separately examine expanding the benchmark pool and duplicating existing results. Additional evidence improves held-out prediction, while density weighting reduces sensitivity to complete duplication. Further analyses identify limitations arising from reporting masks, initialization, and protocol differences.

Critical strengths

The experiments clearly separate the value of adding benchmarks from the effect of duplicating them. This makes the results easier to interpret and strengthens the empirical contribution.

The paper explains how missing scores and initialization affect rankings. These insights are useful for researchers building benchmark aggregation methods.

The study addresses a practical problem using real, sparse benchmark reports. This makes its findings relevant to how researchers actually construct model comparisons.

Critical weaknesses

The predictive benefit of BoB over simpler alternatives remains unclear. BoB and equal-weight equating achieve nearly identical held-out correlations, and both benefit similarly from expanding the benchmark pool. This makes it difficult to determine what semantic weighting adds beyond improved stability under duplication.

The stability benefit does not hold consistently under sparse reporting. At 25% copy coverage, BoB changes rankings more than equal weighting. Since WildScores has only 10.3% observed cells and many related benchmarks have low reporting overlap, this raises concerns about how well the stability gains translate to realistic reporting conditions.

Protocol inconsistencies make the retention gain harder to interpret. After conflicting columns are removed, the gain becomes smaller and is inconclusive after correction. The restricted analysis also changes the evaluation population, making it difficult to isolate the effect of protocol differences.

Similar descriptions do not necessarily imply redundant evidence. Related benchmarks may contain different questions or provide independent measurements, so semantic density weighting could reduce the influence of informative tests. The current experiments do not fully establish whether description similarity captures actual redundancy in the benchmark pool.

Questions for authors

BoB and equal-weight equating achieve nearly identical predictive performance in Table 3. Are there settings where density weighting consistently improves held-out prediction? What practical advantage does BoB offer over simpler alternatives beyond stability under benchmark duplication?

Across the 25%-coverage scenarios, how often does BoB perform worse than equal weighting?

Is the deterioration concentrated in particular benchmarks or models with low reporting coverage?

After removing columns with unresolved protocol differences, the retention gain becomes smaller and is inconclusive after correction. How much of the original improvement comes from these columns?

Can the effect be evaluated while keeping the model population and target benchmarks fixed?

How closely does description similarity agree with actual benchmark overlap, such as shared questions or source datasets?

Are heavily downweighted benchmarks generally redundant, or do some provide independent evidence?

1 comment

Thank you for the questions. Short answers follow. The general response describes the one change we made to the weighting, and below we call the two versions the old and the new weighting.

(Q1) We found no setting where any weighting predicts held-out benchmarks consistently better than equal weights. This is expected when many related measures are averaged [5]. What density weighting offers is that a benchmark listed many times gains less extra weight. With the new weighting this holds in every copy test we ran, also when few models report the copies. Beyond this, we found no further advantage of the weighting itself. The other gains in the paper come from using all reported benchmarks, and they also hold with equal weights (Appendices I.8 and I.9).

(Q2) The old weighting was worse than equal weights in 11 of the 24 cases at 25% coverage. The new weighting is better than equal weights in all 24 (mean rank displacement 0.151 against 0.684, Appendix I.8). These tests run on the WildScores matrix itself, with its sparse reporting, and the real cases in the general response point the same way in most instances.

(Q3) The old weighting was worse in all AIME and GPQA cases and in some SWE-bench, BFCL and MMMU cases. The effect was larger for models that report few benchmarks. This fits the cause we identified. A model that reports only one of several related benchmarks gets too little weight for it when the weight is shared by all models (Appendices E.2 and I.4).

(Q4) When we remove the columns with protocol differences, both the selected set and the full set predict better, and the gap between them shrinks from 0.0508 to 0.0241. This shows that the result depends on which columns are included. It does not tell us how much of the original gain was caused by protocol differences (Section 4.1).

(Q5) Yes. The matched comparison keeps the same models and the same target benchmarks (66 targets). In that comparison the gain of 0.0241 is not statistically clear after correction for multiple comparisons (Holm-adjusted p=0.6093, Section 4.1).

(Q6) We could test this on BIG-bench, where shared inputs are documented. Among subtasks of the same parent task, description similarity finds pairs that share inputs far more often than chance (average precision 0.146 against a base rate of 0.018). BIG-bench has no shared inputs across parent tasks, and WildScores does not record shared questions, so we could not test those cases. Description similarity shows no clear link to how similarly models score once overall ability is removed, with an interval that includes zero (Appendix I.3).

(Q7) No, they are not redundant in general. The benchmarks with the lowest weights are the six RULER context lengths. They use one task design but measure different things, and BoB keeps all six. With equal weights RULER counts as 6 benchmarks, and with the old weighting as about 2.6. This makes the ranking depend less on how many lengths happen to be listed. If only one RULER length were listed, the models that report several lengths would move 2.20 places on average with equal weights, 0.75 with the old weighting and 0.97 with the new one (Appendix I.9). On the other hand, Table 28 shows that adding more RULER lengths helps prediction more with equal weights on average. We do not claim that low-weight benchmarks are redundant (Section 5 and Appendix E.4).

We hope these answers are clear. BoB uses all the reported results and limits how much a benchmark gains from being listed many times.

References

[5] Wilks, S. S. (1938). Weighting systems for linear functions of correlated variables when there is no dependent variable. Psychometrika, 3, 23–40.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their careful reviews.

We noticed that all three reviews ask what BoB offers when its held-out prediction is close to that of equal-weight equating. The weighting in BoB is there to control how much each benchmark counts, and prediction is not where its effect shows. This matters because of how results are reported. Model developers report different benchmarks, and the same benchmark often appears several times, in different versions, settings or lengths. In WildScores, benchmark families with more than one entry cover more than half of the columns (79 of 148). With equal weights, a benchmark that is listed six times counts six times. This problem has been pointed out for an index built from reported scores [1], and existing fixes work on complete score tables, where every model has a score on every benchmark [2, 3]. BoB keeps all the reported scores and gives repeated benchmarks less extra weight.

The revision changes one step of the method. In the submission, each benchmark had one weight, shared by all models. The submission already reported that this breaks down when only a few models report the repeated entries (Section 4.2). The revision shows why. One weight per benchmark cannot be right both for a model that reports all the repeats and for a model that reports only one of them (Appendix E.2). So the revision computes the weight separately for each model, using only the benchmarks that model reports. The weights still come from benchmark descriptions and use no score values. Nothing else in the method changes, and all submitted results, including this breakdown, remain in the paper (Section 3.2 and Appendix I.8). The revised paper calls the old and new versions the column form and the reporting-aware form.

To see what this change does, we add copies of a benchmark and measure how many places models move in the ranking. Copies add no new information, so less movement is better.

Models that also report the copiesEqual weightsOld weightingClone-robust weights [4]New weighting
All1.950.630.140.27
75%1.610.660.330.24
50%1.200.710.500.20
25%0.680.760.650.15

The old weighting and the clone-robust weights [4] both lose most of their benefit when few models report the copies, and the old weighting becomes worse than equal weights in the last row. The new weighting moves the ranking less than equal weights in all 88 copy tests we ran (Appendix I.8). In a real case, a third party re-measured thirteen WildScores benchmarks, and each re-measurement was given the same description as the benchmark it measures. Adding them raises the share of the total weight held by these benchmarks from 33.8% to 44.1% with equal weights, and from 33.7% to 35.9% with the new weighting. The old weighting behaves similarly here (Appendix I.9).

This stability comes at no clear cost in accuracy. Held-out prediction depends on which scores are used and hardly on how they are weighted, which is a known property of weighted averages of related measures [5]. On the WildScores strict panel, the difference between the new weighting and equal weights has a 95% interval from -0.0045 to 0.0029, which includes zero. In the other comparisons the differences are small and go in both directions (Appendix I.9).

The paper makes two further contributions. First, using all reported benchmarks predicts held-out benchmarks better than using a small selected set, with equal weights and with both weightings. The gain appears again on a BIG-bench score matrix that we had not used before (Spearman gains of 0.035 on WildScores and 0.082 on BIG-bench with the old weighting). Second, WildScores provides the scores, their sources and which model reports which benchmark, for 148 benchmarks and 354 model configurations.

The method has clear limits, and we see a way forward. The weighting only reduces weight for entries whose descriptions are almost the same, such as versions, settings and re-measurements of one benchmark. It does not balance broad areas such as coding against math. One description gives each benchmark a single position. A natural next step is to describe a benchmark by all of its questions, so that two benchmarks share weight to the extent that their questions overlap. This needs the text and an embedding of every question, which many benchmarks do not release and which is costly at this scale, and a weighting rule that stays comparable between benchmarks with few and with many questions. We leave this to future work. No weighting we tested predicts consistently better than equal weights. The new weighting was developed during this discussion and is tested on exact copies and two real cases.

We hope this makes clear what BoB offers. It is a score that uses all the reported results and limits how much a benchmark gains from being listed many times.

References

[1] Ho, A., Denain, J.-S., Atanasov, D., Albanie, S., and Shah, R. (2025). A Rosetta Stone for AI Benchmarks. arXiv 2512.00193.

[2] Balduzzi, D., Tuyls, K., Perolat, J., and Graepel, T. (2018). Re-evaluating Evaluation. NeurIPS.

[3] Liu, S., Gemp, I., Marris, L., Piliouras, G., Heess, N., and Lanctot, M. (2025). Re-evaluating Open-ended Evaluation of Large Language Models. ICLR.

[4] Berriaud, D., Wangsutthitham, S., and Wattenhofer, R. (2026). Breaking the Illusion of Artificial Consensus: Clone-Robust Weighting in Arbitrary Metric Spaces. arXiv 2602.24024.

[5] Wilks, S. S. (1938). Weighting systems for linear functions of correlated variables when there is no dependent variable. Psychometrika, 3, 23–40.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.