acceptodds
Withdrawn from ICLR 2027

What Does Benchmark Capability Coverage Establish? An Audit of 16,217 Benchmarks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

New benchmarks are justified by gaps: no existing benchmark, it is claimed, tests some combination of capabilities. What does a coverage map of the benchmark ecosystem establish? We find that its conclusions depend on choices that maps leave implicit: how finely capabilities are distinguished, how many benchmarks must test a combination, what a benchmark's labels assert together, and which reference model defines “unusual”. We make these choices explicit in a coverage-audit instrument whose logic is checked in Lean 4: given benchmark assignments, a vocabulary resolution and a reference model, it reports supported combinations, relative deficits and their witnessing benchmarks. We instantiate it as an atlas of 16,217 benchmarks labelled with 754 capabilities and measure the labels against two human raters. Coverage is concentrated beyond what benchmark breadth and capability popularity explain: 564 benchmarks witness every covered coarse capability pair, against 812 under a margin-preserving null, and Papers-with-Code's own task tags show the same pattern. Which associations and modality restrictions look unusual changes with the reference model. The atlas yields an exploratory queue of deficit candidates with their witnesses. For one under-supported capability triple it found, we show that an executable benchmark can be built and diagnosed with demand-removal controls. Apparent gaps are audit targets, not certified absences. We release the vocabulary, labels, Lean development and pipeline.

Primary area:
datasets and benchmarks
Keywords:
evaluation of large language models, benchmark coverage, LLM-as-judge, measurement validity, formal verification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VMcHOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper claims to measure how much information a new benchmark adds vs. all (or a large number of known) existing benchmarks.

Critical strengths

I like the core idea of the paper, if it were executed well.

Critical weaknesses

This paper is in my opinion unreadable. I strongly suspect the writing was heavily AI-generated. Even if not, the writing is not clear and not up to the standards of science, let alone a top-tier AI conference. It is simply not written in a way where one can fully understand what is going on. There are frequent references, even as early as the abstract and first few sentences, to concepts that have not been introduced or explained. My suspicion is that is because the AI is referring to topics in its context, either in its own reasoning traces or things it has discussed with the authors. The AI use section admits heavy reliance on AI for much of (most of?) the process that led to this paper, and it shows.

A sample size of two human raters is not a large enough to tell signal from noise.

Questions for authors

How much did you use AI to write the paper? Did it do the initial drafts, or did you? Did you let it rewrite sentences? Paragraphs? Whole sections? If this represents your writing, please work on the skill of more clearly describing what you have done and not making reference to lots of not-yet-explained concepts.

How much did you use AI for the rest of the science? Have you checked all the code that it wrote? Are you an expert in the statistical methods such that you know if they were properly done? Etc?

Reviewer_qeZ8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I am really struggling to understand the research question(s) and motivation(s) posed by this paper given how hard it is for me to read. It looks like AI generated with little human assistance. On a high level, the paper examines coverage of benchmarks and tries to quantify coverage. It argues that coverage depends on the granularity of the tasks measuring capability within a given domain, the number of supporting benchmarks, co-occurring labels and others. It is really hard to understand the solution or the results.

Critical strengths
  • Tries to measure how much coverage do benchmarks have in terms of capabilities and if there are gaps but the research questions the paper addresses as well as the solution is not easy to parse.
Critical weaknesses
  • Please clarify the research questions and claims in a human understandable way and try to give context as much as possible about why your approach and coverage-audit framework is strong/offers some useful insight.
Questions for authors
  • The paper needs a lot more human assistance and iterations.
Reviewer_sKKfOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines what benchmark–capability coverage maps can legitimately establish. It identifies four consequential analytical choices: the granularity of the capability taxonomy, the minimum number of benchmark witnesses required for support, the interpretation of co-assigned capability labels as jointly exercised demands, and the reference model used to identify unusual associations. The authors formalize coverage claims and their limits (with conditional properties checked in Lean 4), and construct a documentation-derived atlas of 16,217 benchmark records and 754 capability categories. At an evidence bar of five, the proportion of pairs co-assigned at least once that count as supported is 65.6% for coarse categories and 35.9% for fine categories. A greedy cover of all supported coarse pairs needs 564 benchmark records, versus 812 on average under a fixed-margin randomization. The paper further analyzes reference-model dependence, releases 115 exploratory relative-deficit candidates, and constructs one executable benchmark from an under-supported three-capability combination. I see its principal contribution as a framework for auditing claims about documented coverage, rather than an evaluation of model performance or individual benchmark quality.

Critical strengths
  1. Well-framed measurement question. Claims that a new benchmark fills a gap are common but often under-specified. Explicitly separating resolution, bar, joint interpretation, and reference model provides a valuable reporting framework (Sections 1–2).
  2. Scale with meaningful checks. The analysis covers 16,217 records; reports human agreement and labeling uncertainty; uses fixed-margin null models and sensitivity checks; and provides witnesses and near-threshold cases. The concentration result has a separate corroborating analysis with Papers-with-Code task tags, although the two rely on overlapping documentation (Sections 2–4).
  3. Honest scope and reproducibility. The authors acknowledge that the labels concern intended demands, not validated item-level measurement; a coverage gap is an audit target, not proof of absence. They release labels, vocabulary, formal development, and pipeline, and discuss unsuccessful downstream tests (Section 6; Appendix J).
Critical weaknesses
  1. Unweighted coverage need not indicate adequate evaluation. A record receives one unit of support when assigned the target capabilities, irrespective of annotation correctness, scoring reliability, diversity, sample size, discriminative power, or overlap with other records. The paper already acknowledges that intended-demand counts and evaluation adequacy are different; the remaining concern is whether the headline concentration and practical deficit recommendations are robust to reasonable quality criteria or collapsing near-duplicate benchmark families (Sections 2–3; Appendix F). Size alone would not be an appropriate quality proxy.
  2. Uncertainty in individual pair assignments and joint demands. The closed-label leave-one-benchmark-out F1 is 0.692 against the intersection of two author-raters' gold, with human–human κ = 0.640. The paper additionally reports 0.597 raw F1 on a 75-benchmark subset not used in design choices; because the label representations differ, these scores are not directly interchangeable. Pair coverage can compound errors in individual labels. A useful 40-pair spot check was judged by one author, blind to pair class but not project membership, using LLM-prepared evidence sheets. It assessed threshold-defined holes, not the 115 relative-deficit candidates (Table 1; Sections 2.3–4; Section 6).
  3. Taxonomy and documentation effects. Many capability cards are hypothesized, including numerous fine-grained categories, so measured gaps partly depend on the ontology. The coarse/fine contrast persists after removing speculative cards, which partly addresses this. Nevertheless, sparse benchmark documentation, correlated labeller errors, and nested/duplicated records can alter witness counts and inferred co-assignment. More explicit sensitivity tests would help bound these effects (Sections 2 and 6; Appendices C–F).
  4. Utility for selecting new evaluations remains unestablished. The new executable benchmark demonstrates feasibility, but its controls do not isolate a three-way interaction, and it is only one manually selected case. The authors report a negative retrospective result versus a simpler pair-gap baseline and propose prospective evaluation as future work (Section 5; Appendix J).
Questions for authors
  1. Benchmark quality and duplication: Although the paper acknowledges that coverage does not imply adequacy, could the authors comment on how benchmark quality and potential duplication might affect the main findings? A small sensitivity analysis, if feasible, would strengthen the results.

  2. Validity of identified deficits: How confident are the authors that the identified deficits reflect genuinely under-evaluated capability combinations rather than labeling errors or incomplete documentation? A few additional item-level checks of the top candidates could be helpful.

  3. Practical usefulness: How much additional value does the proposed framework offer over simpler approaches to identifying benchmark gaps? A brief discussion of how this could be evaluated would help clarify its practical contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.