acceptodds
Withdrawn from ICLR 2027

Benchmark Visibility from Annotations Alone: An Auditable Lower Bound on Blind Exploitability

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Whether a benchmark can see the capability it is cited for is a property of the corpus, and the field establishes it by running systems that ought to fail: late, expensive, and one corpus at a time. We show that much of the same information is computable from the annotations alone: no model, no judge, no frames, which matters because every corpus here keeps its media behind a signed agreement and its annotations public. The instrument is two counts and a curve. Announcement counts items whose gold answer is stated in the question; the blind bound is the best accuracy of an enumerated family of strategies that see the question's form and the answer distribution and nothing else. It is one-sided (it condemns and can never acquit, for a reason no instrument of this kind escapes), and announcement is a detector of the same failure rather than independent evidence, so the two counts must not be added, yet they flag nearly disjoint items. The curve is what the field is missing: the same estimator returns a positive value when the template carries no information at all, a null-induced baseline fixed by the group-size profile alone, and it collapses as the template isolates items; we construct a corpus on which it rises first instead, and which shape a given corpus takes is measured rather than deduced. Either way the same corpus reads very differently under two choices papers rarely state. Three common estimators mistake a corpus's answers for its structure by one mechanism: in-sample scoring and the ordinary nonparametric bootstrap leak the held-out answer back into its own evidence, and leave-one-out inside a group of two fits the single other item. The bootstrap is the dangerous one, because it inflates the bound substantially while looking like rigour, and deduplicating its own draws returns it almost exactly to the point estimate; a fit-and-test split is the repair. We validate the instrument against a documented defect on two corpus versions identical outside the repair, against a third-party repair its own authors performed, and behaviourally on an open-answer corpus, where the separation survives an intervention that moves the surface feature the mask is built from while leaving the question's content alone. Across the corpora we survey, 3 of the 5 option-regime corpora carry a statistically supported blind-exploitability signal under a family-wise max-statistic permutation test that prices the rule family's own selection into the null.

Primary area:
datasets and benchmarks
Keywords:
benchmark diagnosis, blind baselines, shortcut learning, dataset artefacts, annotation-only audit, permutation null, leakage, bootstrap, multimodal evaluation, pre-registration

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_8DWnOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper claims to devise a method evaluating the soundness of a benchmark without having to train models to solve it.

Critical strengths

None

Critical weaknesses

Presentation:

  • The paper is poorly written and hard to follow.
  • A lot of concepts are mentioned without prior definition (even informal). For example I had to read the code to find the definition of what a template is.
  • The abstract contains too much uninformative jargon does not inform the reader on the problem solved, the method used or the main experimental results.
  • In addition to not providing any insight on the methods used in the paper, propositions are basic results of maximums over sets and subsets.
  • The experiments are poorly described: the datasets are not presented, the method is not described, the results are large tables whose columns are not presented, there is no clear experimental takeaway.

Contributions:

  • The fact that some benchmarks can be solved without solving the task they are supposed to evaluate has been known for quite a while, in different domains (privacy leakage, robotics, question answering...). This paper does not provide additional insights on that phenomenon.
Questions for authors

None.

Reviewer_8Ff9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an annotation-only framework for auditing dataset quality and identifying potential shortcuts in multimodal benchmarks, without requiring access to the underlying visual data or running models. The proposed instrument is CPU-only and consists of three main components: (1) an announcement metric that identifies when gold answers are exposed in questions, (2) a blind-exploitability lower bound that measures how well predefined strategies can predict answers without visual inputs, and (3) a template-granularity curve that characterizes how the estimated bound changes under different question groupings.

The authors also introduce theoretical definitions and propositions characterizing the one-sided nature of the bound, the relationship between exposure detectors, and potential leakage or overfitting in different scoring procedures. Through experiments across eight benchmarks (nine configurations), including known-defect controls and additional datasets evaluated after the methodology was fixed, the authors demonstrate how annotation-level statistics can identify dataset artifacts and how estimator and granularity choices can substantially affect the reported results.

Critical strengths
  1. Important and relevant problem. The authors address the problem of understanding benchmark quality and identifying potential dataset shortcuts, which is particularly relevant as multimodal benchmarks become increasingly important for measuring model capabilities. The corpus-side perspective provides a complementary approach to model-based benchmark evaluations.
  2. Lightweight and interpretable auditing framework. The proposed instrument is computationally inexpensive, requires only publicly available annotations, and does not depend on access to the underlying media or model inference. This makes it potentially useful for benchmark developers to identify certain dataset defects before release.
Critical weaknesses
  1. Difficult presentation and lack of accessible experimental context. The paper is challenging to follow, particularly given the number of definitions, propositions, and experimental metrics introduced. The evaluated benchmarks and their underlying tasks are not sufficiently introduced, making it difficult to understand the specific settings, potential defects, and motivations for analyzing each dataset without consulting external resources.
  2. Insufficient justification of experimental design choices. The rationale for selecting the evaluated benchmarks is not clearly motivated. In particular, it is unclear why the initial datasets and subsequently added datasets were chosen, and whether they represent a sufficiently diverse range of benchmark structures and shortcut mechanisms. Similarly, the practical motivations for the selected strategy families and template groupings could be better explained.
  3. Limited interpretation of the empirical findings and generalizability. While the paper reports a large number of quantitative results, it provides relatively limited qualitative analysis of what the identified artifacts imply for each benchmark. For example, the authors could provide concrete examples of flagged questions, explain the types of shortcuts being detected, and discuss how benchmark developers should interpret or act on the resulting measurements. Also, it remains unclear how consistently the identified shortcuts correspond to blind performance in stronger models, or how readily the framework can be extended to other task settings and modalities beyond those evaluated.
Questions for authors
  1. Dataset selection and practical implications: What motivated the selection of the two initially studied configurations and seven subsequently added configurations? Could the authors provide more qualitative or corpus-level analysis explaining the specific defects identified, their implications for benchmark quality, and potential corrective actions?
  2. Behavioral validation and model dependence: How does the choice of model backbone affect the IN/OUT accuracy differences reported in Table 5? Given the mixed results on EgoSchema, to what extent would the authors expect these findings to generalize to stronger frontier models, and what evidence supports that expectation?
  3. Transferability across tasks and modalities: How broadly can the proposed auditing framework be applied beyond the video, visual, and embodied question-answering benchmarks considered here? Which components would transfer directly to other modalities or benchmark formats, and which would require new strategies or grouping definitions?
  4. Interpretation of the reported metrics: How should benchmark developers interpret the magnitude of the blind-exploitability bound and its excess over the permutation null? What practical thresholds, qualitative examples, or follow-up analyses would the authors recommend for determining whether a detected artifact warrants modifying a benchmark?
Reviewer_d9jTOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a measurement to assess whether a benchmark inherently leaks information about its own solution. The authors argue that existing methods have severe drawbacks, while their method their method computes these insights directly from public annotations of the corpus itself.

Critical strengths
  • The authors identify that information leakage from a dataset can be assessed with a simple instrument of to two counts and a curve.
Critical weaknesses
  • Obscure and dense writing style. I found the writing style to be excessively dense, unintuitive and difficult to parse to the point, where it hindered me from understanding the main contribution of the paper. The paper assumes a high level of prior knowledge from the reader. It does not provide sufficient intuitive explanations or analogies to help readers understand core concepts. Additionally, the paper is riddled with unnecessary, almost poetic rhetorical phrases like "None of them is a lie. None of them is a measurement", "not a limitation we accepted but a property we needed", "This is the paper’s sharpest practical claim, and it is partly a theorem and partly a measurement", "The number is not a number until its granularity is stated", or "it condemns and can never acquit". Concrete scientific descriptions are either abstract "The instrument is two counts and a curve" or highly technical and compressed "refining a template can only shrink groups, the leave-one-out estimate is positive under the permutation null before any structure is present, and in the limit each group holds one item and the estimate collapses to zero". I suggest the authors fundamentally revisit the paper's presentation with a focus on making the submission more accessible. I think the paper should first build an intuition of the setting and the problem (e.g., with concrete examples), before then moving to more condensed technical details and terminology.

  • Reference to anonymous publications. The authors cite seven of their own papers by denoting them with "Anonymous authors". This prevents me from consulting these papers without breaching double blindness.

Questions for authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.