acceptodds
Withdrawn from ICLR 2027

When Does a Benchmark Score License a Capability Claim? An Executable Three-State Identification Certificate

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A benchmark score is routinely read as evidence that a model has a capability, yet the same score can be produced by a non-capability side channel: acquisition metadata, option form, question templates. We ask when a score licenses its capability claim, and turn the question into an executable procedure. Given a claim, a frozen model score S, and a pre-declared nuisance class T , our certificate returns FAIL when a witness inflates the score without adding intended evidence, SURVIVE when no admissible witness moves it beyond a pre-registered equivalence margin, and INCONCLUSIVE when the data cannot separate the two at that margin, reported together with the inflation the data can resolve. A SURVIVE is earned by a two-one-sided-test bound calibrated against injected channels of known strength, so “no witness found” can never be mistaken for “benchmark is valid”. A companion identity delimits scoring-side repair: the maximum effective sample size of any channel-neutralizing reweighting equals a mass-weighted overlap efficiency, so localized channels are cheap to repair, while the corpus-wide near-deterministic channels our failures exhibit can only be disclosed. Applied to seven audit units, six of them the benchmarks’ own published headline claims, the verdict changes conclusions in both directions: two claims fail, two are positively preserved, and three are returned INCONCLUSIVE and then separated by a power analysis rather than left open. The certificate is a measurement instrument, not a benchmark-condemning machine.

Primary area:
datasets and benchmarks
Keywords:
benchmark validity, capability claims, equivalence testing, construct validity, evaluation methodology, shortcut learning, statistical certification

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_x7f1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper seems to propose a three-state certificate for a model–benchmark pair and a declared nuisance class. It compares a predictor using the model’s frozen score with an augmented predictor using that score and nuisance information. The resulting performance difference, Δinf\Delta_{\mathrm{inf}}, is evaluated against a declared margin ε\varepsilon, producing FAIL, SURVIVE, or INCONCLUSIVE.

Critical strengths

None.

Critical weaknesses

The paper's prose is written in a way that feels to be significantly LLM generated with very minimal human supervision. The writing is full of colons and uses jargon unnecessarily to the point where it is genuinely uncomfortable to read or understand. For instance, see the below.

The certificate is a measurement instrument, not a benchmark-condemning machine.

Why the green cards are green. The surviving units are not lucky: their resistance is mechanistic, which is what a SURVIVE should ultimately trace to.

The instrument never false-FAILs the clean benchmark, so a SURVIVE is not a trigger-happy pass; its detection power is tied to metric impact; and that power scales with the incumbent’s residual error rate, not with the channel’s strength.

We report at the unit level deliberately—the adjudicated claims collapse onto seven independent measurements, so counting claims would multiply-count a single witness. Four observations. First, the certificate preserves as well as fails: every fresh general-VLM claim clears the equivalence bar or is honestly marked underpowered, so the instrument is not a benchmark-condemning machine

Our contribution on this unit is therefore not the channel’s existence but three things the prior literature does not do: (a) a claim-level, pre-registered adjudication—claim, nuisance class, margin, and splits frozen before any score existed—rather than a dataset-level post-hoc demonstration; (b) a score-conditioned estimand: prior work shows the channel predicts labels (I(Y;Z)>0), whereas the certificate measures what the channel adds above the frozen model’s score, which is what a leaderboard reading needs; and (c) a validation of mechanism-driven selection: we chose ISIC by the pre-registered rule “multi-source acquisition structure,” and the dermatology literature independently documents the same pathology—convergent evidence that the selection rule tracks a real mechanism.

Figure/table captions are excessively long, which shows poor style, and almost an entire page is taken up by a single paragraph in section 6, which makes the reading experience genuinely unpleasant.

It is disappointing to see this submitted to ICLR. The authors should revise this work significantly with human supervision and resubmit if desired.

Questions for authors

There are no questions that the authors can answer whose answers would materially change my assessment of this paper.

Reviewer_N5qgOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes an executable certificate that checks whether a benchmark score truly reflects capability or can be inflated by pre-declared nuisance channels, returning FAIL, SURVIVE, or INCONCLUSIVE.

Critical strengths

With the rapid development of LLMs and related fields, the community increasingly values model scores on relevant benchmarks, as such scores are generally believed to be positively correlated with model capability. The motivation of this paper is therefore highly meaningful.

Critical weaknesses

Weakness 1: The writing is overly verbose. The purpose of a table caption is to briefly describe the content of the table, while the analysis should be placed in the main text. A caption that occupies roughly one third of a page is excessively long, and many other instances of such verbosity can be found throughout the paper. I strongly suggest that the authors read more published academic papers to improve the standard of their writing.

Weakness 2: The experiments are insufficient. The authors' experiments are mainly conducted on a few simple benchmarks. However, across different domains—such as language model tasks, agent control tasks, and interactive tasks—there exist many more representative benchmarks that the authors fail to consider. As a result, the authors overclaim their contribution.

Weakness 3: The proposed method is overly crude. Using simple thresholds to classify a benchmark as FAIL, SURVIVE, or INCONCLUSIVE could in principle be a viable approach, but the authors lack thorough ablation experiments to support their method, which makes the empirical results of the paper appear even less convincing.

Questions for authors

See Weaknesses

Reviewer_LtGgOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks when a benchmark score can support a claim about a model's capability. Given a frozen model score and a predeclared class of nuisance channels, the authors measure how much a predictor that also uses the nuisance information can improve the benchmark's native metric. They turn this measure into a three-state certificate. A claim receives FAIL when an admissible attack produces inflation above a specified margin, SURVIVE when a simultaneous upper confidence bound places the inflation below that margin, and INCONCLUSIVE when neither decision is supported. The procedure separates attacker selection from certification and uses clustered bootstrap intervals to account for dependence between evaluation items.

The paper also introduces a Repair–ESS identity that relates the maximum effective sample size of a channel-neutralizing reweighting to the overlap between nuisance groups and labels. The authors use this result to explain why some scoring-side repairs become inefficient and test an entropy-balancing repair against a held-out nonlinear attacker. Across seven main benchmark-audit units covering video anomaly detection, general vision-language tasks and skin-lesion diagnosis, the certificate returns two FAIL, two SURVIVE and three INCONCLUSIVE verdicts at a margin of 0.02. The paper further evaluates an injected answer-leakage channel and repeats the external medical-imaging audit with BiomedCLIP.

Critical strengths

The strongest part of the paper is the question it addresses and the attempt to make benchmark validity claims testable. A high benchmark score can reflect information outside the capability that the benchmark is meant to measure, and the paper offers a concrete way to examine this concern. I find the distinction between FAIL, SURVIVE and INCONCLUSIVE useful. In particular, SURVIVE requires evidence that the remaining inflation is below a declared margin, instead of treating a nonsignificant attack as evidence of validity. The separation between attacker selection and an independent certification fold is also a sensible design choice. The authors report uncertain results rather than forcing a binary decision, and the power analysis makes some of the limits of the available data visible.

The Repair–ESS identity is another interesting contribution. It quantifies how much effective sample size can be retained when reweighting is required to balance the full conditional label distribution across nuisance groups. This provides a useful connection between benchmark repair and overlap arguments from causal inference. The constructive experiment gives the analysis a concrete setting: the calculated maximum ESS is 0.582 of the original sample, below the declared budget of 0.8, while the implemented repair retains about 0.55 and a held-out nonlinear attacker recovers approximately 96 percent of the original inflation. I also appreciate that the authors report results that limit their claims, including the weak-incumbent problem in the ISIC experiment and the unstable verdicts when Roundabout has only three independent clusters.

Critical weaknesses

I recommend rejection because the statistical guarantees behind the certificate remain uncertain, and several statements about the results do not agree with the reported tables.

First, I found inconsistencies between the main text and the appendix. Section 8 and Appendix A state that all units become INCONCLUSIVE at ε = 0.01, but Table 2 shows that VAU and CLIP on ISIC-2019 remain FAIL. The paper also states that protocol tightening moves verdicts toward weaker claims, although Table 4 shows MMMU moving from INCONCLUSIVE to SURVIVE. This matters because the direction of these changes is used to support the claim that the tightened procedure produces more conservative decisions. Other numerical differences are harder to interpret. Section 6 reports CLIP's baseline AUC as both 0.523 and 0.525, and gives an ISIC lower confidence bound of 0.064, while the description of Table 1 and Appendix Table 2 report 0.072. These values may come from different calculations, but the paper does not explain the distinction. TAD and Video-MME also have upper bounds reported as 0.020, exactly the declared margin, so the unrounded values are needed to check their verdicts. I do not take these discrepancies as proof that the final decisions were computed incorrectly, but they make it difficult to verify the results of a method whose main contribution is statistical certification.

The inferential procedure raises a more direct concern. Equation (2) assigns FAIL when any member of the attacker family has an individual 95 percent lower confidence bound above the margin, while SURVIVE uses a simultaneous upper bound over the family. Although the independent certification fold separates attacker selection from evaluation, it does not by itself control the probability of a false FAIL when several attackers are tested and any one can trigger the verdict. I could not find a correction for this multiple-testing problem. The two reported FAIL effects are large and may survive a correction, but that has not been shown. For SURVIVE, the authors acknowledge that the bootstrap may be inconsistent when the maximum inflation lies on the boundary, which is precisely the setting of their SURVIVE results. They do not compute the alternative Bonferroni bound mentioned in Section 5. Figure 2 provides a useful check with an injected channel, but it tests one leakage mechanism on MMBench with twelve seeds per strength. It does not establish that the confidence bounds have the intended coverage across the attacker families and cluster structures used in Table 1. The Roundabout result is particularly concerning. Its 74 items contain only three independent clusters, and the paper reports a FAIL probability of about 0.33 when the true inflation is below the margin. The authors disclose this limitation, but the certificate does not specify when such data should be considered unsuitable for a reliable verdict.

The theoretical result also needs a more careful connection to the claimed repair limit. Appendix B derives the maximum ESS for reweightings that equalize the full conditional label distribution across nuisance groups within score strata. This is an interesting and useful identity, but complete distribution balancing is stronger than requiring zero inflation under a particular benchmark metric. For example, suppose two equally sized nuisance groups have positive-label rates of 0.7 and 0.9, while the frozen score is constant. A classifier that maximizes accuracy predicts the positive class in both groups, with or without the nuisance information. The nuisance therefore contains additional information about the label, but it does not improve accuracy. In this case no reweighting is needed to eliminate accuracy inflation, although the identity assigns a cost to achieving full distribution balance. The theorem remains valid for its stated balancing constraint, but it does not necessarily give the maximum ESS among all reweightings that eliminate inflation under the native metric. I would like the authors to distinguish these objectives before using the identity to support broader claims about when scoring-side repair is impossible.

My other concern is that the empirical results support a narrower conclusion than capability identification. The proposed inflation measures what a declared nuisance channel adds beyond the frozen model score, not whether the model already uses that channel. As the authors acknowledge, a model that has absorbed a shortcut can receive SURVIVE because there is no additional information left for the attacker. The certificate can therefore bound residual inflation without establishing that the model possesses the intended capability. The experiments do not resolve this distinction. Six of the seven main units use benchmarks whose nuisance structure was already known to the authors. The external FAIL comes from CLIP on ISIC-2019, where the baseline AUC is close to 0.52 and a nuisance-only predictor reaches 0.636. When CLIP is replaced with BiomedCLIP, whose baseline AUC is 0.686, the inflation decreases from 0.120 to 0.020 and the verdict becomes INCONCLUSIVE. This shows that the certificate depends on the frozen scorer, but does not establish that the stronger model has absorbed the nuisance channel. The explanations offered for the SURVIVE results, such as CircularEval blocking answer-position effects, are also not tested through controlled ablations. I appreciate that the paper acknowledges many of these limitations, but the conclusion still treats the absence of further nuisance inflation as a condition for licensing a capability claim.

I find the problem important and the Repair–ESS analysis interesting, but the current evidence does not establish the reliability or scope of the proposed certificate. If the authors can reconcile the reported results, show that the main verdicts hold under valid family-wise confidence bounds, and clarify the relationship between distribution balancing, metric inflation and capability identification, I would be willing to reconsider my assessment.

Questions for authors

These are clarifications and focused checks of results that are central to my assessment.

  1. Could you reconcile the discrepancies between the main text and the appendix? At ε = 0.01, Table 2 retains two FAIL verdicts, although Section 8 and the appendix description state that all units are INCONCLUSIVE. Table 4 also shows MMMU moving from INCONCLUSIVE to SURVIVE, contrary to the statement that all changes under protocol tightening weaken the claims. Please explain the different CLIP AUC values of 0.523 and 0.525 and the ISIC lower bounds of 0.064 and 0.072, including which attacker and evaluation set each refers to. Could you also provide the unrounded confidence bounds and the exact values used to assign the TAD and Video-MME verdicts?
  2. How is the family-wise false-positive probability controlled for FAIL? Equation (2) allows any of K attackers to trigger FAIL through its individual lower confidence bound. If no multiple-testing correction is applied, could you recompute the FAIL verdicts with simultaneous lower bounds or a suitable correction and report whether both remain FAIL?
  3. Could you check the SURVIVE verdicts using confidence bounds that do not rely on the potentially inconsistent boundary bootstrap? In particular, do MMBench and MMMU remain SURVIVE under a Bonferroni-based bound? A controlled simulation with known population inflation at or near ε, reporting the actual false-SURVIVE probability for different numbers of clusters and attacker-family sizes, would help establish whether the claimed error control holds in the settings where the certificate is used. Please also clarify whether the severe calibration failure reported for Roundabout leads to any restriction on when the certificate should be applied.
  4. Does the Repair–ESS identity characterize complete conditional distribution balancing, or all reweightings that eliminate inflation under the benchmark's native metric? These conditions need not be equivalent for accuracy. Could you clarify which statement the theorem establishes and whether the repair-impossibility conclusions remain valid when the objective is only to make the measured inflation zero? Please also specify the weighting constraints used when the single-stratum result is extended across score strata.
  5. What additional evidence supports interpreting SURVIVE as a statement about capability rather than residual nuisance information? The paper acknowledges that a model can already use a nuisance channel and still receive SURVIVE. Could the claim be restricted to robustness against the declared attacker family, or could the authors provide a controlled intervention that distinguishes this case from a model that does not use the nuisance? For the external ISIC experiment, a comparison across additional scorers or a controlled analysis of baseline performance and residual inflation would also help establish how much of the FAIL result is due to the channel rather than the weakness of the incumbent model.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.