acceptodds
Under review as a conference paper at ICLR 2027

When Should AI Evaluation Measure More?

Abstract

When an evaluation result is uncertain, the usual remedy is more of the same evaluation. We argue that evaluation is instead a target-conditioned design problem: an evaluator can change what is measured, how the responses are analyzed, or which target and reporting rule the evidence is meant to support, and these levers should be compared at matched cost. Across vision and language-model response banks, we find that the useful action depends on the bottleneck. In one matched-cost vision comparison, the same added budget reduced target error from 1.10 to 1.05 percentage points when spent on deeper replication of existing conditions, but to 0.56 when spent on target-relevant joint conditions. More broadly, reanalysis can outperform substantially more calls under a plug-in analysis for lower-tail targets but not for mean targets, and a rule accurate for a scalar summary can fail on the condition-level responses behind it. In a held-out check, a comparison frozen on development models selected released estimators that reduced equal-query error on held-out models when a historical calibration bank was available. We formalize the distinction between measurement support, statistical inference and decision targets, and provide Reassess, a toolkit for comparing these alternatives from response-level banks. The practical question is therefore not only how much more to evaluate, but what the next unit of evaluation should change.

Primary area:
datasets and benchmarks
Keywords:
AI evaluation, benchmark evaluation, measurement science, target-conditioned evaluation, efficient evaluation, identifiability, experimental design, uncertainty quantification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hauvOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how an AI evaluation budget should be allocated among additional measurements, statistical reanalysis, and the choice of target and reporting rule. It separates target identification from finite-sample estimation using classical partial-identification and optimal-recovery arguments. Experiments on image-corruption grids compare additional single-corruption measurements with measurements of joint conditions at matched query budgets. For language models, the authors compare estimators of mean and lower-tail performance across prompt variations, including shrinkage and selection using only acquired responses. Further experiments examine calibration across architecture families and the cost of selecting a model or returning an unresolved set. The paper presents the Reassess toolkit and evaluates a frozen estimator-selection procedure on held-out models using an existing historical calibration bank.

Critical strengths

The paper is clear and well-written, and it does an excellent job of explaining the issues and their solutions. The comparisons establish useful distinctions between evaluation problems that require different remedies. Usefully, the paper goes beyond recommending larger evaluations by measuring what alternative uses of the same query budget actually achieve.

Critical weaknesses

The paper demonstrates why evaluation design matters, but does not sufficiently establish how its proposed framework improves evaluation decisions beyond established methods. There is still useful work here: the controlled comparisons, negative results, and explicit accounting of calibration cost, but I waver between weak rejection and weak acceptance. Again my soundness rating reflects my question as to value of new ideas. The evidence that the framework can guide a new evaluation before its answers are known, is limited. Most positive comparisons on the authors’ own banks are retrospective or post hoc.

Questions for authors

Please present further evidence that you have important new methods for new problems.

3 comments

We sincerely thank the reviewer for the thoughtful feedback and for recognizing the value of our matched-budget experiments and transparent reporting. We address the request for stronger methodological and practical evidence below.

[Q1 / W1] Operational Method and Controlled Evidence.

Q1. Please present further evidence that you have important new methods for new problems.

W1. The paper demonstrates why evaluation design matters, but does not sufficiently establish how its proposed framework improves evaluation decisions beyond established methods. There is still useful work here: the controlled comparisons, negative results, and explicit accounting of calibration cost, but I waver between weak rejection and weak acceptance. Again my soundness rating reflects my question as to value of new ideas.

The revised method is an executable reference-free claim audit, not a new concentration inequality. Given a declared frame, target and sampling design, it (i) tests whether measured conditions identify the requested model comparison and supplies a constructive witness for NONID; (ii) checks frame consistency and statistical eligibility; and (iii) returns CERT or NOT_CERT with a target-specific finite-sample bound when eligible. Input refusals are separate from these three scientific outcomes (Sec. 3, pp. 3–4; Fig. 2, p. 5; Appendix B.4, pp. 22–24).

Table R1. Two controlled tests of the implemented method.

Controlled comparisonComparatorReassessDirect conclusion
Same full-support balanced observations, 10 prompt-format runs, δ=2 pp, α=0.05Proved generic A0-H cellwise box: 0/10 CERTTarget-matched route: 6/10 CERTPost hoc, route-specific certification gain on one task
216 matched-information declarations (135 author-constructed defects)Support-aware analyst: 33 valid / 3 materially wrong singleton claims22 valid / 0 wrongThe frame-consistency check prevents three constructed wrong claims

The frame-check ablation is especially informative: among 27 constructed frame contradictions, 24 valid analyst singletons and three wrong ones were all withheld. Deleting the check restores the three wrong claims. The abstention trade-off is explicit (Sec. 5.4, p. 8; Fig. 5d, p. 9; Table 11, p. 36).

Proof scope. Proposition B.5 proves the target-matched pairwise bound and Proposition B.7 the generic ρ=1 cellwise box, conditional on eligible finite-frame sampling and one prespecified look. Independence across conditions is declared, not verifiable by the code. The default A0 box with Serfling correction is empirically checked but outside the proved guarantee (Appendix B.3, pp. 18–21). These are operational improvements in auditing a claim, not evidence that Reassess is uniformly more efficient than standard valid intervals.

[W2] Pre-Response Study Designs and Validation Scope. The evidence that the framework can guide a new evaluation before its answers are known, is limited. Most positive comparisons on the authors’ own banks are retrospective or post hoc.

We added two single-task acquisition studies with the plans, budgets, success criteria and primary look fixed before model responses. On the original prompt-format task, at equal +768 evaluations per model from a 192-evaluation start, the outcomes were:

Table R2. Certification at the single primary look; 10 correlated runs per plan (Sec. 5.2, pp. 6–7; Fig. 3, p. 7).

AllocationCERT / 10Explanation
Replicate 16 measured formats0Missing 32 target formats: NONID
Acquire 32 unmeasured formats3Restores support
Balanced acquisition over 48 formats6Restores support and improves precision
MedMCQA: acquire missing option orders10Separate single-task test at 1,280 evaluations/model

All prompt-format point-estimate winners were already correct on the reference; this establishes reportability, not a change in model selection. After response collection, the certification bound was corrected, reducing full-support balanced outcomes from 8/10 to 6/10, and all results were recomputed. Thus the acquisition design was pre-response, but the final bound was not prospectively validated (Appendix B.3, p. 19).

We also tested target-dependent inference on six newly evaluated MMLU models, with comparisons fixed before their responses:

Table R3. History-informed at 128 versus logit main effects at 256 evaluations per model (Table 1, p. 10; Table 13, p. 40).

TargetDifference in absolute target error [95% interval] (pp)Interpretation
Lower tail (worst 10%)−6.75 [−8.83, −4.66]History-informed estimator lower error
Mean+0.34 [0.02, 0.65]Direction reverses; logit lower error
Lower tail: history-informed vs EB without history+0.14 [−0.61, 0.89]Unresolved

The history-informed method additionally used 344,064 earlier development responses, disclosed separately from the new evaluation budget. This is target-specific evidence, not proof that historical responses explain the tail gain. Five other prespecified follow-up tests did not meet their criteria (Table 5, pp. 27–28). Finally, reference-free next-action prediction was investigated retrospectively without a demonstrated gain; its planned prospective validation was not run (Appendix E.1, pp. 44–45).

Thank you again for your thoughtful and constructive review. We particularly appreciate your recognition of the value of our evaluation-design comparisons and transparent reporting.

Your feedback has been valuable in refining our work, and we will carry these insights into our future research.

We would welcome any final questions or comments if time permits. Thank you sincerely for your time and consideration.

Best regards, The Authors

Reviewer_xtSdOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper addresses how to improve model evaluation (broadly defined) under resource constraints and a given evaluation target. It distinguishes three “levers” analysts might adjust: (1) measurement design (which conditions are measured), (2) statistical estimation, and (3) the choice of target and how results are reported. The paper argues that replicating existing measurements cannot reduce error arising from unmeasured conditions, and recommends instead measuring conditions closer to the target, reanalyzing existing responses with better-suited estimators, or changing the target or reporting rule. These options are illustrated with separate cases, including image classifiers under corruption and language models under prompt variation. The paper also provides Reassess, a tool for assessing such modifications in evaluations.

Critical strengths
  • Though I have concerns about their framing and implications, the analysis appears to be sound and well-conceived, with thoughtful paired comparisons and clear presentation of results.
  • The related works section is relatively thorough, although some clarification would be appreciated in terms of how the proposed framework develops or departs from existing work.
Critical weaknesses
  • Context, scope, and motivation are poorly defined. The abstract and introduction express what was done (or what the objects of analysis are) in vague abstraction (“target-conditioned design problem,” “useful action depends on the bottleneck,” “target-relevant joint conditions”). It would also be worth contextualizing early on what kind of evaluation is being discussed. E.g., an LLM evaluation or audit provides a different context than an image classifier. The kind of context in question potentially changes the meaning or relevance of the outlined “levers,” like “measure,” “infer,” and “decide.” Understanding the context of the problem is made more difficult by the thin references included in the introduction. For example, it states that the paper is concerned with a “classical distinction” without providing references for what this distinction is. The two cited works (Manski; Micchelli & Rivlin) are invoked for “an identification question,” but the introduction doesn’t explain how this is connected to ML evaluation questions.
  • Unclear why the developed framework/procedure is needed. The paper opens with the (contestable) claim that “When an evaluation result is uncertain, the usual remedy is more of the same evaluation.” Is there evidence of previous evaluations coming to problematic conclusions because of this supposed habit? While stating that established procedures are problematic, the paper doesn’t provide evidence of real evaluations going wrong in ways the proposed framework would avoid. The discussion would be a great place to demonstrate what the community gains by adopting the framework, but this is not included.
  • Lack of originality. The framework largely restates familiar notions of measurement, estimation, and estimand selection without redeploying or reframing them in a clearly useful way. Some of the findings read more as general recommendations for practitioners than novel contributions, such as the idea that with sparse data, estimators with shrinkage provide better estimates.
  • Underdeveloped code/data artifact. Although the paper includes code for implementing the framework, it gets little attention in the paper (e.g., no usage example) and appears to be more of an afterthought.
Questions for authors
  • Can you point to published evaluations, such as benchmark papers, leaderboard results, or audits, where relying on replication led to a conclusion that the proposed framework would have changed? A concrete example would make the case for the framework much stronger.
  • Each "lever" is compared against replication on a different target (e.g., the grid mean for measure, a lower-tail mean for infer, per-cell error and model selection for decide), creating the problem that the effects of different interventions are not measured on a consistent scale. How should a practitioner choose between the given levers when all are feasible? Is there a way to compare the impacts of levers directly to each other, on the same target outcome and at matched cost?
5 comments

We sincerely thank the reviewer for the careful assessment of our work and the constructive feedback. We appreciate the recognition of the analysis’s soundness and address the concerns regarding practical value, matched comparisons, and methodological contribution below.

[W2 / Q1] Published Evaluations and Practical Value.

W2. Unclear why the developed framework/procedure is needed. The paper opens with the (contestable) claim that “When an evaluation result is uncertain, the usual remedy is more of the same evaluation.” Is there evidence of previous evaluations coming to problematic conclusions because of this supposed habit? While stating that established procedures are problematic, the paper doesn’t provide evidence of real evaluations going wrong in ways the proposed framework would avoid. The discussion would be a great place to demonstrate what the community gains by adopting the framework, but this is not included.

Q1. Can you point to published evaluations, such as benchmark papers, leaderboard results, or audits, where relying on replication led to a conclusion that the proposed framework would have changed? A concrete example would make the case for the framework much stronger.

We removed the unsupported opening premise that repeating an evaluation is the usual remedy. The present paper instead asks a testable, narrower question: can the collected responses support the specific comparison being reported? (Introduction, pp. 1–2; Sec. 3, pp. 3–4).

We audited published claims and broader interpretations separately (Sec. 5.4, p. 8; Table 10, p. 35):

Source of interpretationsCasesResult
Literal published claims quoted from source evaluations139 assessable; none contradicted
Broader interpretations authored by us2828 NONID with constructive witnesses
Exact finite-split leaderboard comparisons14Exact census findings; not inference-based superiority

We do not claim that a published paper reached a demonstrably false conclusion because of replication. The 28 extensions are our own counterfactual readings, not claims of the source papers.

An additional real-response MMLU analysis illustrates a concrete reporting change: with four models, 96 evaluations per model, and δ=2 pp for the worst-10%-of-48-conditions target, Reassess returned NOT_CERT without accessing reference results. Afterwards, plug-in and EB selections scored 5.92 and 8.13 pp regret—above the declared tolerance. This example shows why withholding a point claim can matter, but does not establish superiority over standard valid statistical intervals; its default A0 route is outside the proved guarantee (Sec. 5.4, pp. 7–8; Appendix D.4, p. 34).

The operational benefit distinctive to Reassess is to report which constraint prevents the claim—missing support, insufficient precision, or an ineligible/inconsistent declaration—rather than treating every unreported winner as merely a wide interval (Fig. 2, p. 5; Appendix B.4, pp. 22–24).

[Q2 / W3] Same-Target Matched-Cost Comparison and Methodological Novelty.

Q2. Each "lever" is compared against replication on a different target (e.g., the grid mean for measure, a lower-tail mean for infer, per-cell error and model selection for decide), creating the problem that the effects of different interventions are not measured on a consistent scale. How should a practitioner choose between the given levers when all are feasible? Is there a way to compare the impacts of levers directly to each other, on the same target outcome and at matched cost?

W3. Lack of originality. The framework largely restates familiar notions of measurement, estimation, and estimand selection without redeploying or reframing them in a clearly useful way. Some of the findings read more as general recommendations for practitioners than novel contributions, such as the idea that with sparse data, estimators with shrinkage provide better estimates.

We agree that results across different targets cannot be ranked on a common loss scale. To answer the comparable part directly, we added a retrospective replay at one fixed lower-tail model-selection target with six models, eight test task families, the same initial observations, and equal additional evaluations for each allocation plan.

Table R1. Gap-fill minus replication in reference winner regret (pp): positive favors replication (Sec. 5.5, pp. 8–10; Table 1, p. 10; Fig. 10, p. 42).

EstimatorEffect [95% interval] (pp)Outcome
Empirical Bayes+1.40 [0.32, 2.47]Replication better in 6/8 test families
Logit main effects+0.21 [−0.54, 0.97]Unresolved

The result demonstrates how the preferred measurement plan can depend on the estimator, not a comparison or joint optimization of all three levers. The separately reported history-informed MMLU inference uses 344,064 development responses; we do not treat that information as free (Sec. 4, p. 5; Appendix C.2, p. 26).

The method's technical contribution is not a new inequality. Reassess composes (i) a finite-frame support check with a witness, (ii) eligibility and frame-consistency checks, and (iii) a target-specific certification report (Sec. 3; Fig. 2; Appendix B.3–B.4). Two comparisons expose what this adds:

Table R2. Matched-information evidence of the auditing procedure.

ComparisonExisting/simple comparatorReassessWhat this shows
Identical full-support prompt-format observations (10 runs)Proved generic cellwise A0-H: 0 CERTTarget-matched: 6 CERTRoute-specific, post hoc gain on one task—not superiority over all valid methods
216 declarations (81 valid, 135 author-constructed defects)Support-aware analyst: 33 valid / 3 wrong singletons22 valid / 0 wrongFrame check prevents 3 wrong claims, at an abstention cost

For the 27 constructed frame contradictions specifically, 24 analyst singletons were valid and three materially wrong; the check withheld all 27. Removing it restores the three wrong ones (Sec. 5.4, p. 8; Fig. 5d, p. 9; Table 11, p. 36). The constructed cases demonstrate enforceable claim–evidence consistency, not the prevalence of errors in the literature.

[W1 / W4] Evaluation Context, Terminology, and Executable Artifact.

W1. Context, scope, and motivation are poorly defined. The abstract and introduction express what was done (or what the objects of analysis are) in vague abstraction (“target-conditioned design problem,” “useful action depends on the bottleneck,” “target-relevant joint conditions”). It would also be worth contextualizing early on what kind of evaluation is being discussed. E.g., an LLM evaluation or audit provides a different context than an image classifier. The kind of context in question potentially changes the meaning or relevance of the outlined “levers,” like “measure,” “infer,” and “decide.” Understanding the context of the problem is made more difficult by the thin references included in the introduction. For example, it states that the paper is concerned with a “classical distinction” without providing references for what this distinction is. The two cited works (Manski; Micchelli & Rivlin) are invoked for “an identification question,” but the introduction doesn’t explain how this is connected to ML evaluation questions.

W4. Underdeveloped code/data artifact. Although the paper includes code for implementing the framework, it gets little attention in the paper (e.g., no usage example) and appears to be more of an afterthought.

The Abstract and Introduction now begin with the general benchmark-evidence problem and prior work (MMLU, HELM, prompt-format sensitivity, partial identification and evaluation validity) before presenting LMentry as a concrete example in the latter part of the Introduction (pp. 1–2). We define evaluation condition, target, lower tail, measured support, item-and-condition frame, identification and precision on first use, then introduce the formal symbols B, ψ and T in Sec. 3 (pp. 3–4).

The methodology is explicitly tied to the type of evaluation: prompt formatting and option order in language benchmarks, and corruption conditions in vision. Measure changes observed support (B), Infer changes the estimator (ψ), and Decide changes the target/reporting rule (T); the evaluator, not the tool, chooses the next intervention (Fig. 1, p. 2; Sec. 3.4, p. 4).

The revised artifact is a runnable, reference-free CLI, with declarations, machine-readable reports and refusal reasons in Appendix B.4, pp. 22–24. Its three primary example states are visible directly in Fig. 2, p. 5:

StatusActual recorded prompt-format reportMeaning
NONID16 of 48 formats measured, run 4Constructive witness: two completions yield different acceptable models
NOT_CERT48-format focused allocation, run 4; regret bound 12.294 ppIdentified but imprecise relative to δ=2 pp
CERT48-format balanced allocation, run 2; bound 1.119 ppPhi-4-mini certified within δ=2 pp

The anonymous code ZIP reproduces those recorded reports, one refusal, and all 50 primary-look +768 prompt-format reports (0/3/6 from partial support, 0/6 from full support), with 35 passing CLI tests (Appendix F, p. 45). The ZIP does not reproduce every other experiment; the paper states this limitation explicitly (Reproducibility Statement, p. 11). The final target-matched proof is conditional on its declared sampling assumptions; the default A0 route remains outside the proved guarantee (Sec. 3.3–3.4; Appendix B.3).

Reviewer_X9k5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that an uncertain evaluation result can have three different causes, unmeasured conditions, a poor estimator, or the wrong target for the decision, and that the remedies (measure new conditions, reanalyse the responses in hand, change the target or reporting rule) should be compared at matched cost rather than defaulting to more of the same. It supports this with classical identification results, matched-cost illustrations on vision corruption grids, MMLU prompt conditions, released PromptEval banks and a held-out tinyBenchmarks check, and releases Reassess, a tool for running such comparisons from response-level banks.

Critical strengths

The framing is sound and useful: a confidence interval can shrink while the target remains unidentified by the measured conditions, so replication, new measurement and reanalysis are different interventions and should be priced against each other. The reporting is exemplary: every comparison is labelled prespecified, retrospective or post hoc, negative results are kept, the held-out rule was frozen before reading outcomes, and the scope statement (Section 8) is candid. The tool and the reproduction of released PromptEval and tinyBenchmarks numbers are commendable.

Critical weaknesses

The paper has an interesting idea but I do not consider it ready for publication, for two reasons that concern the design of the study rather than its details. Most importantly, it lacks a compelling demonstration. The illustrations show, on convenience panels with a dense reference already in hand, facts a statistician would predict (measuring joint cells beats extrapolating from marginals; the lowest of noisy means is biased; the estimator should match the target); the single prespecified test (blur×gamma) missed its criterion, and the held-out check replicates tinyBenchmarks’ own result. In the practitioner’s situation, without a dense reference, the tool returns NOT_ASSESSABLE, and as far as I can see the paper never tests whether anything computable in that situation predicts which lever pays.

Second, the presentation is opaque: a reader cannot tell from the abstract or Figure 1 what was shown, and the body is a sequence of numbers and caveats without a clear flow. I believe the work should be redone around one prospectively tested case on an evaluation conclusion that matters, and rewritten around it, rather than revised in a rebuttal. One candidate the paper already cites is to test whether the typical spreads and rank-flip frequencies reported by Sclar et al. (2024) survive a shrinkage estimate of the tail on the same calls (their extreme examples, scored on 1,000 items per format, will). Ideally, however, you would pick a case where the extreme-value bias could be decisive, that is, where worst cases are searched over many conditions with few items each, as in Sclar et al.’s 500-format, 250-item feature analysis or in jailbreak and agentic evaluations with a few hundred prompts.

If the case study attaches an explicit loss to the decision (for instance, the cost of selecting the wrong model or of misjudging a safety threshold), then ‘when to measure more’ has a proper answer: measure while the expected reduction in decision loss exceeds the cost, and the three levers can be compared on expected loss reduction per unit cost, forecast before the budget is spent and checked afterwards. Without that, the title’s question, when more evaluation is worth its cost, remains unanswered.

Questions for authors

(1) Is there any quantity computable without the dense reference, for instance the practitioner-mode sensitivity diagnostic or a small pilot of joint cells, that predicts across your existing banks where measurement or reanalysis paid and where it did not? A retrospective check would be informative. (2) How much of the 1.05 versus 0.38 pp gap on blur×JPEG survives when the reference excludes the queried items, as in your MMLU design?

Suggestions: Redo the empirical part around the most compelling case you can find. Rewrite the abstract and introduction around the chosen case, define the vocabulary (support, bank, axes, checkerboard, arm, lever) on first use, add one table listing every comparison with its status (prespecified, retrospective, post hoc, secondary) and outcome, and cite Miller (2024), ‘Adding error bars to evals’, for the inference lever.

5 comments

We thank the reviewer for recognizing the importance of distinguishing identification from statistical precision and for appreciating our transparent reporting. We have retained these principles and added targeted analyses, practical auditing capabilities, and clearer empirical demonstrations.

Questions for Authors

Q1. Can we predict which evaluation intervention pays without a dense reference?

We investigated this retrospectively across 16 task families and 1,280 starting states. Reference-blind forecasts showed weak associations with realized gains (Spearman 0.03–0.06 for plans acquiring evaluations) and no established improvement over the best fixed policy. A learned next-action selector also underperformed simple alternatives (Appendix E.1, pp. 43–44).

Although optimal next-action prediction remains open, Reassess now provides reference-free claim auditing, returning NONID with a witness, NOT_CERT, or CERT from observed responses alone. This replaces the original NOT_ASSESSABLE behavior and identifies whether coverage or precision limits a declared claim (Sec. 3; Fig. 2, p. 5; Appendix B.4).

Q2. Does the blur×JPEG advantage survive on images excluded from evaluation?

Yes. On 2,545 never-queried ImageNet-V2 images, axis replication has +0.421 [0.339, 0.523] pp greater target error than four joint cells (B4), retaining approximately 83% of the corresponding full-reference contrast. Against all 16 target cells (D16, post hoc), the difference is +0.520 [0.428, 0.628] pp, retaining approximately 77%. Both contrasts favor alternative measurement in 9 of 10 recipe blocks (Table 1, p. 10; Fig. 7 and Table 12, p. 37).

Critical Weaknesses

W1. Lack of compelling prospective evidence. We added two concrete single-task studies with acquisition plans, budgets, and success criteria fixed before responses. At equal additional cost, replication, missing-condition acquisition, and balanced allocation yielded 0/3/6 certificates across ten correlated runs; MedMCQA achieved 10/10 under missing-condition acquisition (Sec. 5.2; Fig. 3, p. 7). These interventions changed whether the already-correct winner could be certified. The final bound was corrected after data collection, so we do not present these as prospective tests of that corrected route.

W2. Extreme-value bias with many conditions and few items. Following the suggestion, we analyzed 81 released tasks with eight items per format. We found −30.2 pp selected-worst bias and 4.04 pp exploratory model-choice regret. Select-then-remeasure reduced worst-format regret to 1.43 pp under the nominal design, although the improvement did not extend to the prespecified lower-tail target (Sec. 5.3; Fig. 4, p. 8; Appendix D.3). The specific Sclar et al. 500-format study lacked the released item-level responses needed for direct replication.

W3. Presentation and clarity. We substantially rewrote the Abstract and Introduction, replaced Figure 1 with a study overview, defined the central terminology on first use, and added an evidence registry documenting every comparison's status and outcome (Table 5, pp. 27–28). We also added the requested Miller (2024) citation (Sec. 2).

These revisions strengthen the empirical motivation and establish an executable claim-auditing contribution while clearly distinguishing it from the open problem of prospective evaluation-plan selection.

Reviewer_X9k5OpenReview ↗

Response to rebuttal

Thank you for the candid response. It fully answers my second question: the advantage of measuring joint blur×JPEG cells also holds on images that were never queried. I also appreciate the worst-format analysis on 81 tasks, the clearer introduction and the evidence table.

I am keeping my score, though. The answer to my first question is negative: as you say yourselves, no rule that works without the dense reference did better than the best fixed plan, and the planned prospective test was not run. The paper is now about claim auditing instead. Reviewers xtSd and hauv asked in their reviews what the framework changes in practice, and I think that question is still open for the new version: by your own account, none of the published claims you audited was contradicted, and in the new studies the point estimate already picked the right model.

In sum, the revision is clearer and more careful about what it claims, but it is really a new paper, which I could not review again in full depth, and its practical value is not yet shown. If I have missed a result where the audit makes a difference, please point me to it.

We thank the reviewer for the thoughtful follow-up. We appreciate the recognition of the image-disjoint validation sparse worst-format analysis, and presentation improvements.

We would like to highlight a real-response example in the revised manuscript that we did not emphasize sufficiently.


1. A real evaluation where auditing changes the reporting decision

MMLU model-selection example (Appendix D.4, p. 34; Appendix B.4, pp. 22–24)

We audited a comparison of four models using 96 evaluations per model, with a declared target of the worst-10% mean across 48 prompt conditions, a regret tolerance of δ=2 percentage points, and α=0.05.

Without accessing reference outcomes, Reassess returned NOT_CERTIFIED_AT_CURRENT_PRECISION (NOT_CERT) and ABSTAIN, issuing no single-model conclusion.

After finalizing the observed-only report, we evaluated the point-estimate choices against the reference.

Table R1. Practical reporting impact on a real MMLU evaluation. All methods use the same observed responses. Reference regret is computed only after the audit report is finalized.

Decision ruleReporting decisionReference regret
Plug-in point estimate (first tie-break)Select one model5.92 pp
Plug-in point estimate (alternative tied candidate)Select one model8.13 pp
Empirical-Bayes point estimateSelect one model8.13 pp
Reassess (reference-free audit)NOT_CERT / ABSTAINNo singleton claim

The plug-in estimates were tied between two candidates, and both candidates exceeded the declared 2 pp tolerance. The empirical-Bayes choice also exceeded this tolerance.

Thus, unlike our prompt-format studies where the point-estimate winner was already correct, this example shows a concrete change in the reporting decision: the audit withheld a single-model conclusion that would have been materially wrong under the declared target.


2. What is the operational contribution beyond standard statistical principles?

We also added a controlled comparison against a support-aware analyst, using identical information across 216 declarations (81 valid controls and 135 author-constructed defects).

Table R2. Auditing outcomes under matched information (Figure 5d, p. 9; Table 11, p. 36).

MethodValid singletonMaterially wrong singletonNo singleton
Point winner190260
Support-aware analyst333180
Reassess220194

The frame-consistency check prevents the analyst's three materially wrong singleton reports under frame contradictions, at the cost of fewer valid singleton conclusions (22 versus 33).

These defects were constructed by us; we do not present them as the prevalence of errors in published evaluations. The contribution illustrated here is executable enforcement of claim–evidence consistency, not a new statistical inequality.


We agree that the practical significance of the MMLU example was easy to miss in Appendix D.4. We are prepared to foreground a concise version in Section 5.4 of the main text as a presentation-only revision, retaining the full declaration and technical details in Appendices B.4 and D.4, without introducing new experiments or changing the paper's claims.

We hope this clarification helps distinguish the demonstrated contribution—auditable decisions about whether a benchmark comparison can be reported—from the separate, unresolved problem of selecting the optimal next evaluation intervention.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revised manuscript and supplement

We thank the reviewers for their constructive feedback.

  • Presentation and continuity. We substantially revised the Abstract, Introduction, figures, and organization, clarifying the central question: Which model comparisons can benchmark measurements justify? The core framework—measurement map BB, estimator ψ\psi, target TT, and Measure / Infer / Decide—remains unchanged, together with the original vision and language studies (Secs. 1–3, 5.1–5.5; Figs. 1–2).

  • Additional evidence and practical auditing. We added targeted analyses and an executable, reference-free claim audit. At equal additional evaluation budgets, replication, missing-condition acquisition, and balanced allocation yielded 0/3/6 certificates across ten correlated runs on one task (Sec. 5.2; Fig. 3, p. 7). We also foregrounded a real MMLU example where auditing withheld a single-model conclusion that point-estimate choices would not have justified (Sec. 5.4, pp. 7–8), alongside a controlled frame-consistency analysis (Fig. 5, p. 9; Table 11, p. 36).

  • Scientific transparency. The acquisition plans and success criteria were fixed before model responses, while the certification bound was corrected after data collection and all affected results were recomputed (Appendix B.3, p. 19). The revised supplementary material provides detailed evidence and an evidence-status registry (Table 5, pp. 27–28).

We address each reviewer's specific questions in their respective review threads.

We have made final refinements to the manuscript to improve clarity and better highlight the practical contributions, without introducing new experiments or changing the results.

We have also updated and reorganized our responses into concise, issue-focused comments, each directly addressing the reviewers' concerns with key evidence, quantitative results, and references to the revised manuscript.

As the discussion period approaches its end, we would greatly appreciate it if the reviewers could revisit our revised manuscript and responses, if time permits. We welcome any remaining questions or feedback.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.