- Rating
- 2(−1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how an AI evaluation budget should be allocated among additional measurements, statistical reanalysis, and the choice of target and reporting rule. It separates target identification from finite-sample estimation using classical partial-identification and optimal-recovery arguments. Experiments on image-corruption grids compare additional single-corruption measurements with measurements of joint conditions at matched query budgets. For language models, the authors compare estimators of mean and lower-tail performance across prompt variations, including shrinkage and selection using only acquired responses. Further experiments examine calibration across architecture families and the cost of selecting a model or returning an unresolved set. The paper presents the Reassess toolkit and evaluates a frozen estimator-selection procedure on held-out models using an existing historical calibration bank.
Critical strengths
The paper is clear and well-written, and it does an excellent job of explaining the issues and their solutions. The comparisons establish useful distinctions between evaluation problems that require different remedies. Usefully, the paper goes beyond recommending larger evaluations by measuring what alternative uses of the same query budget actually achieve.
Critical weaknesses
The paper demonstrates why evaluation design matters, but does not sufficiently establish how its proposed framework improves evaluation decisions beyond established methods. There is still useful work here: the controlled comparisons, negative results, and explicit accounting of calibration cost, but I waver between weak rejection and weak acceptance. Again my soundness rating reflects my question as to value of new ideas. The evidence that the framework can guide a new evaluation before its answers are known, is limited. Most positive comparisons on the authors’ own banks are retrospective or post hoc.
Questions for authors
Please present further evidence that you have important new methods for new problems.
3 comments
[W2] Pre-Response Study Designs and Validation Scope. The evidence that the framework can guide a new evaluation before its answers are known, is limited. Most positive comparisons on the authors’ own banks are retrospective or post hoc.
We added two single-task acquisition studies with the plans, budgets, success criteria and primary look fixed before model responses. On the original prompt-format task, at equal +768 evaluations per model from a 192-evaluation start, the outcomes were:
Table R2. Certification at the single primary look; 10 correlated runs per plan (Sec. 5.2, pp. 6–7; Fig. 3, p. 7).
| Allocation | CERT / 10 | Explanation |
|---|---|---|
| Replicate 16 measured formats | 0 | Missing 32 target formats: NONID |
| Acquire 32 unmeasured formats | 3 | Restores support |
| Balanced acquisition over 48 formats | 6 | Restores support and improves precision |
| MedMCQA: acquire missing option orders | 10 | Separate single-task test at 1,280 evaluations/model |
All prompt-format point-estimate winners were already correct on the reference; this establishes reportability, not a change in model selection. After response collection, the certification bound was corrected, reducing full-support balanced outcomes from 8/10 to 6/10, and all results were recomputed. Thus the acquisition design was pre-response, but the final bound was not prospectively validated (Appendix B.3, p. 19).
We also tested target-dependent inference on six newly evaluated MMLU models, with comparisons fixed before their responses:
Table R3. History-informed at 128 versus logit main effects at 256 evaluations per model (Table 1, p. 10; Table 13, p. 40).
| Target | Difference in absolute target error [95% interval] (pp) | Interpretation |
|---|---|---|
| Lower tail (worst 10%) | −6.75 [−8.83, −4.66] | History-informed estimator lower error |
| Mean | +0.34 [0.02, 0.65] | Direction reverses; logit lower error |
| Lower tail: history-informed vs EB without history | +0.14 [−0.61, 0.89] | Unresolved |
The history-informed method additionally used 344,064 earlier development responses, disclosed separately from the new evaluation budget. This is target-specific evidence, not proof that historical responses explain the tail gain. Five other prespecified follow-up tests did not meet their criteria (Table 5, pp. 27–28). Finally, reference-free next-action prediction was investigated retrospectively without a demonstrated gain; its planned prospective validation was not run (Appendix E.1, pp. 44–45).
Thank you again for your thoughtful and constructive review. We particularly appreciate your recognition of the value of our evaluation-design comparisons and transparent reporting.
Your feedback has been valuable in refining our work, and we will carry these insights into our future research.
We would welcome any final questions or comments if time permits. Thank you sincerely for your time and consideration.
Best regards, The Authors
We sincerely thank the reviewer for the thoughtful feedback and for recognizing the value of our matched-budget experiments and transparent reporting. We address the request for stronger methodological and practical evidence below.
The revised method is an executable reference-free claim audit, not a new concentration inequality. Given a declared frame, target and sampling design, it (i) tests whether measured conditions identify the requested model comparison and supplies a constructive witness for NONID; (ii) checks frame consistency and statistical eligibility; and (iii) returns CERT or NOT_CERT with a target-specific finite-sample bound when eligible. Input refusals are separate from these three scientific outcomes (Sec. 3, pp. 3–4; Fig. 2, p. 5; Appendix B.4, pp. 22–24).
Table R1. Two controlled tests of the implemented method.
The frame-check ablation is especially informative: among 27 constructed frame contradictions, 24 valid analyst singletons and three wrong ones were all withheld. Deleting the check restores the three wrong claims. The abstention trade-off is explicit (Sec. 5.4, p. 8; Fig. 5d, p. 9; Table 11, p. 36).
Proof scope. Proposition B.5 proves the target-matched pairwise bound and Proposition B.7 the generic ρ=1 cellwise box, conditional on eligible finite-frame sampling and one prespecified look. Independence across conditions is declared, not verifiable by the code. The default A0 box with Serfling correction is empirically checked but outside the proved guarantee (Appendix B.3, pp. 18–21). These are operational improvements in auditing a claim, not evidence that Reassess is uniformly more efficient than standard valid intervals.