- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper studies whether cross-model behavioural comparisons remain defensible when elicitation designs change. It audits published benchmarks and introduces a design-relative framework distinguishing uniform dominance, mean ordering and single-design claims. The framework includes nuisance averaging, framing-dependent intervals, a breakdown-tilt measure, and a procedure classifying comparisons as certified, noise-limited, or design-limited. Experiments cross models, tasks, elicitation designs, and repeated responses, with additional item-budget, judge, and frontier-model analyses. The central contribution is a framework intended to distinguish genuine design sensitivity from instability caused by finite sampling.
Critical strengths
-
The paper usefully separates claims and estimands. Distinguishing single-design performance, a specified average, and dominance across all enumerated designs makes explicit what a leaderboard comparison does and does not establish. The breakdown-tilt measure provides an interpretable sensitivity summary instead of another aggregate robustness score.
-
The experiments make a substantial effort to distinguish sensitivity from measurement noise. The crossed design, repeated observations, item-splitting analysis and larger-item follow-up go well beyond merely demonstrating prompt-induced rank changes. The paper also appropriately acknowledges that the published record often lacks enough information to attribute reversals to design.
Critical weaknesses
-
The 'designed-limits' verdict is not sufficiently established by the stated tests (Section 4.3 and Appendices B-C). Rejection of the permutation null provides evidence against design exchangeability, but does not by itself establish that the true pairwise contrast changes sign across designs. A pair could have mostly very small positive contrasts and a few large positive contrasts: design dependence may be detectable, noisy estimates may reverse near zero, and the overall mean gap may exceed 0.04, though the true ordering remains uniform. The paper itself notes that DIS and the split-sample floor are different functionals and that floor conservativeness requires per-design separartion. Restricting the average gap doe not supply that condition. Consequently, the inference that additional sampling cannot resolve a comparison labelled design-limited needs stronger justification. Calibration under a design-invariant null does not settle the heterogenous, same-sign case.
-
The mapping from published claims to uniform dominance is sometimes too strong (Section 4.2 and Appendix F). For example, a statement that some models favor their developers' products does not inherently assert strict pairwise dominance under every permissible elicitation. The conclusion that seven of eight audited papers require L1 therefore depends on an interpretation that needs more explicit justification. This matters because the paper's criticism of existing conclusions is stronger than its demonstration of sensitivity: showing that a result changes across designs does not automatically establish that the original paper claimed invariance across those designs.
Questions for authors
-
How does the full triage procedure behave under heterogeneous but strictly same-sign true contrasts, including many near-zero contrasts? In particular, what is the false 'design-limited' classification rate in that setting? Can the existing data support both a positive and a negative true contrast, with appropriate uncertainty assessment, for the comparisons labelled design-limited?
-
Can you distinguish genuinely design-uniform claims in the audited papers from average, prevalence, or explicitly conditional claims, and explain how this distinction affects the seven-of-eight conclusion?