Design, Not Noise: Certifying Cross-Model Claims in Black-Box Behavioural Audits of Language Models

Behavioural benchmarks (economic games, sycophancy, dark patterns) report a leaderboard: one number per model, under one elicitation design chosen by the authors. We show that the published record cannot adjudicate its own leaderboards. Across 28 fully crossed model$\times$design tables from the six of eight published behavioural benchmarks that print one, more than one model attains rank 1 in 70.6% of the 17 tables whose crossed facet the benchmark itself calls irrelevant, and 25.8% of their 612 pairwise orderings reverse on one of the papers' own variants, rising to 55.0% of the 60 that vary a prompt paraphrase (all from GAMA-Bench, the only benchmark publishing one). No crossed table publishes repetitions, so no reversal can be attributed to design rather than noise. We supply the missing instrument: a design-relative estimand with an identified interval, a three-level certificate hierarchy with valid tests, a closed-form breakdown tilt, and a replicate floor that triages every claim as certified, noise-limited or design-limited. Run over 9 models $\times$ 4 domains $\times$ 96 crossed designs, each domain yields 73–94 distinct full leaderboards and uniform dominance certifies 0–8% of pairs; seven 2026-frontier models are no more stable (8 of 84 pairs certified) and show no detectable capability attenuation ($+0.023$, model-clustered se $0.050$). Decidability is an item budget: taking the worst-served domain from 16 items to 96 drops its noise floor $0.294\to 0.129$ while the design signal holds, and its undecidable claims from 26 of 36 to 10. We release the corpus, the toolkit and a reporting standard: a point estimate and an identified interval per model, a certificate per claim. All code, data and raw responses accompany this submission as supplementary material and will be released at https://github.com/anonymous-iclr2027/design-relative-behaviour upon acceptance.