acceptodds
Under review as a conference paper at ICLR 2027

The Structural Content of a Transfer Relation: A Benchmark's Labelled Dimensions Predict Less About Each Other as They Are Split Further

Abstract

A benchmark that reports P labelled dimensions side by side invites P(P-1) ordered transfer readings — a system ahead on this dimension is ahead on that one — and suites are increasingly refined into more dimensions on a fixed item budget. What refinement does to those readings has never been measured: the pair count grows quadratically while each pair is measured on fewer items, so refinement multiplies the claims and divides the evidence. The relation's apparent size is confounded too, since comparing whatever systems are to hand mostly measures a base rate rather than transfer. On eight public leaderboard corpora and over a thousand models — MMLU, BIG-Bench Hard and six more, using no artifact of our own — we split a relation's description length exactly into a base-rate term and a structural term, isolate the second by matching ability, and send the dimension count to infinity at a pinned item budget. Out of sample, label-predictability runs from 6.0% on MMLU to 53.1% across one leaderboard's own benchmarks, so there is no single number for what a benchmark's labels predict; refinement then drives it down as a power law with exponent -1.223 in the number of splits, against an analytic no-structure baseline of -1, and read backwards on MMLU's taxonomy it falls from 9.98% to 6.02% as the dimension count goes from 17 to 57, while pair heterogeneity rises. A minimum-description-length statistic shows the structural term is real on all eight corpora nonetheless, paying 11.58 times its cost on MMLU against 0.82 times for a permutation null. The refinement limit has a closed form, and so a design rule computable before any model is run: a suite needs at least 7 dimensions to report a label-predictability of 6% at all, and at least 11 to report 1%.

Then back it, or bet against it.

Related papers

Open the market on this paper to see 7 more related papers.