acceptodds
Withdrawn from ICLR 2027

The Structural Content of a Transfer Relation: A Benchmark's Labelled Dimensions Predict Less About Each Other as They Are Split Further

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A benchmark that reports P labelled dimensions side by side invites P(P-1) ordered transfer readings — a system ahead on this dimension is ahead on that one — and suites are increasingly refined into more dimensions on a fixed item budget. What refinement does to those readings has never been measured: the pair count grows quadratically while each pair is measured on fewer items, so refinement multiplies the claims and divides the evidence. The relation's apparent size is confounded too, since comparing whatever systems are to hand mostly measures a base rate rather than transfer. On eight public leaderboard corpora and over a thousand models — MMLU, BIG-Bench Hard and six more, using no artifact of our own — we split a relation's description length exactly into a base-rate term and a structural term, isolate the second by matching ability, and send the dimension count to infinity at a pinned item budget. Out of sample, label-predictability runs from 6.0% on MMLU to 53.1% across one leaderboard's own benchmarks, so there is no single number for what a benchmark's labels predict; refinement then drives it down as a power law with exponent -1.223 in the number of splits, against an analytic no-structure baseline of -1, and read backwards on MMLU's taxonomy it falls from 9.98% to 6.02% as the dimension count goes from 17 to 57, while pair heterogeneity rises. A minimum-description-length statistic shows the structural term is real on all eight corpora nonetheless, paying 11.58 times its cost on MMLU against 0.82 times for a permutation null. The refinement limit has a closed form, and so a design rule computable before any model is run: a suite needs at least 7 dimensions to report a label-predictability of 6% at all, and at least 11 to report 1%.

Primary area:
datasets and benchmarks
Keywords:
benchmark evaluation, dimension tables, transfer relations, minimum description length, item-response theory, evaluation methodology, pre-registration, benchmark meta-evaluation, MMLU, BIG-Bench Hard

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_r2EfOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how benchmark granularity affects the predictability of relative model performance. Bad expression, disjoint logic and poorly presented figures indicate low overall quality and raise strong suspicions that this paper may be the product of AI automated research.

Critical strengths

None.

Critical weaknesses
  1. The manuscript contains many vague expressions, such as “What this paper is not, and exactly what it shares” (page 2, line 100), “The shared propositions travel in that direction only, and neither paper’s evidence is evidence for the other’s claim” (page 2, line 107), and “The shortest true summary” at the beginning of the conclusion (page 9, line 443). The language is broad but imprecise, and the narrative logic is highly disjointed. These issues make the manuscript very difficult to follow.

  2. The figures are poorly presented. In Figure 1, many symbols are introduced without explanation, and the caption does not clarify the figure’s intended message, making it hard to understand. In Figure 2, the legend appears to be duplicated, and the meaning of the dashed line is not explained. In Figure 3, the “mmlu” label in the left panel extends outside the plotting area, while the tick spacing in the right panel also appears unusual, which is also not explained.

  3. The layout of the tables and figures in the appendix is disorganized: for example, the horizontal axis label overlaps with the legend in Figure 4, and Table 8 extends beyond the normal width. Taken together, these issues indicate poor overall quality and raise strong suspicions that the paper may be the product of AI automated research.

Questions for authors

See Critical Weaknesses above.

Reviewer_ty2MOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies how much structural information benchmark labels provide about cross-dimensional comparative performance. It distinguishes pair heterogeneity, obtained from an entropy, obtained from the prediction error of a learned partition. Using public benchmark matrices, the authors show that naïve cross-dimensional agreement is dominated by general model ability; after ability matching, residual pair heterogeneity remains optimal. The paper further studies refinement of MMLU at fixed item budget, both using natural hierarchy and by randomly splitting subject items, finding decreasing measurable Lambda with increasing granularity.

Critical strengths

The distinction between pair heterogeneity and label-predictability is useful and, in my view, the paper's strongest conceptual contribution. The general-ability confound is demonstrated clearly.

Critical weaknesses

My main concern is construct validity. The paper repeatedly describes its observational comparative-concordance relation as transfer, although no intervention on one dimension is performed. This makes some conclusions sound causal when the measured object is associational. Second, the obligation relies on the contestable assumption that merely publishing dimensions implicitly asserts every ordered cross-dimensional generalization. The authors acknowledge this assumption, but several headline claims present it more strongly. Third, most of the forward-refinement decline appears to be the expected statistical consequence of dividing a fixed item budget. It is unclear whether the difference is statistically meaningful. The natural MMLU hierarchy supports the same direction but provides only two resolvable granularities. I therefore find the general refinement law interpretation stronger than the current evidence warrants.

Finally, I would like a clearer treatment of effective sample size. Millions of pairwise comparisons are generated from hundreds of models, so model-pair observations are strongly dependent. A model-level bootstrap and discussion of how this affects the MDL break-even calculation would substantially strengthen the paper.

Questions for authors
  1. Is the fitted refinement exponent statistically distinguishable from the sampling-noise baseline?
  2. How does dependence among model-pair comparisons affect the effective sample size and the reported MDL payoff?
  3. How robust are D and Lambda to the matching tolerance, SE cutoff, and median threshold?
  4. What notion of transfer is supported by observational cross-dimension concordance rather than an intervention?
  5. Is the obligation a mathematical claim or an interpretive assumption about benchmark reporting?
  6. What evidence supports generalizing the refinement result beyond MMLU?
Reviewer_HMgNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether increasingly fine-grained benchmark dimensions provide reliable information about the relationships between model performance across different tasks.

Critical strengths
  1. The problem studied in this paper is relevant and important.
Critical weaknesses
  1. The paper does not clearly articulate its central research questions or the significance of the problem being investigated. Considerable effort is required to understand the objectives, key claims, and intended contributions. The Introduction provides insufficient motivation and limited engagement with closely related literature, making it difficult to identify the specific research gap and understand how this work advances existing knowledge.
  2. The paper should precisely define what constitutes "transfer" between benchmark dimensions and explain how it differs from statistical correlation or agreement in model rankings. More importantly, the authors need to justify why such cross-dimension transfer is meaningful, what it reveals about model capabilities, and why measuring it matters for benchmark evaluation or design.
  3. The theoretical analysis relies heavily on counting (P(P-1)) ordered transfer relations and uses this assumption to derive information-theoretic bounds and practical design rules. However, the transfer measure defined in Section 4 is based on agreement in relative model performance across benchmark dimensions. This agreement appears symmetric: the relationship between dimensions A and B is identical to that between B and A. Moreover, these pairwise relationships are derived from the same underlying model-performance data and are not necessarily statistically independent. The paper should clarify why the transfer relationships are treated as directional information units, justify the assumptions underlying the counting and entropy arguments, and examine how dependencies between dimensions affect the validity of the theoretical conclusions.
  4. The experimental design is very difficult to follow. Section 4 only presents a brief description. The paper does not justify model selection, thresholding.
Questions for authors
  1. Can you add an example to show why the "transfer" notion is meaningful, and show what it can reveal about model capabilities.
  2. Can you provide a clearer, reproducible experimental protocol and investigate how sensitive the conclusions are to these choices.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.