acceptodds
Under review as a conference paper at ICLR 2027

Tied, Not Won: Testing Claims That One Model Scales Better Than Another

Abstract

Many papers conclude that one architecture, training objective, or pretraining corpus "scales better" than another. Because a scaling rate is the exponent of a fitted power law, such a conclusion is a claim about the difference between two exponents. In practice it is drawn from a handful of model sizes, and neither the exponents nor their difference come with a standard error. We supply the missing statistics. Using only the per-scale results that the original papers released, we refit each power law, check that the refit reproduces the published exponent, and test whether the gap between two exponents is distinguishable from zero. Across widely cited comparisons from eight released model suites, most claimed differences in scaling rate are statistical ties. The conclusion that the vanilla Transformer scales best among ten architectures does not hold against its close contenders, and nearly all pairs of pretraining corpora in the DataDecide suite differ in loss level but not in rate. The probability that the claimed winner would reverse, computed from the same test, separates these ties from large, well-established differences. A closed-form bound on the smallest detectable gap explains the pattern: the small sets of model sizes in common use cannot resolve gaps of the size being claimed. A survey of recent scaling papers further shows that most do not release the data needed to check their own comparisons. We close with simple reporting practices that make comparative scaling claims testable.

Primary area:
foundation or frontier models, including LLMs
Keywords:
scaling laws, statistical testing, reproducibility, meta-science, experimental design, power analysis, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TCm6OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work examines comparative scaling claims through a statistical lens. Specifically, it considers previous works that make claims of the type "A scales better than B" on the basis of fitting a Chinchilla-form power law to scaling and comparing exponents in the resulting fit. For some of the considered works, the authors refit the scaling laws, based on the provided data points, then calculate the standard errors on the resulting fit parameters, including the exponents. They report the extent to which confidence intervals overlap, and the likelihood of rejecting the null hypothesis (being able to distinguish alpha_A from alpha_B) and of the reported directionality (alpha_A > alpha_B) being correct. They also calculate the theoretical detection floor for a particular gap given the setup (model size range, number of independent points, etc). Finally, the authors include recommendations for future comparative scaling law works to allow for responsible auditing.

Critical strengths

S1. The unreliability of scaling laws is now well-established, but the specific perspective addressed by this work has not been investigated in any peer-reviewed work to my knowledge. Such audits, especially ones targeting influential works that are positioned to change the best practices of the entire field, are increasingly crucial as open-science is threatened by the closed nature of most large-scale labs. The arguments made by this work are relatively straightforward and simple to understand even for researchers with minimal undergraduate-level statistics.

Critical weaknesses

W1. For this work to really achieve impact, I think there should be some more consideration of the practical implications of scaling law comparisons. While theoretical floors and confidence intervals are important information to consider, empirical research work does not generally follow directly from statistical theory. I would like to see, e.g. potential recommendations of action available to an author who finds that their scaling law comparison is limited in statistical power. Please see Q2 for some examples of questions around discussions I would like to see.

Questions for authors

Q1. The Chinchilla scaling law form includes an exponent on both the parameters N and data budget D. Your discussion appears to target the exponent on N only (alpha). Do you consider beta in your work? And if not, how would your discussion change if you incorporate B?

Q2. From W1 – How can an author of a scaling law comparison work appropriately adjust their setup if they reach the end of experimentation and find overly-wide or heavily overlapping confidence intervals? How do we as a field deal with the common criticism that a scaling law work is "too small scale" and the pressure to add a single "hero run" at the cost of more statistical power that might be achieved by adding more small-scale experiments? The underlying goal of many scaling law comparison works is not so much about exponents in an infinite-scaling regime, as it is about a target scale in the 10E25 to 10E30 FLOP range – how should this work be interpreted if the goal is shifted from the absolute value of alphas to the goal of correctly predicting L_A vs. L_B at a target scale?

Reviewer_cUGeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper finds comparisons of scaling exponents in published literature and tests wether these difference (of exponents) are statistically indistinguishable. It also derives a minimum detectable gap for this difference and finds that most audited comparisons in released results are not resolved thus recommends reporting of uncertainty or confidence with these scaling coefficients.

Critical strengths
  1. The question is important and requires research - as models are getting bigger, and it's crucial to distinguish if the performance is good at a fixed scale or a better scaling rate.
  2. The method does not require retraining, which is a very practical approach to this problem. External auditors can use such methods to verify released results.
Critical weaknesses
  1. The given justification for the given uncertainty estimates is not close to being enough. For ex. residual scatter around a fit line can be curvature rather than random noise. The fit test also ignores covariance between methods (for example two curves can behave similarly at small scales, thus those cant be trated as independent uncertainty). Independence analysis needs ablations.
  2. The study has several contradictions. The text claims that refits repoduce the published numbers and preserve ranking however Universal changes from 0.50 to 0.83 and ranks above transformer - this is called a "mis-extraction" in Table 7 0 but is maintained in the paper. The numbers for detection floor also appear inconsistent. For transformer GLU, the reported SE gap is 0.0947 but 80% power floor is 0.145. Via calculation this should be 0.265 before small sample correction. This may have some explanation but to the reviewer it seems incorrect.
  3. The main claim, or the headline is overstating. Many comparisons depend on dataset ranking or training without identifying an original claim of superior rate. Additionally "differ in level not in rate" requires some evidence of equivalence within a margin - non significance does not establish the claimed.
  4. The interpretation said for reversal probability is not clear. Fitting sloped around their estimated does not (by itself) establish that the Probability that the true ordering is wrong or if the future experiment will reverse it. These are different quantities - so terminology should match the quantity.
Questions for authors

My main concerns are with the assumptions

  1. Do the main conclusions still exist with joint fits or pair fits. Can authors provide bootstrap results?
  2. Can the authors verify the Tay refits with published numbers and resolve the flagges issue? Also explain discrepency between reported SE and floor?
  3. Can authors confirm which audited comparisions have explicit published claims of better scaling?
Reviewer_cRoMOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines whether published claims that one method scales better than another are supported by uncertainty estimates on differences between fitted scaling exponents. It reanalyzes 29 comparisons from eight sources, reporting 22 unresolved differences, six separations, and one ambiguous comparison. It also proposes a reversal-probability summary, derives minimum detectable exponent differences, and surveys the availability of data needed to audit scaling claims. The central message is that comparative scaling claims should include uncertainty estimates and account for statistical power.

Critical strengths
  1. The distinction between performance at a particular scale and the rate of improvement with scale is useful and relevant to experimental design.

  2. Reanalyzing released results is a valuable approach that can improve scientific practice without requiring expensive new training runs.

  3. The recommendations to release per-scale measurements and report uncertainty on exponent differences are practical and well motivated.

  4. The conclusion appropriately acknowledges that an unresolved difference does not establish equality of scaling rates.

Critical weaknesses

Post-rebuttal update, 15 November 2026: I retain 1: Clear rejection. I appreciate the authors' engagement, but the available manuscript does not yet resolve the substantive concerns about the numerical reanalysis, detection-floor calculation, and interpretation of uncertainty. My detailed assessment is provided in the discussion with the authors and committee. The original concerns below are retained for context.

  1. Inconsistencies between original and refitted exponents undermine the principal case study.

Sections 3.1 and 4.1 describe reproducing the original exponents as a prerequisite for testing their differences. However, Tables 2 and 3 report substantial discrepancies: Universal Transformer changes from 0.50 to 0.833, Funnel from 0.47 to 0.596, and Performer from 0.25 to 0.487. Section 4.2 states that the refit preserves the ranking, but then reports that Universal and Funnel overtake the Transformer. Table 7 also describes the Universal result as “mis-extracted,” although it remains in other reported results.

These discrepancies could reflect differences in point selection or fitting procedures, rather than arithmetic errors, but they must be resolved. In particular, Table 2 combines published point estimates with standard errors obtained from different refits. The validity of those standard errors for the original estimators is not established.

  1. The detection-floor example appears inconsistent with the reported standard error.

For Transformer versus GLU, Section 3.3 reports a gap of approximately 0.0454 and a standard error of 0.0947. The usual normal approximation for 80% power at a two-sided 5% significance level gives a minimum detectable difference of approximately (1.96 + 0.842) × 0.0947 = 0.265, whereas Table 7 reports 0.145.

Equation 5 uses a natural-log scale through W = R ln(10), while the Tay exponents are slopes against log10 FLOPs. This suggests a possible mismatch in units. The gap, standard error, and detection floor must be expressed in consistent coordinates. Resolving this discrepancy is necessary to assess the quantitative experimental-design recommendations.

  1. The uncertainty model requires stronger justification.

Equation 1 assumes independent slope estimates, while standard errors are estimated from residual scatter around each fitted curve. This scatter can include shared curvature and systematic changes across scale, rather than only independent stochastic variation. In general, the variance of the slope difference also includes a covariance term.

Although the paper discusses pairing theoretically, it does not establish that the headline findings are robust to an appropriate joint analysis, particularly for DataDecide’s common size grid. Residual diagnostics and sensitivity to shared structure and functional-form assumptions would help determine whether the reported uncertainty measures the intended quantity.

  1. The interpretation of reversal probability is insufficiently justified.

Section 3.2 samples slopes from Student-t distributions centered on the estimates and interprets the resulting sign frequency as the probability that the claimed ordering is wrong. A frequentist sampling distribution does not, by itself, assign probabilities to the true ordering of fixed parameters. A posterior interpretation would require an explicit probabilistic model and prior assumptions. A probability of reversal in a future experiment is a different quantity again.

The paper should specify its intended interpretation and justify the calculation accordingly. Section 4.4 correctly acknowledges that separating ties from separations using the same underlying evidence is a consistency check; it does not independently validate the probability interpretation.

Questions for authors
  1. What explains the discrepancies between the original and refitted Tay exponents? Which exact points and fitting procedures underlie each estimate, and why are the refit standard errors appropriate for the published estimates in Table 2?

  2. Can the authors reconcile the Transformer–GLU detection floor of 0.145 with the reported gap standard error of 0.0947, explicitly tracking logarithm conventions and units?

  3. How do the principal results change under a joint or paired analysis that accounts for covariance and shared departures from the fitted scaling law?

  4. Does reversal probability represent posterior uncertainty about the true ordering, a prediction about a repeated experiment, or a confidence-distribution summary? What assumptions justify that interpretation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.