- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This work examines comparative scaling claims through a statistical lens. Specifically, it considers previous works that make claims of the type "A scales better than B" on the basis of fitting a Chinchilla-form power law to scaling and comparing exponents in the resulting fit. For some of the considered works, the authors refit the scaling laws, based on the provided data points, then calculate the standard errors on the resulting fit parameters, including the exponents. They report the extent to which confidence intervals overlap, and the likelihood of rejecting the null hypothesis (being able to distinguish alpha_A from alpha_B) and of the reported directionality (alpha_A > alpha_B) being correct. They also calculate the theoretical detection floor for a particular gap given the setup (model size range, number of independent points, etc). Finally, the authors include recommendations for future comparative scaling law works to allow for responsible auditing.
Critical strengths
S1. The unreliability of scaling laws is now well-established, but the specific perspective addressed by this work has not been investigated in any peer-reviewed work to my knowledge. Such audits, especially ones targeting influential works that are positioned to change the best practices of the entire field, are increasingly crucial as open-science is threatened by the closed nature of most large-scale labs. The arguments made by this work are relatively straightforward and simple to understand even for researchers with minimal undergraduate-level statistics.
Critical weaknesses
W1. For this work to really achieve impact, I think there should be some more consideration of the practical implications of scaling law comparisons. While theoretical floors and confidence intervals are important information to consider, empirical research work does not generally follow directly from statistical theory. I would like to see, e.g. potential recommendations of action available to an author who finds that their scaling law comparison is limited in statistical power. Please see Q2 for some examples of questions around discussions I would like to see.
Questions for authors
Q1. The Chinchilla scaling law form includes an exponent on both the parameters N and data budget D. Your discussion appears to target the exponent on N only (alpha). Do you consider beta in your work? And if not, how would your discussion change if you incorporate B?
Q2. From W1 – How can an author of a scaling law comparison work appropriately adjust their setup if they reach the end of experimentation and find overly-wide or heavily overlapping confidence intervals? How do we as a field deal with the common criticism that a scaling law work is "too small scale" and the pressure to add a single "hero run" at the cost of more statistical power that might be achieved by adding more small-scale experiments? The underlying goal of many scaling law comparison works is not so much about exponents in an infinite-scaling regime, as it is about a target scale in the 10E25 to 10E30 FLOP range – how should this work be interpreted if the goal is shifted from the absolute value of alphas to the goal of correctly predicting L_A vs. L_B at a target scale?