acceptodds
Under review as a conference paper at ICLR 2027

How Large a Break Can Hide in a Scaling Law?

Abstract

Scaling laws guide extrapolation from a small set of training runs, but a smooth fit does not establish that the exponent is stable. We study the detection of a continuous change in exponent when the breakpoint is unknown. For designs with 4–20 model sizes, comparing a scanned-breakpoint statistic with a fixed-break F distribution gives false-positive rates of 18.1–32.0% at a nominal 5% level. Simulating the complete scan under the null restores rates to 5.0–5.3%. We characterize the test’s invariance to the baseline exponent and noise scale, and quantify detection power over sample size, noise, and break location. Across two decades of model size, designs with 6–8 observations and 2–5% relative noise require exponent changes of 0.1–0.4 on the tested grid to reach 80% power. Power decreases when the break moves toward either end of the range. A six-model Pythia case study illustrates a separate issue: adding a loss floor reduces log-scale residual sum of squares by a factor of 53, showing how smooth curvature can produce a strong apparent break under a pure power law. We provide a reproducible calibration and power-analysis procedure, and recommend reporting detectable changes alongside fitted exponents.

Primary area:
learning theory
Keywords:
scaling laws, change-point detection, breakpoint testing, statistical power, hypothesis testing, model scaling, Pythia

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_HcpNOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether small scaling studies are able to detect any change in exponent at some unknown breakpoint. It also calibrates the breakpoint search through simulation and validates detection power across different sample sizes, noise levels, and break locations. It uses an example to show how curvature from a loss floor can produce a break in a pure power law.

Critical strengths

Broadly, the paper is well presented and easy to follow

  1. Since a good fit over a few model sizes is not enough to establish that the exponent is stable, the question does have relevance.
  2. The presented analysis in the paper is clear, and within the stated model, the calibration is also justified. The paper is also able to correctly distinguish detection power from a confidence bound on the exponent change.
Critical weaknesses

My main concerns are regarding the novelty and some claims not substantiated in the paper

  1. Taking into account the breakpoint for breakpoint search and calibrating the procedures seem to be fairly established approaches. Related work on Monte Carlo testing for join point regression, such as Kim et al. (2000) [1], is not discussed in the paper. It is not clear to me what the contribution is beyond applying existing methods to selected scaling designs.
  2. The changes which are detectable depend a lot on the chosen test, but there is no clean ablation, and no alternate tests are compared. For example, in a simple calculation with 6 sizes, 2% noise, and an exponent change of 0.1, power is about 62% for the scan statistic versus 75% for an average fixed-break F statistic. If the reported threshold describes this particular procedure, than the detection limit is on the design. Can authors explain this? 3)The practical analysis is not complete for the paper's said example. For example, adding a flow reduces Pythia's residual sum of squares by around 53 times, showing that the pure power-law null is not appropriate. While the authors acknowledge that their power tables do not apply to this curve, it is not clear why, and it is a weakness of the paper - can authors explain?
  3. Every conclusion depends on independent Gaussian log errors with constant variance, and there is no ablation to check sensitivity to these assumptions.
  4. Several crucial numerical choices are not explored sufficiently. For example, the effect size grid is coarse, so reported detectable changes can differ by a factor of, let's say, 2 between adjacencies. Also, the 4-point result depends on nearly interpolating scan with a critical value around 5.7 × 10^5. The paper does acknowledge grid sensitivity but is not evaluating which limits how broadly these results can guide study design.

[1] Kim, H.-J., Fay, M. P., Feuer, E. J., & Midthune, D. N. Permutation tests for joinpoint regression with applications to cancer rates. Statistics in Medicine, 19(3):335–351, 2000.

Questions for authors

Please look at the weaknesses for some questions. The rest are below.

  1. What is the proposed procedure adding beyond the existing Collaborated Breakpoint tests?
  2. Is there any evidence that scaling studies use fixed break F reference?
  3. See W2 and W5: What motivates the chosen statistic and the scan grid?
  4. See W3: What changes are required to test an exponent break under the floor model?
  5. See W 4: How should a practitioner choose sigma when evaluating uncertainty, variation in training etc can all contribute to residual scatter?
5 comments
AuthorsOpenReview ↗

response to Reviewer HcpN (part 1/2)

(1/2)

Thank you for the detailed review, in particular for pointing us to Kim et al. (2000) and for asking which assumptions the conclusions depend on. Both points led to new analyses in the revision.

W1 and Q1 (what this adds beyond the established joinpoint test). Kim et al. (2000) is now cited and included in a head-to-head comparison. We put four procedures on the same designs, each at its own calibrated critical value so the comparison is at equal size: the scanned maximum, ave-FF and exp-FF (Andrews–Ploberger), and the scanned maximum with Kim's per-dataset permutation calibration (B=199B=199), over 20,000 simulations and 800 permutation datasets. Under Gaussian noise of constant variance all four hold their level. When the noise scale grows along the design, which is what happens when larger runs are evaluated on fewer tokens, the scanned test's size rises from 0.050 at n=4n=4 to 0.136 at n=20n=20, and Kim's permutation calibration rises to 0.128, because the residuals of the null fit are not exchangeable once their scale varies with model size. The same holds for the functional form (below): residual permutation calibrates the search but still tests the pure-power null. Thus, while Kim et al.'s permutation test and our Gaussian scan are equivalent under homoscedastic pure-power nulls, design-level simulation is necessary once noise varies with scale or a loss floor is present.

Q2 (whether scaling studies use a fixed-break FF reference). The fixed-break reference is what a practitioner obtains by default when a hinge column is added at an estimated breakpoint and the fit is read from an ordinary least-squares routine, which reports the tt/FF p-value as if the breakpoint had been fixed in advance. Published broken-scaling analyses (e.g. Caballero et al., 2023) estimate break locations as free parameters of a smoothly broken power law and compare fit quality, without a calibrated test of whether a break is present. Table 1 quantifies the Type-I error inflation of the naive reference (18.4–32.0% at a nominal 5%), and Section 6 compares against properly calibrated joinpoint tests (Kim et al., 2000; Andrews and Ploberger, 1994).

W2 and Q3 (alternative statistics, the ablation, and the scan grid). Your calculation agrees with ours, and the comparison is now in the paper. At equal size under Gaussian noise, ave-FF dominates the scanned maximum at the smallest detectable changes, by 5–10 points of power at δ=0.05\delta=0.05 for n≥6n\ge6 (for example n=8n=8: 0.48 versus 0.40). We therefore report calibrated thresholds for sup-FF, ave-FF and exp-FF, recommend ave-FF, and state that a reported detectable change belongs to the statistic that produced it (Section 6). The scan grid of 60 equally spaced candidates between the second and the second-to-last design points keeps the hinge inside the design, so the segmented model is identified at every candidate; because the calibration simulates the same grid, the reported size holds for whichever grid is used.

Continued in part 2/2 below.

AuthorsOpenReview ↗

response to Reviewer HcpN (part 2/2)

(2/2)

W4 and Q5 (non-Gaussian and non-constant-variance errors, and the choice of σ\sigma). Table 3 also gives size under Student-t5t_5 errors (0.052–0.068 for the scan) and under the growing-variance model above. Section 9 now answers how a practitioner should choose σ\sigma: state whether the noise scale is constant along the design, distinguish evaluation uncertainty from run-to-run variation, and calibrate under the structure assumed. The invariance of Proposition 1 holds for any constant σ>0\sigma>0 but not for a σ\sigma that varies with NN.

W3 and Q4 (the Pythia analysis, and why the power tables did not apply). Section 7 now carries out the full size-and-power analysis under the loss-floor model. We simulate floor-only curves and apply two tests. The pure-power scan at its own calibrated critical value rejects floor-only data 0.14–0.87 of the time, and the rate rises with the number of model sizes (0.36 at six, 0.87 at twelve, at 2% noise), as additional sizes increase power against genuine floor-induced curvature. The same scan calibrated inside the floor family (the null fits (E,A,α)(E,A,\alpha), the alternative adds a hinge, and the critical value is simulated under the floor null on the same design) returns the level to 0.040–0.063. Floor-aware power at twelve sizes and 2% noise is 0.39 at δ=0.2\delta=0.2 and 0.88 at δ=0.3\delta=0.3 (Table 4, Figure 3b). These are the numbers that apply to a curve such as Pythia, where a floor is present; at its six sizes the floor-aware detectable change at 2% noise exceeds 0.4.

W5 (grid resolution and the four-point case). Table 2 keeps the submitted effect-size grid so that it can be compared cell by cell with the submission, and the floor analysis uses the finer grid $\delta\in\lbrace 0.05,0.1,0.15,0.2,0.3,0.4\rbrace $. The four-point result is now stated explicitly: at n=4n=4 no tested change reaches 80% power and the largest power we measure at any tested change is 0.11. The large critical value at n=4n=4 follows from the design: with three fitted parameters and a scanned breakpoint, the segmented fit nearly interpolates four points.

On reproduction. Every table in the revision is regenerated by the released code. The submitted Tables 1 and 2 reproduce within Monte Carlo error from code/repro_main.py (for example the fixed-break rates 0.320/0.184/0.192/0.210/0.234 against the submitted 0.320/0.181/0.188/0.216/0.242), and the Pythia fit reproduces at α^=0.177\hat\alpha=0.177, E^=2.345\hat E=2.345, with the residual sum of squares falling by a factor of 51.7.

Thank you again for pushing us to test heteroscedasticity, residual permutation, and floor calibration directly; we hope these new results address your concerns.

Thank you again for your detailed review. Since our replies of 13 November, the revision uploaded on 17 November adds two results that bear on your W3 and Q4, and we summarise them here in case they are useful before the discussion closes.

Section 8 now calibrates the floor-aware scan on the actual Pythia design: the six measured, unequally spaced sizes and the fitted floor, which is about 90% of the loss at the largest size, at four noise levels (new Table 6). On this design the test holds its level (0.038–0.053), the measured curve has calibrated pp-values of 0.09–0.12 at every noise level, and a pure-power scan calibrated on the same design rejects 0.87–1.00 of floor-only curves. No exponent change up to 0.8 reaches 80% power, so this calibration replaces the figure in our earlier reply (a detectable change above 0.4), which had been inferred from the general floor designs of Table 4.

Section 9 now converts the detectable change into an error margin for the extrapolated law. For eight sizes over two decades at 2% noise, an undetected slowdown of 0.1 in the exponent leaves the actual loss one decade beyond the largest size 18% above the forecast, while the model-size exponent reported by Kaplan et al. (2020) predicts a 16% reduction over that decade.

The equal-size comparison with the permutation calibration of Kim et al. (2000) and with ave-FF and exp-FF under Gaussian, Student-tt and growing-variance noise remains in Section 6 and Table 3. If any of your concerns are still open, we would be glad to address them.

Reviewer_Ka1FOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper correctly shows that the scanned-breakpoint statistic requires scan-aware calibration under its specified Gaussian pure-power model. My independent simulations reproduce the Table 2 thresholds for n≥6n \ge 6 and closely match the Table 3 power estimates. For Table 1, I also reproduce the calibration pattern and numerical results within Monte Carlo error. The scan test itself uses standard change-point machinery, so the paper’s main contribution is a scaling-specific sensitivity analysis and reporting recommendation.

The Pythia case study shows that adding a loss floor can explain much of the observed curvature without requiring a sharp exponent break; it does not establish that no break is present, and the paper does not evaluate break detection under the floor model. In a separate simulation using the Pythia design, applying the pure-power calibrated cutoff to a floor-only curve flags nearly all instances as rejections. This is model-misspecification rejection under the paper’s pure-power test, not a Type-I error under the stated null. Median-statistic matching indicates a signal comparable to a hinge effect near the paper’s reported detection limit. This creates a serious interpretation risk if those limits are transferred without model-specific calibration, although it does not invalidate the paper’s conditional calibration result. Overall, the work is technically sound within its stated scope, but its practical value for empirical scaling-law analysis remains unestablished.

Critical strengths
  • S1. The scan-aware calibration is correct, consequential, and independently verifiable. Under the paper’s stated pure-power null, scanning over candidate breakpoints and then using a fixed-breakpoint F reference inflates the false-positive rate to 18–32% at a nominal 5%. Calibrating the complete scan by simulation restores the nominal level. I reran the stated protocol and reproduced Tables 1–3 within Monte Carlo error. This is a useful correction for breakpoint testing, and it is the main reason my recommendation is weak reject rather than reject.

  • S2. The finite-sample sensitivity analysis offers a useful conditional reference point. Tables 2–3 quantify the changes the scanned test can detect under the paper’s specified pure-power null and hinge alternatives, while Proposition 1 identifies when calibration is invariant to baseline parameters and noise scale. These results give readers a concrete sensitivity map for the tested settings and a starting point for future, model-specific calibrations.

Critical weaknesses
  • W1. The practical target of the sensitivity analysis lacks a direct empirical anchor. The paper’s scaling-specific sensitivity analysis concerns a sharp, continuous hinge in the log–log slope, but neither the paper nor the empirical examples it discusses provide direct evidence that this form of break occurs in real scaling curves. The Pythia case study shows that a loss floor can explain much of the observed curvature; it does not establish that no exponent break is present. The scan-calibration result remains useful, but the paper does not show how often the sharp-hinge target arises or whether reporting the proposed detection limits would change conclusions in an empirical scaling study. This leaves the practical significance of its sensitivity and reporting contributions uncertain, and is the main reason for my weak-reject recommendation.

  • W2. The reported detection limits have not been validated under the loss-floor model highlighted by the paper. Tables 2–3 and the abstract’s MDE figures are conditional on the pure-power model with iid Gaussian log errors. In my separate floor-model simulation, applying the pure-power-calibrated cutoff produces rejection in nearly all tested cases. This is not a Type I error under the paper’s stated null; it shows that rejection under a floor model cannot by itself identify a sharp exponent break. The paper correctly notes that a floor model requires its own calibration, but does not carry out a corresponding size-and-power analysis. Consequently, its MDE figures remain valid for the simulated model family, while their usefulness for interpreting floor-shaped empirical curves is unestablished.

Questions for authors
  • Q1. Section 2 defines the alternative as a continuous, sharp change in log–log slope, but the paper does not provide an empirical scaling curve where such a kink remains supported after accounting for a loss floor or other smooth curvature. Is the sharp hinge intended to represent an observed phenomenon in scaling-law practice, or is it a stylized alternative used to characterize test sensitivity? If it is empirically motivated, what evidence supports it? If it is stylized, what concrete decision in scaling practice is the proposed sensitivity-reporting procedure intended to change?

  • Q2. Sections 6–7 state that the Tables 2–3 thresholds are not confidence bounds for the Pythia curve and that a nonlinear floor model requires its own calibration. If the authors intend readers to transfer the abstract’s “0.1–0.4” limits to empirical curves where a loss floor is plausible, which result in the current submission supports that transfer? Do the authors have, or can they obtain, floor-aware size and power results for the same design—calibrating the scan under a floor-only null and measuring power against floor-plus-hinge alternatives? If such results are close to Tables 2–3, the concern about transfer would be substantially reduced; if they differ materially or do not exist, the headline figures need narrower interpretation.

5 comments
AuthorsOpenReview ↗

response to Reviewer Ka1F

Thank you for reproducing Tables 1–3 independently and for specifying the floor-aware analysis precisely. We ran that study, and it is now Section 7 of the paper.

W2 and Q2 (floor-aware size and power on the same designs). You asked whether calibrating the scan under a floor-only null and measuring power against floor-plus-hinge alternatives gives figures close to Tables 2–3. The null is ln⁡L=ln⁡(E+Ae−αx)\ln L=\ln(E+Ae^{-\alpha x}) with EE at half the loss at the largest size; the alternative adds a continuous hinge in the exponent; the null fits (E,A,α)(E,A,\alpha) and the alternative (E,A,α,δ)(E,A,\alpha,\delta) over a scanned breakpoint, with the critical value simulated under the floor null on the same design from 1,500 draws, evaluated on an independent set so that no reported size is the quantile of the sample that defined it. The results are in Table 4 and Figure 3b.

The pure-power-calibrated cutoff applied to floor-only curves rejects 0.14–0.87 of the time, in line with your own simulation. Rejection is higher at low noise and increases with the number of model sizes: 0.36 at six sizes, 0.64 at eight, 0.87 at twelve, at 2% noise. Additional model sizes increase power against curvature the floor genuinely produces, so more data does not protect against a mis-specified null.

Size is restored by floor-aware calibration. The same scan calibrated inside the floor family holds its level, 0.040–0.063 across the six cells against a nominal 0.05.

Minimum detectable changes are substantially larger. Under the floor-aware calibration, power at twelve sizes and 2% noise is 0.39 at δ=0.2\delta=0.2 and 0.88 at δ=0.3\delta=0.3; at 5% noise it reaches 0.52 only at δ=0.4\delta=0.4. These are materially larger than the pure-power values of Table 2, so the revision reports both and states which applies when.

By your own criterion this is the "differ materially" branch, and the paper now states the floor-aware figures as the ones to quote for a curve where a floor is plausible, rather than transferring the pure-power limits.

W1 and Q1 (is the sharp hinge observed, or a stylised alternative?). It is a stylised alternative used to characterise test sensitivity, and the revision now says so where the alternative is defined (Section 3). The sensitivity analysis serves the decision a scaling study makes when it extrapolates: having fitted an exponent on 6–8 sizes, what size of exponent change could its design have detected? That question is well posed whether or not any particular curve contains a kink, and the answer (0.1–0.4 under a pure-power null, larger under a floor, and no tested change at four sizes) is the number we propose be reported beside the fitted exponent.

On the Pythia curve specifically, the revision no longer stops at "a floor explains the curvature". At six sizes under the floor-aware calibration, the detectable exponent change at 2% noise exceeds 0.4. The curve is therefore explained by an unbroken form with an irreducible term, and its six-size design could not have detected an exponent change of 0.4 or less with 80% power.

Reproduction. Since you reran the protocol, the revision also regenerates the submitted tables from released code: code/repro_main.py returns fixed-break rates 0.320/0.184/0.192/0.210/0.234 against the submitted 0.320/0.181/0.188/0.216/0.242, calibrated rates 0.048–0.052, the same Table 2 cells, and the location pattern (at n=8n=8, 2% noise, δ=0.1\delta=0.1: 0.90 midpoint versus 0.65 at the quarter). The Pythia fit regenerates at α^=0.177\hat\alpha=0.177, E^=2.345\hat E=2.345 nats, α^floor=0.544\hat\alpha_{\text{floor}}=0.544, with the residual sum of squares falling by a factor of 51.7 (the submitted text said 52.8; the released fit, converged from a grid of starting points, gives 51.7 and the figure and text now use that value).

Heteroscedastic noise. The revision also adds a second axis the calibration does not cover: when the noise scale grows with model size, the scan's size rises to 0.136 at twenty sizes and the permutation calibration of Kim et al. (2000) to 0.128, while ave-FF stays at 0.076 and is also the more powerful statistic at equal size (Section 6, Table 3).

Thank you again for the constructive reproduction and floor-simulation suggestion, which substantially strengthened the paper.

Thank you for the response and the additional floor-aware experiments. These results address much of my original W2 concern. I have two remaining questions.

W1 (Practical significance). I understand that the sharp hinge is a stylized alternative. My main concern remains how the proposed sensitivity reporting informs actual scaling-law analysis. The test measures sensitivity to exponent changes within the observed range, but it is unclear how this helps with extrapolation beyond that range. Could the authors give a concrete example of how the reported MDE would affect the interpretation or use of a fitted scaling law?

W2 (Applicability to Pythia). Table 4 assumes a floor equal to 50% of the largest-model loss, while the fitted Pythia floor is approximately 90%. The Pythia model sizes are also unequally spaced, and the training-run noise is unknown. Was the detection limit stated in Section 8 calibrated using the actual Pythia design and fitted parameters, or inferred from Table 4? If the latter, the conclusion should be qualified accordingly.

The additional experiments improve the paper, but my concern about the practical significance of the sensitivity analysis remains.

Thank you for the follow-up. Both questions now have direct answers in the revised PDF.

W1 (how the detectable change informs extrapolation). The detectable change converts into an error margin for the extrapolated law, and the conversion needs no further simulation. An exponent change below the detectable one passes the test, and for a midpoint hinge of size δ\delta in Eq. (1) a single power law fitted to the observed sizes overstates ln⁡L\ln L at a target size xtx_t by δ [(xt−c)+−(a+bxt)]\delta\ [(x_t-c)_+-(a+bx_t)], where a+bxa+bx is the least-squares line through (xi−c)+(x_i-c)_+ on the observed design. The bracket is positive beyond the design, and the test detects δ\delta and −δ-\delta equally well, since TT is unchanged when MyMy changes sign. A concrete case: a study fits eight sizes over two decades at 2% noise, and the calibrated test finds no break. Table 2 gives its detectable change as 0.1. An undetected slowdown of that size (δ=−0.1\delta=-0.1) leaves the actual loss of a model ten times larger than the largest one 18% above the forecast, and of a model a hundred times larger 32% above it; with six sizes the margin one decade out is 38%. The model-size exponent of 0.076 reported by Kaplan et al. (2020) forecasts a 16% loss reduction over one decade, so a slowdown this design does not reliably detect offsets essentially the whole gain the extrapolation predicts. Reporting the detectable change therefore states how far, and with what error band, the fitted law can be used to size a larger run. This is now a paragraph of Section 9 ("Translate the detectable change into an extrapolation margin"), computed by code/mde_extrapolation.py.

W2 (applicability to Pythia). We have now calibrated the Section 8 statement on the actual Pythia design and fitted parameters, and the revised PDF replaces the earlier figure, which had been inferred from Table 4, with that calibration (new Table 6, code/pythia_design_calib.py). The null is the fitted floor model itself on the six measured, unequally spaced sizes, with the floor at 90% of the loss at the largest size, as you note; alternatives add a hinge in the exponent of the reducible term at the midpoint of the log-size range. Because training-run noise is unknown, Table 6 reports four noise levels: the evaluation standard error (0.46%), the residual scale of the floor fit (1.38%), 1% and 2%, with 2,000 null draws for each critical value, an independent 2,000 for each size, and 600 power draws per cell.

On this design the floor-aware test holds its level (0.038–0.053), and the observed floor-aware statistic on the Pythia curve, T=153.0T=153.0, has calibrated p=0.09p=0.09–0.120.12 at every noise level, so the data give no evidence of a break once the floor is modelled. The pure-power scan calibrated on the same design rejects 0.87–1.00 of floor-only curves, in agreement with your own simulation.

No tested change up to 0.8 reaches 80% power: the highest power, at the evaluation noise and δ=0.8\delta=0.8, is 0.47, and at 2% noise it is 0.18. The size of the floor is the reason: the exponent change acts on the reducible term, which is only about a tenth of the measured loss at the largest size. Section 8 now states the conclusion from this calibration: the curve is explained by an unbroken form with an irreducible term, and its design could not have detected an exponent change of 0.8 or less with 80% power. The earlier statement (above 0.4 at 2% noise) is consistent with this result and is replaced by it.

Thank you again for catching the mismatch between Table 4 and the Pythia floor share, and for prompting the explicit extrapolation-margin calculation.

Reviewer_25tHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper uses classical changepoint methods under Gaussianity to study an exponent change in scaling laws and calibrates the test by simulation.

Critical strengths

The problem is well defined, and the method is clearly presented.

Critical weaknesses

The paper provides limited contribution to statistical methodology, and its numerical study provides insufficient evidence of practical value.

Questions for authors

Can the authors further show benefits of using such a changepoint model for practical model validations or predictions?

2 comments
AuthorsOpenReview ↗

response to Reviewer 25tH

Thank you for your review. To demonstrate concrete practical value for model validation and experimental design, the revision adds three actionable findings.

Adding model sizes inflates false positives under misspecified noise or functional forms. On curves that have an irreducible loss and no break, a pure-power scan rejects 0.36 of the time at six model sizes and 0.87 at twelve (2% noise). The same pattern holds when the noise scale grows along the design: the scan's false-positive rate rises from 0.050 at four sizes to 0.136 at twenty. Adding runs is the usual response to a weak scaling study, and in both of these regimes it makes the conclusion less reliable unless the calibration matches the noise model and the functional form.

The established joinpoint procedure does not protect against either. The permutation calibration of Kim et al. (2000) also rises to 0.128 under growing noise, because the residuals of the null fit are not exchangeable once their scale varies with model size. What controls the level is a design-level simulation under the noise model and the functional form assumed, which is the procedure the paper supplies (Section 6, Table 3).

Four model sizes cannot detect an exponent break with this procedure. At n=4n=4 no tested exponent change reaches 80% power; the largest power we measure at any tested change is 0.11. A reader can act on this design conclusion before spending compute.

Rather than proposing a new test statistic, we show how noise heteroscedasticity and loss floors invalidate standard joinpoint permutation tests in scaling studies, provide calibrated procedures for both, and establish ave-FF as the recommended statistic: at equal size it is both more robust to growing noise (0.076 versus 0.136 at twenty sizes) and 5–10 points more powerful at the smallest detectable changes (Section 6).

For predictions and model validation specifically, Section 9 gives the reporting rule: state the functional form and noise structure, calibrate the complete procedure under them, and report the exponent change the design can detect alongside the fitted exponent. On the measured Pythia curve the rule changes the conclusion in two ways: the apparent break is explained by the missing irreducible term, and the six-size design could not have detected an exponent change of 0.4 or less with 80% power (Section 8).

Thank you again for your review. You asked whether the changepoint analysis brings benefits for practical model validation or prediction, and the revision uploaded on 17 November adds a direct answer for prediction.

Section 9 ("Translate the detectable change into an extrapolation margin") shows that the detectable change converts, in closed form, into an error band on the extrapolated loss. A study with eight sizes over two decades at 2% noise whose calibrated test finds no break can still carry an exponent slowdown of 0.1. That slowdown leaves the actual loss of a model ten times larger than the largest one 18% above the forecast, and of a model a hundred times larger 32% above it. The model-size exponent reported by Kaplan et al. (2020) forecasts a 16% reduction over one decade, so the change the design cannot detect is as large as the gain the extrapolation predicts, and reporting it states how far the fitted law can be used to size a larger run.

For model validation, Section 8 calibrates the test on the measured Pythia curve at its actual sizes (new Table 6). Once the loss floor is modelled the curve gives no evidence of a break (calibrated pp-values of 0.09–0.12), while a pure-power test on the same design reports a break on 0.87–1.00 of floor-only curves, which contain none. We would be glad to answer any further question before the discussion closes.

Reviewer_fTLSOpenReview ↗
Rating
3(+1)
Confidence
1(−1)
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Empirical scaling laws are useful in practice for building larger and more capable systems. This paper examines the inference process for estimation of the nature of a scaling law from data. The important idea is to examine how much power (i.e., how much observation) is required to distinguish a single scaling law from a mixture of scaling laws which exhibits a break from one law to another. This allows a number of improvements on the usual procedure, including some practical measures to estimate sensitivity of a scaling study in distinguishing different sorts of scaling curves.

Critical strengths

I am not a statistician, so I cannot really evaluate whether the derivations are novel or important. However, given my limited knowledge and perspective, this seems like an important problem, and this paper appears to tackle it head-on and come up with some interesting methods.

Critical weaknesses

Again, this is not my area of expertise. But it does seem like distinguishing between two possible forms of scaling law (a single curve of particular form vs a transition between one of that form to another) naturally suggests that there might be even more sorts of curves, with yet other forms. I'd think some empirical evidence that these two possibilities cover the space in practice would be useful.

This is applied only to synthetic data. To show that this is useful in practice, it would be helpful if the methods developed were applied to existing data: showing that some studies have sufficient power, that others had insufficient power, perhaps even predicting which are likely to break down. Maybe even looking at follow-on studies and seeing whether they did.

Questions for authors
  1. It appears that the data used for estimating scaling curves is the results of various runs at different network and/or data set sizes. It seems to me, perhaps naïvely, that statistics taken during the training of each of these might be useful. Perhaps more complicated to interpret, but still. I'm wondering if my intuition here is wrong.

  2. Was there some reason for not analyzing historic scaling studies using the methods developed?

3 comments
AuthorsOpenReview ↗

response to Reviewer fTLS

Thank you for these thoughtful questions regarding alternative functional forms, empirical scaling curves, and within-training statistics.

Alternative functional forms (Section 7). The revision makes the functional form an explicit axis of the analysis. Section 7 adds the form that scaling practice uses, L=E+AN−αL=E+AN^{-\alpha}, as a null in its own right: we calibrate the break test inside that family and measure its power against floor-plus-hinge alternatives. The pure-power test rejects floor-only curves 0.14–0.87 of the time, more often as sizes are added, while the floor-aware calibration holds its level at 0.040–0.063 and needs exponent changes of about 0.3 at twelve sizes and 2% noise to reach 80% power. The general lesson is the one your question points to: the break test is a test of a specified null, and each candidate form needs its own calibration. Section 9 states this as step two of the reporting rule, and Section 6 does the same for the noise model, where a scale that grows with model size inflates the scanned test to 0.136 at twenty sizes.

Analysing historic scaling studies (Section 8). Applying the method requires the per-size losses, which most published scaling studies do not release, so we work with a suite whose losses can be measured: the Pythia models, evaluated on a fixed WikiText-103 slice, where the per-window losses give a measured noise level instead of an assumed one. The revision extends that case study in the direction you ask. It states not only that a loss floor explains the curvature a pure-power test calls a break, but also what the six-point design could resolve: under the floor-aware calibration, the detectable exponent change at 2% noise exceeds 0.4. The curve is therefore explained without a break, and its design could not have detected an exponent change of 0.4 or less with 80% power.

Using intermediate training checkpoints (Section 9). Your intuition is reasonable, and the obstacle is specific: intermediate checkpoints of one run are strongly dependent along training, so they add observations without adding independent information about the size exponent, and the independent-noise model of Eq. (1) no longer applies. Using them requires an observation model with within-run correlation and, by the same principle as above, its own calibration. Section 9 lists correlated errors among the structures that define a different power problem.

We hope the floor-calibrated analysis and empirical Pythia evaluation address your questions, and we welcome any further feedback during the discussion period.

Reviewer_fTLSOpenReview ↗

revised ranking

In light of the discussions, I've raised my ranking but lowered my confidence.

Thank you for following the discussion and for raising your rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision

We thank all reviewers for their constructive feedback. In response to the reviews, the uploaded revision calibrates two critical assumptions beyond the breakpoint search (the noise structure and the scaling functional form) and provides full code to regenerate every figure and table.

The submitted results reproduce. Reviewer Ka1F reran the protocol; so did we, from the released code/repro_main.py. The fixed-break rates come back as 0.320/0.184/0.192/0.210/0.234 against the submitted 0.320/0.181/0.188/0.216/0.242, the calibrated rates as 0.048–0.052, Table 2 cell for cell, and the location pattern unchanged. The Pythia fit returns α^=0.177\hat\alpha=0.177, E^=2.345\hat E=2.345 nats and a residual sum of squares smaller by a factor of 51.7 (the submitted text said 52.8; the converged fit gives 51.7 and the revision uses that).

The noise model is an axis the calibration must cover, and the established permutation test does not cover it. Four procedures were run on the same designs, each at its own calibrated critical value: the scanned maximum, ave-FF, exp-FF, and the scanned maximum with the permutation calibration of Kim et al. (2000), which Reviewer HcpN rightly asked about. Under Gaussian noise of constant variance all four hold their level. When the noise scale grows along the design, the scan rises to 0.136 at twenty sizes and the permutation calibration to 0.128, because the null residuals are not exchangeable once their scale varies with model size; ave-FF stays at 0.076. At equal size ave-FF is also 5–10 points more powerful at the smallest detectable changes, so the revision reports calibrated thresholds for all three statistics and recommends ave-FF (Section 6, Table 3, Figure 3a).

The functional form is the other axis, and it is now calibrated. Reviewer Ka1F asked for size and power under a loss floor on the same designs. On floor-only curves the pure-power test rejects 0.14–0.87 of the time, and the rate rises with the number of model sizes (0.36 at six, 0.87 at twelve, at 2% noise), as additional model sizes increase power against genuine floor-induced curvature. The same scan calibrated inside the floor family returns the level to 0.040–0.063 and gives floor-aware detectable changes (power 0.39 at δ=0.2\delta=0.2 and 0.88 at δ=0.3\delta=0.3 at twelve sizes and 2% noise), which are larger than the pure-power values (Section 7, Table 4, Figure 3b).

The Pythia case study now answers the question it raised. Beyond showing that a floor explains the curvature a pure-power test calls a break, the revision states what the six-size design could resolve: under the floor-aware calibration the detectable exponent change at 2% noise exceeds 0.4, so that design could not have detected an exponent change of 0.4 or less with 80% power (Section 8).

Four model sizes cannot detect a break of any size with this procedure: no tested change reaches 80% power and the largest power measured at any tested change is 0.11. This was implicit in the submitted Table 2 and is now stated.

Overall, valid exponent-stability testing requires calibrating the breakpoint search, the noise structure, and the functional form together. The revision provides the calibration procedure across all three axes and recommends reporting the resulting minimum detectable exponent change alongside fitted exponents.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.