- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies whether small scaling studies are able to detect any change in exponent at some unknown breakpoint. It also calibrates the breakpoint search through simulation and validates detection power across different sample sizes, noise levels, and break locations. It uses an example to show how curvature from a loss floor can produce a break in a pure power law.
Critical strengths
Broadly, the paper is well presented and easy to follow
- Since a good fit over a few model sizes is not enough to establish that the exponent is stable, the question does have relevance.
- The presented analysis in the paper is clear, and within the stated model, the calibration is also justified. The paper is also able to correctly distinguish detection power from a confidence bound on the exponent change.
Critical weaknesses
My main concerns are regarding the novelty and some claims not substantiated in the paper
- Taking into account the breakpoint for breakpoint search and calibrating the procedures seem to be fairly established approaches. Related work on Monte Carlo testing for join point regression, such as Kim et al. (2000) [1], is not discussed in the paper. It is not clear to me what the contribution is beyond applying existing methods to selected scaling designs.
- The changes which are detectable depend a lot on the chosen test, but there is no clean ablation, and no alternate tests are compared. For example, in a simple calculation with 6 sizes, 2% noise, and an exponent change of 0.1, power is about 62% for the scan statistic versus 75% for an average fixed-break F statistic. If the reported threshold describes this particular procedure, than the detection limit is on the design. Can authors explain this? 3)The practical analysis is not complete for the paper's said example. For example, adding a flow reduces Pythia's residual sum of squares by around 53 times, showing that the pure power-law null is not appropriate. While the authors acknowledge that their power tables do not apply to this curve, it is not clear why, and it is a weakness of the paper - can authors explain?
- Every conclusion depends on independent Gaussian log errors with constant variance, and there is no ablation to check sensitivity to these assumptions.
- Several crucial numerical choices are not explored sufficiently. For example, the effect size grid is coarse, so reported detectable changes can differ by a factor of, let's say, 2 between adjacencies. Also, the 4-point result depends on nearly interpolating scan with a critical value around 5.7 × 10^5. The paper does acknowledge grid sensitivity but is not evaluating which limits how broadly these results can guide study design.
[1] Kim, H.-J., Fay, M. P., Feuer, E. J., & Midthune, D. N. Permutation tests for joinpoint regression with applications to cancer rates. Statistics in Medicine, 19(3):335–351, 2000.
Questions for authors
Please look at the weaknesses for some questions. The rest are below.
- What is the proposed procedure adding beyond the existing Collaborated Breakpoint tests?
- Is there any evidence that scaling studies use fixed break F reference?
- See W2 and W5: What motivates the chosen statistic and the scan grid?
- See W3: What changes are required to test an exponent break under the floor model?
- See W 4: How should a practitioner choose sigma when evaluating uncertainty, variation in training etc can all contribute to residual scatter?
5 comments
response to Reviewer HcpN (part 2/2)
(2/2)
W4 and Q5 (non-Gaussian and non-constant-variance errors, and the choice of ). Table 3 also gives size under Student- errors (0.052–0.068 for the scan) and under the growing-variance model above. Section 9 now answers how a practitioner should choose : state whether the noise scale is constant along the design, distinguish evaluation uncertainty from run-to-run variation, and calibrate under the structure assumed. The invariance of Proposition 1 holds for any constant but not for a that varies with .
W3 and Q4 (the Pythia analysis, and why the power tables did not apply). Section 7 now carries out the full size-and-power analysis under the loss-floor model. We simulate floor-only curves and apply two tests. The pure-power scan at its own calibrated critical value rejects floor-only data 0.14–0.87 of the time, and the rate rises with the number of model sizes (0.36 at six, 0.87 at twelve, at 2% noise), as additional sizes increase power against genuine floor-induced curvature. The same scan calibrated inside the floor family (the null fits , the alternative adds a hinge, and the critical value is simulated under the floor null on the same design) returns the level to 0.040–0.063. Floor-aware power at twelve sizes and 2% noise is 0.39 at and 0.88 at (Table 4, Figure 3b). These are the numbers that apply to a curve such as Pythia, where a floor is present; at its six sizes the floor-aware detectable change at 2% noise exceeds 0.4.
W5 (grid resolution and the four-point case). Table 2 keeps the submitted effect-size grid so that it can be compared cell by cell with the submission, and the floor analysis uses the finer grid $\delta\in\lbrace 0.05,0.1,0.15,0.2,0.3,0.4\rbrace $. The four-point result is now stated explicitly: at no tested change reaches 80% power and the largest power we measure at any tested change is 0.11. The large critical value at follows from the design: with three fitted parameters and a scanned breakpoint, the segmented fit nearly interpolates four points.
On reproduction. Every table in the revision is regenerated by the released code. The submitted Tables 1 and 2 reproduce within Monte Carlo error from code/repro_main.py (for example the fixed-break rates 0.320/0.184/0.192/0.210/0.234 against the submitted 0.320/0.181/0.188/0.216/0.242), and the Pythia fit reproduces at , , with the residual sum of squares falling by a factor of 51.7.
Thank you again for pushing us to test heteroscedasticity, residual permutation, and floor calibration directly; we hope these new results address your concerns.
Thank you again for your detailed review. Since our replies of 13 November, the revision uploaded on 17 November adds two results that bear on your W3 and Q4, and we summarise them here in case they are useful before the discussion closes.
Section 8 now calibrates the floor-aware scan on the actual Pythia design: the six measured, unequally spaced sizes and the fitted floor, which is about 90% of the loss at the largest size, at four noise levels (new Table 6). On this design the test holds its level (0.038–0.053), the measured curve has calibrated -values of 0.09–0.12 at every noise level, and a pure-power scan calibrated on the same design rejects 0.87–1.00 of floor-only curves. No exponent change up to 0.8 reaches 80% power, so this calibration replaces the figure in our earlier reply (a detectable change above 0.4), which had been inferred from the general floor designs of Table 4.
Section 9 now converts the detectable change into an error margin for the extrapolated law. For eight sizes over two decades at 2% noise, an undetected slowdown of 0.1 in the exponent leaves the actual loss one decade beyond the largest size 18% above the forecast, while the model-size exponent reported by Kaplan et al. (2020) predicts a 16% reduction over that decade.
The equal-size comparison with the permutation calibration of Kim et al. (2000) and with ave- and exp- under Gaussian, Student- and growing-variance noise remains in Section 6 and Table 3. If any of your concerns are still open, we would be glad to address them.
response to Reviewer HcpN (part 1/2)
(1/2)
Thank you for the detailed review, in particular for pointing us to Kim et al. (2000) and for asking which assumptions the conclusions depend on. Both points led to new analyses in the revision.
W1 and Q1 (what this adds beyond the established joinpoint test). Kim et al. (2000) is now cited and included in a head-to-head comparison. We put four procedures on the same designs, each at its own calibrated critical value so the comparison is at equal size: the scanned maximum, ave-F and exp-F (Andrews–Ploberger), and the scanned maximum with Kim's per-dataset permutation calibration (B=199), over 20,000 simulations and 800 permutation datasets. Under Gaussian noise of constant variance all four hold their level. When the noise scale grows along the design, which is what happens when larger runs are evaluated on fewer tokens, the scanned test's size rises from 0.050 at n=4 to 0.136 at n=20, and Kim's permutation calibration rises to 0.128, because the residuals of the null fit are not exchangeable once their scale varies with model size. The same holds for the functional form (below): residual permutation calibrates the search but still tests the pure-power null. Thus, while Kim et al.'s permutation test and our Gaussian scan are equivalent under homoscedastic pure-power nulls, design-level simulation is necessary once noise varies with scale or a loss floor is present.
Q2 (whether scaling studies use a fixed-break F reference). The fixed-break reference is what a practitioner obtains by default when a hinge column is added at an estimated breakpoint and the fit is read from an ordinary least-squares routine, which reports the t/F p-value as if the breakpoint had been fixed in advance. Published broken-scaling analyses (e.g. Caballero et al., 2023) estimate break locations as free parameters of a smoothly broken power law and compare fit quality, without a calibrated test of whether a break is present. Table 1 quantifies the Type-I error inflation of the naive reference (18.4–32.0% at a nominal 5%), and Section 6 compares against properly calibrated joinpoint tests (Kim et al., 2000; Andrews and Ploberger, 1994).
W2 and Q3 (alternative statistics, the ablation, and the scan grid). Your calculation agrees with ours, and the comparison is now in the paper. At equal size under Gaussian noise, ave-F dominates the scanned maximum at the smallest detectable changes, by 5–10 points of power at δ=0.05 for n≥6 (for example n=8: 0.48 versus 0.40). We therefore report calibrated thresholds for sup-F, ave-F and exp-F, recommend ave-F, and state that a reported detectable change belongs to the statistic that produced it (Section 6). The scan grid of 60 equally spaced candidates between the second and the second-to-last design points keeps the hinge inside the design, so the segmented model is identified at every candidate; because the calibration simulates the same grid, the reported size holds for whichever grid is used.
Continued in part 2/2 below.