- The central claim rests on six points and a model comparison that the data cannot resolve. The capacity-only vs capacity+shape ranking comes from leave-one-out on the same six points used to choose the model. The in-sample R^2 is identical (0.942) for both models, so the difference is leverage on single folds. The paper's own noise reference says the model-to-model difference is within seed noise (the 410M seed standard deviation of ln R is 0.356, Table 3, against a MALE gap of 0.16). The only real out-of-sample test (6.9B and 12B) slightly favours capacity+shape (0.443/0.233 vs 0.448/0.239), and in the last-two window capacity wins only 2 of 6 folds (Table 1). No test or interval is given for the model comparison. So the headline ("capacity model is the best description") is not supported beyond "R decreases with size in Pythia." The word "prespecified" is also overstated: Supp. Note 1 describes an earlier version with a different window, model form and metric that reached the opposite conclusion, and the 6.9B and 12B points were already seen then.
- The metric is confounded by what each model writes, and the obvious control is missing. Each model's displacement is averaged over its own generated answers (different text, length, correctness and token positions), and there is no fixed-text control (acknowledged in the abstract). With six models under 3B, teacher-forcing one shared text through all of them is cheap. Without it, a reader cannot tell whether the trend is a property of the architecture or of what bigger models say. The result is also not connected to any downstream quantity (loss, accuracy, capability) that would show why the exponent matters, and the window is a free parameter that moves it (exponent -0.58 to -0.75 and MALE 0.208 to 0.512 across three windows).
Notes on the Presentation, Informativeness and Soundness scores
Presentation (2): The paper reads like several papers compressed into one. The main text carries the capacity fit, the cross-family comparison, the time-axis result, the pooling artefact and the detection null, and the appendix holds thirteen supplementary notes, many of them "historical", "raw", "withdrawn" or "superseded". Each analysis states a different window, convention (raw vs corrected) and fit, so it is hard to keep track of which number applies to which claim. Cross-references are wrong in several places (for example Sec. 6 is cited for the Kaggle archive loss). Nearly every sentence carries a hedge, and several results are restated four or five times. The prose is careful and the reported numbers can be followed, so the score is 2, but it takes real effort.
Informativeness (1): The headline is that late-layer cosine displacement falls with Pythia size, which the paper itself cites as already observed (Lawson et al., 2025). The new part is a six-point slope whose value moves from -0.58 to -0.75 depending on the window. The identifiability points (collinear (d, L) designs give an imprecise shape coefficient) are standard regression facts restated for this case. The pooled-axis density-minimum caution (Sec. 4, Fig. 5) is mildly useful, though it is a caution about a statistic nobody uses.
Soundness (2): Where the reported numbers can be recomputed from the paper's tables they are internally consistent, and the claims are hedged to match. The concerns are about what six noisy points can support and about the "prespecified" framing, not about arithmetic errors. Several concerns need new data (fixed text, more seeds), so this is borderline 1.
Response to Reviewer Xcwu
Thank you for the review. You noted that the cautionary results help practitioners who build detectors or thresholds on this geometry.
Q1. How robust is the result with six points and two parameters?
How to read SE 0.093. It is the classical OLS SE conditional on the six size coordinates (four residual df), assuming a linear mean and independent equal-variance errors. Training dispersion grows with size (410M-to-70M variance ratio 62, Appendix J.3). Alternatives without it: HC3 gives SE 0.185 and interval [−1.259, −0.230]. Leave-one-out slopes span −0.914 to −0.701. A random-effects model assigning the 410M dispersion to each larger size gives −0.717 [−1.010, −0.423] (Appendix J.3). All three exclude zero. Read 0.093 as the precision of the fitted line at these six checkpoints, not as training-run uncertainty. Under the random-effects assumption only the ranking of the size models stays uncertain: capacity has the lower held-out error in 89.8% of draws (advantage interval [−0.137, +0.381]; Appendices J.3, T).
Q2. What is the methodological novelty? We contribute a quantitative audit of an internal geometric quantity: what transfers, what does not, and which measured states, not configuration, record the difference. (i) Within a recipe the trend survives identical tokens, a second task and 22 shapes (held out: post hoc, Appendix T; Q1); (ii) across families, forecasts fixed before measurement lost to a size-blind constant (§4). (iii) The coefficient comment maps which measured conditions move the coefficients, with validity ranges. (iv) Four further comments (new; mostly in Appendices O, R–T) claim only measurements: the write-size/direction split of the size trend, per-factor direction terms, the null model's depth slope, a sealed test of a training-time coefficient change, and blind tests. Three elements (item-noise ceiling, seed-noise ruler, common-mode identity) are now in §2.3 and Appendix O, two failed blind tests in §6 and Appendix R. We claim no new estimator, identity or null model; of the identity we claim only its use to separate the common-mode part of the readout. The ingredients are known: the quantity is ShortGPT's Block Influence; Lawson et al. (2025) show qualitatively that its mean-centred version falls with size; coefficients as functions of factors and cross-family forecasts have precedents for loss and benchmarks (Clark et al. 2022; Ruan et al. 2024; Choshen et al. 2025). Appendix N cites them.
For practitioners who set thresholds on this geometry