acceptodds
Under review as a conference paper at ICLR 2027

Auditing Capacity Trends in Transformer Residual-Stream Geometry

Abstract

Interpreting scaling relationships in transformer residual-stream geometry requires separating measurement conventions, estimation precision, and predictive performance. We audit these factors using released model checkpoints, organizing the analysis around identifiability, reproducibility, trend diagnostics, and internal consistency. Our primary analysis measures late-layer cosine displacement over self-generated TriviaQA answers from six Pythia checkpoints under a prespecified reanalysis protocol. Among constant, capacity-only, and capacity-plus-shape models, the capacity-only model achieves the lowest mean leave-one-model-out absolute log prediction error. With capacity represented by , its fitted exponent is , with a conditional ordinary-least-squares standard error of . Predictive errors remain substantial: the geometric-mean symmetric multiplicative error factor is , reaching in the worst fold. Across three specified analysis windows, the capacity model retains the lowest mean leave-one-out error, although fold-level rankings differ and fitted exponents range from approximately to . Adding a width-to-depth shape term does not improve the mean prediction metric in these comparisons; this finding does not establish the absence of shape effects. Historical extrapolation to two larger Pythia checkpoints further illustrates the limits of predictive accuracy. These results support a family- and window-dependent capacity trend and demonstrate how measurement choices affect its interpretation. Because models generate different text and we do not include fixed-text controls, the reported relationships characterize task-conditioned, self-generated trajectories rather than isolated causal effects of architecture.

Primary area:
interpretability and explainable AI
Keywords:
scaling laws, residual stream geometry, transformer, representational geometry, identifiability, reproducibility, measurement methodology, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_XcwuOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper's primary analysis measures late-layer cosine displacement in self-generated TriviaQA answers across six Pythia checkpoints using a prespecified reanalysis protocol. Of the constant, capacity-only, and capacity-plus-shape models, the capacity-only model yields the lowest mean leave-one-model-out absolute log prediction error.

Critical strengths

The cautionary results are useful to practitioners who build detectors or thresholds on residual-stream geometry.

Critical weaknesses
  1. The evaluation is not sufficient. The main result rests on six points with two parameters. The exponent's conditional SE (0.093) assumes independent homoscedastic errors, which the paper itself says the design does not support (Section 2.4).
  2. The novelty is limited. The individual diagnostics are standard statistical and measurement practices.
Questions for authors
  1. How robust is the main result given only six data points and two fitted parameters? How should the exponent’s conditional SE (0.093) be interpreted when its assumptions of independent, homoscedastic errors are unsupported by the design, as acknowledged in Section 2.4?
  2. What is the methodological novelt?
1 comment
AuthorsOpenReview ↗

Response to Reviewer Xcwu

Thank you for the review. You noted that the cautionary results help practitioners who build detectors or thresholds on this geometry.

Q1. How robust is the result with six points and two parameters?

  • Identical text. When all eight Pythia checkpoints read one shared continuation, the six-size slope is −0.740 against −0.744 on each model's own text (Table 3, p. 5).
  • Readout task stability (same checkpoints). The line, frozen before the GSM8K data were read, predicts them with a typical error factor of 1.35, against 3.11 for a constant (Table 4).
  • Larger sizes. At Pythia-6.9B and 12B, outside the fit, displacement keeps falling, below the six-point line in the interior window (§3.1, Appendix U).
  • 22 more architectures. On a single-recipe Gemstones grid, width and depth predict held-out architectures within a geometric-mean factor of 1.23 (= exp(0.203), leave-one-out MALE 0.203; capacity alone 0.759; constant 0.867; post hoc, Appendix T). This is interpolation: toward wider models the fit is worse than a constant (1.078 vs 0.998), toward larger ones barely better (0.969 vs 1.026).
  • Independent training runs. Substituting each of nine separately trained 410M checkpoints in turn gives slopes −0.7480 to −0.7437, with capacity first in all nine (Appendix Q.4; limited evidence, J.3).
  • Data-only retraining. The deduplicated-Pile release of all six sizes (same initialization, different data) gave slope −0.620 on the same shared text; the difference from −0.740, +0.120 [+0.118, +0.121], lies inside a ±0.156 band (assumed, not measured; Appendix T.4), with the same candidate ranking (new; prospective).

How to read SE 0.093. It is the classical OLS SE conditional on the six size coordinates (four residual df), assuming a linear mean and independent equal-variance errors. Training dispersion grows with size (410M-to-70M variance ratio 62, Appendix J.3). Alternatives without it: HC3 gives SE 0.185 and interval [−1.259, −0.230]. Leave-one-out slopes span −0.914 to −0.701. A random-effects model assigning the 410M dispersion to each larger size gives −0.717 [−1.010, −0.423] (Appendix J.3). All three exclude zero. Read 0.093 as the precision of the fitted line at these six checkpoints, not as training-run uncertainty. Under the random-effects assumption only the ranking of the size models stays uncertain: capacity has the lower held-out error in 89.8% of draws (advantage interval [−0.137, +0.381]; Appendices J.3, T).

Q2. What is the methodological novelty? We contribute a quantitative audit of an internal geometric quantity: what transfers, what does not, and which measured states, not configuration, record the difference. (i) Within a recipe the trend survives identical tokens, a second task and 22 shapes (held out: post hoc, Appendix T; Q1); (ii) across families, forecasts fixed before measurement lost to a size-blind constant (§4). (iii) The coefficient comment maps which measured conditions move the coefficients, with validity ranges. (iv) Four further comments (new; mostly in Appendices O, R–T) claim only measurements: the write-size/direction split of the size trend, per-factor direction terms, the null model's depth slope, a sealed test of a training-time coefficient change, and blind tests. Three elements (item-noise ceiling, seed-noise ruler, common-mode identity) are now in §2.3 and Appendix O, two failed blind tests in §6 and Appendix R. We claim no new estimator, identity or null model; of the identity we claim only its use to separate the common-mode part of the readout. The ingredients are known: the quantity is ShortGPT's Block Influence; Lawson et al. (2025) show qualitatively that its mean-centred version falls with size; coefficients as functions of factors and cross-family forecasts have precedents for loss and benchmarks (Clark et al. 2022; Ruan et al. 2024; Choshen et al. 2025). Appendix N cites them.

For practitioners who set thresholds on this geometry

  • A threshold does not port across families. The 90% interval for an unseen family's level is about ±2.25 log units (raw readout; SD 3 df; post hoc); a centred refit: ±0.94 (raw ±2.30), mostly via GPT-Neo (Appendix U.7(vi)). GPT-Neo-125M, with Pythia's data, token count and shape, has 26 times lower raw displacement than Pythia-160M (§4, p. 6).
  • Calibrating per family is cheap. One forward pass of the target on fixed text estimates its self-generated level within 5% (12 of 16 checkpoints; development; p. 5).
  • What geometry tells you about answers. Geometry did not improve error detection over probabilities (AUROC −0.0049, p. 9). Related edits of one late-layer write raised confidence while lowering the correct option's probability in two hosts and did the reverse in two others, so no consistent direction (Table 16, p. 22); in a later readout (development; Appendix T), accuracy moved by at most 1.35 points, all intervals including zero (one orthogonal control edit: −1.1 points).
Reviewer_ijnzOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper measures a "cosine displacement" R = 1 - cos(h_j, h_{j+1}) between consecutive residual-stream states, averaged over a late window of blocks (floor(0.6L) to L-2), over self-generated TriviaQA answers (200 items, 5-shot, Eq. 1-3). It fits a log-log model of the median level against N_proxy = 12 d^2 L on six Pythia checkpoints (70M to 2.8B). Three nested models are compared by leave-one-model-out mean absolute log error (MALE): constant, capacity-only, and capacity plus log(d/L). The capacity model wins (0.490 vs 0.650 for capacity+shape and 1.322 for constant), with exponent -0.744 (conditional OLS SE 0.093) and a geometric-mean error factor of 1.63 (worst fold 2.69) (Abstract, Table 1). Two held-out checkpoints (Pythia-6.9B and 12B) are used for historical extrapolation. The paper adds design-conditioning diagnostics (eigenvalue ratio 10.95), a seed-dispersion reference at 410M, a descriptive comparison with Qwen2.5, Qwen3 and SmolLM2 (flat, non-monotone), a note that pooling "depth" and "time" displacement creates a spurious density minimum near 0.25, and a pre-registered detection study that returns a null (adding geometry features to single-pass probability baselines changes AUROC by about -0.005). Throughout, the claims are hedged to "a limited, window-sensitive trend in self-generated trajectories, not an architectural law."

Critical strengths
  1. Honest calibration of claims. Almost every result is stated at the level the evidence supports, with limits listed (single-run models, one seed size, confounded cross-family comparison). The paper also reports its own reversals and withdrawals (Supp. Notes 1, 4, 5, 10), which is a green flag and unusual.
  2. Useful methodological habits. Reporting out-of-sample error instead of in-sample R^2, reporting the design spectrum before fitting, and reporting a seed-dispersion reference (Appendix D) are good practice. The reported arithmetic is internally consistent where it can be recomputed from the tables.
Critical weaknesses
  1. The central claim rests on six points and a model comparison that the data cannot resolve. The capacity-only vs capacity+shape ranking comes from leave-one-out on the same six points used to choose the model. The in-sample R^2 is identical (0.942) for both models, so the difference is leverage on single folds. The paper's own noise reference says the model-to-model difference is within seed noise (the 410M seed standard deviation of ln R is 0.356, Table 3, against a MALE gap of 0.16). The only real out-of-sample test (6.9B and 12B) slightly favours capacity+shape (0.443/0.233 vs 0.448/0.239), and in the last-two window capacity wins only 2 of 6 folds (Table 1). No test or interval is given for the model comparison. So the headline ("capacity model is the best description") is not supported beyond "R decreases with size in Pythia." The word "prespecified" is also overstated: Supp. Note 1 describes an earlier version with a different window, model form and metric that reached the opposite conclusion, and the 6.9B and 12B points were already seen then.
  2. The metric is confounded by what each model writes, and the obvious control is missing. Each model's displacement is averaged over its own generated answers (different text, length, correctness and token positions), and there is no fixed-text control (acknowledged in the abstract). With six models under 3B, teacher-forcing one shared text through all of them is cheap. Without it, a reader cannot tell whether the trend is a property of the architecture or of what bigger models say. The result is also not connected to any downstream quantity (loss, accuracy, capability) that would show why the exponent matters, and the window is a free parameter that moves it (exponent -0.58 to -0.75 and MALE 0.208 to 0.512 across three windows).

Notes on the Presentation, Informativeness and Soundness scores

Presentation (2): The paper reads like several papers compressed into one. The main text carries the capacity fit, the cross-family comparison, the time-axis result, the pooling artefact and the detection null, and the appendix holds thirteen supplementary notes, many of them "historical", "raw", "withdrawn" or "superseded". Each analysis states a different window, convention (raw vs corrected) and fit, so it is hard to keep track of which number applies to which claim. Cross-references are wrong in several places (for example Sec. 6 is cited for the Kaggle archive loss). Nearly every sentence carries a hedge, and several results are restated four or five times. The prose is careful and the reported numbers can be followed, so the score is 2, but it takes real effort.

Informativeness (1): The headline is that late-layer cosine displacement falls with Pythia size, which the paper itself cites as already observed (Lawson et al., 2025). The new part is a six-point slope whose value moves from -0.58 to -0.75 depending on the window. The identifiability points (collinear (d, L) designs give an imprecise shape coefficient) are standard regression facts restated for this case. The pooled-axis density-minimum caution (Sec. 4, Fig. 5) is mildly useful, though it is a caution about a statistic nobody uses.

Soundness (2): Where the reported numbers can be recomputed from the paper's tables they are internally consistent, and the claims are hedged to match. The concerns are about what six noisy points can support and about the "prespecified" framing, not about arithmetic errors. Several concerns need new data (fixed text, more seeds), so this is borderline 1.

Questions for authors

Clarification requests

  1. Please tabulate ln R per model for the main window, and the per-fold LOO errors. The main text gives only the fitted slope and aggregate errors, so the fit cannot be reproduced from the paper alone.
  2. The paper says capacity+shape is better on the two held-out points under the frozen window, while under include-final the held-out points sit above the line and under the frozen window their residuals are negative (Fig. 2 caption, Supp. Note 3). Which model and window does the paper want the reader to believe for extrapolation?
  3. Table 2 and Methods list SmolLM2 as base weights with plain few-shot prompting, but Supp. Notes 3 and 4 and the detection study treat SmolLM2-1.7B as one of "five instruction-tuned models". Which is it for each analysis?
  4. The Reproducibility statement says the checker re-derives headline values from shipped raw trajectories of the six-model grid, but Appendix B says four Pythia trajectory archives were "lost to a Kaggle versioning error described in Sec. 6". Sec. 6 does not appear to describe it. What exactly is shipped?

Suggestions (answers could move my score) 5. Run the fixed-text control: the same text through all six (and 6.9B, 12B) Pythia models, same window, same fit. Does the slope survive? 6. Measure the seed floor at 1B and above (PolyPythias stops at 410M, but a few seeds at 1B would help) or, if not possible, model the noise as heteroscedastic. The floors you measured are 2.6% (70M), 5.5% (160M) and 31.5% (410M), so the constant-variance OLS assumption behind SE 0.093 is contradicted by your own data. 7. Test cross-family slope differences directly (a family x ln N interaction with a noise estimate) rather than describing Pythia as "monotone" and the others as "flat". For Qwen3 the fitted slope is about -0.03 over 21x: please give its uncertainty and say whether it is distinguishable from -0.58. 8. Report how the three windows' exponents and MALE change if the window is defined by a fixed number of absolute layers rather than a fraction of L. For 70M the main window is only two transitions (L=6, floor(3.6)=3 to 4).

2 comments
AuthorsOpenReview ↗

Response to Reviewer ijnz (1/2): Questions 1–8

Thank you for the detailed review. This comment answers Q1–Q8; the next one addresses the weaknesses. Page, table and section numbers refer to the revised PDF.

Q1. Table 1 (p. 4), main interior window, leave-one-out absolute log errors:

\text{Pythia} & \text{ln median R} & \text{Constant} & \text{Capacity} & \text{+ shape} \cr \hline \text{70M} & −0.683 & 2.131 & 0.991 & 1.267 \cr \text{160M} & −1.101 & 1.630 & 0.494 & 0.510 \cr \text{410M} & −2.288 & 0.206 & 0.140 & 0.367 \cr \text{1B} & −2.733 & 0.328 & 0.512 & 0.949 \cr \text{1.4B} & −3.761 & 1.562 & 0.446 & 0.447 \cr \text{2.8B} & −4.190 & 2.077 & 0.357 & 0.359 \end{array}$$ **Q2.** For extrapolation we use the late interior window, where the two size models agree at 6.9B and 12B to within 0.006 log units (signed residuals −0.448/−0.239 for capacity, −0.443/−0.233 with shape). The window, not the model, sets the sign: with the final transition in fitting and scoring the residuals are +0.226/+0.153 and +0.249/+0.180 (Table 21, p. 37). At 6.9B, adding the final block raises the observed log level by 1.674 but the prediction by only 1.000 (Appendix J.2). Across families we measure the target instead (§4; largest models in the general comment). **Q3.** SmolLM2-135M, 360M and 1.7B **Base** with plain few-shot prompts form the geometry family grid. SmolLM2-1.7B-**Instruct** with a chat template entered the detection study, the pooled-axis analysis, the five-target transfer comparison (submitted Note 3) and the mixed fit (Note 4); Note 4's 3B→7B step used Qwen2.5 (Appendix L, Table 23, p. 50). **Q4.** The original supplement contains per-answer displacement arrays for the six fit checkpoints and the two larger ones (8 × 200 answers), 2,000 detection trajectories and a checker (p. 9). The four lost archives are the separate attention/MLP branch trajectories at 160M, 410M, 1B and 2.8B; their per-answer summaries survive. The whole-block capacity archive is complete, so the six-point fit is reproducible from it (Appendix I.3, p. 35; Appendix G). The loss is described in Appendix I.3 (p. 35). **Q5.** All eight Pythias read the same 200 prompts and one shared 410M continuation per prompt, with the same FP32 replay, positions and window. The six-size slope moved from −0.7441 to −0.7397 (paired difference +0.0043, item interval [+0.0021, +0.0079]); the other two windows changed by +0.0020 and +0.0007 (Table 3, p. 5). Omitting the donor checkpoint gives −0.739 (p. 4). In Pythia the decline therefore does not come from differences in what each model writes. **Q6.** Documented independent Pythia training runs at 1B and above are not released, so we followed your alternative and modelled heteroscedasticity. Nine training runs at each of 70M, 160M and 410M give log SDs 0.0452, 0.0839 and 0.3563 in your raw include-final convention (your 2.6%, 5.5% and 31.5% are median relative deviations of the same runs); the 410M-to-70M variance ratio is 62.2 [24.1, 269.0] (Appendix J.3, p. 38). The 410M dispersion is the largest at any size we measured; in the interior window it is 0.344 (Appendix Q.4). DataDecide-1B, another family, gives 0.039 within recipe (Appendix C.2). For the slope: HC3 SE 0.185, interval [−1.259, −0.230]. Assigning the 410M dispersion to each of 1B, 1.4B and 2.8B gives a random-effects slope of −0.717 [−1.010, −0.423]. In the same parametric resampling, capacity has the lower held-out error in 89.8% of draws (central interval for its advantage [−0.137, +0.381]; Appendices J.3, T). The decline is clear; the ranking stays uncertain under this assumption. We also retested with a second full training of all six sizes, the released deduplicated-Pile Pythia models (same initialization, different data), on the same shared text and FP32 replay. The slope was −0.620; its difference from −0.740, +0.120 (item interval [+0.118, +0.121]), lies inside a ±0.156 band (assumed, not measured; Appendix T.4; the last-two-block window gives about 0.173, outside it), and the candidate ranking was unchanged (new; criteria fixed before measurement; Appendix T). Individual sizes moved more (1.4B: +1.16 log units). What public seeds can verify: §6; Appendix O.2 (pp. 56–57). **Q7.** In the include-final window, family × ln N regression on ten models: Pythia slope −0.576, Qwen3 −0.034, difference +0.542, model-point 95% interval [+0.310, +0.774] (six residual df); omitting any one checkpoint leaves +0.397 to +0.639. Qwen3 alone: [−0.332, +0.264], and dropping 8B moves its slope to +0.020 (§4, p. 5; Appendices H.2, U). Under that common-variance model the two slopes are distinguishable; the sign within Qwen3 is not determined. **Q8.** With exactly two interior transitions placed early, midway and late, slopes are −0.334/−0.451/−0.748 and capacity MALEs 0.083/0.244/0.512 (Appendices Q.4, U). Every fixed-count slope is negative, so averaging more blocks in deeper models does not explain the sign; location sets the magnitude.
AuthorsOpenReview ↗

Response to Reviewer ijnz (2/2): Weaknesses

This comment addresses the weaknesses; Q1–Q8 are in the previous comment, the largest models in the general comment.

A model comparison the data can resolve. On 22 Gemstones architectures trained with one recipe, out-of-sample error resolves them clearly (post hoc; Appendix T): leave-one-architecture-out MALE 0.867 for a constant, 0.759 for capacity, 0.203 for capacity plus shape. With 5 random anchors predicting the other 17, capacity plus shape beat capacity in 99.5% of draws. This is interpolation: toward wider models the fit is worse than a constant (1.078 vs 0.998), toward larger ones barely better (0.969 vs 1.026); the coefficient comment discusses why. The six Pythia releases cannot make this comparison, as you note: both models have R² 0.942 (p. 4), and at 6.9B and 12B they differ by at most 0.006 log units (Table 21). The revised headline is therefore that displacement falls with size within a recipe and depends on shape (in sample), not that capacity alone is the best description (abstract; §3).

"Prespecified." The revision no longer uses the word for the six-point analysis. The text states that its window, aggregation and candidates were fixed for a reanalysis after earlier exploration (§2.1, p. 3) and that 6.9B and 12B had been seen (Appendix U). Forecasts fixed before their targets were read are scored separately against a size-blind constant: GSM8K (Eq. 7 prints the frozen coefficients; capacity won, MALE 0.302 vs 1.134, §3.2); three DataDecide sizes reading Pythia token prefixes, OPT-6.7B and Qwen3-14B (the constant was closer each time; §4, Appendix F.3); and four families never used in fitting (configuration did no better than the development mean, §4). These are our own records, not an external registry; the frozen coefficients are printed (Eq. 7, §3, Appendix Q.8; Table 12), so each forecast can be recomputed (not all target data are uploaded; Appendix N.3). Later blind tests: blind-test comment.

Connection to a downstream quantity. Three tests.

  1. Capability. At fixed architecture and scale, 25 DataDecide-1B training-data recipes spanned 1.505-fold in displacement; Spearman correlation with the released ten-task average was −0.141 (exact p = 0.50; p. 9, Appendix C.2); a sealed centred follow-up (Appendix T): also none.
  2. Confidence, by intervention. On 964 ARC questions, moving one late block's write toward its input in Qwen2.5-0.5B lowered choice entropy by 0.096 nats and raised correct-option loss by 0.124 nats. Qwen2.5-1.5B and Pythia-160M (a weight edit) reversed; Falcon3-1B matched Qwen2.5-0.5B (Table 16, p. 22); cf. entropy neurons (Stolfo et al. 2024). A rotation that keeps displacement fixed also changed entropy and loss in Falcon3, so displacement magnitude alone does not set the effect (Appendix C.3). In a later readout of saved outputs (development; Appendix T), the edits moved accuracy by at most 1.35 points across four hosts, every simultaneous 95% interval including zero (one orthogonal control edit lowered accuracy by 1.1 points on one host).
  3. Detection. Adding eleven geometric features to probability features changed detection AUROC by −0.0049 across nine cells (pp. 9, 30).

So related edits of the late write change answer confidence with no consistent direction and no detected accuracy change. We claim no detection or correction benefit. Why the exponent matters: across families it does not transfer, and one fixed-text pass supplies the target's level.

Beyond Lawson et al. They showed higher adjacent-layer similarity at larger Pythia sizes after mean subtraction; we credit that direction (§1). The quantity itself is ShortGPT's Block Influence (arXiv 2403.03853), used there to rank layers for pruning. What we add: the identical-text control, task transfer of a frozen line, shape dependence on a single-recipe grid (in sample; held-out: post hoc, Appendix T), and a measured size of the family offset. For an unseen family it is about ±2.25 log units at 90% (raw readout; SD 3 df; post hoc; a centred refit: ±0.94 (raw ±2.30), mostly via GPT-Neo; Appendix U.7(vi)), and GPT-Neo-125M, with Pythia's data, token count and shape, has 26 times lower displacement than Pythia-160M (§4, p. 6); centring out the common mode shrinks that gap from −3.26 to −0.48 log units (partial support; Table 13, Appendix O.3). The raw uncentred readout is not invariant: in four LayerNorm models, a function-preserving shift of the residual stream changed raw ln R by −1.25 to −2.68 while the centred readout was stable (prospective, amended gate; Appendix D). Two further comments (new; partly in §5 and Appendices B.5, O.3, T) split the size trend into write-size and direction parts, and test one coefficient change during training under seal (part passed, part failed).

Presentation. The main text is now organised by factor (§§3–6); the include-final window is used only for your Q7 slope difference (Appendix U).

Reviewer_b7FyOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper attempts to interpret scaling relationships in transformer residual-stream geometry. They use 6 checkpoints from one QA task to compare 3 models of capacity. The results show limited narrow potential fit in some cases, which do not extend. Generalization and application are not presented.

Critical strengths

The paper is well written.

Critical weaknesses

The results appear insufficient and do not generalize.

Questions for authors

Why are only Pythia models and only the internal late-layer compared? Why these three candidate capacity models, and where does using the one that partially fits help? What gives any confidence in the results scaling to larger and more sophisticated base models?

1 comment
AuthorsOpenReview ↗

Response to Reviewer b7Fy

Thank you for the review. References are to the revised PDF.

Why Pythia, and why the late layers? These choices are controls. Pythia is a public ladder with data, order, tokens and tokenizer fixed; width, depth and learning rate change. The late interior window stops before the final block, whose update into the output state can be unusually large (§2.1): its median rotation is 0.20–0.79 vs 0.003–0.061 for the final normalization (Appendix I.1), and including it adds 1.67 and 1.49 log units at 6.9B and 12B (Appendix J.2). The result is not confined to that window: two transitions placed early, midway or late give slopes −0.334, −0.451 and −0.748, and including the final block gives −0.576 (Table 2; Appendices Q.4, U). We also measured Gemstones, Qwen3, SmolLM2, OPT, OLMo-2, GPT-Neo and DataDecide. The decline holds within Pythia and Gemstones (Gemstones: raw readout, mainly depth); in Qwen3 the slope is near zero and window-dependent (§4), a difference we measure but do not explain. Late layers build the state the output reads: editing one late write changes confidence (Application).

Why these three candidates, and where does the partially fitting one help? They are nested: one typical level; plus capacity (12·width²·depth); plus log(width/depth) at fixed capacity. Where they help:

  • Predicting an unmeasured size within a recipe. On six Pythia sizes, leave-one-out MALE was 0.490 for capacity against 1.322 for the constant (Table 1, p. 4). Frozen and moved to GSM8K, it had a typical error factor of 1.35 instead of 3.11 (Table 4).
  • When shape varies. On 22 Gemstones architectures, adding shape raises in-sample R² from 0.337 to 0.951 (p. 4); held out one at a time, MALE is 0.203 against 0.759 for capacity and 0.867 for a constant (post hoc; Appendix T). This holds inside the grid; toward wider models the fit is worse than a constant (1.078 vs 0.998), toward larger ones barely better (0.969 vs 1.026), as the coefficient comment discusses.

What gives confidence for larger, more capable base models?

  • The measurement scales. A forward pass on fixed text estimated self-generated displacement within 5% at three of four 6.9–14B checkpoints in the PDF (development: Pythia-12B, Qwen3-8B/14B; −5.4% at Pythia-6.9B; Table 20) and at OLMo-2-7B (Appendix T).
  • Precision per reading. Item noise, and so each cell's ceiling, agreed with the bootstrap for mean-type readouts in three blind tests on unseen models up to 1.5B (lower bounds ≥ 0.90; C5t marginal; 0.95 criteria not met on the seven unseen models; Tables 25–26), but failed on 300 new questions (§6; Appendix O.1); seed noise is larger (p. 8).
  • The direction holds beyond the fit range. At Pythia-6.9B and 12B displacement keeps falling, below the six-point line in the interior window (residuals −0.448, −0.239; §3.1, Appendix U).
  • Absolute numbers do not carry across families. Forecasts fixed before measurement at DataDecide, OPT-6.7B and Qwen3-14B lost to a size-blind constant (§4). For an unseen family the 90% interval is about ±2.25 log units (raw readout; post hoc; SD 3 df; §4, p. 6). For a new family we measure the target.

Two families never measured before. For Salamandra (2B→7B) and RedPajama-INCITE (3B→7B), the readout, anchor rule and pass criteria were fixed before any activation was read (new; prospective; Appendix T.4). Ten fixed-text questions reproduced the 100-question level of each 7B model within 0.7% (Salamandra) and 0.3% (RedPajama-INCITE) against a 5% criterion, at 45–53% of the full cost. Measuring only the smaller model, a within-family step rule frozen from 12 earlier families gave log errors −0.209 and +0.246 at 7B (constant: −0.542, +0.556; configuration: −0.950, +0.148); by the pre-set criterion (within 25% and better than both) Salamandra passed (one checkpoint) and RedPajama-INCITE missed (+27.9%). Salamandra was read through a tokenization path replacing about 5% of native tokens, same for anchor and target; a native-tokenization re-measure, specified after scoring, gave the same verdicts (anchor-rule error −0.216 vs −0.209; levels within 0.011 ln). RedPajama-INCITE used native tokenization.

Application. Three uses. (1) Cheap: one forward pass on fixed text, no generation (above). (2) Within a recipe, measured architectures estimate a width or depth change inside their shape range (Gemstones above; post hoc). (3) Understanding: related edits moving one late-layer write toward its input on 964 ARC questions changed choice entropy and correct-option loss, with no consistent direction across hosts (Table 16, p. 22). In a later readout of the saved outputs (development; Appendix T), the edits moved top-choice probability by −19.4 to +19.6 points across four hosts and accuracy by at most 1.35 points, every simultaneous 95% interval including zero (one orthogonal control edit lowered accuracy by 1.1 points on one host). No detection or correction benefit is claimed (AUROC −0.0049, p. 9).

Comments on the paper

8 comments
AuthorsOpenReview ↗

Summary of the revision for all reviewers

We thank all three reviewers. The reviews ask whether the size trend reflects the model or the text it writes, what six Pythia points can support, and whether anything generalizes or is useful. Pages, tables and sections refer to the revised PDF.

Main result. The paper is a quantitative audit: what transfers, what does not, and which measured states, not configuration, record the difference. Within one recipe, late-layer displacement is fit in sample by width and depth and survives a change of text and of task; across families the coefficients do not transfer, and we measure by how much.

Checks beyond the original fit

  1. Identical text. The six-size slope is −0.744 on each model's own text and −0.740 when all eight Pythias read one shared continuation (Table 3, p. 5).
  2. Task stability of the readout (same six checkpoints). The frozen line predicts GSM8K with MALE 0.302, against 1.134 for a constant (Table 4, p. 5).
  3. 22 single-recipe architectures (Gemstones). Adding shape raises in-sample R² from 0.337 to 0.951 (p. 4). Out of sample (post hoc; Appendix T), leave-one-architecture-out MALE is 0.867 / 0.759 / 0.203 for constant / capacity / capacity plus shape. This is interpolation: toward wider models the fit is worse than a constant (1.078 vs 0.998), toward larger ones barely better (0.969 vs 1.026).
  4. Across families. Forecasts fixed before measurement at DataDecide, OPT-6.7B and Qwen3-14B lost to a size-blind constant (§4); for an unseen family the 90% interval is about ±2.25 log units (raw readout; post hoc; SD 3 df; §4, p. 6).

Largest models. We measured twelve 6–14B checkpoints from eight families (Pythia, OPT, GPT-J, Qwen2.5, Qwen3, OLMo-2, BLOOM†, llm-jp-3†); eight are in the PDF (pp. 22, 29). † = Appendix T, forecast fixed before measurement; ‡ = Appendix T, post hoc.

\text{Model (width×depth)} & \text{Result} \cr \hline \text{Pythia-12B (5120×36)} & \text{fixed text → own text −0.71\% (p. 34); six-point line 1.27–1.31× too high (p. 26)} \cr \text{Qwen3-14B (5120×40)} & \text{line through four smaller Qwen3 +5.35\%, through two +70\% (p. 29); fixed text −2.36\% (p. 34)} \cr \text{OPT-13B (5120×40)} & \text{line through OPT-2.7B and 6.7B +6.8\%†} \cr \text{OLMo-2-7B (4096×32)} & \text{fixed text → own text +3.4\%†} \cr \text{Pythia-6.9B, OPT-6.7B, OLMo-2-7B (all 4096×32)} & \text{raw levels differ by 2.3 log units, about tenfold (p. 22)} \end{array}$$ *What generalizes (measured):* one fixed-text forward pass estimates the own-text level within 5% at three of four large checkpoints in the PDF (development; Pythia-12B, Qwen3-8B/14B; Pythia-6.9B −5.4%; p. 34) and at OLMo-2-7B†; within a family, a line through measured smaller sizes can forecast a larger one; mean-centred single-sibling forecasts reach MALE 0.12 on nine 1.7–12B targets (post hoc route, then sealed; the pre-chosen raw route missed). *What differs:* at 6–7B, 66–77% of Pythia's raw gap to OPT and GPT-J is common-mode†; configuration-only forecasts failed on ten large checkpoints (R²ext −0.05, p. 22); at OPT-6.7B, Qwen3-8B/14B and OLMo-2-7B†, Pythia's capacity formula predicts 73–86% too low (pp. 44, 48; Appendices T, U); Qwen2.5 rises from 1.5B to 7B‡ (one step). At llm-jp-3-13b, measured after its forecast was fixed, the centred nearest-sibling forecast was within +0.023 ln of the measured level (band ±0.126)†; the far-anchor (440M) route erred by +0.193 (one checkpoint, low power)†. *Scope:* base models up to 14B; TriviaQA text. **Practical use we claim.** (1) Cheap to measure: one forward pass on fixed text, no generation; the fixed-text level estimates the self-generated level within 5% at 12 of 16 checkpoints (development; p. 5). (2) Within a recipe, a few anchors estimate the effect of a width or depth change inside the anchors' shape range (5 random Gemstones anchors predict the other 17 with median MALE 0.257, post hoc; Appendix T). (3) Understanding: related edits moving one late-layer write toward its input made Qwen2.5-0.5B and Falcon3-1B more confident while lowering the correct option's probability, and two other hosts reversed, so no consistent direction (Table 16, p. 22; cf. entropy neurons, Stolfo et al. 2024; Appendix N). In a later readout of saved outputs (development; Appendix T), the edits moved accuracy by at most 1.35 points across four hosts, every simultaneous 95% interval including zero (one orthogonal control edit lowered accuracy by 1.1 points on one host). We claim no error detection or correction. **Reading path.** §§3–4 (Tables 1–4) answer the review questions; §5 is an extension. A separate comment, "Why local fits stop transferring", maps which conditions move the coefficients. Four more are new and optional ("What the size trend is made of", "What moves a coefficient during training", "What held in blind tests" 1/2, 2/2); no reply depends on them. Parts are now in the PDF (§§2.3, 5, 6; Appendices O, R–U).
AuthorsOpenReview ↗

Why local fits stop transferring: the coefficients are condition-dependent, and we measure how they move

For all reviewers: why size predictions hold inside a recipe's range and fail toward wider and larger models. All are retrospective analyses of families already seen; forecasts fixed in advance are reported separately (§§3.2 and 4; the sealed test in §5.2; §§5.3 and 6).

Prior work. The quantity is ShortGPT's Block Influence (arXiv 2403.03853; related: Gromov et al., arXiv 2403.17887). Scaling coefficients as functions of factors have precedents for loss (Clark et al. 2022, Porian et al. 2024, Li et al. 2025, Han et al. 2026; arXiv 2202.01169, 2406.19146, 2506.10972, 2605.07546), as does cross-family forecasting (Ruan et al. 2024, arXiv 2405.10938; Choshen et al., ICML 2025, arXiv 2410.11840); Sharma & Kaplan (arXiv 2004.10802) tie the loss exponent to data dimension. We apply this workflow to an internal geometric quantity, with coefficient-difference relations, validity ranges and sealed cross-family forecasts. Appendix N now cites them.

1. A shared form, moving coefficients. Within one recipe (Gemstones), log displacement is close to additive in log width and log depth (in-sample R² 0.951, self-generated text; §3); Pythia adds a width×depth interaction (−0.92, Table 6). Across families, recipes, size ranges and stages the coefficients change measurably.

2. Which factor moves which coefficient

  • Size range → capacity slope. Refitting Gemstones on its narrower or smaller half gives capacity coefficients −0.29 and −0.26, against −0.62 on the full grid (post hoc; Appendix T).
  • Depth growth along the width path → effective width slope (item 4).
  • Learning-rate recipe → depth slope. Halving the nominal learning rate changed the depth slope by +0.077, −0.033 and −0.147 at widths 256, 384 and 1280 (98B tokens; Table 18). It also moves with training (E.2); sealed extrapolations across tokens failed (blind-test comment; Appendix T.4).
  • Common-mode state → its recipe response: about 1.36 times the change in the window-start common-mode state (MSE −84%; 29 of 32 segments; needs both states measured; E.3). Post hoc: with window-mean states the leading 2 is algebraic; Gemstones gives 1.94–1.98, Pythia 1.96, SmolLM2 2.14, DataDecide 2.55 (§5; Appendix O.3).
  • Window → depth coefficient: about −1.33 in a fixed relative window, −0.93 in the original (E.1).
  • Window-start write-to-state ratio → level: 69% of ln R variance over 41 checkpoints, 78% over 13 family means (exploratory; B.5; a known identity, credited in the decomposition comment).

3. Why a local fit fails when a factor leaves the anchors' range. A fit inside one range carries that range's coefficients. Inside the Gemstones grid, held-out architectures are predicted well (leave-one-out MALE 0.203). With the narrow-half coefficient, predictions for wider models are about one log unit too high, worse than a constant (1.078 vs 0.998); with the small-half fit, larger models are barely better (0.969 vs 1.026). Reverse directions work: wider→narrower 0.194 vs 0.840, larger→smaller 0.449 vs 1.158 (all post hoc; Appendix T). Pythia-6.9B and 12B also lie below the six-point line in the interior window; including the final block reverses the sign (§3.1, Appendix U). The upward extrapolation failures are consistent with coefficient drift.

4. The difference relation. For two same-family anchors and a target, let Δb_eff be the change in effective width slope between the two consecutive segments, and D(ln L) the matching change in log depth per log width (§5.1):

Δb_eff ≈ −0.035 − 0.926 · D(ln L)

Leave-one-family-out over four development families: MSE vs constant drift −49.6%; llm-jp-3 worsened. With both constants fixed, five further archived families (nine targets) gave −92.9%: four improved, OPT worsened. Pythia supplies 84% of the gain; the other four alone still show −84% (Tables 7, 17). Returning a target's slope needs the two anchors.

5. Why they work, and where. If displacement is locally additive in ln d and ln L along both segments, the change in width slope equals the depth coefficient times D(ln L). The algebra is standard; we measure the constant, its transfer and failures. Limits:

  • Only 1 of 9 test targets lies inside the development range of D(ln L), [0, 0.56].
  • Qwen2.5-7B and OPT-6.7B have almost the same exposure (−0.286, −0.292) but opposite changes (+0.385, −0.133).
  • The recipe factor is condition-specific: predicting a held-out width from the other two is worse than no change (MSE 0.0189 vs 0.0095). Stage changes are not predicted from starting states; for its common-mode analogue, a relation also using target-stage states passed part of a sealed test and failed part (training-coefficient comment; Appendix T). The 24B values rest on one partially admitted package (E.2).
  • The level share partly restates Eq. 2; configuration does not predict the ratio across families (B.5), but one fixed-text pass per family measures it (§4).
AuthorsOpenReview ↗

What the size trend is made of: a decomposition with a known identity as bookkeeping, and three measurements (post hoc; the common-mode identity and slopes are now in §§2.3, 5 and Appendix O.3)

For all reviewers. Apart from Measurement 1 (Appendix B.5) and the common-mode identity with its within-family slopes (now §§2.3, 5 and Appendix O.3), everything here is a retrospective analysis of data seen before (post hoc; not in the PDF), with each number recomputed by a separate script. It is optional reading; no reply to an individual reviewer depends on it.

What we do not claim. R is ShortGPT's Block Influence. With ρ = |write|/|state| and α = cos(state, write), ln R ≈ 2 ln ρ + ln(1−α²) − ln 2 for small ρ is the law of cosines; ShortGPT v3 (Appendix A) bounds the cosine from below by the update-to-state norm ratio for randomly initialised pre-norm models, and the orthogonal-write case is in Figliolia & Millidge (arXiv 2606.16112). The random-walk null model (state variance growing linearly with depth, so ρ_j² ≈ 1/j, late-layer R ≈ 1/(2L), depth slope −1) follows from De & Smith (2020, arXiv 2002.10444) and Xiong et al. (2020, arXiv 2002.04745); an informal blog note by Liu (2026) states 1−cos ∼ 1/l under random initialization. We use these only as bookkeeping; Appendix N cites them.

Measurement 1: how the variance splits. Over 41 checkpoints the window-start ratio ρ₀ carries 0.69 of the variance of ln R and the direction profile 0.15; over 13 family means, 0.78 and 0.14 (Appendix B.5; exploratory). Because the 2 is algebraic, only the shares are informative, and configuration does not predict ρ₀ across families.

Measurement 2: direction terms by factor. Write a factor's slope on ln R as m × (its slope on ln ρ) + a direction term. For size and tokens (raw readout), m is the exact multiplier at measured ρ, 1.46–1.80 rather than 2 (2 is the small-ρ limit; measured multipliers agree with this formula in four families and not in one, so this is unresolved). Size: −0.043 [−0.062, −0.025] pooled over five families, no detected heterogeneity (p = 0.14; the pool mixes fixed-token and release-path estimates). Training tokens: not homogeneous, −0.029 in Gemstones against +0.07 to +0.08 in OLMo-2, Pythia and SmolLM2 (p = 1.5e−9). Learning rate and depth are estimable in one family only, width in none. For depth (Gemstones, 22 shapes, centred, m = 2), the ln R slope −0.91 = 2 × (−0.59) + 0.27: the ρ part is steeper than the total, and the direction term +0.27 [+0.22, +0.31] gives part of it back, 87% of it through the finite-ρ correction of the exact formula, not ln(1−α²) (on the raw readout with the exact multiplier, the depth term is +0.05 [−0.04, +0.17]). This is consistent with Appendix B.2 ("depth effects run entirely through write size") and is an accounting split, not evidence that α plays no role: in raw coordinates about 0.10 of the term (raw depth slope −1.30) runs through α.

Measurement 3: the null model's depth slope. In four Gemstones models of width 256 (depths 23–80), ρ scales with relative depth with exponent 0.499 (leave-one-shape-out 0.488–0.513), compatible with the null's ½. Inside the paper's late window, however, the local exponent is 0.576: the window slope of ln ρ, −0.601, is −0.522 from the null, −0.098 from the profile of write sizes along layers, −0.001 from write correlations and +0.020 from the embedding. The null does not give the level: family means lie 1.1–2.5 times above it. So it is a reference line for the depth slope, not an explanation of levels or family differences; one family, one width.

The 2 in the common-mode relation. If the shared-mean share s_cm stays constant across the window, the leading term of raw minus centred ln R is exactly 2Z (Z = ½ ln(1−s_cm)): the 2 is algebra, and departures from it cannot all be assigned to common-mode motion. Appendix O.3 gives the derivation; it does not cover the per-token averaging of the readout, and the leading term fails as s_cm → 1. With window-mean states the measured slopes are 1.94–1.98 (Gemstones), 1.96 (Pythia), 2.14 (SmolLM2) and 2.55 (DataDecide, across recipes; 1.72 within a recipe lineage); the heterogeneity rests on DataDecide alone (p = 0.33 without it). Appendix E.3's 1.36 uses the window-start state, a different regressor. GPT-Neo's two checkpoints, almost entirely one shared direction (shares 0.98 and 0.96), are the only such case: with them the cross-family slope is 1.59, without them 2.03, and one family cannot separate a changed relation at high share from a family-specific effect.

What we contribute. Measurements 1–3 (2 and 3 post hoc, not in the PDF), and the use of the 2Z identity to separate the common-mode part of the readout (Appendix O.3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.