acceptodds
Under review as a conference paper at ICLR 2027

The Model Shape Behind Scaling Laws: A Gradient Superposition Perspective

Abstract

Neural scaling laws guide resource allocation, but how should a fixed parameter budget be divided between depth and width? Motivated by inverse-width representation-error scaling under strong superposition, we study this allocation through the network's end-to-end response. The Neural Feature Ansatz links weight and gradient geometry, motivating the Frobenius norm of the input average gradient outer product (AGOP) as an observable of response strength and overlap. By reproducing Anthropic's toy-model double-descent experiment, we find that and test reconstruction loss follow an approximately affine relationship across the full sample-size sweep and share an interpolation peak. Through Gaussian minimum-norm regression experiments, we further show that this correlation arises from a fitted-noise amplification term shared by prediction error and . By pretraining language models spanning different shapes and parameter budgets, we find a strong positive correlation between loss and among the best shapes selected by validation loss. This supports interpreting effective parameter allocation as preserving task-relevant computation while reducing the amplification of error responses. Our results identify error-response amplification as a mechanism linking gradient superposition to generalization, providing a functional account of depth-width allocation and efficient parameter utilization.

Primary area:
interpretability and explainable AI
Keywords:
Scaling Law, Mechanistic Interpretability, Feature Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qGA7OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper shows how a fixed parameter budget should be divided between depth and width by proposing the Frobenius norm of the input average gradient outer product as a measure of response strength and overlap. It reports correlations between the norm and loss in sparse reconstruction and language models, gives a regression example where both share a fitted-noise term, and fits a preferred-depth trajectory across budgets

Critical strengths
  1. The reviewer sees the importance of the problem - as it is crucial to allocating parameters between width and depth. A good solution has practical value.
  2. The sweep done by authors across a large set of configurations is useful emperical work.
Critical weaknesses

Novelty is one of the concerns, additionally the experiments don't justify the said claims very well:

  1. Within-budget correlations do not sufficiently separate the proposed explanation from a simpler depth effect, since shallow models have much higher loss (and F) giving them substantial influence on Pearson correlation with unlogged F.
  2. The mechanism is not established in the nonlinear models. The regression calculation identifies a shared fitted-noise term, but the toy targets are noiseless and the corresponding error component is not identified in the Transformers.
  3. The toy correlation is not convincing across the full trajectory. By removing n=200,500 is reducing pearson correlation from about 0.98 to 0.48.
  4. The measurement is not sufficiently stable for fine comparisons between shapes, since doubling the number of examples changes F by roughly 14 to 21% and reverses the ordering of depths 24 and 32. Tangent coordinates also change which depth minimizes F, and these checks cover only one budget.
  5. The depth result clearly needs stronger validation, since the exponent 0.459 comes from a descriptive validation loss fit without reported uncertainty. Also, 112/152 configs have only one seed, and loss difference is very small.
  6. The novelty over prior work on input sensitivity and double descent [1,2] is not clear to me. The paper needs to explain what the AGOP overlap measure adds beyond these results and the simpler trace diagnostic (Q3)

[1] Gamba, M., Englesson, E., Björkman, M., & Azizpour, H. Deep Double Descent via Smooth Interpolation. Transactions on Machine Learning Research (TMLR), 2023. [2] Dherin, B., Munn, M., Rosca, M., & Barrett, D. G. T. Why Neural Networks Find Simple Solutions: The Many Regularizers of Geometric Complexity. NeurIPS, 2022.

Questions for authors
  1. See W1 - can the author to establish what (F) explains beyond depth, width and trace, particularly among competitive shapes?
  2. See W2 - Appnedix E requires assumptions about error directions and gradient cross terms that are not verified. Can this be justified?
  3. See W6, Can authors explain what the AGOP overlap measure adds beyond these results and the simpler trace diagnostic?
  4. What justifies describing the relationship as approximately affine across the full trajectory, given the influence of the two peak configurations?
  5. What connects the independently fitted depth law to the gradient analysis? Does the proposed mechanism constrain the exponent 0.459?
Reviewer_u3HcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper studies the Frobenius norm (denoted F) of a model's input average gradient outer product. The paper begins with analysis of a toy linear model setting, finding that F correlates with loss throughout the training trajectory in a double descent experiment. The result is then explain theoretically by connecting F to a fitted noise amplification term. The paper then presents a set of language modeling experiments with sweeps over parameter count and width/depth allocation using parameter counts ranging from 0.3M to 50M. A method to fairly compare F across model widths is introduced, the paper then finds a correlation between language modeling loss and F. Using the checkpoints from the language modeling sweep a preferred depth trajectory is fit to predict the optimal model width for a given parameter budget.

Critical strengths

The section dealing with the toy model is strong, AGOP norm F is convincingly found to empirically correspond with loss, and the connection is explained theoretically.

The language modeling experiment includes a large sweep of architectural configurations and show a convincing correlation between language modeling loss and AGOP norm F.

Critical weaknesses

The paper does not draw a convincing connection between the AGOP norm F and the allocation of width and depth in a model. The paper contains good study of F in a toy setting and then convincingly argues that F is correlated with language modeling loss in a larger experimental sweep. The paper then analyzes how the allocation of model width and depth corresponds to language modeling loss and fits a preferred depth trajectory to guide the allocation. However, a connection between F and the width/depth allocation is not established, outside of the fact that F is correlated with loss, and loss is connected to the width and depth allocation. The paper claims on lines 126-127 that "If shape at matched P mainly changes error-response amplification, lower-loss shapes should occupy a lower-F region of the same budget", but does not demonstrate a relationship between model shape and error-response amplification in the language modeling setting. It remains unclear what the AGOP norm perspective has to tell us about model architecture. As written, the paper's AGOP norm material and width/depth material form separate lines of work.

Questions for authors

What is the relationship between F and model architecture for checkpoints with similar loss but different width/depth configs? Relationships between width/depth and loss are well established in the literature, your paper finds that AGOP norm is correlated with loss, so analysis of an empirical connection between model architecture to the AGOP norm / error-response amplification must be account for that.

Reviewer_HENFOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper claims a small Frobenius norm of the input average gradient outer product (AGOP) inversely correlates with better generalization, demonstrates it on a toy autoencoder experiment, and argues it also holds in a language model experiment.

Critical strengths

The paper states a clear hypothesis and runs several experiments to test it.

Critical weaknesses
  1. The paper significantly overclaims its theoretical contribution. Proposition 1 is the classical ridgeless decomposition in linear regression with no ties to depth-width trade-off that is of interest to this paper, and there is no direct connection to the "superposition" literature, which are inherently about nonlinear models.
  2. The only non-toy experiments on language model is weak. The model has O(10M) parameters and is trained on O(1B) tokens, which is nowhere near the interpolation regime where the proposed generalization metric bears relevance. Instead, the loss is mostly approximation or optimization error. There is little room for a fitted-noise variance term, which only matters when the model can fit noise along poorly constrained directions.
Questions for authors

Can you report the training and validation loss separately for the language model experiments to see if there is in fact any generalization gap that your theory is trying to characterize?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.