acceptodds
Under review as a conference paper at ICLR 2027

Understanding Network Size Scaling through Linear Region Geometry

Abstract

Neural network test losses are observed to evolve as power laws with the network size, but there is no theory that can fully predict their parameters after training. We present an empirical study of the learned properties that describes the scaling by exploiting the linear region partition of ReLU networks. We show that trained neural networks do not constitute ideal piecewise-linear approximators and identify that the regression loss is dominated by deviations of regional affine offsets from their optimal values. This offset error is described locally as the first-order variation of the target over a characteristic regional scale, and its mean depends approximately on the square of that same scale averaged over high-gradient regions. These results connect neural scaling to measurable properties of the piecewise-affine structure of networks and target geometry.

Primary area:
learning theory
Keywords:
Neural scaling laws, ReLU networks, linear regions, piecewise-affine approximation, approximation theory, neural network geometry.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KiEEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the neural scaling laws of piecewise-linear networks. By decomposing the regional MSE of trained ReLU MLPs into an affine offset error, an affine slope error, and a non-affine remainder, the authors find that the offset error dominates the total loss and decays the slowest. They introduce a characteristic "crossing distance" and show that the offset error follows a quadratic power law with respect to this distance when restricted to the top 5-10% highest-gradient regions.

Critical strengths

None

Critical weaknesses

The core theoretical mechanism seems to be tautological and lacks predictive power. The scope of this paper is very limited: I do not know whether the insight of this paper still holds if we consider attention layer or other activation function such as SwiGLU, which is widely used in modern LLMs.

The empirical scaling results appear sensitive to several post-hoc choices, including the selection of high-gradient regions and the removal of larger networks when fitting the power law. It is unclear whether the reported exponents reflect a genuine scaling mechanism or are largely consequences of these choices.

Questions for authors
  1. Since the crossing distance is constructed from the network's predictions, Eq. (11) holds regardless of how the network is trained. Would the same relationship hold if we simply perturbed the output bias of a trained network while keeping its linear regions unchanged? If so, what learning-specific mechanism does this analysis actually reveal?

  2. Eq. (11) already implies that the squared offset error scales quadratically with the crossing distance when gradients are approximately fixed. Isn't the observed quadratic law therefore expected, especially after selecting high-gradient regions?

  3. All experiments use small ReLU MLPs on low-dimensional synthetic regression tasks. It is unclear whether the insights extend to modern architectures involving attention or smooth activations such as SwiGLU, where the piecewise-affine decomposition no longer directly applies. Which parts of the proposed mechanism are expected to generalize, and what evidence supports this expectation?

4 comments

We thank the reviewer for valuable feedback. We respond below to their questions and critical weakness points and describe the changes we have made to the paper in response to them. We would be happy to address any remaining issues during the discussion period.

[W1] Our paper does not propose a theoretical mechanism to predict network scaling, our goal is to present a new descriptive framework that can allow for an examination of which network and data features affect the scaling. We hope that the restructure of the content of the paper and the new empirical results that we have added in Sections 4.2 and 4.3 help illustrate this point. We fully acknowledge that our description is limited to neural networks that can be represented as piecewise-linear functions, but this is in line with previous studies in the literature, which restrict their studies to simple setups with fully connected neural networks or even simpler frameworks such as random feature linearized models. We cite several of these works in Section 2 (lines 86-88). Nevertheless, we plan an extension to setups with smooth nonlinear operations for future work.

[W2] We disagree with the assertion that our results are due to the result of post-hoc choices. The removal of larger networks is done to reproduce the limit of infinite data and compute that is often considered in the literature, and the selection of high-gradient regions is done after observing that the largest fraction of the error is concentrated in that part of the target space and can be used as a reliable variable for the scaling. In order to increase the transparency of our results, we have included a study of the validity of the first-order characterization in Appendix B and a more detailed description of our fitting method in Section 3.2 (lines 237-240).

[Q1] Yes, Eq. (11) (Eq. (14) in the new draft version) holds regardless of how the network is trained, but our main result regarding that parametrization is that the offset error can be reliably approximated by the first-order term in the expansion. That condition may not hold if we apply a perturbation to the output bias. Beyond that, we do not intend to use Eq. (11) as a predictive tool, but rather to justify the crossing distance as a descriptive scaling variable.

[Q2] Yes, we rely on that expectation for that part of the argument. However, our goal with that characterization is to use it to make two non-trivial observations: 1) the crossing distance distribution can scale in a heterogeneous way and that affects the scaling when you have an irregular gradient distribution, 2) the total offset error can be characterized as a constant approximation error as a function of the median crossing distance over the high-gradient regions. We have rewritten parts of Section 4.2 to make this point clearer.

[Q3] We plan to explore how this description can be extended to transformers and networks with smooth activations in future work. Our preliminary studies indicate that the scaling exponents are higher for MLPs with GeLU activations, but we cannot explain that change using our framework directly due to the lack of a region partition. We have edited Section 5 to remark this limitation explicitly (lines 533-535). Furthermore, we now show in the new Section 4.3 that our conclusions can be extended to CNNs, which represent more complex piecewise-affine architectures. Still, we believe that a full description of the scaling at the piecewise-linear regime in terms of network and data features would still be very valuable, if anything as an intermediate step towards understanding the scaling of more complex architectures.

As the discussion period is coming to a close, we would like to kindly ask whether our responses and additional experiments have satisfactorily addressed your concerns. We would be happy to provide any further clarification if needed.

Thank you for the detailed response and the revision. I appreciate the additional experiments and analyses. However, my main concerns remain. The quadratic scaling is largely expected from the Taylor expansion, as you now acknowledge. Moreover, since the crossing distance is constructed using the prediction error, I am still not convinced that the proposed framework provides much new insight beyond describing the observed errors.

Although the revised paper presents more empirical results, it is unclear what we can actually learn from these observations about neural network scaling. I understand that theoretical research in deep learning can be complex, but the complexity of a new framework should be justified by the insights it provides. This paper introduces considerable conceptual machinery, yet it remains difficult to identify what new understanding of neural network scaling we actually gain from it. I therefore struggle to see the value of introducing such a complicated framework to the community, and I will maintain my original recommendation.

We thank the reviewer very much for their response. We completely understand the reservations mentioned about the insights of our work, but we do not believe that the quadratic scaling expectation invalidates or makes obvious any of the other contributions that we make and we have to insist that the crossing distance is not computed using the prediction error. As a final plea, we would like to list here the contributions that our paper makes so that our perspective is made clear:

  1. We present a decomposition of the regression loss into a sum of terms that quantify the deviation of the network region offsets and slopes from their ideal conditions, and a remainder term that captures the nonlinear features of the target function. We show through this decomposition alone that the loss and its scaling is dominated by the offset error.
  2. We demonstrate that the offset error can be characterized locally through a linear expression that depends on measurable features of both the network and the target.
  3. We show that the target gradient distribution plays a pivotal role in the scaling of the offset error. More specifically, we demonstrate that the average region size can be used as a scaling variable as the gradient distribution becomes more constant, and that its exponents get reduced as the gradient width grows, while also explaining what factors produce that evolution.
  4. We demonstrate that the scaling can be determined just by examining the regions in the high-gradient limit.
Reviewer_vA6EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how well ReLU networks approximate arbitrary functions as a function of network size. Specifically, the paper decomposes approximation error into the gap between the network and the best piecewise affine approximation, and the remaining gap. The first gap is shown to be dominant, and roughly constant within each affine region. This gap is shown to be linked to a measure of region size, allowing a full characterization of the scaling of approximation error with respect to network size.

Critical strengths
  1. The primary strength is the paper's novelty: the paper proposes an approximation theory-based approach to deriving a scaling exponent wrt model scale. This is a fresh perspective that connects the literature on viewing ReLU networks as piecewise affine functions to the scaling law literature. Moreover, the results are clean, intuitive and provide testable predictions about scaling exponents.
Critical weaknesses
  1. The biggest weakness is that experiments are only conducted on a simple, synthetic dataset. I understand that a synthetic dataset may be necessary for most of the empirical analysis, but this casts doubt on whether the results are an artifact of the particular quadratic-like function that the ReLU networks fit, or are more general principles. I suspect the qualitative results are fairly general, but it would be good to have some indications of this from real data.
  2. Less critical, but the theory introduced in this paper is not formalized with theorems and proof. Addressing this point makes the first concern less important: it shifts the paper's contributions away from empirics and towards theory.
Questions for authors
  1. Can the experiments in the paper be performed on real datasets, or at least more complex synthetic datasets?
3 comments

We thank the reviewer for valuable feedback. We respond below to their questions and critical weakness points and describe the changes we have made to the paper in response to them. We would be happy to address any remaining issues during the discussion period.

[W1] We believe that this is a very fair criticism, and we address it by extending our analysis to two new target types that do not exhibit this quadratic-like structure: Gaussian random fields, and a PDE dataset that models the one-dimensional viscous Burgers equation. We indeed find in the updated Section 4.2 and the new Section 4.3 that the main conclusions from our study can be extended to a diverse range of datasets and architectures.

[W2] The descriptions and results presented in our paper are meant to be purely empirical, so we believe that formalization efforts through theorems and proofs fall beyond our scope. We hope that the expanded results we have included in this updated version of the paper better justify this decision.

[Q1] Yes, we have extended our analysis to both more complex synthetic datasets and to real datasets, as we describe in our W1 reply. We believe that our new results in Sections 4.2 and 4.3 illustrate that the main conclusions of our initial analysis can be mostly extended to a wider range of target features. Furthermore, we also show that it is possible to identify and describe the effects of concrete data features on the offset error scaling.

Thank you for the additional experiments. I think they are valuable and I am willing to reconsider my assessment.

However, I am concerned that the experiments are still quite small scale and conducted on synthetic problems. Would the same results hold in more practical settings (even something like MNIST)? I understand that the experiments may require a generative model rather than just a fixed dataset; perhaps one can be trained on an existing dataset?

Thank you very much for reconsidering your assessment about our paper. Concerning an extension of our results to a dataset such as MNIST, yes, it would be possible to reproduce our region analysis by using a generative model that models its logit distribution. We did a preliminary study in this direction by training a sequence of MLPs on the MNIST dataset and measuring the scaling of the test cross entropy with respect to the median region size. We find a scaling exponent of 0.84±0.030.84 \pm 0.03, which seems to indicate that the scaling is set by an offset error analogue for the cross entropy loss with hard labels, for which we expect a LCE∼sRL_{CE} \sim s_R scaling law. However, this is a very early result, we would still need to validate that a generative model is faithful enough to reproduce the target and fully adapt our loss decomposition to a classification task.

We would also like to remark that the benefit of our method is not that it produces the same results for every task or dataset, but rather that it is a general tool to explore which properties of both the network and the data influence the scaling. In that sense, we believe that our approach can be readily applied to a wide range of targets.

Reviewer_RCk7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This study connects the piecewise-affine structure of learned networks and its properties to the model-size scaling law of the test loss. The authors consider multi-layer networks with piecewise-linear (ReLU) activations on regression tasks, which partition the input space into linear regions.

Within each region RR, they decompose the regression error into (i) an offset error, (ii) a slope error, and (iii) a non-affine remainder, and measure each term. They construct datasets from Pfaffian target functions, train 3-hidden-layer ReLU MLPs, and fit each term to a power law in quantities such as the median region size to compare the exponents. They show that the offset error dominates the loss and that it is described not by the region size but by the product of the target gradient and the crossing distance. Moreover, they show that when the median crossing distance over high-gradient regions is used as the scale, the offset error approximately follows a quadratic power law in it.

Critical strengths

(1) Based on the region partition induced by piecewise-linear activation functions, the paper directly measures, on trained networks, the ideal affine approximation assumed by existing explanations of scaling laws, and shows that this assumption does not hold.

(2) By binning the regions according to ∣∇f(ξR)∣|\nabla f(\xi_R)| to control for the effect of the gradient, the paper shows that the exponent of 2 is recovered when the crossing distance is measured in the high-gradient regions where the error actually arises. Moreover, a weighted sum of the per-bin power laws reproduces the total offset error within 12%.

Critical weaknesses

(1) Lack of motivation.

The authors point out that previous work relies on idealized assumptions and simplified settings, but for the spectral and solvable-model line of work they do not sufficiently explain which specific assumptions are problematic and why. Moreover, their own experiments also use a simplified setting, namely low-dimensional synthetic regression with small MLPs, so it should be explained why their criticism of prior work does not apply to this study as well.

(2) Limited validation of the first-order description.

The claim in Figure 1 that the first-order approximation accurately reproduces the offset error rests on a qualitative comparison for a single network on a single dataset. It would be better to report quantitative metrics such as R2R^2 or the relative error across all datasets and model sizes, and to show the effect of the R0R_0 cut as well.

Questions for authors

Q1. What were the criteria for selecting the datasets? Is there a particular reason for considering only the gauss and tanh target families?

Q2. Is the approximately quadratic power law of the offset error with respect to the median crossing distance over high-gradient regions also observed in experiments with other datasets?

Q3. What results would be obtained in experiments with other model architectures that use piecewise-linear activation functions (e.g., CNNs)?

6 comments

We thank the reviewer for valuable feedback. We respond below to their questions and critical weakness points and describe the changes we have made to the paper in response to them. We would be happy to address any remaining issues during the discussion period.

[W1] We have edited Section 2 (lines 83-91) to point out the shortcomings of established theoretical descriptions and added two sentences to the introduction justifying our target choice. The main difference between existing theories and our approach is that we seek to examine which features of the region partition and the target function affect the scaling of real trained networks. Working with low-dimensional synthetic datasets streamlines our analysis, but there is no technical limitation to extend our approach to more complex architectures or datasets, as we demonstrate through our new results in Sections 4.2 and 4.3. However, we also acknowledge that our description is limited to settings where neural networks can be represented as piecewise-affine functions and we have full access to the target functions. We have added a couple of sentences to Section 5 (lines 533-536) in our paper pointing this out explicitly.

[W2] We have added a new Appendix B where we study in detail the validity of the first-order description and the effects of the R0R_0 filter we work with. We measure for all our networks trained on the gauss, tanh and GRF datasets the error of our approximation and the fraction of regions and error magnitude that is kept after applying a R0R_0 cut. We also study the effect of different cuts, their impact at different network widths, and how they affect the scaling exponents. Our main conclusions are that most of the offset errors are well approximated by the first-order characterization, and that their scaling can be reliably described by the crossing distance distribution after a R0<0.5R_0 < 0.5 filter.

[Q1] The main motivation for selecting the datasets was the pretense of studying the relation between the scaling and the Pfaffian parameters, which provide a formal notion of complexity for the target functions. The choice of gauss and tanh targets was done to cover two distinct gradient distribution structures, which cannot be specified through the Pfaffian characterization. We have now expanded the scope of our analysis in this direction by considering two extra dataset types: gaussian random fields (GRF) and the Burgers equation from PDEBench, which cover a wider range of target features.

[Q2] Yes, we show that is the case for the new datasets considered in our experiments in the updated Figure 3, in Table 16 and in Table 20.

[Q3] In order to answer this question, we have trained a set of MLPs and CNNs on the Burgers dataset in the new Section 4.3. Our main observation is that the description we work with can be extended to CNNs without issue and that all the results concerning the offset error scaling are qualitatively similar between the two architectures. We also observe that for the CNN almost all of the loss is comprised by the offset error.

As the discussion period is coming to a close, we would like to kindly ask whether our responses and additional experiments have satisfactorily addressed your concerns. We would be happy to provide any further clarification if needed.

Dear Authors,

Thank you for your detailed responses and revisions.

[W1] Background and Related Work

I understand the motivation for studying features of the region partition. However, I think the discussion of why simplifications are problematic still needs more detail. For example, it is unclear why assuming

  • spectral kernel regression
  • random feature models

is problematic, and why addressing these limitations motivates the study of linear region partitions.

In my understanding, this work has an independent motivation distinct from the existing studies. I consider this a clear strength, as it offers a new perspective. In the related work section, I suggest explicitly identifying the relevant prior studies rather than broadly describing their limitations as "simplified settings."

[W2] Approximation Validation

I have confirmed that the offset error is well approximated by the first-order characterization.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all the reviewers very much for their time and constructive feedback. We have made substantial changes to the paper in order to address all their comments. Our main updates are the following:

  • Extension of our analysis to a wider range of datasets. We present additional measurements on two types of datasets: Gaussian random field (GRF) target functions and a dataset built from the PDEBench repository which models the one-dimensional viscous Burgers equation. Furthermore, we also study variants of our original Pfaffian targets, and extend their input dimension range to 7.
  • Study of the dependence of the scaling on target features and network depth. We add a new paragraph to Section 4.2 where we study the dependence of the description of the scaling provided by our frameworks on concrete features of the target functions and also on the network depth. We identify and describe a clear relation between the width of the target gradient distributions and the scaling of the offset.
  • Extension of our analysis to more architectures. We add Section 4.3, which compares the scaling of MLPs and CNNs on the Burgers dataset. Additionally, we also study how the scaling changes with network depth by training MLPs with 3 and 5 layers.
  • Better analysis documentation. We add Appendix A, where we expand the discussion of technical aspects of our analysis, which includes network optimization, fitting strategy and how we extend our analysis to the Burgers dataset in the absence of a closed form expression for the target.
  • Dedicated study of the first-order error description. We add Appendix B, where we study in detail the validity of our first-order description and the impact of the R0R_0 cut on our measurements.
  • Relocation of some results to appendices. We move the discussion of the relation between the crosssing distance and region size distributions to Appendix C, and our analysis of the offset error, gradient and crossing distance distributions to Appendix D. We have made this choice because the relative importance of those parts of our analysis has been diminished in light of the new results that we have incorporated into the paper, and we believe that putting them outside of the main text highlights the main contributions of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.