- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates the neural scaling laws of piecewise-linear networks. By decomposing the regional MSE of trained ReLU MLPs into an affine offset error, an affine slope error, and a non-affine remainder, the authors find that the offset error dominates the total loss and decays the slowest. They introduce a characteristic "crossing distance" and show that the offset error follows a quadratic power law with respect to this distance when restricted to the top 5-10% highest-gradient regions.
Critical strengths
None
Critical weaknesses
The core theoretical mechanism seems to be tautological and lacks predictive power. The scope of this paper is very limited: I do not know whether the insight of this paper still holds if we consider attention layer or other activation function such as SwiGLU, which is widely used in modern LLMs.
The empirical scaling results appear sensitive to several post-hoc choices, including the selection of high-gradient regions and the removal of larger networks when fitting the power law. It is unclear whether the reported exponents reflect a genuine scaling mechanism or are largely consequences of these choices.
Questions for authors
-
Since the crossing distance is constructed from the network's predictions, Eq. (11) holds regardless of how the network is trained. Would the same relationship hold if we simply perturbed the output bias of a trained network while keeping its linear regions unchanged? If so, what learning-specific mechanism does this analysis actually reveal?
-
Eq. (11) already implies that the squared offset error scales quadratically with the crossing distance when gradients are approximately fixed. Isn't the observed quadratic law therefore expected, especially after selecting high-gradient regions?
-
All experiments use small ReLU MLPs on low-dimensional synthetic regression tasks. It is unclear whether the insights extend to modern architectures involving attention or smooth activations such as SwiGLU, where the piecewise-affine decomposition no longer directly applies. Which parts of the proposed mechanism are expected to generalize, and what evidence supports this expectation?
4 comments
As the discussion period is coming to a close, we would like to kindly ask whether our responses and additional experiments have satisfactorily addressed your concerns. We would be happy to provide any further clarification if needed.
Thank you for the detailed response and the revision. I appreciate the additional experiments and analyses. However, my main concerns remain. The quadratic scaling is largely expected from the Taylor expansion, as you now acknowledge. Moreover, since the crossing distance is constructed using the prediction error, I am still not convinced that the proposed framework provides much new insight beyond describing the observed errors.
Although the revised paper presents more empirical results, it is unclear what we can actually learn from these observations about neural network scaling. I understand that theoretical research in deep learning can be complex, but the complexity of a new framework should be justified by the insights it provides. This paper introduces considerable conceptual machinery, yet it remains difficult to identify what new understanding of neural network scaling we actually gain from it. I therefore struggle to see the value of introducing such a complicated framework to the community, and I will maintain my original recommendation.
We thank the reviewer very much for their response. We completely understand the reservations mentioned about the insights of our work, but we do not believe that the quadratic scaling expectation invalidates or makes obvious any of the other contributions that we make and we have to insist that the crossing distance is not computed using the prediction error. As a final plea, we would like to list here the contributions that our paper makes so that our perspective is made clear:
- We present a decomposition of the regression loss into a sum of terms that quantify the deviation of the network region offsets and slopes from their ideal conditions, and a remainder term that captures the nonlinear features of the target function. We show through this decomposition alone that the loss and its scaling is dominated by the offset error.
- We demonstrate that the offset error can be characterized locally through a linear expression that depends on measurable features of both the network and the target.
- We show that the target gradient distribution plays a pivotal role in the scaling of the offset error. More specifically, we demonstrate that the average region size can be used as a scaling variable as the gradient distribution becomes more constant, and that its exponents get reduced as the gradient width grows, while also explaining what factors produce that evolution.
- We demonstrate that the scaling can be determined just by examining the regions in the high-gradient limit.
We thank the reviewer for valuable feedback. We respond below to their questions and critical weakness points and describe the changes we have made to the paper in response to them. We would be happy to address any remaining issues during the discussion period.
[W1] Our paper does not propose a theoretical mechanism to predict network scaling, our goal is to present a new descriptive framework that can allow for an examination of which network and data features affect the scaling. We hope that the restructure of the content of the paper and the new empirical results that we have added in Sections 4.2 and 4.3 help illustrate this point. We fully acknowledge that our description is limited to neural networks that can be represented as piecewise-linear functions, but this is in line with previous studies in the literature, which restrict their studies to simple setups with fully connected neural networks or even simpler frameworks such as random feature linearized models. We cite several of these works in Section 2 (lines 86-88). Nevertheless, we plan an extension to setups with smooth nonlinear operations for future work.
[W2] We disagree with the assertion that our results are due to the result of post-hoc choices. The removal of larger networks is done to reproduce the limit of infinite data and compute that is often considered in the literature, and the selection of high-gradient regions is done after observing that the largest fraction of the error is concentrated in that part of the target space and can be used as a reliable variable for the scaling. In order to increase the transparency of our results, we have included a study of the validity of the first-order characterization in Appendix B and a more detailed description of our fitting method in Section 3.2 (lines 237-240).
[Q1] Yes, Eq. (11) (Eq. (14) in the new draft version) holds regardless of how the network is trained, but our main result regarding that parametrization is that the offset error can be reliably approximated by the first-order term in the expansion. That condition may not hold if we apply a perturbation to the output bias. Beyond that, we do not intend to use Eq. (11) as a predictive tool, but rather to justify the crossing distance as a descriptive scaling variable.
[Q2] Yes, we rely on that expectation for that part of the argument. However, our goal with that characterization is to use it to make two non-trivial observations: 1) the crossing distance distribution can scale in a heterogeneous way and that affects the scaling when you have an irregular gradient distribution, 2) the total offset error can be characterized as a constant approximation error as a function of the median crossing distance over the high-gradient regions. We have rewritten parts of Section 4.2 to make this point clearer.
[Q3] We plan to explore how this description can be extended to transformers and networks with smooth activations in future work. Our preliminary studies indicate that the scaling exponents are higher for MLPs with GeLU activations, but we cannot explain that change using our framework directly due to the lack of a region partition. We have edited Section 5 to remark this limitation explicitly (lines 533-535). Furthermore, we now show in the new Section 4.3 that our conclusions can be extended to CNNs, which represent more complex piecewise-affine architectures. Still, we believe that a full description of the scaling at the piecewise-linear regime in terms of network and data features would still be very valuable, if anything as an intermediate step towards understanding the scaling of more complex architectures.