- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper shows how a fixed parameter budget should be divided between depth and width by proposing the Frobenius norm of the input average gradient outer product as a measure of response strength and overlap. It reports correlations between the norm and loss in sparse reconstruction and language models, gives a regression example where both share a fitted-noise term, and fits a preferred-depth trajectory across budgets
Critical strengths
- The reviewer sees the importance of the problem - as it is crucial to allocating parameters between width and depth. A good solution has practical value.
- The sweep done by authors across a large set of configurations is useful emperical work.
Critical weaknesses
Novelty is one of the concerns, additionally the experiments don't justify the said claims very well:
- Within-budget correlations do not sufficiently separate the proposed explanation from a simpler depth effect, since shallow models have much higher loss (and F) giving them substantial influence on Pearson correlation with unlogged F.
- The mechanism is not established in the nonlinear models. The regression calculation identifies a shared fitted-noise term, but the toy targets are noiseless and the corresponding error component is not identified in the Transformers.
- The toy correlation is not convincing across the full trajectory. By removing n=200,500 is reducing pearson correlation from about 0.98 to 0.48.
- The measurement is not sufficiently stable for fine comparisons between shapes, since doubling the number of examples changes F by roughly 14 to 21% and reverses the ordering of depths 24 and 32. Tangent coordinates also change which depth minimizes F, and these checks cover only one budget.
- The depth result clearly needs stronger validation, since the exponent 0.459 comes from a descriptive validation loss fit without reported uncertainty. Also, 112/152 configs have only one seed, and loss difference is very small.
- The novelty over prior work on input sensitivity and double descent [1,2] is not clear to me. The paper needs to explain what the AGOP overlap measure adds beyond these results and the simpler trace diagnostic (Q3)
[1] Gamba, M., Englesson, E., Björkman, M., & Azizpour, H. Deep Double Descent via Smooth Interpolation. Transactions on Machine Learning Research (TMLR), 2023. [2] Dherin, B., Munn, M., Rosca, M., & Barrett, D. G. T. Why Neural Networks Find Simple Solutions: The Many Regularizers of Geometric Complexity. NeurIPS, 2022.
Questions for authors
- See W1 - can the author to establish what (F) explains beyond depth, width and trace, particularly among competitive shapes?
- See W2 - Appnedix E requires assumptions about error directions and gradient cross terms that are not verified. Can this be justified?
- See W6, Can authors explain what the AGOP overlap measure adds beyond these results and the simpler trace diagnostic?
- What justifies describing the relationship as approximately affine across the full trajectory, given the influence of the two peak configurations?
- What connects the independently fitted depth law to the gradient analysis? Does the proposed mechanism constrain the exponent 0.459?