- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies the Gaussian popoulation loss landscape of bias-free, shallow NNs in the teacher-student setting. More specifically, they construct a plain network with spurious local minima. They then show that adding a learned skip connection removes those spurious local minima when the student and the teacher have some extra assumptions. This is a loss-landscape paper not a training dynamics paper. The paper also shows a feature subspace result. More specifically, at local minima with nonnegative student output weights, active student features must lie in the teacher’s feature span. Additianally they show an effective width result, that is, for two-dimensional plain NNs, critical points of the population loss with positive student output weights have at most twice as many distinct feature directions as teacher neurons.
Critical strengths
I think the main strength is that the authors prove that spurious local minima are effectively removed for their defined skip network under clearly stated assumptions. This applied across the population loss landscape and not only near a good solution.
The authors construct a fixed three-neuron teacher for which they show that spurious local minima for the plain student NN can exist and that they persist at every larger student width. Together with the main result above this shows that adding width and adding the linear skip can fundamentally change the landscape.
Critical weaknesses
The main guarantee is about Gaussian population loss for a bias-free, shallow network with planar teacher features and positive teacher output weights. It also covers only local minima with nonnegative student output weights. This is quite limiting in my opinoin. Additionally, it does not analyze gradient training which is primarily used in practice, and the finite sample result talks only about minima stable over a fixed-radius ball. This makes it hard to identify how much the landscape result explain the benefit of skip connections when we optimize in practice.
Questions for authors
-
Tinaz et al. prove GD convergence for a linear target despite non-strict saddles, while Zhang et al. prove efficient GD learning for generic Gaussian multi-index targets in a different setting. What does ruling out spurious local minima in your skip network tell us about training? Is there some training failure in the plain network that adding the skip is expected to prevent?
-
The finite sample result assumes that no change to the network parameters in a ball of fixed radius can make the loss smaller. Is this something we should expect at solutions found by "normal" training? How does required sampling accuracy change as the radius shrinks?
Zhang, B., Wang, Z., Fu, H. and Lee, J., 2026, April. Neural networks learn generic multi-index models near information-theoretic limit. In International Conference on Learning Representations
2 comments
Thank you for the detailed response. Your explanation helps clarify the specific training failure that the skip connection prevents: getting trapped at a local minimum with positive loss. I also appreciate the distinction between removing these minima and guaranteeing convergence of GD.
However, my main concerns remain. The guarantee still relies on restrictive assumptions. Although showing that a skip connection removes spurious local minima under these assumptions is meaningful, it remains unclear how much this explains the benefits of skip connections during standard GD training.
For the finite sample result, my concern is that it requires an additional stability condition. Since it has not been shown that ordinary training finds solutions satisfying this condition, its implications for training in practice remain unclear.
Overall, I appreaciate your response but it does not sufficiently change my assessment of the scope and practical relevance of the results. For this reason, I am keeping my score.
Thank you for your careful review and for emphasizing the distinction between landscape guarantees and training dynamics.
1. Experiments and optimizer behavior. We ran the requested experiments and found that adding the learnable skip connection facilitates convergence to a fit, which indicates that our results may hold more generally. We describe this in more detail in the general response.
2. What training failure does the skip prevent? Theorem 11 identifies an obstruction to training: a suboptimal local minimum despite realizability. At width three, the proof certifies positive curvature in a neighborhood of the plain-network minimum: sufficiently small-step GD started nearby can converge to positive loss in the paper's mass–direction coordinates. Splitting units preserves local minimality at every larger width. Theorem 10 excludes positive-loss local minima in the stated nonnegative-weight regime after adding the learned linear skip. The skip absorbs the linear mismatch that is coupled to the hidden features in the plain loss.
Thus, if training converges to a local minimum covered by the theorem, it cannot be trapped there at positive loss. Establishing that GD reaches such a minimum, including avoidance of non-strict saddles, remains a separate dynamics problem.
Comparison with the cited convergence results. Tinaz et al. prove finite-sample GD convergence for a linear target using exactly two ReLU hidden units, a prescribed small random initialization in their signed two-neuron parameterization, and suitable step sizes (Theorem 2). Their analysis establishes saddle avoidance and a convergence rate in that setting. Our counterexample gives a nonlinear planar teacher with positive-loss plain-network minima at every student width n≥3.
Zhang et al. study polynomial target functions that depend on a fixed number of input directions, using networks with smooth activation functions. Their central Assumption 5 requires a full-rank second-order signal after label preprocessing, with bounded condition number; Algorithm 1 uses a specified initialization and two-stage layer-wise GD. They prove efficient approximate learning for this class, allowing constant ranks above two. Our ReLU landscape theorem imposes no analogous initialization or spectral-conditioning requirement, but retains the planar-teacher and sign assumptions in Points 2–3 and does not replace their convergence guarantee.
3. Fixed-radius stability and sampling accuracy. A prescribed stability radius is stronger than ordinary local minimality; we do not show that standard training produces such solutions. In general, smaller radii require higher sampling accuracy. We did not focus on developing or optimizing explicit radius-dependent accuracy or sample-complexity bounds.
We hope this clarifies the connection to optimization and the precise limits of the finite-sample result.