acceptodds
Under review as a conference paper at ICLR 2027

Removing spurious minima for planar features by skip connections

Abstract

Understanding loss landscapes is central to explaining neural-network training, yet their structure remains only partially understood even in simple models. We study the Gaussian population loss of shallow, bias-free ReLU networks in the teacher–student setting, which provides a simple model for studying essential aspects such as feature learning and overparameterization. For teachers with positive output weights and planar features, we show that including a learned linear skip removes all spurious local minima with nonnegative student output weights, provided that the student is at least as wide as the teacher. In contrast, without the skip, we construct a fixed teacher with positive output weights and only three hidden neurons in input dimension two whose spurious local minima persist at every student width at least three. Thus, a learned linear skip can remove spurious minima that persist under arbitrary overparameterization. We further show that a positive output weight student always learns the subspace spanned by the teacher features: student features at local minima with nonnegative student output weights lie in the span of the teacher features. For ReLU networks in two dimensions, even heavily overparameterized students have effective width controlled by the teacher width: every critical point with positive student output weights has at most twice as many distinct student feature directions as teacher neurons. Finally, we transfer the benignity result to empirical minima over parameter balls of any prescribed radius, with the required sampling accuracy depending on that radius.

Primary area:
learning theory
Keywords:
Loss landscapes, ReLU Networks, Spurious local minima, Linear skip connections, Overparameterization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hspQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the Gaussian popoulation loss landscape of bias-free, shallow NNs in the teacher-student setting. More specifically, they construct a plain network with spurious local minima. They then show that adding a learned skip connection removes those spurious local minima when the student and the teacher have some extra assumptions. This is a loss-landscape paper not a training dynamics paper. The paper also shows a feature subspace result. More specifically, at local minima with nonnegative student output weights, active student features must lie in the teacher’s feature span. Additianally they show an effective width result, that is, for two-dimensional plain NNs, critical points of the population loss with positive student output weights have at most twice as many distinct feature directions as teacher neurons.

Critical strengths

Removing spurious local minima under assumptions.\textbf{Removing spurious local minima under assumptions.} I think the main strength is that the authors prove that spurious local minima are effectively removed for their defined skip network under clearly stated assumptions. This applied across the population loss landscape and not only near a good solution.

Separation.\textbf{Separation.} The authors construct a fixed three-neuron teacher for which they show that spurious local minima for the plain student NN can exist and that they persist at every larger student width. Together with the main result above this shows that adding width and adding the linear skip can fundamentally change the landscape.

Critical weaknesses

Limited connection to actual training.\textbf{Limited connection to actual training.} The main guarantee is about Gaussian population loss for a bias-free, shallow network with planar teacher features and positive teacher output weights. It also covers only local minima with nonnegative student output weights. This is quite limiting in my opinoin. Additionally, it does not analyze gradient training which is primarily used in practice, and the finite sample result talks only about minima stable over a fixed-radius ball. This makes it hard to identify how much the landscape result explain the benefit of skip connections when we optimize in practice.

Questions for authors
  1. Tinaz et al. prove GD convergence for a linear target despite non-strict saddles, while Zhang et al. prove efficient GD learning for generic Gaussian multi-index targets in a different setting. What does ruling out spurious local minima in your skip network tell us about training? Is there some training failure in the plain network that adding the skip is expected to prevent?

  2. The finite sample result assumes that no change to the network parameters in a ball of fixed radius can make the loss smaller. Is this something we should expect at solutions found by "normal" training? How does required sampling accuracy change as the radius shrinks?

Zhang, B., Wang, Z., Fu, H. and Lee, J., 2026, April. Neural networks learn generic multi-index models near information-theoretic limit. In International Conference on Learning Representations

2 comments

Thank you for your careful review and for emphasizing the distinction between landscape guarantees and training dynamics.

1. Experiments and optimizer behavior. We ran the requested experiments and found that adding the learnable skip connection facilitates convergence to a fit, which indicates that our results may hold more generally. We describe this in more detail in the general response.

2. What training failure does the skip prevent? Theorem 11 identifies an obstruction to training: a suboptimal local minimum despite realizability. At width three, the proof certifies positive curvature in a neighborhood of the plain-network minimum: sufficiently small-step GD started nearby can converge to positive loss in the paper's mass–direction coordinates. Splitting units preserves local minimality at every larger width. Theorem 10 excludes positive-loss local minima in the stated nonnegative-weight regime after adding the learned linear skip. The skip absorbs the linear mismatch that is coupled to the hidden features in the plain loss.

Thus, if training converges to a local minimum covered by the theorem, it cannot be trapped there at positive loss. Establishing that GD reaches such a minimum, including avoidance of non-strict saddles, remains a separate dynamics problem.

Comparison with the cited convergence results. Tinaz et al. prove finite-sample GD convergence for a linear target using exactly two ReLU hidden units, a prescribed small random initialization in their signed two-neuron parameterization, and suitable step sizes (Theorem 2). Their analysis establishes saddle avoidance and a convergence rate in that setting. Our counterexample gives a nonlinear planar teacher with positive-loss plain-network minima at every student width n≥3n\ge3.

Zhang et al. study polynomial target functions that depend on a fixed number of input directions, using networks with smooth activation functions. Their central Assumption 5 requires a full-rank second-order signal after label preprocessing, with bounded condition number; Algorithm 1 uses a specified initialization and two-stage layer-wise GD. They prove efficient approximate learning for this class, allowing constant ranks above two. Our ReLU landscape theorem imposes no analogous initialization or spectral-conditioning requirement, but retains the planar-teacher and sign assumptions in Points 2–3 and does not replace their convergence guarantee.

3. Fixed-radius stability and sampling accuracy. A prescribed stability radius is stronger than ordinary local minimality; we do not show that standard training produces such solutions. In general, smaller radii require higher sampling accuracy. We did not focus on developing or optimizing explicit radius-dependent accuracy or sample-complexity bounds.

We hope this clarifies the connection to optimization and the precise limits of the finite-sample result.

Thank you for the detailed response. Your explanation helps clarify the specific training failure that the skip connection prevents: getting trapped at a local minimum with positive loss. I also appreciate the distinction between removing these minima and guaranteeing convergence of GD.

However, my main concerns remain. The guarantee still relies on restrictive assumptions. Although showing that a skip connection removes spurious local minima under these assumptions is meaningful, it remains unclear how much this explains the benefits of skip connections during standard GD training.

For the finite sample result, my concern is that it requires an additional stability condition. Since it has not been shown that ordinary training finds solutions satisfying this condition, its implications for training in practice remain unclear.

Overall, I appreaciate your response but it does not sufficiently change my assessment of the scope and practical relevance of the results. For this reason, I am keeping my score.

Reviewer_3VhkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proves that skip connections remove all spurious minima in two-layer ReLU NNs in a teacher-student setting. While the considered theoretical problem is a rather niche area (given the assumptions used in the proofs; see below), to my knowledge, this is one of the first results showing provable benefits of skip connections in this setting.

Critical strengths
  • One of the first results showing provable benefits of skip connections in the two-layer teacher-student setting.
  • The proofs include some technical novelty in their interlacing arguments.
Critical weaknesses
  • The considered setting is rather limited: besides the two-layer NN architecture with non-negative output weights (also used by prior work), the main results only apply to "planar features", i.e., all teacher feature vectors are constrained to a two-dimensional subspace. It is difficult for me to build intuition about how this assumption could be connected to NN training practice.
  • It seems difficult to generalize the proof technique to "non-planar features", and the authors do not provide synthetic results indicating that the main results indeed generalize beyond planar features.
Questions for authors
  • Can you provide intuitions on why "planar features" reflect some aspects of practical NN training?
  • Can you provide some synthetic experiments showing the generality of the results?
  • Just asking casually, do you think LLM-assisted NN theoretical studies could go beyond analyzing existing toy models (e.g., two- or three-layer NNs) to developing new toy models that better reflect LLM/agent practice?

I am willing to increase the score if the first two questions are addressed.

1 comment

Thank you for your review and for highlighting the landscape result and the interlacing argument.

1. Experiments and generality. We ran the requested experiments and found that adding the learnable skip connection facilitates convergence to a fit, which indicates that our results may hold more generally. We describe this in more detail in the general response.

2. Why planar features? Point 2 explains their intended role, now stated explicitly in the Introduction: planar teacher features give mathematical tractability through circular ordering and interlacing, as a first step toward arbitrary teacher feature rank. We do not posit coplanarity as a property required by practice. The theorem allows arbitrary ambient dimension, and the student must discover the target subspace without being initialized inside it. The arbitrary-rank conjecture was already stated in the conclusion. Point 3 distinguishes the teacher-span confinement theorem, which already holds in arbitrary rank, from the planar benignity and effective-width results.

3. LLM-assisted development of theories covering the latest models. We share the interest in models closer to LLM and agent practice. Our approach begins with tractable cases, uses them to discover structure, and then seeks general statements. In this project, studying two-neuron teacher–student networks helped generate hypotheses that led to the present arbitrary-width theorems. We hope our bottom-up approach can eventually inform deeper networks and architectures such as transformers.

We hope the new synthetic evidence and explanation of planarity's mathematical role address your two main concerns.

Reviewer_hTdBOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the loss landscapes of shallow ReLU networks in a Gaussian teacher–student setting. The authors prove that adding a learned linear skip connection eliminates spurious local minima under planar teacher features and non-negative student output weights. They further construct a counterexample showing that spurious minima can persist under arbitrary overparameterization, establish feature-learning and effective-width properties, and provide a finite-sample extension under additional stability assumptions.

Critical strengths
  1. The paper establishes a nontrivial guarantee that a learned linear skip connection eliminates spurious local minima under clearly specified conditions.

2.The construction showing that spurious minima persist under arbitrary overparameterization provides a convincing theoretical distinction between increasing network width and modifying network architecture.

Critical weaknesses
  1. Limited practical relevance. The main theoretical results rely on restrictive assumptions, including planar teacher features, non-negative output weights, and shallow ReLU networks. Their implications for modern deep networks remain unclear.

2.Lack of empirical validation. The paper provides no systematic training experiments, even on simple synthetic datasets, to investigate whether the theoretical findings translate into improved optimization behavior.

Questions for authors
  1. Could the authors provide numerical experiments, at least on synthetic teacher–student networks, comparing optimization behavior with and without skip connections? Such evidence would help establish the practical significance of the theoretical results.

  2. Skip connections have been extensively studied since ResNet, and Zhou et al. (2021) already investigated related theoretical properties. Could the authors clarify the fundamental new insights and their significance beyond these existing results?

  3. Can the authors provide theoretical or empirical evidence that the main conclusions extend beyond planar features?

1 comment

Thank you for your careful review and your questions about empirical evidence, prior work, and generality.

1. Experiments. We ran the requested experiments and found that adding the learnable skip connection facilitates convergence to a fit, which indicates that our results may hold more generally. We describe this in more detail in the general response.

2. New insights beyond ResNet and Zhou et al. ResNet proposes residual connections for deep networks and shows empirical benefits, without deep theoretical results. We would again like to emphasize the difference of their proposed identity skip connections and our linear skip connections. Our contribution is the precise architectural separation: a learned linear skip excludes positive-loss local minima that persist under arbitrary widening of the plain network for the same fixed teacher.

Zhou et al. (2021) exclude nonzero-loss stationary points within a specified low-loss sublevel and prove GD convergence from there, under teacher angular-separation and norm bounds (Theorems 4–5). They allow arbitrary teacher rank. Their Theorem 6 also provides global convergence from specified initialization procedures with additional width requirements. These guarantees do not exclude spurious minima elsewhere in parameter space. While our setup is comparable, we have slightly different assumptions on the signs of weights. More importantly, while convergence to a global optimum under certain assumptions as deduced in Zhou et al. is desirable, showing that there are no local minima is structurally much stronger. We consider our structural result as a step towards a more general structural result about spurious minima.

Our planar theorem covers every local minimum in the stated sign regime, without a low-loss condition, prescribed initialization, or separation margin for the teacher units, for every student width n≥mn\ge m. This complements their convergence guarantees with an exclusion of spurious minima throughout the landscape. The centered-loss reformulation is credited to prior work; the new content is this exclusion and its contrast with the persistent plain-network traps.

We hope the new experiments and these clarifications address your concerns about empirical relevance, novelty, and generality.

Comments on the paper

2 comments

Thank you to all three reviewers for their careful assessments. We collect the shared points here and address the individual questions in our reviewer-specific replies.

While our theoretical contribution proves that skip connections remove spurious minima under the assumption of planar features, this is meant as a first big step. We added experiments demonstrating that this planarity assumption may not be necessary. To achieve this, we used various optimizers to find (local) minima; this is non-trivial due to numerical issues, in particular "flat gradients" and poorly conditioned regions. The experiments are not meant to provide new results concerning general training behaviour, but one can indeed observe that there is more structure to be discovered. The experiments appear in Section 7 and Appendix D and we provide a compute report in Appendix E.

1. Experiments. Our experiments indicate that also for higher rank, skip connections imply that there are no spurious minima. Due to numerical issues, computing the (local) minima is actually non-trivial, and so the experiments cannot show the exact fit but at least an improvement of the fit. Namely, we record substantial fitting gains from learned skips on fully random higher-rank teachers in dimension 800. At matched teacher/student widths m=n=20,50,100m=n=20,50,100, the skip models fit at least 511 of 512 trials, while plain ReLU fits 117, 0, and 0, respectively. Overparameterization also helps plain ReLU: widening from n=20n=20 to 3030 at m=20m=20 raises fits from 117 to 512, and widening from n=50n=50 to 8080 at m=50m=50 raises them from 0 to 300.

We want to remark that these experimental runs probe optimization behaviour, which is linked to the loss landscape, but cannot prove the existence or absence of critical points or spurious local minima.

Each setting uses 512 paired random teacher/student initializations, shared across the models; teachers are also shared across student widths. A fit requires normalized loss at most 10−1010^{-10}. Joint skip trains the linear skip weights; profiled skip is exactly centered loss, obtained by analytically minimizing the weights of the linear skip in every optimization step.

Fully random higher-rank teachers, d=800d=800.

(m,n)(m,n)PlainJoint skipCentered / profiled skip
(20,20)(20,20)117512512
(20,30)(20,30)512512512
(50,50)(50,50)0512512
(50,80)(50,80)300499510
(100,100)(100,100)0511512

Planar teachers, d=2d=2. Within the theorem's assumptions, both skip formulations improve fit counts in every reported setting.

(m,n)(m,n)PlainJoint skipCentered / profiled skip
(3,3)(3,3)446504507
(5,5)(5,5)170421427
(5,10)(5,10)485512512
(20,20)(20,20)95661
(20,30)(20,30)413465484

We choose trust-region Newton as optimizer, as gradient descent heavily underperforms in this setting. Our working hypothesis is that nearly flat or poorly conditioned regions slow first-order training, while curvature information helps Newton make progress.

2. Why planarity? Our aim is to understand whether a learned linear skip removes spurious minima for positive teachers of arbitrary feature rank. The Introduction now explicitly states that planarity provides a mathematically tractable first step: circular ordering enables the interlacing argument. We do not assume practical features are coplanar. Ambient dimension is arbitrary, and student features need not initially lie in the teacher plane.

The conclusion already states the arbitrary-rank conjecture (Conjecture 14). Substantial attempts to find higher-rank counterexamples have been unsuccessful, which motivates further study but of course does not prove the conjecture. Extending the geometric argument, including eventually to deeper models, is a further research objective.

We see that our results rely on some assumptions that are a priori rather restrictive. However, the progress on the theoretical understanding of spurious minima has been achieved only in small steps also in previous other work. In this context, our work actually adds new structural insights, and the new experiments (added for the rebuttal) indicate that we are actually on the right track with our structural insights.

Unfortunately, the reviewers did not interact (much) with us in the rebuttal phase, so that we could not address any follow-up questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.