acceptodds
Under review as a conference paper at ICLR 2027

Teacher Geometry Shapes Learnability in Teacher-Student Networks

Abstract

Teacher-student systems, in which a teacher neural network generates targets so that a student neural network can learn to implement the same function, are widely used to study learning. However, the structure of the teacher weights is often overlooked by assuming normally-distributed parameters. This hides substantial variations in how learnable different teachers are. We formalize learnability as the success rate of converging to the global minimum and study how it is affected by overparameterization, teacher geometry, initialization distribution, in the context of a bounded data domain. We analytically reduce and systematically characterize the loss landscape to identify two mechanisms through which teacher geometry shapes learnability in small, tractable ReLU systems: (i) In single-neurons systems, the convergence to this minimum is governed by the initial similarity between teacher and student neurons. If the neurons are initially too dissimilar, the student settles into a sub-optimal minimum, which we name out-of-bounds, where the training dynamics push the ReLU activation threshold outside of the bounded data domain. (ii) In two-neuron systems, the alignment between the two teacher neurons determines the existence of suboptimal local minima and shapes their basins of attraction. Motivated by these findings, we empirically study the trainability of larger systems by constructing teacher distributions with different types of alignment between nodes. We discover that increasing alignment between teacher nodes not only reduces learnability but also increases the rate of convergence to out-of-bounds minima. Moreover, convergence rates to these sub-optimal minima can be reduced by increasing the output-layer learning rate, improving learnability. Finally, by training networks on a set of physics regression problems, we confirm that problems that require aligned-nodes solutions tend to have an increased rate of convergence to out-of-bounds minima.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
teacher-student networks, ReLU networks, loss landscapes, learnability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NpsjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how the learnability of teacher-student neural networks depends on the geometry of the teacher neurons. Both the teacher and the student are single-layer neural networks with the same input dimension. The student network may have more neurons that the teacher. When both teacher and student have only a single neuron, the authors show that the initial dissimilarity between the teacher- and student-neurons could lead to divergence of learning. When both teacher and student have two neurons, the authors show that a teacher with maximally dissimilar neurons can allow easier learning. Similar observations are made when the teacher has an even a larger number of neurons. Finally, the paper shows that increasing the learning rate of the readout layer while decreasing that of the hidden biases can improve learning success. Most of the results are obtained through numerical simulations of these networks.

Critical strengths
  1. Studying how the similarity or dissimilarity between teacher neurons can affect the learnability of the student seems to be a new angle.
Critical weaknesses
  1. The significance of the main findings for larger networks seems questionable. Specifically, if the teacher network has more than two layers, it is unclear how one can still determine whether neurons are similar or dissimilar. Generalization to real networks (e.g., transformers) seem to be even more difficult.

  2. The experimental setup of this paper has little connection to practice. For theory, it is understandable that we study a simple model. However, the experiment in the paper also uses a NN with one hidden layer, which is rarely used in practice.

  3. Most of the results for two or more neurons are based on experimental observations (except Lemma 1, but it is unclear to the reader what its significance is). In this sense, the technical depth of the paper is somewhat limited.

Questions for authors
  1. Does the main conclusion of this paper provide any insight when the teacher network is any real networks (such as a well-optimized language transformer or diffusion transformer)?

  2. Where is Figure 2E that the paper refers to at the end of Section 3?

2 comments

Thank you for recognizing the novelty of our research question, and the soundness of our study. Below, we address your questions.

Question 1

Our analyses allow us to make predictions about which networks would be easier to fit than others based on their internal geometry. This should be possible to do in deep networks by considering the similarity of the different neurons in each layer separately. Connecting these results to transformers is an interesting question for future research!

Question 2

Thanks for raising this point. This was actually supposed to be Figure 2D. We have updated it in the manuscript.

If you have any more questions, please let us know. If not, we would be grateful if you would consider raising your score!

I thank the authors for the response. However, it does not seem to address my concerns on the technical depth and on the relevance to more complex networks. I wish to keep my score.

Reviewer_TFpzOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how teacher-network geometry affects learnability in teacher-student networks. Through theoretical loss-landscape analysis and controlled experiments, the authors show that neuron alignment and readout signs influence convergence to global and local minima. They further demonstrate that differential learning rates can improve optimization success and explore related geometric properties in physics-inspired regression tasks.

Critical strengths

The paper addresses a meaningful question about the factors influencing neural network learnability beyond model capacity. The loss-landscape analysis provides an interpretable explanation of optimization failures, while controlled experiments across different teacher geometries and network configurations support the findings. The differential learning rate experiments also offer practical insights into improving training success.

Critical weaknesses
  1. The paper defines learnability solely in terms of convergence to near-zero training loss, without evaluating generalization performance. It remains unclear whether higher optimization success rates across teacher geometries also translate into better performance on unseen data.
  2. The experiments mainly rely on synthetic teacher networks. The physics-inspired experiments primarily examine correlations between function geometry and learned network properties, without directly evaluating how these geometric factors affect optimization success rates.
Questions for authors
  1. Could the authors evaluate generalization performance across different teacher geometries to determine whether the proposed learnability metric reflects performance on unseen data?
  2. Could the authors directly evaluate how function geometry affects optimization success rates on the physics-inspired regression tasks?
  3. Could the authors further validate the findings on other real-world datasets and more complex neural network architectures?
2 comments

Thank you for your review. We are grateful that you understand the value of our research question. Below, we answer each of your questions in turn.

Question 1

Given that we consider a large number of samples and the inputs are low-dimensional, the data should cover the input range well. It is an established methodology in teacher-student studies not to consider generalization separately.

Question 2

Training losses are not directly comparable across different equations. However, we now provide an additional analysis that links the amount of OOB neurons to performance within an equation (Figure 27).

Question 3

We agree that this is an important next step, but consider it out-of-scope for the current work.

If you have any more questions, please let us know. If not, we would be grateful if you would consider raising your score.

Thank the authors for the reply. However, the authors do not provide more experimental results. I would like to keep my score.

Reviewer_eLVtOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how the geometry of a teacher network affects a student network’s ability to fit its outputs. It defines “learnability” as the probability of reaching zero training loss across random initializations. For shallow ReLU networks, it analyzes simplified one- and two-neuron loss landscapes and compares teachers with different weight directions and readout signs. Its main empirical claim is that more aligned teacher neurons are harder to fit and more often produce out-of-bounds student neurons: units that become permanently inactive or permanently linear on the training data. It also reports improvements from changing parameter-specific learning rates and correlations between target-function curvature and trained-network geometry on physics equations.

Critical strengths

Differences in optimization difficulty between equally expressive teacher networks are worth investigating. Varying teacher directions and readout signs could support an informative discussion of what conventional randomly sampled teacher benchmarks conceal.

Critical weaknesses
  1. This is a badly written paper, starting from the abstract.

Teacher-student systems, in which a teacher neural network generates targets so that a student neural network can learn to implement the same function, are widely used to study learning. However, the structure of the teacher weights is often overlooked by assuming normally-distributed parameters. This hides substantial variations in how learnable different teachers are.

What does this even mean? What parameters are normally distributed? Is the initialization before training normal or do we just draw the full teacher normal by iid weights? Where does this even appear? If it's initialization, then we will train the teacher first allegedly so why does that matter? If it's the full network, why are we even studying it in "teacher-student" settings if the teacher is a random function? I have no idea what's going on, and we are only on the second line of the abstract. Do the authors know what they are writing about?

  1. There are missing figures everywhere. Figure 2D is forgotten, and the plot is empty. Line 283 references a nonexistent figure 2E. Figure 4B and 4E don't exist. Appendix F.5 contains “Figure ??F,” and the success-versus-input-dimension plot described in Appendix F.4.1 and the Figure 12 caption is not present in that figure.

  2. The analysis does not establish the main mechanism claimed by the paper. The simplifying assumptions remove the phenomenon the paper wants to explain. Appendix E.1 explicitly states that Gaussian population inputs eliminate out-of-bounds minima. The readouts are optimized instantaneously, and the two-neuron analysis additionally removes biases. These results therefore do not directly explain failures in the finite-data, biased networks trained in the experiments.

Appendix D does establish the existence of an always-active local minimum, but this does not explain how teacher alignment determines its basin of attraction or the probability of reaching it. Moreover, Appendix E.4 assumes that the same-sign two-neuron result transfers to the optimized-readout setting, while Section 3.2 presents the result as established. Section 6’s statement that fast readouts make local minima disappear also needs correction: changing finite positive learning rates changes trajectories, not the stationary points of a fixed loss. The paper needs to distinguish these claims and establish what its analysis adds beyond the single-neuron literature it already cites.

  1. The experimental protocol does not support the reported interpretation as convergence probabilities. Appendix B.1 states that runs without saved output are excluded, while all saved runs are included without an additional convergence criterion. This is a serious problem when the central quantity is the probability of convergence. Table 1 reports only 267, 302, and 289 completed ReLU runs at teacher width 16 for the three distributions, against its stated 1,500-run counts. Excluding unfinished runs can substantially bias the comparison, particularly if difficult teachers are more likely to time out. Likewise, a saved endpoint above the success threshold is not necessarily a local minimum.

Several mechanistic claims also lack adequate controls. Figure 6A simultaneously accelerates readout weights and the output bias while slowing the hidden biases. It demonstrates the effect of this combined intervention, not the separate effects attributed to its components. Comparisons with the Gaussian teacher baseline additionally change biases, weight norms, and readout magnitudes; Appendix B.2 states that output normalization is not applied during training. Consequently, those comparisons do not isolate geometry alone.

  1. The reported evidence is internally inconsistent and sometimes weaker than the claims attached to it. These are not merely cosmetic problems. Section 5 states that same-sign, similar-direction teachers produce fewer out-of-bounds minima, whereas Figure 5B shows the opposite, consistent with the abstract. Figure 16 is described as an AUROC analysis, but its axes report Spearman correlations and include negative values. The reader cannot determine which analysis supports the claimed predictive relationship.

Finally, the physics experiment establishes correlations with alignment and out-of-bounds counts, not convergence to known suboptimal minima. Its curvature descriptors do not significantly correlate with training MSE in the reported plots. It therefore does not establish that the proposed geometry makes these problems less learnable. The abstract should distinguish these associations from a validated explanation of optimization failure.

Questions for authors
  1. Which result establishes how teacher geometry affects out-of-bounds convergence probabilities under finite data and finite learning rates? What specifically is new beyond the existing single-neuron analyses, and can the same-sign two-neuron claim be justified under the actual optimized-readout assumptions?

  2. How do the conclusions change when every attempted run is included in the denominator and unfinished runs are explicitly accounted for? Please provide convergence diagnostics and training-budget sensitivity, separate the learning-rate interventions, and control the non-geometric differences in the Gaussian-baseline comparison.

  3. Can the authors reconcile Section 5 with Figure 5B, clarify whether Figure 16 reports AUROC or Spearman correlation, and supply the missing dimensionality plot? What evidence connects the physics correlations to optimization failure rather than merely to properties of the fitted representations?

2 comments

Thank you for your review. We are grateful that you have taken the time and effort to engage with our paper in detail.

Based on your feedback, we have fixed minor issues and improved clarity of the manscript. Below, we address each of your points in turn.

Writing in the abstract

The parameters we were referring to in this sentence are the teacher weights. In the teacher-student setting, teachers are considered fixed and generate labels for a student network. Our research question is how the choice of teacher parameters affect the difficulty of the learning problem. To our knowledge, most existing literature will consider a setting like drawing teacher weights from a normal distribution or set weights to be diagonal/orthogonal, without considering teacher structure in more detail.

Figures

There appears to have been a software- and OS-dependent rendering issue in the PDF: Figures 2D, 4B and 4E did not open in MacOS Safari. However, Chrome or Acrobat Reader should have shown the missing panels even on MacOS, and the updated version should work on all systems. We have also fixed the incorrect references and supplied the panel 12E.

Relevance of the analysis to the empirical setting

In Section 3, we use an analytical reduction to study the loss landscape of tractable, simplified systems and compare these results to empirical simulations. In the single-neuron case, to the best of our knowledge the specific finding we present in Proposition 1 about the absence of spurious local minima in the reduced population loss is novel. In the two-neuron case, the structure of the interior minima transfers to the optimized-readout setting because critical points are preserved under the change of learning rate. We observe some similarity with the finite-data setting empirically (see Figure 13). In both cases, the primary benefit is visualization in two dimensions. We have edited the manuscript to make these contributions and their scope clearer.

Convergence

Handling potential confounding factors is important to this study. We have expanded Table 1 which shows that the failed runs are concentrated on a few settings. We now systematically limit our analyses to the runs where we have >99% data. With regards to convergence, the framework we are using is optimized to find critical points, and prior work uses a similar methodology. Notwithstanding, we have inserted a new Appendix G that analyzes convergence properties by continuing a subset of runs for 1 hour and all of them for 1 minute and studying the loss dynamics. Although we cannot rule out that neurons will still shift with more training, based on these results, we believe that the risk of a confound is minimal in this case.

We now also supply the separated learning rate interventions in Figure 17. This shows that the effect is strongest when both interventions are applied as in Figure 6A.

Thanks for for pointing us to the description of output normalization. We have added more details to the training procedure. We do indeed apply normalization as well at the start of training by normalizing the weight vectors to 1 and adjusting the readout weights so that all teachers train under the same conditions.

Evidence and claims

Indeed, same-sign, similar-direction teachers produce more out-of-bounds minima. The dominant effect is due to the readout signs. We have adjusted this claim, and now additionally supply the per-setting AUROC scores in Figure 16.

On the AI Feynman dataset, we believe that despite standardization, the loss values across different equations are not comparable due to the very different shapes of the functions. However, we supply a new analysis in Figure 27, showing that, for a given equation, the amount of OOB neurons correlates with higher final loss. The effect is larger for smaller initializations, which lead to a larger number of OOB neurons.

Final remarks

Thank you again for your comments. We sincerely hope that you can take a look at our revised PDF and consider our responses. Let us know if you have any further questions or comments, as we would be very eager for further feedback. If you find it addresses your concerns, we would be extremely grateful if you would raise your scores!

With the end of the rebuttal period approaching, please let us know if you have any additional questions or concerns. We would be extremely, extremely grateful to engage more with you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.