acceptodds
Under review as a conference paper at ICLR 2027

Layers as Lenses: A Feature Learning Narrative for Gated Deep Networks

Abstract

End-to-end gradient-based training of multi-layer architectures has been immensely successful across machine learning tasks, yet principled explanations for why, when, and how it discovers task-relevant structure in data remain incomplete. Existing viewpoints based on representational power, input-space folding, optimization conditioning, the neural tangent kernel, and hierarchical feature learning offer useful insights, but leave important aspects of end-to-end feature learning unresolved. We propose a new narrative for gated deep networks trained by gradient descent. From the viewpoint of any given layer, the other layers act as a lens that assigns importance weights to the input, facilitating the emergence of features without global scope that might otherwise be overlooked. Gradient methods then update each layer to fit data that have been importance-weighted by a lens formed by the remaining layers. In this view, individual neuron parameters serve a dual role: separator for data lensed by other layers, and component of the lens for other layers. We hypothesize that the resulting positive feedback is a key mechanism underlying feature learning in gated deep networks. We support this narrative with theoretical analysis and empirical results on the recently proposed Deep Linearly Gated Network (DLGN), an architecture that combines elements of deep linear and ReLU networks. In particular, our analysis provides predictive control over gradient-descent trajectories from initialization in the DLGN setting, and naturally leads to a notion of hierarchical feature discovery in the feature-learning regime. We also present evidence suggesting that aspects of this narrative may extend to ReLU networks.

Primary area:
learning theory
Keywords:
Feature Learning, Deep Learning, Learning Dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_z1eSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a theory of feature learning during training for gated deep networks trained on data generated via a Oblique Decision Tree.

They use the model-data framework from Banerjee et al 2025, which is both complex enough to represent interesting training scenarios and simple enough to derive analytical feature predictions.

The theory is divided into single path (width=1) and multipath neural networks. For single paths, they derive how the NTK features change during training, and provide intuition with a work out example. For multipath, the main result is that a L2 penalty on the gates can reduce the error. They further identify two effects: a hierarchy-seeking effect that can promote movement toward task-relevant internal features, and a self-opposing effect that can amplify movement along spurious directions.

They also show that they can predict solution from random initialization, using a heuristic score that give different weights to any possible solution.

Experiments with ReLU networks suggest that the phenomena studied in the paper also apply to them.

Critical strengths

The paper is clearly written, easy to follow, and well organized.

Being able to predict which solution training will reach is very interesting, and, as far as I'm aware, novel.

The fact that the theory predicts an intervention (l2 penalty) that will help reduce test error is also important.

Critical weaknesses

The theory is limited as the theorems mainly focus on a width 1 network (single path). However, by analogy with other deep learning theory literature that studies two layer neural networks (i.e., depth 1), this might not be an issue.

Real data often doesn’t come from ODT, hence this data regime might question the impact of the results.

There are no real data experiments.

Questions for authors

Could the author find real data that have an approximate ODT structure and see whether their results extend to this setting?

1 comment

Thank you for the review.

The ODT is just a test-bed for evaluating hypothesis. We do not claim it is powerful enough to model real data, but that decision trees are powerful data models is almost folklore and we just borrow its reputation. The decision tree labelling function is already significantly more complex than typical labelling functions studied in deep learning theory.

Reviewer_eTE8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies feature learning in Deep Linearly Gated Networks on synthetic data labeled by an oblique decision tree. The authors propose a “lens” interpretation, where each layer changes the importance of different examples for learning in other layers. They analyze local interactions between gates in a single-path model and use these results to motivate an explanation of hierarchical feature discovery. Experiments examine the order in which features emerge, prediction of the final solution from initialization, and the effects of regularization and pruning. Additional experiments in ReLU networks explore whether similar behavior occurs beyond the gated model.

Critical strengths

I find the goal of understanding hierarchical feature learning interesting and relevant. In particular, studying how learning one feature can help the network discover others is a worthwhile direction.

Critical weaknesses
  1. I find the positioning relative to existing theory insufficient and incomplete. There is a substantial literature describing end-to-end feature learning dynamics through dynamical mean-field theory, multilayer mean-field limits and Tensor Programs, including work on deep networks, transformers, and MoE architectures that the authors fail to ackowledge (I'll make a short, non conclusive, list below). The authors should explain what their analysis adds relative to these approaches. Studying a structured decision-tree task is a choice whose broader significance must come from a demonstrated mechanism or prediction that extends beyond this setting, which I don't find present here.
  2. I find the conclusions broader than what the theory establishes. The results describe particular interactions between gates, but I do not see how these are enough to explain the overall learning dynamics or the order in which features emerge.
  3. I am not convinced that weak alignment with individual teacher directions means that a neuron has learned something irrelevant. It could simply represent a combination of useful directions. This makes the interpretation of some experiments less clear to me.
  4. The experiment predicting the final solution seems rather specific, since it uses selected initializations and tuned coefficients. I would like to understand how representative this result is of what happens during training more generally.
  5. Improvements from regularization and pruning: I do not see why they specifically support the proposed mechanism. More generally, I find the connection between the theoretical explanation and the experimental evidence insufficiently established.

Relevant (very short) papers list: Yang, G., and Hu, E. J. (2021). Sirignano, J., and Spiliopoulos, K. (2022). Bordelon, B., and Pehlevan, C. (2022). Nguyen, P.-M., and Pham, H. T. (2023). Montanari, A., and Urbani, P. (2025). Barboni, R., de Hoop, M. V., Furuya, T., and Peyré, G. (2026).

Questions for authors

See above.

1 comment
  1. Thanks for the references. We will add discussions on them in a future version. Broadly, the oblique-decision tree is clearly a multi-index function, but is not a low-degree polynomial of the multi-index. Mean field theory is rather intricate for multi-index functions even with low degree polynomials. The fact that relu nets and DLGNs learn these labelling functions rather easily might mean that there is an easier path to analyse it.

  2. Indeed, the discussion on how these results make the order of feature emergence is not discussed in detail. Broadly however, it relies on the larger coverage of nodes closer to the root. In more detail, the hierarchy-seeking result says "if w_k moves along a leaf node, then the tendency of w_l to move along an ancestor node increases". The root node is an ancestor to more nodes than its children, and hence it is likely to get its push earlier.

  3. In general, yes the neurons do learn combinations of useful directions, but we claim that these are sub-optimal -- any neuron whose half-space boundary does not align with the ODT boundaries is a neuron not performing to its potential.

  4. The main control for endpoint prediction results is the simple proximity based predictor -- "predict the solution closest to init". It corresponds to c_{HS}=c_{SO}=0. While these constants are validated, the performance of the predictor is quite robust to their variations. The full table is given in the appendix.

  5. We indeed do not know if the improvements are due to the mechanism. As we have no way of isolating only the hierarchy seeking behaviour or the self-opposing from gradient descent. This is only a potential explanation at this point.

Reviewer_21ZEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors study feature learning in an analytically tractable nonlinear neural network model, the Deep Linearly Gated Network (DLGN), trained on data labeled by an Oblique Decision Tree. For single-path DLGNs, they identify “lens” and “blinder” factors in the gradient, interpreting each neuron as both a separator and part of a mechanism that reweights the data seen by other neurons. They use this decomposition to prove two local second-order theorems. one describing ancestor–leaf interactions that can introduce a learning signal along internal-node directions absent from the initial gradient, the other describing self-opposing interactions along directions orthogonal to the ODT directions. They then investigate several empirical applications of the structure of their theorems, including to endpoint prediction (predicting the weight-solution that a given initialization converges to), the temporal ordering of learning of hierarchical features, and generalizations to multi-path DLGN's and ReLU networks.

Critical strengths
  • A concrete and interpretable local mechanism: I find the main strength of the paper to be the ideas behind the local mechanism, and its use to understand how latent features of the data (in this case, the hierarchical tree structure of the hidden nodes) can be learned implicitly by the network. Theorem 3.1 identifies how interaction with another feature can create a learning signal along a direction lacking initial first-order label correlation. Together with the self-opposition result, this provides a useful mathematical account of competing interactions in a model where the relevant features can be inspected directly. I also appreciate the ODT data model as a good balance between tractability and structure.

Other strengths:

  • Particularly the introduction and some of the intuitions around feature learning are explained in a clear and compelling way.
  • The idea of training a linear classifier to predict endpoint convergence is a nice one.
  • The precise L^2 regularization scheme (two-phase training/pruning) is potentially interesting
Critical weaknesses
  • Much of the exposition connecting the various ideas and results, for example, why Theorem 3.1 is an application of the main gradient decomposition from the previous section, is left for the appendix (if stated at all), and not put into the main paper, making the logic difficult to follow
  • In several of their empirical findings, the authors don't sufficiently argue that the effect is coming from their proposed mechanism. For example, the endpoint selection doesn't have any kind of control suggesting that the gradient-dynamics-related features they deduce from their theorem help more than other gradient-dynamics-related features; and in the ReLU interventions, does suppressing an ancestor impede descendants through the proposed hierarchical learning process?
  • The authors don't put their own work sufficiently in context, precisely comparing to antecedent literature studying second order gradient properties, their implications for feature learning in neural networks, and other things like regularization schemes that are antecedents to theirs, other works proposing temporal learning order for hierarchical features, etc. This makes it difficult for readers (and myself) to judge the novelty of the work.
Questions for authors
  • It seems that the paper's Theorem 3.1 follows from the following property of the network function: let fW(x)=F(t1,…,tL), ti=wi⊤xf_W(x)=F\left(t_1, \ldots, t_L\right), ~ t_i=w_i^{\top} x denote the network input output function, then Theorem 3.1 follows from the assumption that ∂2F∂tk∂tℓ>0\frac{\partial^2 F}{\partial t_k \partial t_{\ell}} > 0 Therefore it applies to models like F(t)=F(t)= softplus (∑rtr)\left(\sum_r t_r\right) that have positive mixed derivatives but depend only on the effective vector ∑rwr\sum_r w_r, and don't have a deep hierarchy or distinct-neuron specialization. Does this change the interpretation of their theorem in any way?
  • Can the authors provide more information about what analyses in the literature are most similar to theirs? This would include their main theorem and framework, as well as their applications (eg regularization, temporal learning order of hierarchical features, endpoint prediction from initialization, etc)
  • Quantitative support for the interventions and regularization interpretation in ReLU interventions: Can the ReLU intervention results be reported quantitatively, including variability and comparisons that distinguish descendant-specific effects from general disruption of learning? For §4.1, can off-span components be measured directly? Low cosine similarity to every individual teacher direction does not establish a component outside their span; a mixture of teacher directions can have the same appearance.
  • Likewise, can controls be provided for the endpoint prediction results, giving evidence that the proposed mechanisms are responsible for the effect?

Addressing these could definitely warrant moving from weak reject to accept.

1 comment

We thank the reviewer for the detailed review.

  1. The softplus of sum of linear projections is indeed a good approximation of the single path DLGN at close to init, and similar theorems could potentially be made under more restricitve conditions, but clearly even the notion of distinct neurons seeking the hierarchy make no sense there. In the single path DLGN setting, the hierarchy-seeking theorem says "if w_k moves along a leaf node, then the tendency of w_l to move along an ancestor node increases". We can actually say the same about w_k and w_k itself, instead of w_k and w_l, but the derivative changes are weaker here.

  2. The line of work by Abbe et al. Learning staircase polynomials is the closest similar line of work in terms of goals. The setting is simpler there with boolean polynomials, and single hidden layer learning. End-point prediction from initialisation is not a goal we have seen elsewhere, but seems like a natural choice to check if we have really understood the learning dynamics of a non-convex objective.

  3. The main control for endpoint prediction results is the simple proximity based predictor -- "predict the solution closest to init". It corresponds to c_{HS}=c_{SO}=0. While these constants are validated, the performance of the predictor is quite robust to their variations. The full table is given in the appendix.

  4. Yes, a quantitative reporting is possible, but is not much more detailed than the sentence given. We will add it in a revised version. Essentially, if we stop parameter movement along u_1, the model still learns u_0, u_2 and descendants of u_2, but not descendants of u_1.

Reviewer_4oo8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a "lens" view of feature learning, where the other layers reweight the data seen by a given layer during training. This is studied on Deep Linearly Gated Networks trained on synthetic data labelled by an oblique decision tree.

Critical strengths

None

Critical weaknesses
  • Clarify the similarities with the “Globally Gated Deep Linear Networks” paper by Li and Sompolinsky, published in NeurIPS 2022 (https://openreview.net/pdf?id=hqRwcqzegr7). The contributions appear to be similar in how the models are defined, yet I can see some difference in how the gates operate. I believe a paragraph connecting them is necessary.
  • From what I understand, the model for the single path case at the beginning of Section 3 is basically a single layer of perceptrons with sigmoidal activations that is multiplied instead of taking the weighted sum. The multi-path case considers all possible paths in a multilayer network. However, even here each gate looks directly at the input and not at the output of the previous layer, so the depth seems to live only in the linear part, while the gating is still a single layer of sigmoidal units. I think this matters for the main story of the paper. The "lens" seen by a given layer is just the product of the other gates, each of which is an independent function of the input, so the feedback between layers that the authors describe is really a coupling between parallel units rather than between stacked layers. The paper uses some specific jargon like “lens” to describe what seems to be the product of activation function values (eq. 2). The term shows up in the gradient of the logistic loss. I don’t mind introducing terms if they help the reader understand the role of specific components or mathematical objects, but I fail to see why this is needed here.
  • I suspect the “OBLIQUE DECISION TREE” described in section 2.3 has something to do with the way this type of neural network model operates, but this is not easy to understand given the way the paper is written (which refers too much to Banerjee et al., 2025 without giving enough context). I am not really sure what the contribution is supposed to tell us. It seems to be a highly artificial setup where the data, somehow, fits the structure of an oblique decision tree. The authors justify this by saying that real data often have some hierarchical structure that trees can approximate, but I could not find evidence of this in the paper (and I really think this is artificial). The authors then analyze the learning dynamics across this structure. I am not sure what kind of mechanisms are revealed here and why they are useful in a more general sense.
  • If the paper wants to make general claims about the learning dynamics in deep learning architectures, then I think the authors should make an effort to clarify what is their contribution in that direction. What is the general mechanism that is revealed here? The type of models they consider seem to have little to no impact in deep learning, or at least I don’t see any in the paper. For instance, recent papers that investigate the learning dynamics of various deep learning architectures reveal the presence of critical phenomena on a variety of tasks and data. In order to make the paper interesting, the authors should consider expanding their analysis to a more serious experimental setting, demonstrating how (and why) the architecture they are considering is relevant in more realistic settings. See:
  1. A. Ly and P. Gong. Optimization on multifractal loss landscapes explains a diverse range of geometrical and dynamical properties of deep learning. Nature Communications, 16(1):3252, 2025. doi: 10.1038/s41467-025-58532-9
  2. X.-Y. Zhang and C. Tang. Heavy-tailed update distributions arise from information-driven self-organization in nonequilibrium learning. Proceedings of the National Academy of Sciences, 122(51):e2523012122, 2025. doi: 10.1073/pnas.2523012122
Questions for authors

See my comments in the "Critical Weaknesses" section.

1 comment

Thanks for the feedback. Reply to specific points below.

  1. Thanks for the reference on GGDLGNs. They seem interesting but significantly more complex than the DLGNs studied here, where the non-linear interaction between the different parameters are explicitly brought out. This simplicity is what enables the study of feature learning dynamics under gradient descent (this is not the goal at all in the GGDLN paper). Will add a paragraph on this and other similar architectures referred in the GGDLN paper

  2. In the single path DLGN, parameter interaction is indeed an explicit product of sigmoids of linear functions. The multi-path DLGN is much more complex with path interactions. The reason we study this far simpler object is that it shows similar behaviour to real and complex architectures like the ReLU network on the ODT and several other datasets. The main object of study is feature learning under gradient descent. Deep networks are interesting because they perform this efficiently. The fact that the multi-path DLGN can be expressed as sum of exponentially many interacting single path DLGNs, makes the single path DLGN interesting. Even the single path DLGN optimisation is highly non-convex due to the symmetries and multiple local/global minima. We do not see how one can claim to understand SGD with ReLU nets/multi-path DLGNs if one cannot even explain single path DLGN parameter trajectories under population gradient flow.

  3. The decision tree labelling is arguably far more general and powerful than most typical settings analysing feature learning -- which include parity, single-index functions, low-degree polynomials of multi-index functions etc. If the ODT setting is too artificial for a theoretical analysis, we would have to nullify almost all deep learning papers on feature learning theory. The fact that the DLGN is naturally a nice sum of polyhedral-indicator functions does indeed help in analysis, but that is a feature and not a bug. If we do not understand how any correct parameters will look like, how can one measure progress towards the solution when the loss is not a reliable indicator due to its highly non-convex nature?

  4. The general claims are simple and stated several times in the paper. "parameters perform several roles (e.g. lens and separator) and a positive feedback between the two is crucial for feature learning". A statement as general as this is necessarily vague. The single path DLGN theorems with the ODT setting are a concrete version of the above statement.

As for a justification of the architecture under the microscope here, we dedicate an entire section in the paper. We acknowledge the artificiality of the architecture, but an analysis at this level is just not possible for an architecture like the ReLU net, and is meaningless for an architecture like the deep linear network. The DLGN is competitive on standard feature vector datasets, and comparable versions of the architecture to (say) the ReLU ResNet gives an accuracy within a few percentage points, and beats linear/kernel methods handily (not emphasised in the paper her purposefully). This is all to just say that "we know ReLU nets perform feature learning, and feature learning is essential, and DLGNs also perform feature learning and are simpler. Hence we study feature learning in DLGNs first."

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.