acceptodds
Under review as a conference paper at ICLR 2027

Neural LoFi with Backward Coupling: A Spectral Theory of Cross-Layer Feature Learning

Abstract

During joint training, backpropagation allows later layers to reshape earlier representations, a feedback mechanism absent from purely layer-wise learning. We extend Neural Low-Degree Filtering (Neural LoFi), a recently proposed layer-wise spectral surrogate for feature learning, to capture this interaction. Starting from the exact backpropagation gradients, we derive a local coupling between a neuron’s incoming weights and its corresponding next-layer column. In a controlled small-initialization regime with smooth activations and fixed surroundings, these dynamics amplify singular modes of a cross-layer correlation operator. Each growing mode pairs an earlier feature with an upper-layer response through shared neuron coefficients. This motivates a backward-coupling algorithm in which upper-layer sensitivities generate sample-dependent effective labels for earlier spectral steps, while corrected features refine upward transmission. A function-space variational principle characterizes the selected feature–response pairs by maximizing the label correlation of their product under kernel-norm constraints. For three-layer networks, we derive self-consistent continuous-time dynamics for the filtered hidden-layer features, the activation kernels, and the network output under the proposed backward coupling. In a hierarchical multi-index model, we establish a feature-recovery separation: backward coupling recovers lower-layer directions invisible to the original forward label-weighted statistic. The linear specialization recovers residual-dependent singular-mode dynamics, while paired modes provide a dynamical mechanism for linked low-rank covariance components consistent with the rainbow-network picture. Experiments with fully connected and convolutional networks on binary CIFAR-10 show improved accuracy over forward Neural LoFi at matched widths.

Primary area:
learning theory
Keywords:
deep learning, multi-layer, hierarchical, spike, theory

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UEQMOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper extends Neural Low-Degree Filtering (Neural LoFI) by introducing backward coupling between adjacent layers. The proposed method uses upper-layers to guide lower-layer feature selection via cross-layer correlation operator and its singular modes. The authors provide theoretical analysis of local coupled dynamics, continuous-time kernel evolution and feature recovery in a hierarchical network. The authors also conduct experiments on synthetic data and binary CIFAR-10 and show improvements over forward Neural LoFI.

Critical strengths
  • The paper addresses an important and well-motivated question: how end-to-end training enables feature learning beyond layer-wise methods. The proposed backward-coupling mechanism offers an intuitive interpretation of how upper-layer information can guide feature selection in lower layers and the theoretical analysis gives it a principled basis.

  • On binary CIFAR-10 BackLoFi improves over forward Neural LoFI for both fully-connected and convolutional networks, and shows largest gain when data are scarce.

  • The authors are candid in reporting variants that do not work and further analyzing alternative update rules. I appreciate this practice and do not consider it as a shortcoming for a theory-oriented paper.

Critical weaknesses
  • My major concern is about the fairness of experimental comparison. Appendix H & F indicate that spectral configurations and GD hyperparameters are selected via mean test error. BackLoFI also has more tunable hyperparameters than forward LoFI. Given the improvement at n=50K is not large (+1.2% for FC and +1.1% for CNN), the authors should clarify the contribution more clearly.

  • The relationship to end-to-end gradient descent is not solidly estabilished. The GD baselines uses full batch for training without momentum, weight decay or other data augmentation methods like random cropping.

  • The experiments are limited to one binary CIFAR-10 datasets with 3-layer networks. For theory-oriented paper I do not think leaderboard results are expected, but evaluating at least one harder setting (e.g. multi-class CIFAR-10 or deeper networks) would show whether the method generalizes or fails, which can lead to a more meaningful failure analysis.

Questions for authors
  • Could the authors redo the selection on a held-out validation set and report test error only for the selected settings? Please also report the number of hyperparameter searched for BackLoFi, Forward LoFi and GD.
  • App. G.4 shows that test error is still decreasing at the final step for larger datasets. Could the authors report GD with a longer training time, and better include momentum and weight decay. This can provide a fair comparison between BackLoFi and end-to-end training.

I would raise my score if the authors address these concerns.

2 comments
AuthorsOpenReview ↗

Response to Reviewer UEQM

We thank the reviewer for recognizing the backward-coupling mechanism and its theoretical basis. We ran all the requested controls; every change is marked in orange in the revised manuscript and listed at the end of this response.

1. Model selection. We reran every experiment of Figure 3 with the 10,000 CIFAR-10 test images split once into a validation set and a test set of 5,000 images each: every configuration, for every method, is selected on the validation set, and the new Figure 3 reports its error on the test set. The qualitative picture does not change and the main points remain: LoFi with backward coupling improves on Forward LoFi, especially for large training sets.

Regarding the hyperparameters, the goal of Figure 3 is not to search for better ones or to optimize classification accuracy, but to identify the mechanism that makes gradient descent on all layers more effective than layer-wise learning. BackLoFi installs only that mechanism, at fixed network width and with no gradient step: its two additional hyperparameters set how much of the backward coupling is installed, and without them it is exactly Forward LoFi. The improvement therefore comes from learning different features through cross-layer coupling, not from increased capacity or a larger search. For completeness, Appendix H (Table 1) now lists every hyperparameter searched.

2. Gradient-descent baseline. We believe it is important to distinguish a theoretical comparison of learning mechanisms from a benchmark of optimized training procedures. The paper asks which features joint training discovers, and its theory describes plain gradient flow, as do the frameworks it builds on: deep linear networks (Saxe et al., 2014), the neural tangent kernel (Jacot et al., 2018; Du et al., 2019; Arora et al., 2019), mean-field theory (Mei et al., 2018; Chizat & Bach, 2018; Rotskoff & Vanden-Eijnden, 2018) and dynamical mean-field theory (Bordelon & Pehlevan, 2022).

The reference dynamics is therefore full-batch gradient descent without momentum, weight decay or data augmentation: momentum and weight decay change the dynamics, and augmentation changes the learning problem, so none of them is described by the theory. Against an optimizer with these ingredients, the gap between BackLoFi and gradient descent would also contain their effect; against plain gradient descent, it measures how much of what joint training learns is captured by the backward coupling alone, which is the question of the paper.

This is why theories of training dynamics are tested against the dynamics they describe, even on real data: kernel descriptions of training are compared with networks trained by gradient descent on MNIST and CIFAR-10 (Lee et al., 2019; Atanasov et al., 2022), lazy and feature learning are contrasted with gradient flow and no regularization (Geiger et al., 2020; Paccolat et al., 2021; Petrini et al., 2023), and the theory of the edge of stability is developed and tested entirely in full-batch training (Cohen et al., 2021; Damian et al., 2023; Cohen et al., 2025). As Cohen et al. (2021) note, networks are rarely trained this way in practice, but full-batch gradient descent is the dynamics on which practical optimizers are built.

Separately, we increased the number of GD steps for the points in Figure 3,7 and 8. On the fully connected network we continued the same runs to at least 10,000 steps (Figure 7): the error stays almost constant from step 3,000 onwards for all training-set sizes. For the convolutional network, we increased the total number of GD steps to 10,000 for lr=0.003,0.005lr=0.003,0.005. This improves the performance of GD but keeps the qualitative picture of the previous Figure 3.

3. Scope. Our central contribution is theoretical and is not specific to binary CIFAR-10. Equation (22) establishes a population-level separation between forward and backward feature discovery; Theorem 3 (Appendix E) proves hidden-subspace recovery with oracle sensitivities, and Proposition 5 provides a sufficient condition for transferring this recovery to learned scores.

Nevertheless, we extended the experiments to two more challenging settings (new Figure 10, Appendix G.6), and the conclusions remain unchanged.

CINIC-10: On this harder image dataset, with many images drawn from ImageNet, BackLoFi retains essentially the same advantage over Forward LoFi: 15.2% versus 16.3% at n=50,000n=50{,}000, a 1.1-point improvement. It tracks GD up to n=10,000n=10{,}000; GD reaches 13.7% at n=50,000n=50{,}000.

Deeper network: On a four-layer fully connected network, BackLoFi improves over Forward LoFi at every training-set size, reaching 11.1% versus 12.3% at n=50,000n=50{,}000. Adding a fourth layer slightly worsens Forward LoFi but leaves BackLoFi essentially unchanged. As in the three-layer case, BackLoFi tracks GD up to n=5,000n=5{,}000, while GD performs better with more data.

We highlighted all the changes to the manuscript in orange in the new version.

I thank the authors for the additional experiments addressing my main concerns. Therefore I raise my score.

Reviewer_KBWsOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper extends forward Neural LoFi to incorporate feedback from later layers into earlier feature selection. Starting from backpropagation, it derives coupled dynamics for a neuron’s incoming weights and outgoing column and shows that, under controlled small-initialization assumptions, these dynamics amplify paired singular modes of a cross-layer correlation operator. This motivates BackLoFi, which uses upper-layer sensitivities to guide lower-layer spectral corrections. Synthetic experiments and binary CIFAR-10 comparisons support the usefulness of the proposed mechanism.

Critical strengths

The paper provides a clear and mathematically tractable account of cross-layer feature learning. The coupling between incoming rows and outgoing columns explains how learning what a neuron detects is coordinated with learning how its activation influences subsequent layers. The spectral interpretation, variational formulation, and evolving-kernel description develop this mechanism from complementary perspectives. I find this a substantive extension of the forward LoFi framework as it identifies how representations learned above can change which features are discoverable below.

The hierarchical example is apt and experiments seem complete. The construction identifies hidden directions that are absent from the forward population statistic but become detectable after injecting upper-layer sensitivity. The oracle-sensitivity recovery theorem establishes this separation under explicit assumptions, while the synthetic experiments examine both prediction and feature recovery. The matched-width image experiments provide complementary evidence that the correction is useful beyond the controlled theoretical setting. Together, these results connect the proposed mechanism to an observable learning advantage and support acceptance.

Critical weaknesses

The only weakness is perhaps the paper builds on Neural LoFi, which itself is a very recent paper (NeurIPS 2026) and perhaps needs more time for the community to judge its usefulness. This is barely a weakness, in my opinion.

Questions for authors

No question that would materially affect my assessment.

1 comment
AuthorsOpenReview ↗

Response to Reviewer KBWs

We thank the reviewer for the careful reading and for recognizing the contribution of the coupled spectral mechanism and the feature-recovery separation. We will improve the exposition to make the connection to Neural LoFi and the scope of our assumptions clearer.

Reviewer_hscXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The authors extend Neural LoFI to capture feedback from later layers during "joint training". They propose BackLoFi and give some theoretical results and empirical evaluations.

Critical strengths

None.

Critical weaknesses
  • Writing/Exposition: The paper is presented/written in a dense way that makes it difficult for even expert readers to parse. There is insufficient motivation /clarity when introducing ideas. Loaded/unclear terms like "joint training" are often used without sufficient context. For example, sentences like "With a learned upper representation, the effective readout becomes sample dependent. To isolate the coupling, freeze the surrounding maps ...and start the row–column pair small" make reading extremely difficult.

  • Relevance: The premise of the paper is to extend a very recent method (Neural Lofi) that is generally little known/used (3 citations on google scholar at the time of writing this review). The extension involves heavy machinery, but it is unclear to that this is of sufficient relevance/significance for the ICLR audience/conference.

Questions for authors

N/A

1 comment
AuthorsOpenReview ↗

Response to Reviewer hscX

Exposition. We are surprised by the criticism of terminology, since both terms singled out in the review are standard or explicitly defined. Joint training means training all layers together by backpropagation, as opposed to layer-wise; the abstract introduces it through this contrast, and the term is routine (Bengio et al., 2013; Bietti et al., NeurIPS 2022; Glasgow, ICLR 2024). Readout is the standard name for the weights through which a representation is read out into the output, from reservoir computing (Jaeger, 2001; Maass et al., 2002) to the theory of two-layer networks (Zavatone-Veth & Pehlevan, 2021; Cui et al., 2024; Barbier et al., 2025). Effective only indicates that, for a neuron jj in a hidden layer, this role is played by what the layers above read from it: the factor ⟨ωℓ+1,j,gℓ+1(x)⟩\langle\boldsymbol\omega_{\ell+1,j},\mathbf g^{\ell+1}(\mathbf x)\rangle in Eq. (4), defined in the sentence before the quoted one and used in this sense in the paper we extend (Dandi et al., 2026). It is the backpropagated signal of Rumelhart et al. (1986), and in deep linear networks the product of the downstream weights (Saxe et al., 2014).

We have rewritten the quoted passage so that it states explicitly what changes under joint training (Section 2, in orange). If the reviewer has other constructive criticism on how to improve the clarity, we would be glad to hear it.

Relevance. The citation count of a paper released a few months ago does not measure significance, and our question is not tied to that method: how do later layers shape the features learned by earlier ones under end-to-end training (backward feature correction, Allen-Zhu & Li, 2023)? Theorem 1 derives from backpropagation the amplification of paired singular modes across adjacent layers, the nonlinear counterpart of Saxe et al. (2014). Eq. (22) and Theorem 3 exhibit features whose forward statistic vanishes and which are recovered only through the layer above (Figure 2). Figure 3 shows the same mechanism on CIFAR-10. This is a theory of how hierarchical representations are learned, which is within the core scope of ICLR.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.