- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper extends Neural Low-Degree Filtering (Neural LoFI) by introducing backward coupling between adjacent layers. The proposed method uses upper-layers to guide lower-layer feature selection via cross-layer correlation operator and its singular modes. The authors provide theoretical analysis of local coupled dynamics, continuous-time kernel evolution and feature recovery in a hierarchical network. The authors also conduct experiments on synthetic data and binary CIFAR-10 and show improvements over forward Neural LoFI.
Critical strengths
-
The paper addresses an important and well-motivated question: how end-to-end training enables feature learning beyond layer-wise methods. The proposed backward-coupling mechanism offers an intuitive interpretation of how upper-layer information can guide feature selection in lower layers and the theoretical analysis gives it a principled basis.
-
On binary CIFAR-10 BackLoFi improves over forward Neural LoFI for both fully-connected and convolutional networks, and shows largest gain when data are scarce.
-
The authors are candid in reporting variants that do not work and further analyzing alternative update rules. I appreciate this practice and do not consider it as a shortcoming for a theory-oriented paper.
Critical weaknesses
-
My major concern is about the fairness of experimental comparison. Appendix H & F indicate that spectral configurations and GD hyperparameters are selected via mean test error. BackLoFI also has more tunable hyperparameters than forward LoFI. Given the improvement at n=50K is not large (+1.2% for FC and +1.1% for CNN), the authors should clarify the contribution more clearly.
-
The relationship to end-to-end gradient descent is not solidly estabilished. The GD baselines uses full batch for training without momentum, weight decay or other data augmentation methods like random cropping.
-
The experiments are limited to one binary CIFAR-10 datasets with 3-layer networks. For theory-oriented paper I do not think leaderboard results are expected, but evaluating at least one harder setting (e.g. multi-class CIFAR-10 or deeper networks) would show whether the method generalizes or fails, which can lead to a more meaningful failure analysis.
Questions for authors
- Could the authors redo the selection on a held-out validation set and report test error only for the selected settings? Please also report the number of hyperparameter searched for BackLoFi, Forward LoFi and GD.
- App. G.4 shows that test error is still decreasing at the final step for larger datasets. Could the authors report GD with a longer training time, and better include momentum and weight decay. This can provide a fair comparison between BackLoFi and end-to-end training.
I would raise my score if the authors address these concerns.
2 comments
I thank the authors for the additional experiments addressing my main concerns. Therefore I raise my score.
Response to Reviewer UEQM
We thank the reviewer for recognizing the backward-coupling mechanism and its theoretical basis. We ran all the requested controls; every change is marked in orange in the revised manuscript and listed at the end of this response.
1. Model selection. We reran every experiment of Figure 3 with the 10,000 CIFAR-10 test images split once into a validation set and a test set of 5,000 images each: every configuration, for every method, is selected on the validation set, and the new Figure 3 reports its error on the test set. The qualitative picture does not change and the main points remain: LoFi with backward coupling improves on Forward LoFi, especially for large training sets.
Regarding the hyperparameters, the goal of Figure 3 is not to search for better ones or to optimize classification accuracy, but to identify the mechanism that makes gradient descent on all layers more effective than layer-wise learning. BackLoFi installs only that mechanism, at fixed network width and with no gradient step: its two additional hyperparameters set how much of the backward coupling is installed, and without them it is exactly Forward LoFi. The improvement therefore comes from learning different features through cross-layer coupling, not from increased capacity or a larger search. For completeness, Appendix H (Table 1) now lists every hyperparameter searched.
2. Gradient-descent baseline. We believe it is important to distinguish a theoretical comparison of learning mechanisms from a benchmark of optimized training procedures. The paper asks which features joint training discovers, and its theory describes plain gradient flow, as do the frameworks it builds on: deep linear networks (Saxe et al., 2014), the neural tangent kernel (Jacot et al., 2018; Du et al., 2019; Arora et al., 2019), mean-field theory (Mei et al., 2018; Chizat & Bach, 2018; Rotskoff & Vanden-Eijnden, 2018) and dynamical mean-field theory (Bordelon & Pehlevan, 2022).
The reference dynamics is therefore full-batch gradient descent without momentum, weight decay or data augmentation: momentum and weight decay change the dynamics, and augmentation changes the learning problem, so none of them is described by the theory. Against an optimizer with these ingredients, the gap between BackLoFi and gradient descent would also contain their effect; against plain gradient descent, it measures how much of what joint training learns is captured by the backward coupling alone, which is the question of the paper.
This is why theories of training dynamics are tested against the dynamics they describe, even on real data: kernel descriptions of training are compared with networks trained by gradient descent on MNIST and CIFAR-10 (Lee et al., 2019; Atanasov et al., 2022), lazy and feature learning are contrasted with gradient flow and no regularization (Geiger et al., 2020; Paccolat et al., 2021; Petrini et al., 2023), and the theory of the edge of stability is developed and tested entirely in full-batch training (Cohen et al., 2021; Damian et al., 2023; Cohen et al., 2025). As Cohen et al. (2021) note, networks are rarely trained this way in practice, but full-batch gradient descent is the dynamics on which practical optimizers are built.
Separately, we increased the number of GD steps for the points in Figure 3,7 and 8. On the fully connected network we continued the same runs to at least 10,000 steps (Figure 7): the error stays almost constant from step 3,000 onwards for all training-set sizes. For the convolutional network, we increased the total number of GD steps to 10,000 for lr=0.003,0.005. This improves the performance of GD but keeps the qualitative picture of the previous Figure 3.
3. Scope. Our central contribution is theoretical and is not specific to binary CIFAR-10. Equation (22) establishes a population-level separation between forward and backward feature discovery; Theorem 3 (Appendix E) proves hidden-subspace recovery with oracle sensitivities, and Proposition 5 provides a sufficient condition for transferring this recovery to learned scores.
Nevertheless, we extended the experiments to two more challenging settings (new Figure 10, Appendix G.6), and the conclusions remain unchanged.
CINIC-10: On this harder image dataset, with many images drawn from ImageNet, BackLoFi retains essentially the same advantage over Forward LoFi: 15.2% versus 16.3% at n=50,000, a 1.1-point improvement. It tracks GD up to n=10,000; GD reaches 13.7% at n=50,000.
Deeper network: On a four-layer fully connected network, BackLoFi improves over Forward LoFi at every training-set size, reaching 11.1% versus 12.3% at n=50,000. Adding a fourth layer slightly worsens Forward LoFi but leaves BackLoFi essentially unchanged. As in the three-layer case, BackLoFi tracks GD up to n=5,000, while GD performs better with more data.
We highlighted all the changes to the manuscript in orange in the new version.