acceptodds
Under review as a conference paper at ICLR 2027

Increasing Width Allows Greedy Layer-wise Training to Rival End-to-End Backpropagation in Self-Supervised Learning

Abstract

End-to-end backpropagation has been the dominant mode of training in deep learning, allowing for the coordination of parameter updates across layers of a neural network. Prior studies have explored alternative – and, in some cases, simpler – training mechanisms, showing that they can sometimes achieve performance similar to backpropagation. However, the architectural conditions under which locally optimized networks, which avoid end-to-end backpropagation of error, can learn representations comparable to those learned through end-to-end training remain unclear. We aim to answer this question in the context of self-supervised learning, an important framework for large-scale pretraining in artificial intelligence. Here, we investigate how network width and depth affect the efficacy of greedy layer-wise and end-to-end self-supervised training in convolutional networks. We find that in wider networks, the benefits of end-to-end backpropagation over greedy layer-wise training shrink: in relatively shallow and very wide networks, we even observed higher performance in models trained with greedy layer-wise training. Subsequent analysis of the representations formed by these networks shows that very wide greedy-trained networks exhibit more favorable representational geometry than do networks trained end-to-end with backpropagation. This work shows that width can compensate for restricted credit assignment and identifies differences in representational geometry as a potential mechanism for their improved performance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised learning, greedy layer-wise training, local learning, credit assignment, network width, representational geometry

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_3hQMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how network width affects greedy layer-wise self-supervised learning relative to end-to-end backpropagation. Using Conv4/Conv8 models with Barlow Twins and SimCLR, the authors show that increasing width disproportionately benefits greedy training and can largely close, or occasionally reverse, the performance gap. They further relate this behavior to differences in representation geometry measured by SSF and SNF.

Critical strengths
  • The paper presents a clean and systematic empirical study of the interaction between network width and greedy layer-wise training.
  • The representation-geometry analysis provides a useful perspective beyond downstream accuracy alone.
Critical weaknesses
  • The conceptual novelty is limited. Prior work has already shown that local or greedy learning can approach end-to-end training with sufficiently expressive architectures or modules. The paper mainly adds an empirical width‑scaling observation, without a new theoretical or causal explanation for why width mitigates restricted credit assignment.
  • The empirical scope is narrow. The main results are limited to shallow Conv4/Conv8 models on CIFAR‑10/100, and the additional SimCLR/CIFAR‑100 experiments use a single seed. Moreover, the strongest result relies on extremely wide networks, making it unclear whether the effect remains meaningful under realistic model or compute budgets.
Questions for authors
  • Does the main result hold for modern architectures such as ResNets and on larger-scale datasets?
  • Can the authors provide parameter/FLOP/memory-matched comparisons between greedy and end-to-end training?
  • Does the reported crossover remain when checkpoints are selected using validation performance and E2E optimization is tuned specifically for the widest models?
  • Does the trend persist with linear probing or transfer evaluation, rather than only kNN?
  • Can the authors provide stronger evidence that SSF/SNF is explanatory rather than simply correlated with width and downstream performance?
2 comments
AuthorsOpenReview ↗

Response to reviewer 3hQm (Part 1/2)

We thank the reviewer for the in-depth review and comments. Below we address each weakness.

Weakness 1: limited conceptual novelty

We agree that local learning approaching end-to-end training is not new. Our contribution is a controlled study showing that, with the training procedure held fixed, widening the network shrinks the gap between greedy layer-wise and end-to-end training. Though we don’t provide a theoretical or causal explanation, our study is a scientific exploration of a novel and unexpected phenomenon and thus deserves further investigation.

We do, however, propose a potentially causal variable in improved representational geometry. Our additional control experiments (see general response to all reviewers) show that both local learning and freezing are required for improved representational geometry and the increase in accuracy.

Further, our revised Related Work connects our results to recent theory showing that strictly layer-wise training can learn useful hierarchical features (Daniely, 2026; Dandi et al., 2026).

Weakness 2: narrow empirical scope

We agree that the scope is narrow. However, our goal is not to make a broad claim regarding improvement for all networks and architectures. As discussed in the general response, our goal is a controlled, scientific exploration and study of a novel width effect. We simply identify width as an under-explored variable in local learning. And, as mentioned in the general response, preliminary experiments with ResNet-18 networks trained blockwise greedily saw a similar narrowing of the gap between end-to-end and greedy layer-wise training as width increased. This suggests that the results may generalize to other architectures, which we hope future work explores.

Though the additional SimCLR and CIFAR-100 results use a single seed, both show the same trend across all eight widths, consistent with our three-seed CIFAR-10 results, whose seed-to-seed standard deviation at widths of 8x and above is at most 0.6 points, much smaller than the effects we report.

Finally, we would argue that these networks do train under realistic model and compute budgets. Specifically, extremely wide is extreme as a multiplier, not in absolute size or cost. For example, greedy layer-wise training of the Conv4 32x width network (our largest) took about 8 hours on an A100 GPU. Further, although the crossover in performance happens at the widest networks, much of the narrowing itself happens at the smaller networks. An example of this is the Conv4 at 8x width, which has ~25 million parameters, less than a ResNet-50, which narrowed the gap from 4.3 points at 1x width to 1.9 points at 8x width (kNN). Conv8 at 4x width has about 19 million parameters and narrows the gap with end-to-end training from 12.1 points at 1x to 4.2 points (kNN).

AuthorsOpenReview ↗

Response to reviewer 3hQm (Part 2/2)

Questions:

Below, we address each question.

1. Modern architectures and datasets

Our preliminary experiments using ResNet-18s trained blockwise greedy showed a similar narrowing trend with width. This suggests that our result may generalize to other architectures (see general response to all reviewers). Our largest tested dataset is CIFAR-100, where we see a similar trend as well.

2. Compute comparisons

We now report measured training FLOPs and memory for every model (Appendix A.7, Tables 9-10). At the same width, greedy training uses 4.0x (Conv4) and 4.8x (Conv8) fewer training FLOPs than end-to-end training and, at the widest widths, about half the training memory. At equal training FLOPs, greedy training is more accurate than end-to-end training at Conv4 32x (78.2% vs. 77.3%) and matches it at 16x (Table 10). Here, "equal training FLOPs" compares greedy layer-wise training’s final accuracy with end-to-end training’s accuracy at the point in its training where it has used the same total FLOPs as the full greedy run (about epoch 250 for Conv4). Alternatively, measured at end-to-end training’s peak: reaching its best epoch (77.6% at Conv4 32x) takes roughly the same compute as greedy training’s entire run (1.1x on average), and greedy training still ends higher (78.2%).

3. Checkpoint selection & end-to-end optimization

Though we did not save checkpoints, we provide best epoch on the test set, which is an upper bound (at least as favourable to end-to-end training as any validation-based choice). This gives 77.6% at Conv4 32x, still below greedy layer-wise training’s final 78.2% (three seeds). In regards to hyperparameter tuning, under maximal update parameterization (Appendix A.5), the gap narrows as before (see general response to all reviewers) and the crossover survives (kNN).

4. Linear probing

Yes, we implement linear probing and see that the trend still persists (see general response to all reviewers). Under linear probes, greedy Conv4 at 32x comes within 0.5 points of end-to-end training rather than surpassing it, while greedy Conv8 at 16x surpasses end-to-end training in all three seeds.

5. SSF/SNF causality

We read this as two questions. First, whether the geometry change is specific to greedy training. Second, whether geometry plays a causal role in downstream accuracy.

To the first question, our new control experiments (see general response to all reviewers) isolate variables within the training procedure at fixed widths (Section 4.4, Appendix A.6). Neither freezing alone (end-to-end training with progressive freezing) nor local objectives alone (simultaneous local training) at the widest widths raises SNF or SSF to the same degree as greedy layer-wise training. Only through the combination of both, which occurs in greedy layer-wise training, do we see improvements in representational geometry. This fits our previous results showing that greedy training builds SNF and SSF stage by stage, while in end-to-end training both peak early and then decline (Fig. 4c,d). The geometry difference is therefore produced by the greedy procedure, not by width alone.

To the second question, because we do not manipulate geometry directly, we are unable to establish an explicit causality. However, recent theory establishes that SSF and SNF govern classification error (Wakhloo et al., 2026). Further, recent work has shown that directly increasing SNF with a regularizer improves accuracy (Kirti & Zylberberg, 2026).

We thank the reviewer again for the constructive review.

Reviewer_XXPzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The goal of the paper is to study the impact of the width of convolutional networks on the performances of greedy layer-wise training compared to the classical end-to-end backpropagation, and to see whether the greedy version is able to match or outperform end-to-end training. The authors train Conv4 and Conv8 networks in two different self-supervised settings on CIFAR-10 (and CIFAR-100) while keeping the number of epochs consistent across training. The performances obtained are compared with a kNN probe. They find that increasing width helps greedy training much more than end-to-end training, closing the gap and even reversing it for the widest Conv4 network. They also analyse the representational geometry of the learned representations as a possible explanation for this behaviour.

Critical strengths

The problem is well posed and cleanly designed with the same architectures, hyperparameters and a clear protocol making the results easy to compare and interpret.

The analysis of the results goes beyond accuracy by looking at the representational geometry through SSF and SNF.

Critical weaknesses

The models used are quite small and we do not have insight about the scaling of the results on bigger models. The Conv8 greedy never matches the performances of the end-to-end model.

The results could also be interpreted as end-to-end needing smaller models to obtain the same performance as greedy, which would greatly reduce the gain from the greedy method.

The hyperparameters are chosen for width x1 and x8, but they might be under-tuned for x16 and x32, especially for end-to-end models that might be harder to train efficiently.

The compute is not reported (greedy should be faster), so the same number of epochs is not the same training budget; it would be interesting to have at least one experiment at equal budget and not equal epochs.

Questions for authors

In Figure 2c (and 4c,d), we see that for the widest model (Conv4 x32), end-to-end training only needs around 250 epochs. Does greedy perform similarly if we reduce the number of epochs, or does it need more epochs to learn efficiently?

Only evaluated with kNN probing; does the performances translate to other type of probe such as training a linear probe (or a small MLP head)?

Any intuition about the reasons behind the differences in the geometry between end-to-end and greedy?

Did you study the impact of the width at a fixed parameter count (by reducing the depth)?

All experiments use small Conv4 and Conv8 models, did you try bigger models and/or other architectures?

3 comments
AuthorsOpenReview ↗

Response to reviewer XXPz (Part 1/2)

We thank the reviewer for their insightful comments. We address all weaknesses below.

Weakness 1: Small models

Regarding model size, the goal of our paper is to provide a controlled study of an unexpected and novel width effect. To do this, we swept width over a 128-fold range with everything else fixed, which was computationally quite expensive; we thus use shallow convolutional networks (see the general response and Question 5). Note that the wide models themselves are not small but rather just shallow: our widest Conv4 and Conv8 encoders have 396M and 300M parameters, roughly 12 to 15x the size of a ResNet-50.

In addition, the kNN probe for Conv8 at 16x shows near convergence with end-to-end training (−0.23 ± 0.25 points). In fact, with our newly added linear probe, we actually see greedy layer-wise Conv8 at 16x width surpass end-to-end training (+1.00 ± 0.79 points; greedy ahead in all three seeds; Appendix A.4).

Weakness 2: End-to-end needing smaller models

The reviewer’s intuition is correct that end-to-end needs smaller models to obtain the same performance as greedy layer-wise training. However, a fairer comparison is equal training compute and we show that greedy layer-wise training at 32x and end-to-end training at 16x cost about the same total training FLOPs (2,167 vs. 2,166 PFLOP; Appendix A.7). At this equal cost, greedy training is about a point more accurate under the kNN probe (though 0.4 points less accurate under the linear probe).

Beyond this, we motivate the exploration of settings in which local learning is effective due to the many cases where backpropagation is impractical. Greedy layer-wise learning suits hardware where global communication is costly, such as on-chip or neuromorphic learning, or when training is split across devices. Though greedy layer-wise training may not currently be as efficient or effective as end-to-end backpropagation, it is valuable to scientifically explore variables that may improve its performance.

Weakness 3: Hyperparameter tuning

The reviewer brings up a very valid concern. To address this, we repeated the full Conv4 width sweep (0.25x–32x, end-to-end and greedy, seed 0) under the maximal update parametrization (μP; Yang et al., 2022), using the authors’ mup package with 1x as the base width (Appendix A.5, Fig. 7). We used a coordinate check to confirm our implementation, finding that after four training steps, the change in each hidden layer’s output grows 26–34x from 1x to 32x width under the standard parametrization, but stays relatively constant (1.05–1.14x) under μP. Our results hold under μP: the gap between greedy and end-to-end training narrows with width. Under the kNN probe, the gap drops from -4.4 points at 1x to +3.0 points at 32x. We therefore argue that our results do not rely on poor parameterization of the widest networks.

Weakness 4: Compute reporting

We now report measured training FLOPs and memory for every model (Appendix A.7, Tables 9-10). At the same width, greedy training uses 4.0x (Conv4) and 4.8x (Conv8) fewer training FLOPs than end-to-end training and, at the widest widths, about half the training memory. At equal training FLOPs, greedy training is more accurate than end-to-end training at Conv4 32× (78.2% vs. 77.3%) and matches it at 16x (Table 10).

AuthorsOpenReview ↗

Response to reviewer XXPz (Part 2/2)

Below, we address all questions asked by the reviewer. We apologize for the lengthy response.

Question 1: Greedy with reduced epochs

No, our previous (unreported) experiments found that greedy layer-wise training requires its full budget to accomplish similar performance. For example, Conv4 32x reaches 69.5% accuracy with 60 epochs per layer (240 in total) and 75.3% with 125 (500 in total), compared with 77.2% and 76.0% for end-to-end training after the same numbers of epochs (seed 0). However, a greedy epoch costs about 4x fewer FLOPs, so greedy layer-wise training’s full 1,000-epoch run costs about the same as end-to-end training’s first 250 epochs. At this equal cost, greedy training is ahead (78.2% vs. 77.3%). Thus, while end-to-end training is more efficient in terms of total epochs at 32x width, greedy layer-wise training is more efficient in total compute.

Question 2: Other probing

The reviewer raises another important concern. To address this, we trained linear probes for the primary Conv4 and Conv8 results using the standard fixed-feature protocol of Kornblith et al. (2019) (Appendix A.4). The narrowing gap holds: under linear probes, greedy Conv4 at 32x comes within 0.5 points of end-to-end training rather than surpassing it, while greedy Conv8 at 16x surpasses end-to-end training in all three seeds.

Question 3: Intuition for geometry differences

This is a good question. We are unable to point to a specific mechanism but can point to some interesting findings that lead to the following intuition.

The Barlow Twins loss rewards any feature that is invariant to augmentation and decorrelated from other features, whether or not it is related to the class (see the toy example in our response to Reviewer Ry51). End-to-end training optimizes every layer jointly against this single objective for the whole run. With enough capacity, it may keep lowering the loss by recruiting additional invariant but class-irrelevant directions and by reshaping earlier layers to serve the final objective, which may dilute class structure. Consistent with this, at 32x width, the end-to-end network's SSF and SNF peak early in training and then decline (Fig. 4c,d). Freezing earlier layer may preserve class-related geometry that later optimization would otherwise interfere with.

However, our new freezing controls (see general response to all reviewers, Section 4.4, and Appendix A.6) found that both freezing and the local objective together were required for the increases in SNF and SSF (just freezing alone was not enough). Thus, the local objective must contribute to class-related geometry in some way. Intuitively, in greedy layer-wise training, each layer's local objective may make it extract the most augmentation-invariant structure available in its input. This would make sense to be class identity given that class is among the most consistently invariant signals, especially since the image augmentations applied are designed to preserve it.

Question 4: Fixed parameter count

Unfortunately, we did not study the impact of width at a fixed parameter count. This is a very interesting question that we see as a valuable direction for future work.

Question 5: Bigger models

Note again that although our models are shallow, they are not necessarily small (see response to weakness 1). In addition, as mentioned in the general response to all reviewers, preliminary experiments with ResNet-18 networks trained blockwise greedily saw a similar narrowing of the gap between end-to-end and greedy layer-wise training as width increased. The gap decreased from 11.4 points at 0.25x width to 2.2 points at 4x width, where end-to-end plateaued after 2x width while greedy layer-wise training continued to improve at 4x width. This suggests the effect may extend to deeper, residual architectures, which we mention in our discussion is an important direction for future work.

We thank the reviewer again for the constructive review.

AuthorsOpenReview ↗

Brief summary of rebuttal for Reviewer XXPz

Dear Reviewer XXPz,

As the discussion period closes soon, we wanted to briefly summarize our rebuttal and how it addresses each of your concerns.

  1. Model size and Conv8 never matching end-to-end

Our widest models are not small but shallow: the widest Conv4 and Conv8 encoders have 396M and 300M parameters, roughly 12–15x the size of a ResNet-50. Keeping them shallow is what allows a 128-fold width sweep. Under the newly added linear probes (Appendix A.4), greedy Conv8 at 16x exceeds end-to-end training (+1.00 ± 0.79 points, greedy ahead in all three seeds). We also added a preliminary ResNet-18 width sweep (Appendix A.9), where the gap shrinks from 11.4 to 2.2 points (linear probe) between 0.25x and 4x.

  1. End-to-end needing smaller models for the same performance

Though end-to-end training may need smaller models for similar performance, our finding is most relevant where end-to-end backpropagation is impractical, such as on-chip learning or training split across devices with costly communication. In these settings, width offers a way to recover performance lost to restricted credit assignment. At equal compute (i.e., greedy Conv4 at 32x vs. end-to-end Conv4 at 16x), the two are comparable in performance (greedy is about one point higher under kNN and 0.4 points lower under the linear probe).

  1. Hyperparameter tuning for the widest models

We repeated the full Conv4 width sweep (0.25x–32x) under maximal update parametrization (Yang et al., 2022) with a verified coordinate check (Appendix A.5, Fig. 7). The gap narrows with width as before, showing that the narrowing is not caused by the parameterization.

  1. Compute and equal-budget comparison

We now report training FLOPs and peak memory for every model (Appendix A.7, Tables 9-10). As the reviewer anticipated, greedy training is substantially cheaper: at the same width it uses 4.0x (Conv4) and 4.8x (Conv8) fewer training FLOPs, and about half the peak memory at the widest widths. At equal training compute, greedy is more accurate at Conv4 32x (78.2% vs. 77.3%) and matches end-to-end at 16x.

We would be grateful to know whether these revisions address your concerns, and, if so, whether they change your assessment. Thank you again for your time and constructive review.

Reviewer_Ry51OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper central object of study is the gap between layer-wise and end-to-end gradient descent. They authors ask the question of if and when the simpler layer-wise procedure can achieve similar or better performances than the most common end-to-end GD. The layer-wise procedure is more motivated by the biological plausibility of layer-wise learning.

The authors show empirically that the gap closes (and sometimes becomes negative) in the regime of wide networks. The authors additionally study the representational geometry of the different models finding different behaviours between the two types of training.

Critical strengths

I enjoyed reading this paper. I think the question is interesting and the setting is quite original.

I also appreciated how carefully the authors motivate most of their experimental choices. The comparisons are generally well controlled, the methods are clearly described.

Overall, I think this is a good empirical contribution. Even without a complete theoretical explanation, the observed phenomenon is interesting and deserves further investigation.

Critical weaknesses

The main concern I have is the parametrization and optimization of the networks as width increases.

The paper considers a very large range of widths while using essentially the same optimization setup. In particular, the authors use Adam with a fixed learning rate, selected using relatively narrow networks, and keep the number of training epochs fixed.

There is an extensive theoretical literature on scaling networks and how to optimize them, notably the maximal update parametrization introduced by Yang et al., which provides a principled way of scaling initialization and learning rates with width while preserving nontrivial feature learning.

As the authors compare networks with different widths under a fixed training budget, I wonder whether part of the observed advantage could be attributed to the networks are not being trained in an appropriate way.

This is the main limitation. Ideally, I would like to see whether the central result remains true under muP.

The proposed explanation for the results is interesting but not totally convincing

In Section 4.2, the authors suggest that the difference between greedy and end-to-end training could come from the fact that minimizing the self-supervised objective is not equivalent to maximizing the classification performance. I find this explanation plausible, and Figure 3 provides some evidence in this direction. However, I do not think the experiments are sufficient.

The representational geometry analysis is also interesting, even if I would have liked to see a more systematic test of the mechanism: maybe across different seeds.

Finally, although the appendix is quite detailed, if I am not mistaken the code is not available.

The connection to the recent theoretical literature on layer-wise can be improved.

I think the paper would benefit from a more substantial connection to recent theoretical work on how neural networks build representations across layers.

In particular, I would encourage the authors to discuss Dandi et al. (2026), "Deep Learning as Neural Low-Degree Filtering: A Spectral Theory of Hierarchical Feature Learning". Dandi et al. propose a theoretical framework motivated by the early dynamics of gradient descent where they show how the features and (most of the) performances of greedy training can be recovered from a layer-wise training. Another relevant work is Daniely (2026), "Deep Networks Learn Deep Hierarchical Models", which provides theoretical guarantees for learning hierarchical structures using layer-wise SGD.

Questions for authors
  • Have the authors considered using maximal update parametrization when increasing network width? Even a limited experiment at a few widths would strengthen my confidence in the conclusions.

  • Section 4.2 and representational geometry. Could the authors provide more evidence supporting the hypothesis that the mismatch between the self-supervised loss and downstream classification accuracy explains the advantage of greedy training? Is it possible to see this in a toy example.

  • Would it be possible to provide an anonymous version of the code?

  • In Section 3.3, where signal–signal and signal–noise factorization are introduced, I think explaining a simple example that shows how different representation geometries affect eqs (1) and (2) would be helpful.

2 comments
AuthorsOpenReview ↗

Response to reviewer Ry51

We thank the reviewer for the positive and constructive review, and for identifying the parameterization question, which we agree was an important gap. We address all weaknesses and questions below.

Weakness 1: parameterization

Following the reviewer’s suggestion, we repeated the full Conv4 width sweep (0.25x–32x, end-to-end and greedy, seed 0) under the maximal update parametrization (μP; Yang et al., 2022), using the authors’ mup package with 1x as the base width (Appendix A.5, Fig. 7). We used a coordinate check to confirm our implementation, finding that after four training steps, the change in each hidden layer’s output grows 26–34x from 1x to 32x width under the standard parametrization, but stays relatively constant (1.05–1.14x) under μP. Our results hold under μP: the gap between greedy and end-to-end training narrows with width. Under the kNN probe, the gap drops from -4.4 points at 1x to +3.0 points at 32x.

To check whether our learning rate, which was selected on narrow networks, disadvantages the widest end-to-end networks, we also trained both methods with a higher learning rate of 10^-2.5 at 1x and 32x (not reported in the paper). This improved both methods at 32x, and greedy training remained ahead under the kNN probe (78.3% vs. 76.4%).

On the fixed epoch budget: wide end-to-end training peaks around epoch 300 and then declines, so increasing the epoch count wouldn’t improve performance. Further, end-to-end’s best epoch selected on the test set is below greedy layer-wise training’s final accuracy (77.6% vs. 78.2%).

Weakness 2: minimizing the self-supervised objective is not equivalent to maximizing the classification performance

We wanted to begin by clarifying that our claim in Section 4.2 is narrower than the idea that the mismatch between the objective and classification performance explains why greedy layer-wise training performs better. Instead, we simply argue that a lower Barlow Twins loss does not imply higher accuracy, which makes it possible for greedy training to outperform end-to-end training despite reaching a higher loss. We propose representational geometry as a potential explanation. We provide a few pieces of evidence for these claims:

  1. Within a single run, from epoch 300 to 600, Conv4 32x end-to-end training lowers its loss from 23.5 to 16.3 while its kNN accuracy falls from 77.2% to 75.9%.
  2. Previous preliminary experiments with Conv8 greedy layer-wise training with 250 epochs per layer, rather than 125, (not reported in the paper) reached a much lower objective loss but had similar or even lower kNN accuracy.
  3. See the toy example in Q2 (below)

Regarding geometry across seeds, our final-layer SSF and SNF differences in Fig. 4a and 4b report the means over three seeds. We also perform new freezing controls (see general response to all reviewers and Appendix A.6). We find that the gains in representational geometry (SNF and SSF) appear only when freezing and local objectives are both present. We additionally run the analysis with the 16x width for the freezing control across 3 seeds for robustness; the result holds for all three seeds.

Weakness 3: connection to recent theoretical literature

We have revised the related works section to improve our connection to recent theoretical literature. We included references to both Dandi et al. (2026) and Daniely (2026). We thank the reviewer for bringing these papers to our attention.

Questions:

  1. Yes, we implemented maximal update parametrization and found that our results hold (see above).
  2. We clarify our claim above. Still, consider a toy example where there are inputs with three features: a class feature, a nuisance feature unrelated to the class, and a feature that augmentation changes. Augmentation leaves the class and nuisance features unchanged, so any linear encoder that ignores the third feature reaches the minimum Barlow Twins loss of zero, whether it uses the class feature or the nuisance feature. An encoder using the class feature will reach near 100% accuracy while an encoder using the nuisance feature will reach significantly lower accuracy (chance). Which solution is found therefore depends on how the network is trained, not on the loss.
  3. Yes, we will post an anonymized repository as a comment on this page before the end of the discussion period.
  4. We appreciate this point. We have included a toy example (Appendix A.8) with two-dimensional Gaussian classes with SSF and SNF calculations.

We thank the reviewer again for the constructive suggestions, which substantially strengthened the paper.

AuthorsOpenReview ↗

Follow up to reviewer Ry51

As promised, we have included an anonymized code repository that can be found here: https://anonymous.4open.science/r/width-greedy-ssl-D847/

We thank the reviewer again for their constructive and positive review.

Reviewer_FTTnOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper investigates the effects of increasing width of layers on greedy layer-wise training and end-to-end training. The analysis involves Conv4 and Conv8 architectures trained using the Barlow Twins and SimCLR objective on CIFAR-10 and CIFAR-100 across a range of widths - 0.25x to 32x. Their results show that greedy layer-wise training and end-to-end training have matching performance at 16x width with kNN accuracy gap between the 2 decreasing with increasing width. The paper also shows that at 32x on Conv4, greedy layer-wise training has slightly higher kNN accuracy than end-to-end training, although this is partly because of deteriorating performance of end-to-end training after about 200 epochs. To investigate this behavior with respect to width, the authors analyze signal–signal factorization (SSF) and signal–noise factorization (SNF) of the models and found that wider greedy-trained networks exhibit a more favourable class-based geometry. Based on these observations, the authors suggest that increasing network width can "compensate for restricted credit assignment" and enable locally trained networks to "learn representations of comparable quality to those learned through end-to-end backpropagation".

Critical strengths

Interesting and relevant research question: The research question addressed in this work - can greedy layer-wise learning approach the performance of end-to-end (E2E) backpropagation? is very relevant. The paper explores this across a range of widths on Conv4 and Conv8 models and reveals the trend that while both greedy layer-wise learning and end-to-end backpropagation improves with increasing width, the performance gap reduces with greedy improving better as models grow wider. This observation is particularly relevant for training setups where end-to-end gradient propagation is undesirable or impractical.

Critical weaknesses
  1. Limited experimental scope and insufficient evidence of generalizability: The experiments are restricted to CIFAR-10 and CIFAR-100 using custom Conv4 and Conv8 architectures trained using Barlow Twins and SimCLR objectives. While the observed trend is interesting, these settings are insufficient to support the paper's broader conclusions. Furthermore, representation quality is primarily measured via kNN accuracy and SSF and SNF are computed using class identity information. While these represent class-discrimination ability, they do not establish general representation quality or information preservation across layers. Additional architectures, datasets, and evaluations, such as linear probing and layer-wise information-retention analyses would be required to back the broader claims.

  2. Insufficient isolation: The paper claims that increasing width of greedy trained models compensates for restricted credit assignment. However, greedy training involves sequential freezing and intermediate SSL objectives and sequential layer freezing can itself improve generalization due to its regularizing effects. Without isolated controls such as end-to-end training with progressive freezing, it is difficult to conclude that the improvements are from increasing width and not regularization. This is crucial as it is the main claim of the paper.

Questions for authors
  1. While greedy matched E2E best performance on Barlow Twins trained conv4 (2a and 5b), it does not in any other results. However the trend of reducing gap exist. Does greedy match E2E best performance on conv8 models eventually beyond 32x? If yes, is there a relationship between depth and width?

  2. Could the authors provide more evidence that increasing width reduces information loss across successive greedy stages? Maybe via layer-wise information-retention or representation-stability analyses or linear probing?

  3. How would the methods perform under matched memory or compute budgets? Since local learning is motivated by reduced memory and gradient computation, can the authors report the peak memory and total flops to peak accuracy for both methods?

  4. Would this behaviour with increasing width persist on residual networks where gradient propagation is easier?

  5. Conv4 greedy training optimizes each layer for 250 steps and conv8 for 125. Could this optimization time per layer be the reason for conv4 matching E2E as entioned in question 1?

4 comments
AuthorsOpenReview ↗

Response to reviewer FTTn (Part 1/2)

We thank the reviewer for the detailed review and for the suggestions for specific control experiments. Below we have responded to each critical weakness.

Weakness 1: Limited experimental scope & generalizability

We agree that our experiments cover relatively shallow convolutional networks on CIFAR, and our claims are limited to this setting. We want to emphasize our goal of providing a scientific description and exploration of a novel and unexpected phenomenon. We do not claim broad generalizability to all architectures and describe this as an important direction for future work.

Further, the paper's contribution is a controlled observation sweeping width over a 128-fold range. This is computationally demanding, justifying our use of shallow convolutional neural networks. As mentioned in the general response to all reviewers, our preliminary results using blockwise ResNet-18 networks demonstrated a similar narrowing of the gap between end-to-end and greedy layer-wise training, suggesting that the results may generalize to other architectures, which we hope future work explores.

In regards to evaluation, we added both requested analyses:

  • We train linear probes using the standard protocol of Kornblith et al. (2019) (Appendix A.4). We find that our results hold under a linear probe as well.
  • Layer-wise information retention (see our response to Question 2).

Weakness 2: Insufficient isolation

We agree that our main claim would be more strongly supported with isolated control experiments. As such, we ran the suggested end-to-end training with progressive freezing control as well as a simultaneous local training control (Section 4.4 and Appendix A.6). In the progressive freezing control, each progressive layer is frozen after 250 epochs, matching the greedy layer-wise schedule. In the simultaneous local training control, the network uses the same local losses as greedy layer-wise training but with stop-gradients and no sequential freezing. Neither freezing nor local objectives alone fully reproduce greedy training's width trend: progressive freezing performs like ordinary end-to-end training at every width, and simultaneous local training recovers only about half of greedy training's gain from width. In addition, both freezing and local objectives are needed for gains in SSF and SNF. We ran both controls with one seed at every width, and repeated them with two more seeds at 16x for robustness. This shows that the width effect is not explained by freezing acting as a regularizer.

AuthorsOpenReview ↗

Response to reviewer FTTn (Part 2/2)

Questions

Below, we address each question from the reviewer.

Question 1: Conv8 beyond 32x & depth vs. width relationship

Our Conv8 network was trained up to 16x, and at 16x greedy already reaches end-to-end: it matches under kNN (−0.23 ± 0.25) and exceeds it under the newly added linear probe (+1.00 ± 0.79, greedy ahead in all three seeds). So, greedy overtakes end-to-end at both depths at the widest width, Conv4 under kNN (+2.5 ± 0.4 at 32x) and Conv8 under the linear probe. Conv8 end-to-end at 32x would not fit on our 40 GB GPUs: its training memory roughly doubles with each doubling of width (8.9 to 20.6 GiB from 8x to 16x). Thus, we are not able to determine if even wider Conv8 greedy layer-wise trained networks could match the best performance of wider end-to-end Conv8 networks.

On depth: the deeper network starts much further behind but closes the gap faster, reaching parity at a similar width (16x). Unfortunately, we are unable to establish a general depth–width law with only two depths.

Question 2: Does width reduce information loss across greedy stages?

We are unable to isolate a specific mechanism for why width helps greedy layer-wise training in this way; we initially hypothesized reduced information loss. As suggested, we ran two analyses:

  1. To study class information, we trained linear probes at every layer on all three seeds (Appendix A.4, Table 8). In Conv4, greedy training does not lose class information across stages at any width: linear-probe accuracy rises at every stage, at every width and in every seed.
  2. To study all information (label-free retention), we measured how much of a layer's features can be recovered linearly from the next layer (ridge regression, held-out R²; Lenc & Vedaldi, 2015). Beyond the first layer, wider networks actually keep less of each layer's features in the next layer. For example, layer 3 recovered from layer 4 had R² 0.93 at 1x vs. 0.72 at 32x. End-to-end training showed the same trend (0.87 vs. 0.66).

This implies that the mechanism for width helping greedy layer-wise training may not be reduced information loss. Still, because this measure only captures linearly recoverable information, this remains an open question, as wider layers could simply transform their inputs rather than discard them. We hope future work further explores this question.

Question 3: Matched memory and compute

We measured training FLOPs (forward and backward, per step) and training-only peak memory for every configuration (Appendix A.7, Tables 9–10).

  • FLOPs: Greedy layer-wise training needs 4.0x fewer training FLOPs than end-to-end training at the same width for Conv4, and 4.8x fewer for Conv8.
  • FLOPs to peak (Conv4 32x): End-to-end training peaks around epoch 300. Reaching its best test-selected epoch (77.6%) takes roughly the same compute as greedy's entire run (mean 1.1x), and greedy layer-wise training ends higher (78.2%).
  • Equal compute: At equal FLOPs, greedy leads at Conv4 32x (78.2% vs. 77.3%), ties at 16x, and trails at smaller widths and on Conv8.
  • Memory: At the widest widths, greedy needs about half of end-to-end's training memory (Conv4 32x: 13.6 vs. 23.6 GiB, 57%; Conv8 16x: 9.2 vs. 20.6 GiB, 45%); at narrower widths the saving is smaller (up to 79% of end-to-end's memory, at Conv4 0.25x).

These results also raise an interesting point: a greedy network costs about the same training compute as an end-to-end network of half its width, and at Conv4 32x it is more accurate than end-to-end training at 16x for the same compute (78.2% vs. 77.2% kNN), motivating further exploration of wide greedy layer-wise networks.

Question 4: Residual networks

Our preliminary results suggest that our effect may persist in residual networks despite residual connections easing gradient propagation in end-to-end training. A ResNet-18 trained greedily block by block (its four residual stages), we saw a similar pattern to our observed result: under a linear probe, the gap decreased from 11.4 points at 0.25x width to 2.2 points at 4x width, as end-to-end training plateaued after 2x width while greedy layer-wise training continued to improve at 4x width. We agree that deeper and more complex networks are an important direction for future work, but this paper focuses on a controlled study of the width effect, which requires sweeping width over a 128-fold range with everything else fixed.

Question 5: Conv8 optimization time

Previous experiments with Conv8 networks with 250 epochs per layer (for a total budget of 2,000 epochs) found that longer per-layer training did not improve Conv8 performance at the widest widths, despite improving the Barlow Twins loss. Per-layer training time therefore does not explain why Conv4 reaches parity.

We apologize for the lengthy response and thank the reviewer for the feedback; the isolation experiment suggestions especially were genuinely well appreciated by the authors.

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my concerns.

W1. Experimental scope: The additional linear probing results strengthen the findings, and the preliminary ResNet-18 experiments are encouraging and I encourage including it in the manuscript. However, the main conclusions remain supported primarily by shallow and custom convolutional architectures on small 32x32 CIFAR datasets. I believe broader architectural validation within CNNs family (say VGG, MobileNet, EfficientNet etc) and data validation (224x224 with more classes as ImageNet used in Siddiqui et al., 2024 etc) is important for establishing the generality of the observed phenomenon even if it is limited to CNNs and classification datasets.

W2. Isolation of the width effect: The additional width sweeps with progressive freezing and simultaneous local training directly address my original concern. These controls provide convincing evidence that the observed width-dependent improvements cannot be explained by freezing-induced regularization alone. I consider this concern addressed.

Regarding my questions: I appreciate the additional analyses of layer-wise information retention, computational efficiency, and optimization time. The finding that wider networks do not necessarily preserve more linearly recoverable information is particularly interesting, but it also leaves the underlying mechanism unexplained. The preliminary ResNet-18 results are promising, although further experiments are needed to determine whether the phenomenon generalizes to deeper architectures.

Overall, the rebuttal strengthens the empirical evidence and addresses several of my original concerns. I am willing to increase my score to 2 (weak reject) but I lean reject primarily due to the limited architectural and evaluation scope and the broader conclusions (CNNs in general) relative to the available evidence (limited to custom CNNs upto 8 layers on CIFAR only).

AuthorsOpenReview ↗

Response to comment by Reviewer FTTn

We thank the reviewer for the follow-up response and for raising their score. We are happy to see that the freezing controls addressed the concern about isolating the width effect.

Following the suggestion by the reviewer, we have included the preliminary ResNet-18 results to the appendix (Appendix A.9, Table 11). These preliminary results indicate that our findings may generalize to more complex architectures. We agree that our evidence is limited to relatively shallow convolutional networks on CIFAR. To reflect this, we have included a more explicit statement in our discussion that we have not tested whether the width effect holds for other architecture families (such as VGG-style networks or efficient convolutional networks or for larger, higher-resolution datasets such as ImageNet). We further that establishing generalization of our findings to these networks and datasets is an important next step.

We thank the reviewer again for the constructive feedback, which substantially strengthened the paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all four reviewers for careful and constructive reviews. Several concerns were shared, so we summarize the new experiments here and answer each reviewer individually below. An updated PDF of our paper has been attached, with new textual additions in blue.

1. Experimental scope

Multiple reviewers shared concerns regarding the limited experimental scope of our experiments, namely our reliance on relatively shallow convolutional neural networks (Conv4 and Conv8). Our study is a scientific exploration of a novel and unexpected phenomenon: with the training procedure held fixed, widening the network closes the gap between greedy layer-wise and end-to-end self-supervised training. To study this, we sweep width over a 128-fold range (from 0.25× to 32×), which is computationally demanding; this is why we use shallow convolutional networks. We argue that our results suggest that width is an important and overlooked feature of neural network architecture that should be further explored, especially in the context of local learning. We do not necessarily claim that this result will hold on all models.

In preliminary experiments with ResNet-18 networks trained blockwise greedy (similar to Siddiqui et al., 2024), we found that width similarly narrowed the gap between greedy layer-wise and end-to-end training. Specifically, under a linear probe, the gap decreased from 11.4 points at 0.25× width to 2.2 points at 4× width, where end-to-end plateaued after 2× width while greedy layer-wise training continued to improve at 4× width. These preliminary experiments (not reported in the paper) suggest that the width effect may persist in deeper networks. Our discussion section notes this limitation and direction for future work.

2. Parameterization of the widest networks

Multiple reviewers suggested that the width effect may, at least in part, be due to our parameter tuning being done on 1× and 8× width networks instead of being explicitly tuned for the widest networks. To address this, we take the suggestion from Reviewer Ry51 to implement maximal update parametrization (μP; Yang et al., 2022). We perform a full width sweep (0.25×–32×) on our Conv4 network along with a coordinate check (Appendix A.5). Our result holds: the gap between greedy layer-wise training and end-to-end training narrows with width, and greedy even maintains its performance advantage in kNN evaluation in the widest networks.

3. Linear probes

We trained linear probes for the primary Conv4 and Conv8 results using the standard fixed-feature protocol of Kornblith et al. (2019) (Appendix A.4). The narrowing gap holds: under linear probes, greedy Conv4 at 32× comes within 0.5 points of end-to-end training rather than surpassing it, while greedy Conv8 at 16× surpasses end-to-end training in all three seeds.

4. Additional controls (end-to-end training with progressive freezing & simultaneous local training)

To isolate what drives this effect, we added two controls at every width (Section 4.4 and Appendix A.6). First, we train end-to-end networks with progressive layer freezing (consecutive layers frozen after 250 epochs each). These networks perform similarly to ordinary end-to-end training, and we thus conclude that freezing alone does not reproduce greedy's width trend. Simultaneous local training, which uses greedy training's layer-wise losses with stop-gradients between layers but trains all layers together (Siddiqui et al., 2024), similarly does not fully reproduce greedy's width trend. We find that a combination of both freezing and local objectives is required to see our observed effect. Likewise, the gains in representational geometry (SNF and SSF) appear only when both are present. We confirmed these conclusions with three seeds at 16× width.

5. Compute and memory

We measured training FLOPs and memory for every model and added FLOP-matched comparisons (Appendix A.7). At the same width, greedy layer-wise training uses 4.0× (Conv4) and 4.8× (Conv8) fewer training FLOPs than end-to-end training and, at the widest widths, about half the training memory. At equal training compute, greedy layer-wise training is more accurate than end-to-end training for Conv4 at 32× (78.2% vs. 77.3%) and matches it at 16×.

We hope that our responses meaningfully address the concerns of the reviewers. We thank you all deeply for your detailed attention to our manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.