- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how network width affects greedy layer-wise self-supervised learning relative to end-to-end backpropagation. Using Conv4/Conv8 models with Barlow Twins and SimCLR, the authors show that increasing width disproportionately benefits greedy training and can largely close, or occasionally reverse, the performance gap. They further relate this behavior to differences in representation geometry measured by SSF and SNF.
Critical strengths
- The paper presents a clean and systematic empirical study of the interaction between network width and greedy layer-wise training.
- The representation-geometry analysis provides a useful perspective beyond downstream accuracy alone.
Critical weaknesses
- The conceptual novelty is limited. Prior work has already shown that local or greedy learning can approach end-to-end training with sufficiently expressive architectures or modules. The paper mainly adds an empirical width‑scaling observation, without a new theoretical or causal explanation for why width mitigates restricted credit assignment.
- The empirical scope is narrow. The main results are limited to shallow Conv4/Conv8 models on CIFAR‑10/100, and the additional SimCLR/CIFAR‑100 experiments use a single seed. Moreover, the strongest result relies on extremely wide networks, making it unclear whether the effect remains meaningful under realistic model or compute budgets.
Questions for authors
- Does the main result hold for modern architectures such as ResNets and on larger-scale datasets?
- Can the authors provide parameter/FLOP/memory-matched comparisons between greedy and end-to-end training?
- Does the reported crossover remain when checkpoints are selected using validation performance and E2E optimization is tuned specifically for the widest models?
- Does the trend persist with linear probing or transfer evaluation, rather than only kNN?
- Can the authors provide stronger evidence that SSF/SNF is explanatory rather than simply correlated with width and downstream performance?
2 comments
Response to reviewer 3hQm (Part 2/2)
Questions:
Below, we address each question.
1. Modern architectures and datasets
Our preliminary experiments using ResNet-18s trained blockwise greedy showed a similar narrowing trend with width. This suggests that our result may generalize to other architectures (see general response to all reviewers). Our largest tested dataset is CIFAR-100, where we see a similar trend as well.
2. Compute comparisons
We now report measured training FLOPs and memory for every model (Appendix A.7, Tables 9-10). At the same width, greedy training uses 4.0x (Conv4) and 4.8x (Conv8) fewer training FLOPs than end-to-end training and, at the widest widths, about half the training memory. At equal training FLOPs, greedy training is more accurate than end-to-end training at Conv4 32x (78.2% vs. 77.3%) and matches it at 16x (Table 10). Here, "equal training FLOPs" compares greedy layer-wise training’s final accuracy with end-to-end training’s accuracy at the point in its training where it has used the same total FLOPs as the full greedy run (about epoch 250 for Conv4). Alternatively, measured at end-to-end training’s peak: reaching its best epoch (77.6% at Conv4 32x) takes roughly the same compute as greedy training’s entire run (1.1x on average), and greedy training still ends higher (78.2%).
3. Checkpoint selection & end-to-end optimization
Though we did not save checkpoints, we provide best epoch on the test set, which is an upper bound (at least as favourable to end-to-end training as any validation-based choice). This gives 77.6% at Conv4 32x, still below greedy layer-wise training’s final 78.2% (three seeds). In regards to hyperparameter tuning, under maximal update parameterization (Appendix A.5), the gap narrows as before (see general response to all reviewers) and the crossover survives (kNN).
4. Linear probing
Yes, we implement linear probing and see that the trend still persists (see general response to all reviewers). Under linear probes, greedy Conv4 at 32x comes within 0.5 points of end-to-end training rather than surpassing it, while greedy Conv8 at 16x surpasses end-to-end training in all three seeds.
5. SSF/SNF causality
We read this as two questions. First, whether the geometry change is specific to greedy training. Second, whether geometry plays a causal role in downstream accuracy.
To the first question, our new control experiments (see general response to all reviewers) isolate variables within the training procedure at fixed widths (Section 4.4, Appendix A.6). Neither freezing alone (end-to-end training with progressive freezing) nor local objectives alone (simultaneous local training) at the widest widths raises SNF or SSF to the same degree as greedy layer-wise training. Only through the combination of both, which occurs in greedy layer-wise training, do we see improvements in representational geometry. This fits our previous results showing that greedy training builds SNF and SSF stage by stage, while in end-to-end training both peak early and then decline (Fig. 4c,d). The geometry difference is therefore produced by the greedy procedure, not by width alone.
To the second question, because we do not manipulate geometry directly, we are unable to establish an explicit causality. However, recent theory establishes that SSF and SNF govern classification error (Wakhloo et al., 2026). Further, recent work has shown that directly increasing SNF with a regularizer improves accuracy (Kirti & Zylberberg, 2026).
We thank the reviewer again for the constructive review.
Response to reviewer 3hQm (Part 1/2)
We thank the reviewer for the in-depth review and comments. Below we address each weakness.
Weakness 1: limited conceptual novelty
We agree that local learning approaching end-to-end training is not new. Our contribution is a controlled study showing that, with the training procedure held fixed, widening the network shrinks the gap between greedy layer-wise and end-to-end training. Though we don’t provide a theoretical or causal explanation, our study is a scientific exploration of a novel and unexpected phenomenon and thus deserves further investigation.
We do, however, propose a potentially causal variable in improved representational geometry. Our additional control experiments (see general response to all reviewers) show that both local learning and freezing are required for improved representational geometry and the increase in accuracy.
Further, our revised Related Work connects our results to recent theory showing that strictly layer-wise training can learn useful hierarchical features (Daniely, 2026; Dandi et al., 2026).
Weakness 2: narrow empirical scope
We agree that the scope is narrow. However, our goal is not to make a broad claim regarding improvement for all networks and architectures. As discussed in the general response, our goal is a controlled, scientific exploration and study of a novel width effect. We simply identify width as an under-explored variable in local learning. And, as mentioned in the general response, preliminary experiments with ResNet-18 networks trained blockwise greedily saw a similar narrowing of the gap between end-to-end and greedy layer-wise training as width increased. This suggests that the results may generalize to other architectures, which we hope future work explores.
Though the additional SimCLR and CIFAR-100 results use a single seed, both show the same trend across all eight widths, consistent with our three-seed CIFAR-10 results, whose seed-to-seed standard deviation at widths of 8x and above is at most 0.6 points, much smaller than the effects we report.
Finally, we would argue that these networks do train under realistic model and compute budgets. Specifically, extremely wide is extreme as a multiplier, not in absolute size or cost. For example, greedy layer-wise training of the Conv4 32x width network (our largest) took about 8 hours on an A100 GPU. Further, although the crossover in performance happens at the widest networks, much of the narrowing itself happens at the smaller networks. An example of this is the Conv4 at 8x width, which has ~25 million parameters, less than a ResNet-50, which narrowed the gap from 4.3 points at 1x width to 1.9 points at 8x width (kNN). Conv8 at 4x width has about 19 million parameters and narrows the gap with end-to-end training from 12.1 points at 1x to 4.2 points (kNN).