acceptodds
Under review as a conference paper at ICLR 2027

ResNets Are Deeper Than You Think

Abstract

The empirical success of residual connections is remarkable: nearly a decade after their introduction, they remain ubiquitous in modern neural network architectures. Their widespread adoption is usually attributed to improved trainability: residual networks train faster, more stably, and often achieve higher accuracy than their feedforward counterparts. While our experiments confirm the improved trainability of residual networks, we argue that this explanation is incomplete. We propose a complementary view: by combining shallow and deep paths in the computation graph of the network, residual connections can confer generalization benefits beyond pure optimization. Testing this hypothesis is surprisingly difficult: Simple side-by-side comparisons of residual and feedforward networks confound architectural prior with trainability, and prior attempts to close the gap do not isolate these effects conclusively. We therefore introduce a post-training partial-linearization setup that molds an already trained network into either a fixed-depth or a variable-depth architecture. In this setting, ordinary trainability differences are unlikely to explain the result. Across our experiments, variable-depth architectures consistently outperform fixed-depth counterparts at comparable nonlinear complexity, supporting the view that residual connections encode an inductive bias beyond numerical trainability.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
residual networks, inductive bias, generalization, neural network architectures, nonlinear depth, partial linearization, trainability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Fx3MOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether residual connections improve generalization through variable-depth structures, beyond their known benefits for trainability. The authors propose a post-training partial-linearization method to compare fixed-depth and variable-depth networks, and find that the latter better preserve accuracy at comparable nonlinear complexity.

Critical strengths
  1. The paper points out that direct comparisons between ResNets and feedforward networks cannot fully separate architectural effects from trainability.

  2. The paper shows that, under certain assumptions, residual blocks cannot be equivalently represented by plain feedforward blocks of the same width.

  3. The paper introduces a post-training partial-linearization method and finds that variable-depth networks preserve accuracy better than fixed-depth networks at comparable nonlinear complexity.

Critical weaknesses

1. The research question and experiments are not fully aligned. The paper aims to investigate whether variable-depth structures improve the generalization of ResNets beyond trainability. However, as the authors acknowledge in Section 5, their experiments actually “test a narrower question”: which structure better preserves accuracy when nonlinearities are removed from an already trained network. This does not directly explain why variable-depth structures lead to better generalization.

2. The experiments do not fully isolate the effect of variable depth. The authors use channel-wise and layer-wise linearization to construct variable-depth and fixed-depth networks, respectively. However, channel-wise linearization is inherently more flexible, as it can selectively preserve nonlinearities in important channels. Therefore, its better performance may come from this flexibility rather than variable depth itself. The current experiments do not clearly separate these two effects.

3. The effect of trainability is not fully ruled out. The authors argue that starting from the same pretrained model can “substantially reduce the trainability confound”. However, both methods still require gradient-based post-training, and they impose different optimization constraints. The paper does not provide enough analysis of this process or show that the two methods are equally easy to optimize. As a result, trainability may still contribute to the observed performance gap.

4. The practical implications are somewhat limited. The paper suggests that variable-depth structures provide a useful inductive bias, but it is unclear how this finding could guide the design of better networks. For example, what kind of path-length distribution works best? How should residual connections be designed under a fixed compute budget? Some discussion or experiments along these lines would make the findings more useful in practice.

Questions for authors

See weaknesses.

1 comment

We thank reviewer Fx3M for their thorough review and constructive feedback. Please allow us to address the open questions raised in the review.

The research question and experiments are not fully aligned.


Answering the larger research question whether variable-depth structures improve the generalization capabilities of deep nonlinear networks beyond trainability is empirically very difficult. As the paper highlighted, plain side-by-side training comparisons are difficult to interpret, even if direct fixes to trainability issues are applied. To our best knowledge, there are no papers in past literature that are able to conclusively answer this question.

The fact that the gap between non-residual and residual architectures still stands (ref. [1] Table 9) even when using advanced methods to solve all numerical issues we described in Section 2.1 however is highly curious, and we believe that this has not been sufficiently be discussed under the point of view we introduce: is it possible that this gap between fixed and variable-depth architectures can never be closed, even given perfect numerics during optimization? We believe that cleanly introducing this research question that had previously suffered a lot of confound in literature, is a major contribution of our paper.

As directly sampling the global minima for a given architecture is numerically infeasible, testing how well a learnt solution can be preserved in a either shape, fixed or variable-depth, appears to be a meaningful proxy for the above research question. The weaknesses (ref. Question 1-3) that the reviewer described are indeed valid limitations of our experimental setup, as discussed in our paper. The empirical evidence we provided alone does not conclusively answer the research question raised, and we agree to discuss this limitation further in the final version.

We believe however, that beyond these limitations, our experiment delivers a meaningful epistemic contribution, as our post-training setup suffers from fundamentally different experimental strengths and weaknesses wrt. direct comparisons [1]. Our setup mostly ignores per design the differences in trainability (see below), as both arms are post-trained from the same trained base network; however, our setup cannot guarantee that we would obtain the same result when training the networks side-by-side from scratch. Our empirical evidence pointing in the same direction as past literature, whilst using a very different empirical approach is meaningful and novel evidence indicating that the performance gap mentioned above might never be closed.

The experiments do not fully isolate the effect of variable depth


The effect of trainability is not fully ruled out

The cited concerns about our experimental setup are valid, but can largely be ruled out by our control experiment (ref. Figure 3): the fact that the gap mostly vanishes when starting from a ResNet clearly suggests that channel-wise linearization being more flexible or the effect of trainability are not the cause for the observed gap (ref. Figure 3 rhs.), as these factors are equally present in the control experiment.

The practical implications are somewhat limited

We agree that the practical implications are not well-explored in our paper. We did run preliminary experiments exploring the optimal residual block depth, and found that the empirical optimum was close to the widely established 2-3 layers. Finding a novel prior on path lengths that out-performs current SOTA benchmarks which have been optimized for many years using the trial-and-error principle is a significant extension of our work, which is why we decided against including practical implications in the scope of this particular paper. However, we are intending to conduct follow-up work in this direction.

[1] Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping; Martens et al.; 2021

Reviewer_hwx3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors argue that while the optimization benefits of residual connections are well studied, their effect on generalization is not. They first show theoretically that residual blocks are not simply a reparametrization of feedforward blocks, specifically showing that with square weights and non-injective activations, a residual block cannot be represented by a feedforward block of the same width and depth. Their central hypothesis is that ResNets benefit from being variable-depth architectures, composed of paths with different numbers of nonlinearities, whereas plain feedforward networks are fixed-depth. To limit trainability confounds, they linearize already trained networks during a post-training phase, either channel-wise (which can yield shallow but variable-depth networks) or layer-wise (which yields shallow fixed-depth networks) and compare the results at matched normalized average path length (NAPL). Starting from a pretrained non-res RepVGG-A2, channel-wise linearization retains accuracy better than layer-wise, with the gap growing as nonlinear capacity decreases, i.e. at higher linearization. In a control with ResNet56 with and without skip connections, the gap seems to be more modest for the residual network, which the authors take as evidence that the advantage comes from the variable-depth structure. Overall, they show that variable-depth linearization keeps accuracy closer to the full model and its more resilient that layer-wise.

Critical strengths
  • The question asked, i.e. whether residual connections help generalization beyond trainability is an important and under-explored question, especially since residual connections still dominate modern architectures, while recent work has started to question some of their advantages.

  • Useful framing: Viewing residual vs. plain networks as variable- vs. fixed-depth architectures, in the spirit of Veit et al. (2016), shifts attention from “skip vs. no skip” to how nonlinear depth is distributed, a perspective that could prove productive beyond this paper.

  • Useful theoretical observation: The paper formalizes the ambiguity in the ResNet original paper's “reformulation” and shows that equal-width residual and plain feedforward blocks span different function classes, with explicit (non-equivalent) reparametrization constructions.

Critical weaknesses
  1. Writing: NAPL, the quantity underlying all main results, is not formally defined and the reader is referred to prior work. Similarly, “nonlinear depth” is used throughout without a concrete definition.
  2. Evaluation is very weak: only one-model + one from scratch training with post hoc linearization. The breadth and rigor of the evaluation seem below what is expected at ICLR.
  3. The theoretical argument seems a bit disconnected from the rest of the analysis (experiments)
  4. Presentation of the results: they are shown only as scatter plots, with no tables. Since the two methods’ runs fall at different NAPL values, gaps at matched capacity can only be estimated visually. Tables with per-run accuracy, NAPL, proportion of disabled units, and mean slope would make the effect sizes easier to read, which could also strengthen the authors’ case.
  5. On experiments and methodology:
  • Choice of "depth" metric: All main results use NAPL, which is not a common metric and its use its not clearly justified. Specifically, NAPL is binary, meaning that any unit with |α − 1| ≥ 0.01 counts as fully nonlinear, however close to linear it is. This matters in practice as the results show: in Fig. 1 (left), layer-wise runs at the same NAPL (e.g. NAPL=4) differ by about 10 acc. points. Also, as a single value summary of a distribution, NAPL also hides differences in maximum depth, depth variance, and active-unit count. Based on the rest of the metrics used only on Appendix, it is clear that the effect size depends strongly on the metric: on ImageNet the NAPL gap is modest (about 2–5 points down to NAPL 6, from a full model NAPL of about 22) and grows large only under severe linearization, while under avg slope (Fig. 6) it shrinks to about 1–2 points. Under the proportion of disabled ReLUs (Fig. 8), which to me seems more fair, the two methods are within about 1.5 points up to about 70% disabled and diverge only beyond about 80%. The claimed advantage of variable depth thus appears mainly once most of the network’s nonlinearity has been removed. Also, the term “nonlinear depth” is never defined, and the experiments do not seem to hold it fixed in an intuitive sense.
  • On the ResNet control experiment: In Fig. 3, the gap for the residual network at NAPL 1–2 is about 6–9 points, as large as or larger than the plain full model’s gap at low NAPL. It’s dismissed as a “quantization artifact,” but the same argument would remove the low-NAPL region where the main effect is highlighted. The two models also differ a lot in quality (about 68.5% vs. 55%), which reintroduces the trainability confound that the authors themselves pointed out earlier.
  • CIFAR-100 results: Channel-wise networks with roughly 90–95% of ReLUs disabled (Fig. 7) outperform the unmodified full model ( approx. 56–58% vs. 54.6%), and accuracy peaks around NAPL 2–3 (Fig. 1, right). Meanwhile, layer-wise networks that retain 20–40% of their ReLUs, i.e. several fully nonlinear layers, perform worse than channel-wise networks with almost none. This is hard to reconcile with an expressivity-based explanation. It may rather be the case that the full model is weak (54.6% is low for CIFAR-100) and that the 60 epoch post-training acts as substantial additional training, where layer-wise linearization potentially disrupts the solution more.

All in all, as mentioned, the paper shows that fine-grained (channel-wise) linearization of a trained plain feedforward network degrades more gracefully than coarse-grained (layer-wise) linearization (with layer-wise linearization being a subset of channel-wise), mostly at high linearization regimes. Whether this reflects an inductive bias of variable-depth networks, rather than extra flexibility, post-training dynamics, or the choice of metric, is not clearly established. I also find the title misleading, if anything, the results are closer to the (shallow) ensemble view of ResNets (Veit et al.) than the “ResNets are deeper than you think”.

Questions for authors

See weaknesses.

1 comment

We would like to thank reviewer hwx3 for their in-depth feedback and thorough review of our work. We will try to address the individual feedback points and questions below.

Writing

The term APL is mathematically defined in [1] page 3, and NAPL merely adds a normalization factor, as otherwise the contribution of the lower layer vanishes (ref. [1] page 4). We agree with the reviewer that the definition of NAPL should be included in the paper for clarity, and we will make sure to add this to the final version.

Evaluation



In our paper, we compare 3 different architectures (standard ResNet with and without residual connections and RepVGG) on 3 different datasets (ImageNet, Cifar10, Cifar100). We argue that there is some diversity in the choice of of datasets and architectures, and that the differences measured are hard to explain by random chance.

Given the focus of our experiments on CNNs on computer vision tasks, we cannot exclude that this effect is limited to CNNs and does not apply to transformer architectures; we agree that this should be further discussed as a limitation. Having a limited GPU resources, we opted for experiments on CNNs and visual data only, attempting to robustly demonstrate the existence and significance of this effect in this specific domain.

Theoretical argument being disconnected from the experiments

The theoretical argument in our paper is rather obvious, and was meant as a motivation for the empirical work. We still believe that it is important to state that variable and fixed-depth networks span a different function space, as the distinction between benefits in trainability and function space prior have often been confounded in past literature (e.g. the original ResNet paper [3]). Also, the theoretical argument makes our empirical hypothesis (i.e. the existence of generalization benefits of variable-depth architectures beyond mere trainability) plausible in the first place.

Presentation of the results

We used scatter plots to present our results, as the final NAPL of the extracted network is not known until after post-training, and regularization strength differs for channel-wise and layer-wise regularization. It is therefore experimentally difficult to obtain a channel-wise and layer-wise regularized network with the exact same NAPL for a meaningful tabular comparison. However, we will make sure to provide tables of the results in the appendix of the final version.

On experiments and methodology

Choice of "depth" metric

Path length distribution is an established metric to measure the "depth" of ResNets (ref. [2]) and generally variable-depth architectures (ref. [1]) in past literature. It is still a heuristic, which is why we presented our results using different metrics in the Appendix. The fact that the gap is small but consistent across all metrics we used is in agreement with results in past literature (ref. [1] Table 9).

 Given the high amount of data points for each experiment and the consistent gap observed, the result being merely statistical noise is highly unlikely.

We would like to highlight that we used a sparsity regularizer (ref. Eq 7) as well as clipping for values above 1 in our experiments. αi\alpha_i values near 1 that are not fully linearized are highly unlikely in this setup.

On the ResNet control experiment

Intuitively, it makes sense that for a binary metric with only 4 free parameters left the resolution eventually drops. We argue that even when ignoring low-APL regions (NAPL 0-4) where this might be the case, the effect is clearly visible and statistically significant in Fig 1 and 3 for NAPL values between 4 and 9.

The performance gap between residual and non-residual architectures can inherently not be closed is a limitation of the setup, which we agree to stress more in the final revision. Still, it is hard to see how a different mechanism than inherently being variable depth would help the ResNet close the gap, given that the presence/absence of residual connections is the only difference in the ablation; we agree however that this is not a conclusive proof.

CIFAR-100 results:

We argue that these results are possible to reconcile with an expressivity-based explanation: when reducing the average depth of a network, two effects are possibly confounded: a reduced expressivity of the network and a smoothening of the loss surface [2] (in particular around the minimizer found). We have found in several additional experiments that it is possible for the the test accuracy to slightly rise with increasing linearization strength, before eventually dropping as a result of a reduced expressive. We do not believe that this effect is contradictory to our hypothesis.

[1] Nonlinear Advantage: Trained Networks Might Not Be As Complex as You Think; Ali Mehmeti-Göpel, Disselhoff; 2023

[2] Residual Networks Behave Like Ensembles of Relatively Shallow Networks; Veit et al.; 2016

Reviewer_uAjnOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses a fundamental questions for the ResNet architecture: Does the increased performance of ResNet comes only from the better training stability or are there beneficial architectural bias also lurking in the improved performance.

The authors answer positively to the latter: Theoretically, they show that the hypothesis space of a ResNet do not coincide with that of a standard MLP with skip connections, showing that the ResNet architecture is not a mere reparametrization that is more stable from a training perspective, but also this architecture changes the hypothesis space. Theoretically, they produce a novel setting separating the effect of training instability from variable-depth architecture (ResNet special property), showing that variable-depth as an advantage over fix-depth (MLP) independently of the improved stability. They show their results both for CIFAR10 and ImageNet.

Critical strengths

The paper does a good job of convincing that part of the ResNet power may reside in the functional space that it parametrize by providing both a clear theoretical setup where a feedforward and ResNet with the same capacity describe different function spaces.

In particular, the paper:

  • offers a clean and simple theoretical argument clearly showing in a particular setting that ResNet is not a reparametrization of a MLP within the same parameter budget
  • offers a novel way of experimentally disentangling the improved stability from the mere architecture: They linearized a pre-trained ResNet in two ways: Chanel-wise linearization produces a variable-depth architecture (similar to ResNet) and layer-wise linearization producing a fixed-depth architecture (similar to a MLP); pos-training the two linearziation produces a setting without training instabilities and where the variable-depth consistently wins over the fixed-depth
Critical weaknesses

I am not finding any critical weakness to the paper. A point that may be strengthen though (which is a suggestion from improvement rather than a critical weakness) is that it would be interesting to provide theoretical reasons, or at least an intuitive discussion as why the improved hypothesis space offered by ResNet is better than the MLP hypothesis space. I am thinking here argument favoring implicit regularization, smoother, flatness, etc (as for instance discussed in https://arxiv.org/abs/2209.13083), or implicitly creating ensembles as in (https://arxiv.org/abs/1605.06431 or https://arxiv.org/abs/2510.03470) applied to the channel-wise linearization.

Questions for authors

Can you please proved a clear mathematical description (beyond the diagram) of why a channel-wise linearization should be thought of as equivalent to a ResNet architecture. (Please correct me if my thinking here is wrong)?

Please discuss possible reasons for the beneficial bias of variable-depth architectures in terms of hypothesis space, increased smoothness, flatness, etc; see critical weaknesses.

2 comments

We thank reviewer uAjn for their review and interest in our work. In the following, we will try to answer the questions raised.

Can you please proved a clear mathematical description (beyond the diagram) of why a channel-wise linearization should be thought of as equivalent to a ResNet architecture. (Please correct me if my thinking here is wrong)?

In our work, we adopt the path length framework: as multiple stacked linear functions can always be represented by a single linear function, we use the average amount of nonlinear units encountered in the paths from network input to output as a measure for the network’s nonlinear complexity.

In this framework, which was popularized by the paper Residual Networks Behave Like Ensembles of Relatively Shallow Networks, the path-length distribution of a ResNet is a binomial distribution (ref. [2] Fig. 6) with μ≈d2\mu\approx \frac{d}{2}, where dd is the network depth (the exact value of μ\mu depends on the depth of the residual block). This is because the output of a residual block is the sum of the residual branch (short, no non-linearity) and main branch (long, 2-3 nonlinearities). In Figure 4 of our work, we show the path-length distribution of a standard ResNet (rhs.) alongside of a partially linearized network (lhs.): we see that the latter has a similar structure to the former (i.e. in most layers some channels remain active), which is not necessarily the case. We do not argue that these path length distributions are perfectly similar, but rather hypothesize that they share the same benefits, as demonstrated by our control experiment (ref. Figure 3).

Please discuss possible reasons for the beneficial bias of variable-depth architectures in terms of hypothesis space, increased smoothness, flatness, etc; see critical weaknesses.

The variable-depth bias can be understood in terms of flatness of the loss surface overall and in particular at the minima found during training: [6] Figure 1 and [3] Figure 1 show visually that residual connections considerably flatten the loss surface in deep nonlinear networks. Using a theoretical argument based on Fourier-theory, [3] show that every additional nonlinear layer in the network increases sharpness, whereas residual connections lowers it. Flat minima have been long hypothesized to yield a better generalization performance [5].

[1] Nonlinear Advantage: Trained Networks Might Not Be As Complex as You Think; Ali Mehmeti-Göpel, Disselhoff; 2023

[2] Residual Networks Behave Like Ensembles of Relatively Shallow Networks; Veit et al.; 2016

[3] Ringing ReLUs: Harmonic Distortion Analysis of Nonlinear Feedforward Networks; Ali Mehmeti-Göpel et al.; 2021

[4] Universal Approximation with Deep Narrow Networks; Kidger, Lyons; 2023

[5] Flat minima; Hochreiter, Schmidhuber; 1997

[6] Visualizing the Loss Landscape of Neural Nets; Li et al., 2017

Thank you for your explanation, which satisfies me. I therefore maintain my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.