- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether residual connections improve generalization through variable-depth structures, beyond their known benefits for trainability. The authors propose a post-training partial-linearization method to compare fixed-depth and variable-depth networks, and find that the latter better preserve accuracy at comparable nonlinear complexity.
Critical strengths
-
The paper points out that direct comparisons between ResNets and feedforward networks cannot fully separate architectural effects from trainability.
-
The paper shows that, under certain assumptions, residual blocks cannot be equivalently represented by plain feedforward blocks of the same width.
-
The paper introduces a post-training partial-linearization method and finds that variable-depth networks preserve accuracy better than fixed-depth networks at comparable nonlinear complexity.
Critical weaknesses
1. The research question and experiments are not fully aligned. The paper aims to investigate whether variable-depth structures improve the generalization of ResNets beyond trainability. However, as the authors acknowledge in Section 5, their experiments actually “test a narrower question”: which structure better preserves accuracy when nonlinearities are removed from an already trained network. This does not directly explain why variable-depth structures lead to better generalization.
2. The experiments do not fully isolate the effect of variable depth. The authors use channel-wise and layer-wise linearization to construct variable-depth and fixed-depth networks, respectively. However, channel-wise linearization is inherently more flexible, as it can selectively preserve nonlinearities in important channels. Therefore, its better performance may come from this flexibility rather than variable depth itself. The current experiments do not clearly separate these two effects.
3. The effect of trainability is not fully ruled out. The authors argue that starting from the same pretrained model can “substantially reduce the trainability confound”. However, both methods still require gradient-based post-training, and they impose different optimization constraints. The paper does not provide enough analysis of this process or show that the two methods are equally easy to optimize. As a result, trainability may still contribute to the observed performance gap.
4. The practical implications are somewhat limited. The paper suggests that variable-depth structures provide a useful inductive bias, but it is unclear how this finding could guide the design of better networks. For example, what kind of path-length distribution works best? How should residual connections be designed under a fixed compute budget? Some discussion or experiments along these lines would make the findings more useful in practice.
Questions for authors
See weaknesses.
We thank reviewer Fx3M for their thorough review and constructive feedback. Please allow us to address the open questions raised in the review.
Answering the larger research question whether variable-depth structures improve the generalization capabilities of deep nonlinear networks beyond trainability is empirically very difficult. As the paper highlighted, plain side-by-side training comparisons are difficult to interpret, even if direct fixes to trainability issues are applied. To our best knowledge, there are no papers in past literature that are able to conclusively answer this question.
The fact that the gap between non-residual and residual architectures still stands (ref. [1] Table 9) even when using advanced methods to solve all numerical issues we described in Section 2.1 however is highly curious, and we believe that this has not been sufficiently be discussed under the point of view we introduce: is it possible that this gap between fixed and variable-depth architectures can never be closed, even given perfect numerics during optimization? We believe that cleanly introducing this research question that had previously suffered a lot of confound in literature, is a major contribution of our paper.
As directly sampling the global minima for a given architecture is numerically infeasible, testing how well a learnt solution can be preserved in a either shape, fixed or variable-depth, appears to be a meaningful proxy for the above research question. The weaknesses (ref. Question 1-3) that the reviewer described are indeed valid limitations of our experimental setup, as discussed in our paper. The empirical evidence we provided alone does not conclusively answer the research question raised, and we agree to discuss this limitation further in the final version.
We believe however, that beyond these limitations, our experiment delivers a meaningful epistemic contribution, as our post-training setup suffers from fundamentally different experimental strengths and weaknesses wrt. direct comparisons [1]. Our setup mostly ignores per design the differences in trainability (see below), as both arms are post-trained from the same trained base network; however, our setup cannot guarantee that we would obtain the same result when training the networks side-by-side from scratch. Our empirical evidence pointing in the same direction as past literature, whilst using a very different empirical approach is meaningful and novel evidence indicating that the performance gap mentioned above might never be closed.
The cited concerns about our experimental setup are valid, but can largely be ruled out by our control experiment (ref. Figure 3): the fact that the gap mostly vanishes when starting from a ResNet clearly suggests that channel-wise linearization being more flexible or the effect of trainability are not the cause for the observed gap (ref. Figure 3 rhs.), as these factors are equally present in the control experiment.
We agree that the practical implications are not well-explored in our paper. We did run preliminary experiments exploring the optimal residual block depth, and found that the empirical optimum was close to the widely established 2-3 layers. Finding a novel prior on path lengths that out-performs current SOTA benchmarks which have been optimized for many years using the trial-and-error principle is a significant extension of our work, which is why we decided against including practical implications in the scope of this particular paper. However, we are intending to conduct follow-up work in this direction.
[1] Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping; Martens et al.; 2021